2025-10-10 14:57:58,994 [384] ERROR DotNetCore.CAP.RedisStreams.RedisStreamManager - Redis error while creating consumer group [cap.queue.mcms.finance.v1] of stream [MCMS.Finance.FinanceAccountDetail] StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=0KiB, 3504ms elapsed, timeout is 3000ms), command=EXISTS, next: EXISTS MCMS.Finance.FinanceAccountDetail, inst: 0, qu: 0, qs: 1, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, last-in: 0, cur-in: 0, sync-ops: 0, async-ops: 22790690, serverEndpoint: 127.0.0.1:16379, conn-sec: n/a, aoc: 0, mc: 1/1/0, mgr: 10 of 10 available, clientName: DevelopmentServer(SE.Redis-v2.8.24.3255), IOCP: (Busy=0,Free=1000,Min=100,Max=1000), WORKER: (Busy=1,Free=32766,Min=100,Max=32767), POOL: (Threads=8,QueuedItems=0,CompletedItems=90577749,Timers=11), v: 2.8.24.3255 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) at DotNetCore.CAP.RedisStreams.RedisStreamManagerExtensions.TryGetOrCreateStreamConsumerGroupAsync(IDatabase database, RedisKey stream, RedisValue consumerGroup) at DotNetCore.CAP.RedisStreams.RedisStreamManagerExtensions.TryGetOrCreateConsumerGroupPositionsAsync(IDatabase database, StreamPosition[] positions, String consumerGroup, ILogger logger)+MoveNext() 2025-10-10 14:59:38,092 [303] ERROR DotNetCore.CAP.RedisStreams.RedisStreamManager - Redis error while creating consumer group [cap.queue.mcms.finance.v1] of stream [MCMS.Finance.FinanceAccountDetail] StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=0KiB, 4236ms elapsed, timeout is 3000ms), command=EXISTS, next: EXISTS MCMS.Finance.FinanceAccountDetail, inst: 0, qu: 0, qs: 1, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, last-in: 0, cur-in: 0, sync-ops: 0, async-ops: 22790694, serverEndpoint: 127.0.0.1:16379, conn-sec: n/a, aoc: 0, mc: 1/1/0, mgr: 9 of 10 available, clientName: DevelopmentServer(SE.Redis-v2.8.24.3255), IOCP: (Busy=0,Free=1000,Min=100,Max=1000), WORKER: (Busy=4,Free=32763,Min=100,Max=32767), POOL: (Threads=9,QueuedItems=0,CompletedItems=90577807,Timers=10), v: 2.8.24.3255 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) at DotNetCore.CAP.RedisStreams.RedisStreamManagerExtensions.TryGetOrCreateStreamConsumerGroupAsync(IDatabase database, RedisKey stream, RedisValue consumerGroup) at DotNetCore.CAP.RedisStreams.RedisStreamManagerExtensions.TryGetOrCreateConsumerGroupPositionsAsync(IDatabase database, StreamPosition[] positions, String consumerGroup, ILogger logger)+MoveNext() 2025-10-10 14:59:38,344 [153] WARN DotNetCore.CAP.Processor.MessageNeedToRetryProcessor - Get messages from storage failed. Retrying... MySqlConnector.MySqlException (0x80004005): Connect Timeout expired. ---> MySqlConnector.MySqlException (0x80004005): The Command Timeout expired before the operation completed. at MySqlConnector.Protocol.Serialization.SocketByteHandler.DoReadBytesAsync(Memory`1 buffer) in /_/src/MySqlConnector/Protocol/Serialization/SocketByteHandler.cs:line 109 at MySqlConnector.Protocol.Serialization.BufferedByteReader.ReadBytesAsync(IByteHandler byteHandler, ArraySegment`1 buffer, Int32 totalBytesToRead, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/BufferedByteReader.cs:line 34 at MySqlConnector.Protocol.Serialization.ProtocolUtility.ReadPayloadAsync(BufferedByteReader bufferedByteReader, IByteHandler byteHandler, Func`1 getNextSequenceNumber, ArraySegmentHolder`1 previousPayloads, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/ProtocolUtility.cs:line 421 at MySqlConnector.Core.ServerSession.ReceiveReplyAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 886 at MySqlConnector.Core.ServerSession.TryResetConnectionAsync(ConnectionSettings cs, MySqlConnection connection, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 650 at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int64 startingTimestamp, Int32 timeoutMilliseconds, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 144 at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int64 startingTimestamp, Int32 timeoutMilliseconds, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 144 at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1103 at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1103 at MySqlConnector.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 590 at DotNetCore.CAP.MySql.DbConnectionExtensions.ExecuteReaderAsync[T](DbConnection connection, String sql, Func`2 readerFunc, DbTransaction transaction, Object[] sqlParams) at DotNetCore.CAP.MySql.MySqlDataStorage.GetMessagesOfNeedRetryAsync(String tableName, TimeSpan lookbackSeconds) at DotNetCore.CAP.MySql.MySqlDataStorage.GetMessagesOfNeedRetryAsync(String tableName, TimeSpan lookbackSeconds) at DotNetCore.CAP.Processor.MessageNeedToRetryProcessor.GetSafelyAsync[T](Func`2 getMessagesAsync, TimeSpan lookbackSeconds) 2025-10-10 14:59:38,344 [376] WARN DotNetCore.CAP.Processor.MessageNeedToRetryProcessor - Get messages from storage failed. Retrying... MySqlConnector.MySqlException (0x80004005): Connect Timeout expired. ---> MySqlConnector.MySqlException (0x80004005): The Command Timeout expired before the operation completed. at MySqlConnector.Protocol.Serialization.SocketByteHandler.DoReadBytesAsync(Memory`1 buffer) in /_/src/MySqlConnector/Protocol/Serialization/SocketByteHandler.cs:line 109 at MySqlConnector.Protocol.Serialization.BufferedByteReader.ReadBytesAsync(IByteHandler byteHandler, ArraySegment`1 buffer, Int32 totalBytesToRead, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/BufferedByteReader.cs:line 34 at MySqlConnector.Protocol.Serialization.ProtocolUtility.ReadPayloadAsync(BufferedByteReader bufferedByteReader, IByteHandler byteHandler, Func`1 getNextSequenceNumber, ArraySegmentHolder`1 previousPayloads, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/ProtocolUtility.cs:line 421 at MySqlConnector.Core.ServerSession.ReceiveReplyAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 886 at MySqlConnector.Core.ServerSession.TryResetConnectionAsync(ConnectionSettings cs, MySqlConnection connection, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 650 at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int64 startingTimestamp, Int32 timeoutMilliseconds, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 144 at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int64 startingTimestamp, Int32 timeoutMilliseconds, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 144 at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1103 at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1103 at MySqlConnector.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 590 at DotNetCore.CAP.MySql.DbConnectionExtensions.ExecuteReaderAsync[T](DbConnection connection, String sql, Func`2 readerFunc, DbTransaction transaction, Object[] sqlParams) at DotNetCore.CAP.MySql.MySqlDataStorage.GetMessagesOfNeedRetryAsync(String tableName, TimeSpan lookbackSeconds) at DotNetCore.CAP.MySql.MySqlDataStorage.GetMessagesOfNeedRetryAsync(String tableName, TimeSpan lookbackSeconds) at DotNetCore.CAP.Processor.MessageNeedToRetryProcessor.GetSafelyAsync[T](Func`2 getMessagesAsync, TimeSpan lookbackSeconds) 2025-10-10 14:59:38,448 [Kestrel Timer] WARN Microsoft.AspNetCore.Server.Kestrel - As of "10/10/2025 06:59:34 +00:00", the heartbeat has been running for "00:00:01.1631859" which is longer than "00:00:01". This could be caused by thread pool starvation. 2025-10-10 14:59:50,422 [Kestrel Timer] WARN Microsoft.AspNetCore.Server.Kestrel - As of "10/10/2025 06:59:43 +00:00", the heartbeat has been running for "00:00:01.3380435" which is longer than "00:00:01". This could be caused by thread pool starvation. 2025-10-10 15:03:00,755 [Kestrel Timer] WARN Microsoft.AspNetCore.Server.Kestrel - As of "10/10/2025 07:02:52 +00:00", the heartbeat has been running for "00:00:02.6618136" which is longer than "00:00:01". This could be caused by thread pool starvation. 2025-10-10 15:05:09,787 [218] WARN DotNetCore.CAP.Processor.MessageNeedToRetryProcessor - Get messages from storage failed. Retrying... MySqlConnector.MySqlException (0x80004005): Connect Timeout expired. ---> MySqlConnector.MySqlException (0x80004005): The Command Timeout expired before the operation completed. ---> System.Net.Sockets.SocketException (125): Operation canceled at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at MySqlConnector.Protocol.Serialization.SocketByteHandler.DoReadBytesAsync(Memory`1 buffer) in /_/src/MySqlConnector/Protocol/Serialization/SocketByteHandler.cs:line 109 at MySqlConnector.Protocol.Serialization.SocketByteHandler.DoReadBytesAsync(Memory`1 buffer) in /_/src/MySqlConnector/Protocol/Serialization/SocketByteHandler.cs:line 109 at MySqlConnector.Protocol.Serialization.BufferedByteReader.ReadBytesAsync(IByteHandler byteHandler, ArraySegment`1 buffer, Int32 totalBytesToRead, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/BufferedByteReader.cs:line 34 at MySqlConnector.Protocol.Serialization.ProtocolUtility.ReadPayloadAsync(BufferedByteReader bufferedByteReader, IByteHandler byteHandler, Func`1 getNextSequenceNumber, ArraySegmentHolder`1 previousPayloads, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/ProtocolUtility.cs:line 421 at MySqlConnector.Core.ServerSession.ReceiveReplyAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 886 at MySqlConnector.Core.ServerSession.ConnectAsync(ConnectionSettings cs, MySqlConnection connection, Int64 startingTimestamp, ILoadBalancer loadBalancer, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 442 at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, Action`4 logMessage, Int64 startingTimestamp, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 427 at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, Action`4 logMessage, Int64 startingTimestamp, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 432 at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int64 startingTimestamp, Int32 timeoutMilliseconds, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 144 at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int64 startingTimestamp, Int32 timeoutMilliseconds, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 144 at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1103 at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1103 at MySqlConnector.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 590 at DotNetCore.CAP.MySql.DbConnectionExtensions.ExecuteReaderAsync[T](DbConnection connection, String sql, Func`2 readerFunc, DbTransaction transaction, Object[] sqlParams) at DotNetCore.CAP.MySql.MySqlDataStorage.GetMessagesOfNeedRetryAsync(String tableName, TimeSpan lookbackSeconds) at DotNetCore.CAP.MySql.MySqlDataStorage.GetMessagesOfNeedRetryAsync(String tableName, TimeSpan lookbackSeconds) at DotNetCore.CAP.Processor.MessageNeedToRetryProcessor.GetSafelyAsync[T](Func`2 getMessagesAsync, TimeSpan lookbackSeconds) 2025-10-10 15:05:09,787 [258] ERROR DotNetCore.CAP.RedisStreams.RedisStreamManager - Redis error while creating consumer group [cap.queue.mcms.finance.v1] of stream [MCMS.Finance.BankStatementDetailConfirmed] StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=0KiB, 5768ms elapsed, timeout is 3000ms), command=XINFO, next: XINFO MCMS.Finance.BankStatementDetailConfirmed, inst: 0, qu: 0, qs: 1, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, last-in: 1, cur-in: 0, sync-ops: 0, async-ops: 22790696, serverEndpoint: 127.0.0.1:16379, conn-sec: n/a, aoc: 0, mc: 1/1/0, mgr: 10 of 10 available, clientName: DevelopmentServer(SE.Redis-v2.8.24.3255), IOCP: (Busy=0,Free=1000,Min=100,Max=1000), WORKER: (Busy=2,Free=32765,Min=100,Max=32767), POOL: (Threads=13,QueuedItems=0,CompletedItems=90577931,Timers=11), v: 2.8.24.3255 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) at DotNetCore.CAP.RedisStreams.RedisStreamManagerExtensions.TryGetOrCreateStreamGroupAsync(IDatabase database, RedisKey stream, RedisValue consumerGroup) at DotNetCore.CAP.RedisStreams.RedisStreamManagerExtensions.TryGetOrCreateStreamGroupAsync(IDatabase database, RedisKey stream, RedisValue consumerGroup) at DotNetCore.CAP.RedisStreams.RedisStreamManagerExtensions.TryGetOrCreateStreamConsumerGroupAsync(IDatabase database, RedisKey stream, RedisValue consumerGroup) at DotNetCore.CAP.RedisStreams.RedisStreamManagerExtensions.TryGetOrCreateConsumerGroupPositionsAsync(IDatabase database, StreamPosition[] positions, String consumerGroup, ILogger logger)+MoveNext() 2025-10-10 15:05:09,787 [57] WARN DotNetCore.CAP.Processor.MessageNeedToRetryProcessor - Get messages from storage failed. Retrying... MySqlConnector.MySqlException (0x80004005): Connect Timeout expired. ---> MySqlConnector.MySqlException (0x80004005): The Command Timeout expired before the operation completed. ---> System.Net.Sockets.SocketException (125): Operation canceled at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at MySqlConnector.Protocol.Serialization.SocketByteHandler.DoReadBytesAsync(Memory`1 buffer) in /_/src/MySqlConnector/Protocol/Serialization/SocketByteHandler.cs:line 109 at MySqlConnector.Protocol.Serialization.SocketByteHandler.DoReadBytesAsync(Memory`1 buffer) in /_/src/MySqlConnector/Protocol/Serialization/SocketByteHandler.cs:line 109 at MySqlConnector.Protocol.Serialization.BufferedByteReader.ReadBytesAsync(IByteHandler byteHandler, ArraySegment`1 buffer, Int32 totalBytesToRead, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/BufferedByteReader.cs:line 34 at MySqlConnector.Protocol.Serialization.ProtocolUtility.ReadPayloadAsync(BufferedByteReader bufferedByteReader, IByteHandler byteHandler, Func`1 getNextSequenceNumber, ArraySegmentHolder`1 previousPayloads, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/ProtocolUtility.cs:line 421 at MySqlConnector.Core.ServerSession.ReceiveReplyAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 886 at MySqlConnector.Core.ServerSession.ConnectAsync(ConnectionSettings cs, MySqlConnection connection, Int64 startingTimestamp, ILoadBalancer loadBalancer, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 442 at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, Action`4 logMessage, Int64 startingTimestamp, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 427 at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, Action`4 logMessage, Int64 startingTimestamp, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 432 at MySqlConnector.Core.ConnectionPool.CreateMinimumPooledSessions(MySqlConnection connection, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 405 at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int64 startingTimestamp, Int32 timeoutMilliseconds, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 144 at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1103 at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1103 at MySqlConnector.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 590 at DotNetCore.CAP.MySql.DbConnectionExtensions.ExecuteReaderAsync[T](DbConnection connection, String sql, Func`2 readerFunc, DbTransaction transaction, Object[] sqlParams) at DotNetCore.CAP.MySql.MySqlDataStorage.GetMessagesOfNeedRetryAsync(String tableName, TimeSpan lookbackSeconds) at DotNetCore.CAP.MySql.MySqlDataStorage.GetMessagesOfNeedRetryAsync(String tableName, TimeSpan lookbackSeconds) at DotNetCore.CAP.Processor.MessageNeedToRetryProcessor.GetSafelyAsync[T](Func`2 getMessagesAsync, TimeSpan lookbackSeconds) 2025-10-10 15:05:09,787 [145] WARN DotNetCore.CAP.Processor.MessageNeedToRetryProcessor - Get messages from storage failed. Retrying... MySqlConnector.MySqlException (0x80004005): Connect Timeout expired. ---> MySqlConnector.MySqlException (0x80004005): The Command Timeout expired before the operation completed. ---> System.Net.Sockets.SocketException (125): Operation canceled at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at MySqlConnector.Protocol.Serialization.SocketByteHandler.DoReadBytesAsync(Memory`1 buffer) in /_/src/MySqlConnector/Protocol/Serialization/SocketByteHandler.cs:line 109 at MySqlConnector.Protocol.Serialization.SocketByteHandler.DoReadBytesAsync(Memory`1 buffer) in /_/src/MySqlConnector/Protocol/Serialization/SocketByteHandler.cs:line 109 at MySqlConnector.Protocol.Serialization.BufferedByteReader.ReadBytesAsync(IByteHandler byteHandler, ArraySegment`1 buffer, Int32 totalBytesToRead, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/BufferedByteReader.cs:line 34 at MySqlConnector.Protocol.Serialization.ProtocolUtility.ReadPayloadAsync(BufferedByteReader bufferedByteReader, IByteHandler byteHandler, Func`1 getNextSequenceNumber, ArraySegmentHolder`1 previousPayloads, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/ProtocolUtility.cs:line 421 at MySqlConnector.Core.ServerSession.ReceiveReplyAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 886 at MySqlConnector.Core.ServerSession.ConnectAsync(ConnectionSettings cs, MySqlConnection connection, Int64 startingTimestamp, ILoadBalancer loadBalancer, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 442 at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, Action`4 logMessage, Int64 startingTimestamp, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 427 at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, Action`4 logMessage, Int64 startingTimestamp, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 432 at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int64 startingTimestamp, Int32 timeoutMilliseconds, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 144 at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int64 startingTimestamp, Int32 timeoutMilliseconds, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 144 at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1103 at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1103 at MySqlConnector.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 590 at DotNetCore.CAP.MySql.DbConnectionExtensions.ExecuteReaderAsync[T](DbConnection connection, String sql, Func`2 readerFunc, DbTransaction transaction, Object[] sqlParams) at DotNetCore.CAP.MySql.MySqlDataStorage.GetMessagesOfNeedRetryAsync(String tableName, TimeSpan lookbackSeconds) at DotNetCore.CAP.MySql.MySqlDataStorage.GetMessagesOfNeedRetryAsync(String tableName, TimeSpan lookbackSeconds) at DotNetCore.CAP.Processor.MessageNeedToRetryProcessor.GetSafelyAsync[T](Func`2 getMessagesAsync, TimeSpan lookbackSeconds) 2025-10-10 15:05:09,787 [279] WARN DotNetCore.CAP.Processor.MessageNeedToRetryProcessor - Get messages from storage failed. Retrying... MySqlConnector.MySqlException (0x80004005): Connect Timeout expired. ---> MySqlConnector.MySqlException (0x80004005): The Command Timeout expired before the operation completed. ---> System.Net.Sockets.SocketException (125): Operation canceled at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at MySqlConnector.Protocol.Serialization.SocketByteHandler.DoReadBytesAsync(Memory`1 buffer) in /_/src/MySqlConnector/Protocol/Serialization/SocketByteHandler.cs:line 109 at MySqlConnector.Protocol.Serialization.SocketByteHandler.DoReadBytesAsync(Memory`1 buffer) in /_/src/MySqlConnector/Protocol/Serialization/SocketByteHandler.cs:line 109 at MySqlConnector.Protocol.Serialization.BufferedByteReader.ReadBytesAsync(IByteHandler byteHandler, ArraySegment`1 buffer, Int32 totalBytesToRead, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/BufferedByteReader.cs:line 34 at MySqlConnector.Protocol.Serialization.ProtocolUtility.ReadPayloadAsync(BufferedByteReader bufferedByteReader, IByteHandler byteHandler, Func`1 getNextSequenceNumber, ArraySegmentHolder`1 previousPayloads, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/ProtocolUtility.cs:line 421 at MySqlConnector.Core.ServerSession.ReceiveReplyAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 886 at MySqlConnector.Core.ServerSession.ConnectAsync(ConnectionSettings cs, MySqlConnection connection, Int64 startingTimestamp, ILoadBalancer loadBalancer, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 442 at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, Action`4 logMessage, Int64 startingTimestamp, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 427 at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, Action`4 logMessage, Int64 startingTimestamp, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 432 at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int64 startingTimestamp, Int32 timeoutMilliseconds, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 144 at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int64 startingTimestamp, Int32 timeoutMilliseconds, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 144 at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1103 at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1103 at MySqlConnector.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 590 at DotNetCore.CAP.MySql.DbConnectionExtensions.ExecuteReaderAsync[T](DbConnection connection, String sql, Func`2 readerFunc, DbTransaction transaction, Object[] sqlParams) at DotNetCore.CAP.MySql.MySqlDataStorage.GetMessagesOfNeedRetryAsync(String tableName, TimeSpan lookbackSeconds) at DotNetCore.CAP.MySql.MySqlDataStorage.GetMessagesOfNeedRetryAsync(String tableName, TimeSpan lookbackSeconds) at DotNetCore.CAP.Processor.MessageNeedToRetryProcessor.GetSafelyAsync[T](Func`2 getMessagesAsync, TimeSpan lookbackSeconds) 2025-10-10 15:05:09,795 [318] WARN DotNetCore.CAP.Processor.MessageDelayedProcessor - Get delayed messages from storage failed. Retrying... MySqlConnector.MySqlException (0x80004005): Connect Timeout expired. ---> MySqlConnector.MySqlException (0x80004005): The Command Timeout expired before the operation completed. ---> System.Net.Sockets.SocketException (125): Operation canceled at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at MySqlConnector.Protocol.Serialization.SocketByteHandler.DoReadBytesAsync(Memory`1 buffer) in /_/src/MySqlConnector/Protocol/Serialization/SocketByteHandler.cs:line 109 at MySqlConnector.Protocol.Serialization.SocketByteHandler.DoReadBytesAsync(Memory`1 buffer) in /_/src/MySqlConnector/Protocol/Serialization/SocketByteHandler.cs:line 109 at MySqlConnector.Protocol.Serialization.BufferedByteReader.ReadBytesAsync(IByteHandler byteHandler, ArraySegment`1 buffer, Int32 totalBytesToRead, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/BufferedByteReader.cs:line 34 at MySqlConnector.Protocol.Serialization.ProtocolUtility.ReadPayloadAsync(BufferedByteReader bufferedByteReader, IByteHandler byteHandler, Func`1 getNextSequenceNumber, ArraySegmentHolder`1 previousPayloads, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/ProtocolUtility.cs:line 421 at MySqlConnector.Core.ServerSession.ReceiveReplyAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 886 at MySqlConnector.Core.ServerSession.ConnectAsync(ConnectionSettings cs, MySqlConnection connection, Int64 startingTimestamp, ILoadBalancer loadBalancer, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 527 at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, Action`4 logMessage, Int64 startingTimestamp, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 427 at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, Action`4 logMessage, Int64 startingTimestamp, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 432 at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int64 startingTimestamp, Int32 timeoutMilliseconds, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 144 at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int64 startingTimestamp, Int32 timeoutMilliseconds, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 144 at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1103 at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1103 at MySqlConnector.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 590 at DotNetCore.CAP.MySql.MySqlDataStorage.ScheduleMessagesOfDelayedAsync(Func`3 scheduleTask, CancellationToken token) at DotNetCore.CAP.MySql.MySqlDataStorage.ScheduleMessagesOfDelayedAsync(Func`3 scheduleTask, CancellationToken token) at DotNetCore.CAP.Processor.MessageDelayedProcessor.ProcessDelayedAsync(IDataStorage connection, ProcessingContext context) 2025-10-10 15:05:09,795 [147] ERROR DotNetCore.CAP.Processor.CollectorProcessor - An error occurred while attempting to delete expired data from table 'cap.published':Connect Timeout expired. MySqlConnector.MySqlException (0x80004005): Connect Timeout expired. ---> MySqlConnector.MySqlException (0x80004005): The Command Timeout expired before the operation completed. ---> System.Net.Sockets.SocketException (125): Operation canceled at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at MySqlConnector.Protocol.Serialization.SocketByteHandler.DoReadBytesAsync(Memory`1 buffer) in /_/src/MySqlConnector/Protocol/Serialization/SocketByteHandler.cs:line 109 at MySqlConnector.Protocol.Serialization.SocketByteHandler.DoReadBytesAsync(Memory`1 buffer) in /_/src/MySqlConnector/Protocol/Serialization/SocketByteHandler.cs:line 109 at MySqlConnector.Protocol.Serialization.BufferedByteReader.ReadBytesAsync(IByteHandler byteHandler, ArraySegment`1 buffer, Int32 totalBytesToRead, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/BufferedByteReader.cs:line 34 at MySqlConnector.Protocol.Serialization.ProtocolUtility.ReadPayloadAsync(BufferedByteReader bufferedByteReader, IByteHandler byteHandler, Func`1 getNextSequenceNumber, ArraySegmentHolder`1 previousPayloads, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/ProtocolUtility.cs:line 421 at MySqlConnector.Core.ServerSession.ReceiveReplyAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 886 at MySqlConnector.Core.ServerSession.ConnectAsync(ConnectionSettings cs, MySqlConnection connection, Int64 startingTimestamp, ILoadBalancer loadBalancer, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 442 at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, Action`4 logMessage, Int64 startingTimestamp, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 427 at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, Action`4 logMessage, Int64 startingTimestamp, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 432 at MySqlConnector.Core.ConnectionPool.CreateMinimumPooledSessions(MySqlConnection connection, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 405 at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int64 startingTimestamp, Int32 timeoutMilliseconds, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 144 at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1103 at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1103 at MySqlConnector.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 590 at DotNetCore.CAP.MySql.DbConnectionExtensions.ExecuteNonQueryAsync(DbConnection connection, String sql, DbTransaction transaction, Object[] sqlParams) at DotNetCore.CAP.MySql.MySqlDataStorage.DeleteExpiresAsync(String table, DateTime timeout, Int32 batchCount, CancellationToken token) at DotNetCore.CAP.MySql.MySqlDataStorage.DeleteExpiresAsync(String table, DateTime timeout, Int32 batchCount, CancellationToken token) at DotNetCore.CAP.Processor.CollectorProcessor.ProcessAsync(ProcessingContext context) 2025-10-10 15:05:09,823 [251] WARN DotNetCore.CAP.Processor.MessageNeedToRetryProcessor - Get messages from storage failed. Retrying... MySqlConnector.MySqlException (0x80004005): Connect Timeout expired. at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1103 at MySqlConnector.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 590 at DotNetCore.CAP.MySql.DbConnectionExtensions.ExecuteReaderAsync[T](DbConnection connection, String sql, Func`2 readerFunc, DbTransaction transaction, Object[] sqlParams) at DotNetCore.CAP.MySql.MySqlDataStorage.GetMessagesOfNeedRetryAsync(String tableName, TimeSpan lookbackSeconds) at DotNetCore.CAP.MySql.MySqlDataStorage.GetMessagesOfNeedRetryAsync(String tableName, TimeSpan lookbackSeconds) at DotNetCore.CAP.Processor.MessageNeedToRetryProcessor.GetSafelyAsync[T](Func`2 getMessagesAsync, TimeSpan lookbackSeconds) 2025-10-10 15:05:09,825 [376] WARN DotNetCore.CAP.Processor.MessageNeedToRetryProcessor - Get messages from storage failed. Retrying... MySqlConnector.MySqlException (0x80004005): Connect Timeout expired. at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1103 at MySqlConnector.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 590 at DotNetCore.CAP.MySql.DbConnectionExtensions.ExecuteReaderAsync[T](DbConnection connection, String sql, Func`2 readerFunc, DbTransaction transaction, Object[] sqlParams) at DotNetCore.CAP.MySql.MySqlDataStorage.GetMessagesOfNeedRetryAsync(String tableName, TimeSpan lookbackSeconds) at DotNetCore.CAP.MySql.MySqlDataStorage.GetMessagesOfNeedRetryAsync(String tableName, TimeSpan lookbackSeconds) at DotNetCore.CAP.Processor.MessageNeedToRetryProcessor.GetSafelyAsync[T](Func`2 getMessagesAsync, TimeSpan lookbackSeconds) 2025-10-10 15:05:09,862 [147] WARN DotNetCore.CAP.Processor.InfiniteRetryProcessor - Processor 'DotNetCore.CAP.Processor.CollectorProcessor' failed. Retrying... MySqlConnector.MySqlException (0x80004005): Connect Timeout expired. ---> MySqlConnector.MySqlException (0x80004005): The Command Timeout expired before the operation completed. ---> System.Net.Sockets.SocketException (125): Operation canceled at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at MySqlConnector.Protocol.Serialization.SocketByteHandler.DoReadBytesAsync(Memory`1 buffer) in /_/src/MySqlConnector/Protocol/Serialization/SocketByteHandler.cs:line 109 at MySqlConnector.Protocol.Serialization.SocketByteHandler.DoReadBytesAsync(Memory`1 buffer) in /_/src/MySqlConnector/Protocol/Serialization/SocketByteHandler.cs:line 109 at MySqlConnector.Protocol.Serialization.BufferedByteReader.ReadBytesAsync(IByteHandler byteHandler, ArraySegment`1 buffer, Int32 totalBytesToRead, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/BufferedByteReader.cs:line 34 at MySqlConnector.Protocol.Serialization.ProtocolUtility.ReadPayloadAsync(BufferedByteReader bufferedByteReader, IByteHandler byteHandler, Func`1 getNextSequenceNumber, ArraySegmentHolder`1 previousPayloads, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/ProtocolUtility.cs:line 421 at MySqlConnector.Core.ServerSession.ReceiveReplyAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 886 at MySqlConnector.Core.ServerSession.ConnectAsync(ConnectionSettings cs, MySqlConnection connection, Int64 startingTimestamp, ILoadBalancer loadBalancer, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 442 at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, Action`4 logMessage, Int64 startingTimestamp, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 427 at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, Action`4 logMessage, Int64 startingTimestamp, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 432 at MySqlConnector.Core.ConnectionPool.CreateMinimumPooledSessions(MySqlConnection connection, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 405 at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int64 startingTimestamp, Int32 timeoutMilliseconds, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 144 at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1103 at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1103 at MySqlConnector.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 590 at DotNetCore.CAP.MySql.DbConnectionExtensions.ExecuteNonQueryAsync(DbConnection connection, String sql, DbTransaction transaction, Object[] sqlParams) at DotNetCore.CAP.MySql.MySqlDataStorage.DeleteExpiresAsync(String table, DateTime timeout, Int32 batchCount, CancellationToken token) at DotNetCore.CAP.MySql.MySqlDataStorage.DeleteExpiresAsync(String table, DateTime timeout, Int32 batchCount, CancellationToken token) at DotNetCore.CAP.Processor.CollectorProcessor.ProcessAsync(ProcessingContext context) at DotNetCore.CAP.Processor.InfiniteRetryProcessor.ProcessAsync(ProcessingContext context) 2025-10-10 15:05:09,931 [147] WARN DotNetCore.CAP.RedisStreams.AsyncLazyRedisConnection - Connection restored back!, (null), for endpoint:127.0.0.1:16379, failure type:None, connection type:Interactive 2025-10-10 15:05:09,915 [256] ERROR NHibernate.AdoNet.AbstractBatcher - Could not execute query: SELECT this_.Id as id1_1_0_, this_.`Version` as Version2_1_0_, this_.`Number` as Number3_1_0_, this_.`Name` as Name4_1_0_, this_.`AccountType` as AccountType5_1_0_, this_.`CardNumber` as CardNumber6_1_0_, this_.`AccountName` as AccountName7_1_0_, this_.`InitBalance` as InitBalance8_1_0_, this_.`Balance` as Balance9_1_0_, this_.`IncomeFee` as IncomeFee10_1_0_, this_.`Status` as Status11_1_0_, this_.`OrderBy` as OrderBy12_1_0_, this_.`LogisticsAccount` as LogisticsAccount13_1_0_, this_.`LogisticSFCustId` as LogisticSFCustId14_1_0_, this_.`UseDepartmentId` as UseDepartmentId15_1_0_, this_.`UseDepartmentName` as UseDepartmentName16_1_0_, this_.`LogisticJDDeptNo` as LogisticJDDeptNo17_1_0_ FROM `FinanceAccount` this_ ORDER BY this_.`OrderBy` asc MySql.Data.MySqlClient.MySqlException (0x80004005): Fatal error encountered during command execution. ---> MySql.Data.MySqlClient.MySqlException (0x80004005): Fatal error encountered attempting to read the resultset. ---> MySql.Data.MySqlClient.MySqlException (0x80004005): Reading from the stream has failed. ---> System.IO.IOException: Unable to read data from the transport connection: Connection timed out. ---> System.Net.Sockets.SocketException (110): Connection timed out at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count) --- End of inner exception stack trace --- at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count) at MySql.Data.MySqlClient.TimedStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, Boolean execAsync) at MySql.Data.MySqlClient.TimedStream.Read(Byte[] buffer, Int32 offset, Int32 count) at MySql.Data.MySqlClient.MySqlStream.ReadFullyAsync(Stream stream, Byte[] buffer, Int32 offset, Int32 count, Boolean execAsync) at MySql.Data.MySqlClient.MySqlStream.LoadPacketAsync(Boolean execAsync) at MySql.Data.MySqlClient.MySqlStream.LoadPacketAsync(Boolean execAsync) at MySql.Data.MySqlClient.MySqlStream.ReadPacketAsync(Boolean execAsync) at MySql.Data.MySqlClient.NativeDriver.GetResultAsync(Int32 affectedRow, Int64 insertedId, Boolean execAsync) at MySql.Data.MySqlClient.Driver.GetResultAsync(Int32 statementId, Int32 affectedRows, Int64 insertedId, Boolean execAsync) at MySql.Data.MySqlClient.Driver.NextResultAsync(Int32 statementId, Boolean force, Boolean execAsync) at MySql.Data.MySqlClient.MySqlDataReader.NextResultAsync(Boolean execAsync, CancellationToken cancellationToken) at MySql.Data.MySqlClient.MySqlDataReader.NextResultAsync(Boolean execAsync, CancellationToken cancellationToken) at MySql.Data.MySqlClient.MySqlCommand.ExecuteReaderAsync(CommandBehavior behavior, Boolean execAsync, CancellationToken cancellationToken) at MySql.Data.MySqlClient.MySqlCommand.ExecuteReaderAsync(CommandBehavior behavior, Boolean execAsync, CancellationToken cancellationToken) at MySql.Data.MySqlClient.MySqlCommand.ExecuteReaderAsync(CommandBehavior behavior, Boolean execAsync, CancellationToken cancellationToken) at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader(CommandBehavior behavior) at NHibernate.AdoNet.AbstractBatcher.DoExecuteReader(DbCommand cmd) 2025-10-10 15:05:09,985 [256] ERROR NHibernate.Util.ADOExceptionReporter - Fatal error encountered during command execution. 2025-10-10 15:05:09,985 [256] ERROR NHibernate.Util.ADOExceptionReporter - Fatal error encountered attempting to read the resultset. 2025-10-10 15:05:09,985 [256] ERROR NHibernate.Util.ADOExceptionReporter - Reading from the stream has failed. 2025-10-10 15:05:09,985 [256] ERROR NHibernate.Util.ADOExceptionReporter - Unable to read data from the transport connection: Connection timed out. 2025-10-10 15:05:09,985 [256] ERROR NHibernate.Util.ADOExceptionReporter - Connection timed out 2025-10-10 15:05:09,987 [256] ERROR NHibernate.Util.ADOExceptionReporter - Fatal error encountered during command execution. 2025-10-10 15:05:09,987 [256] ERROR NHibernate.Util.ADOExceptionReporter - Fatal error encountered attempting to read the resultset. 2025-10-10 15:05:09,987 [256] ERROR NHibernate.Util.ADOExceptionReporter - Reading from the stream has failed. 2025-10-10 15:05:09,987 [256] ERROR NHibernate.Util.ADOExceptionReporter - Unable to read data from the transport connection: Connection timed out. 2025-10-10 15:05:09,987 [256] ERROR NHibernate.Util.ADOExceptionReporter - Connection timed out 2025-10-10 15:05:09,987 [256] ERROR NHibernate.Util.ADOExceptionReporter - Fatal error encountered during command execution. 2025-10-10 15:05:09,987 [256] ERROR NHibernate.Util.ADOExceptionReporter - Fatal error encountered attempting to read the resultset. 2025-10-10 15:05:09,987 [256] ERROR NHibernate.Util.ADOExceptionReporter - Reading from the stream has failed. 2025-10-10 15:05:09,987 [256] ERROR NHibernate.Util.ADOExceptionReporter - Unable to read data from the transport connection: Connection timed out. 2025-10-10 15:05:09,987 [256] ERROR NHibernate.Util.ADOExceptionReporter - Connection timed out 2025-10-10 15:05:10,052 [256] ERROR MCMS.Finance.Common.GlobalExceptionFilter - URL:http://0.0.0.0:5011/FinanceAccount/GetList;UserIP:127.0.0.1,ServerVariables:Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestHeaders;Exception:could not execute query [ SELECT this_.Id as id1_1_0_, this_.`Version` as Version2_1_0_, this_.`Number` as Number3_1_0_, this_.`Name` as Name4_1_0_, this_.`AccountType` as AccountType5_1_0_, this_.`CardNumber` as CardNumber6_1_0_, this_.`AccountName` as AccountName7_1_0_, this_.`InitBalance` as InitBalance8_1_0_, this_.`Balance` as Balance9_1_0_, this_.`IncomeFee` as IncomeFee10_1_0_, this_.`Status` as Status11_1_0_, this_.`OrderBy` as OrderBy12_1_0_, this_.`LogisticsAccount` as LogisticsAccount13_1_0_, this_.`LogisticSFCustId` as LogisticSFCustId14_1_0_, this_.`UseDepartmentId` as UseDepartmentId15_1_0_, this_.`UseDepartmentName` as UseDepartmentName16_1_0_, this_.`LogisticJDDeptNo` as LogisticJDDeptNo17_1_0_ FROM `FinanceAccount` this_ ORDER BY this_.`OrderBy` asc ] [SQL: SELECT this_.Id as id1_1_0_, this_.`Version` as Version2_1_0_, this_.`Number` as Number3_1_0_, this_.`Name` as Name4_1_0_, this_.`AccountType` as AccountType5_1_0_, this_.`CardNumber` as CardNumber6_1_0_, this_.`AccountName` as AccountName7_1_0_, this_.`InitBalance` as InitBalance8_1_0_, this_.`Balance` as Balance9_1_0_, this_.`IncomeFee` as IncomeFee10_1_0_, this_.`Status` as Status11_1_0_, this_.`OrderBy` as OrderBy12_1_0_, this_.`LogisticsAccount` as LogisticsAccount13_1_0_, this_.`LogisticSFCustId` as LogisticSFCustId14_1_0_, this_.`UseDepartmentId` as UseDepartmentId15_1_0_, this_.`UseDepartmentName` as UseDepartmentName16_1_0_, this_.`LogisticJDDeptNo` as LogisticJDDeptNo17_1_0_ FROM `FinanceAccount` this_ ORDER BY this_.`OrderBy` asc] at NHibernate.Loader.Loader.DoList(ISessionImplementor session, QueryParameters queryParameters, IResultTransformer forcedResultTransformer, QueryCacheResultBuilder queryCacheResultBuilder) at NHibernate.Loader.Criteria.CriteriaLoaderExtensions.LoadAllToList[T](IList`1 loaders, ISessionImplementor session) at NHibernate.Impl.SessionImpl.List[T](CriteriaImpl criteria) at NHibernate.Impl.CriteriaImpl.List[T]() at MCMS.Finance.DAO.Hibernate.FinanceAccountDAO.GetObjectListBySearch(FinanceAccountQueryDto queryDto, IPagingInfo pagingInfo, OrderField orderField) at Spring.Reflection.Dynamic.SafeMethod.Invoke(Object target, Object[] arguments) at Spring.Aop.Framework.DynamicMethodInvocation.InvokeJoinpoint() at Spring.Aop.Framework.AbstractMethodInvocation.Proceed() at Spring.Transaction.Interceptor.TransactionInterceptor.Invoke(IMethodInvocation invocation) at Spring.Aop.Framework.AbstractMethodInvocation.Proceed() at Spring.Aop.Framework.DynamicProxy.AdvisedProxy.Invoke(Object proxy, Object target, Type targetType, MethodInfo targetMethod, MethodInfo proxyMethod, Object[] args, IList interceptors) at CompositionAopProxy_6a53b33003ae48a78c9ede31f6406c10.GetObjectListBySearch(FinanceAccountQueryDto queryDto, IPagingInfo pagingInfo, OrderField orderField) at MCMS.Finance.Controllers.FinanceAccountController.GetList(FinanceAccountSearchDto pars) in E:\agent\_work\5\s\MCMS.Finance\Controllers\FinanceAccountController.cs:line 162 at lambda_method83(Closure , Object , Object[] ) at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync() at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync() --- End of stack trace from previous location --- at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync() --- End of stack trace from previous location --- at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)