2025-10-09 00:05:08,030 [216] WARN DotNetCore.CAP.RedisStreams.AsyncLazyRedisConnection - Connection restored back!, (null), for endpoint:127.0.0.1:16379, failure type:None, connection type:Interactive 2025-10-09 00:05:08,030 [261] ERROR DotNetCore.CAP.Processor.CollectorProcessor - An error occurred while attempting to delete expired data from table 'cap.received':The Command Timeout expired before the operation completed. MySqlConnector.MySqlException (0x80004005): The Command Timeout expired before the operation completed. ---> System.Net.Sockets.SocketException (125): Operation canceled at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at MySqlConnector.Protocol.Serialization.SocketByteHandler.DoReadBytesAsync(Memory`1 buffer) in /_/src/MySqlConnector/Protocol/Serialization/SocketByteHandler.cs:line 109 at MySqlConnector.Protocol.Serialization.SocketByteHandler.DoReadBytesAsync(Memory`1 buffer) in /_/src/MySqlConnector/Protocol/Serialization/SocketByteHandler.cs:line 109 at MySqlConnector.Protocol.Serialization.BufferedByteReader.ReadBytesAsync(IByteHandler byteHandler, ArraySegment`1 buffer, Int32 totalBytesToRead, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/BufferedByteReader.cs:line 34 at MySqlConnector.Protocol.Serialization.ProtocolUtility.ReadPayloadAsync(BufferedByteReader bufferedByteReader, IByteHandler byteHandler, Func`1 getNextSequenceNumber, ArraySegmentHolder`1 previousPayloads, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/ProtocolUtility.cs:line 421 at MySqlConnector.Core.ServerSession.ReceiveReplyAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 886 at MySqlConnector.Core.ResultSet.ReadResultSetHeaderAsync(IOBehavior ioBehavior) in /_/src/MySqlConnector/Core/ResultSet.cs:line 37 at MySqlConnector.MySqlDataReader.ActivateResultSet(CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 137 at MySqlConnector.MySqlDataReader.InitAsync(CommandListPosition commandListPosition, ICommandPayloadCreator payloadCreator, IDictionary`2 cachedProcedures, IMySqlCommand command, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 482 at MySqlConnector.Core.CommandExecutor.ExecuteReaderAsync(CommandListPosition commandListPosition, ICommandPayloadCreator payloadCreator, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/CommandExecutor.cs:line 56 at MySqlConnector.MySqlCommand.ExecuteNonQueryAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlCommand.cs:line 309 at DotNetCore.CAP.MySql.DbConnectionExtensions.ExecuteNonQueryAsync(DbConnection connection, String sql, DbTransaction transaction, Object[] sqlParams) at DotNetCore.CAP.MySql.DbConnectionExtensions.ExecuteNonQueryAsync(DbConnection connection, String sql, DbTransaction transaction, Object[] sqlParams) at DotNetCore.CAP.MySql.MySqlDataStorage.DeleteExpiresAsync(String table, DateTime timeout, Int32 batchCount, CancellationToken token) at DotNetCore.CAP.MySql.MySqlDataStorage.DeleteExpiresAsync(String table, DateTime timeout, Int32 batchCount, CancellationToken token) at DotNetCore.CAP.Processor.CollectorProcessor.ProcessAsync(ProcessingContext context) 2025-10-09 00:05:08,030 [105] WARN DotNetCore.CAP.Processor.MessageNeedToRetryProcessor - Get messages from storage failed. Retrying... MySqlConnector.MySqlException (0x80004005): Connect Timeout expired. at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1103 at MySqlConnector.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 590 at DotNetCore.CAP.MySql.DbConnectionExtensions.ExecuteReaderAsync[T](DbConnection connection, String sql, Func`2 readerFunc, DbTransaction transaction, Object[] sqlParams) at DotNetCore.CAP.MySql.MySqlDataStorage.GetMessagesOfNeedRetryAsync(String tableName, TimeSpan lookbackSeconds) at DotNetCore.CAP.MySql.MySqlDataStorage.GetMessagesOfNeedRetryAsync(String tableName, TimeSpan lookbackSeconds) at DotNetCore.CAP.Processor.MessageNeedToRetryProcessor.GetSafelyAsync[T](Func`2 getMessagesAsync, TimeSpan lookbackSeconds) 2025-10-09 00:05:08,193 [359] ERROR DotNetCore.CAP.RedisStreams.RedisStreamManager - Redis error while creating consumer group [cap.queue.mcms.portal.v1] of stream [MCMS.Portal.SetCustomerFirstOrderUserId] StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=0KiB, 3228ms elapsed, timeout is 3000ms), command=EXISTS, next: EXISTS MCMS.Portal.SetCustomerFirstOrderUserId, inst: 0, qu: 0, qs: 1, aw: False, bw: Inactive, rs: DequeueResult, ws: Idle, in: 0, in-pipe: 4, out-pipe: 0, last-in: 349, cur-in: 4, sync-ops: 0, async-ops: 1151, serverEndpoint: 127.0.0.1:16379, conn-sec: 3048801.54, aoc: 0, mc: 1/1/0, mgr: 9 of 10 available, clientName: DevelopmentServer(SE.Redis-v2.8.58.19568), IOCP: (Busy=0,Free=1000,Min=100,Max=1000), WORKER: (Busy=3,Free=32764,Min=100,Max=32767), POOL: (Threads=10,QueuedItems=0,CompletedItems=235206102,Timers=10), v: 2.8.58.19568 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) at DotNetCore.CAP.RedisStreams.RedisStreamManagerExtensions.TryGetOrCreateStreamConsumerGroupAsync(IDatabase database, RedisKey stream, RedisValue consumerGroup) at DotNetCore.CAP.RedisStreams.RedisStreamManagerExtensions.TryGetOrCreateConsumerGroupPositionsAsync(IDatabase database, StreamPosition[] positions, String consumerGroup, ILogger logger)+MoveNext() 2025-10-09 00:05:08,260 [261] WARN DotNetCore.CAP.Processor.InfiniteRetryProcessor - Processor 'DotNetCore.CAP.Processor.CollectorProcessor' failed. Retrying... MySqlConnector.MySqlException (0x80004005): The Command Timeout expired before the operation completed. ---> System.Net.Sockets.SocketException (125): Operation canceled at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at MySqlConnector.Protocol.Serialization.SocketByteHandler.DoReadBytesAsync(Memory`1 buffer) in /_/src/MySqlConnector/Protocol/Serialization/SocketByteHandler.cs:line 109 at MySqlConnector.Protocol.Serialization.SocketByteHandler.DoReadBytesAsync(Memory`1 buffer) in /_/src/MySqlConnector/Protocol/Serialization/SocketByteHandler.cs:line 109 at MySqlConnector.Protocol.Serialization.BufferedByteReader.ReadBytesAsync(IByteHandler byteHandler, ArraySegment`1 buffer, Int32 totalBytesToRead, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/BufferedByteReader.cs:line 34 at MySqlConnector.Protocol.Serialization.ProtocolUtility.ReadPayloadAsync(BufferedByteReader bufferedByteReader, IByteHandler byteHandler, Func`1 getNextSequenceNumber, ArraySegmentHolder`1 previousPayloads, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/ProtocolUtility.cs:line 421 at MySqlConnector.Core.ServerSession.ReceiveReplyAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 886 at MySqlConnector.Core.ResultSet.ReadResultSetHeaderAsync(IOBehavior ioBehavior) in /_/src/MySqlConnector/Core/ResultSet.cs:line 37 at MySqlConnector.MySqlDataReader.ActivateResultSet(CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 137 at MySqlConnector.MySqlDataReader.InitAsync(CommandListPosition commandListPosition, ICommandPayloadCreator payloadCreator, IDictionary`2 cachedProcedures, IMySqlCommand command, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 482 at MySqlConnector.Core.CommandExecutor.ExecuteReaderAsync(CommandListPosition commandListPosition, ICommandPayloadCreator payloadCreator, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/CommandExecutor.cs:line 56 at MySqlConnector.MySqlCommand.ExecuteNonQueryAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlCommand.cs:line 309 at DotNetCore.CAP.MySql.DbConnectionExtensions.ExecuteNonQueryAsync(DbConnection connection, String sql, DbTransaction transaction, Object[] sqlParams) at DotNetCore.CAP.MySql.DbConnectionExtensions.ExecuteNonQueryAsync(DbConnection connection, String sql, DbTransaction transaction, Object[] sqlParams) at DotNetCore.CAP.MySql.MySqlDataStorage.DeleteExpiresAsync(String table, DateTime timeout, Int32 batchCount, CancellationToken token) at DotNetCore.CAP.MySql.MySqlDataStorage.DeleteExpiresAsync(String table, DateTime timeout, Int32 batchCount, CancellationToken token) at DotNetCore.CAP.Processor.CollectorProcessor.ProcessAsync(ProcessingContext context) at DotNetCore.CAP.Processor.InfiniteRetryProcessor.ProcessAsync(ProcessingContext context) 2025-10-09 11:43:08,255 [385] ERROR MCMS.Portal.Common.GlobalExceptionFilter - URL:http://portal.10g10g.com/UserCommonLocations/GetLocationByIP?ip=::1;UserIP:127.0.0.1,ServerVariables:Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestHeaders;Exception:Input string was not in a correct format. at System.Number.ThrowOverflowOrFormatException(ParsingStatus status, TypeCode type) at System.Int64.Parse(String s) at MCMS.Portal.DAO.Hibernate.IpLocationDAO.<>c.b__3_0(String s) in E:\agent\_work\8\s\MCMS.Portal.DAO.Hibernate\IpLocationDAO.cs:line 83 at System.Linq.Enumerable.SelectArrayIterator`2.ToList() at MCMS.Portal.DAO.Hibernate.IpLocationDAO.IpToInt(String ip) in E:\agent\_work\8\s\MCMS.Portal.DAO.Hibernate\IpLocationDAO.cs:line 83 at MCMS.Portal.DAO.Hibernate.IpLocationDAO.GetByIp(String ip) in E:\agent\_work\8\s\MCMS.Portal.DAO.Hibernate\IpLocationDAO.cs:line 26 at MCMS.Portal.Service.Impl.IpLocationService.GetByIp(String ip) in E:\agent\_work\8\s\MCMS.Portal.Service.Impl\IpLocationService.cs:line 131 at _dynamic_MCMS.Portal.Service.Impl.IpLocationService.GetByIp(Object , Object[] ) at Spring.Reflection.Dynamic.SafeMethod.Invoke(Object target, Object[] arguments) at Spring.Aop.Framework.DynamicMethodInvocation.InvokeJoinpoint() at Spring.Aop.Framework.AbstractMethodInvocation.Proceed() at Spring.Transaction.Interceptor.TransactionInterceptor.Invoke(IMethodInvocation invocation) at Spring.Aop.Framework.AbstractMethodInvocation.Proceed() at Spring.Aop.Framework.DynamicProxy.AdvisedProxy.Invoke(Object proxy, Object target, Type targetType, MethodInfo targetMethod, MethodInfo proxyMethod, Object[] args, IList interceptors) at CompositionAopProxy_5639c95a424c408387b61ea027b16411.GetByIp(String ip) at MCMS.Portal.Service.Impl.UserCommonLocationsService.GetLocationByIP(String ip) at _dynamic_MCMS.Portal.Service.Impl.UserCommonLocationsService.GetLocationByIP(Object , Object[] ) at Spring.Reflection.Dynamic.SafeMethod.Invoke(Object target, Object[] arguments) at Spring.Aop.Framework.DynamicMethodInvocation.InvokeJoinpoint() at Spring.Aop.Framework.AbstractMethodInvocation.Proceed() at Spring.Transaction.Interceptor.TransactionInterceptor.Invoke(IMethodInvocation invocation) at Spring.Aop.Framework.AbstractMethodInvocation.Proceed() at Spring.Aop.Framework.DynamicProxy.AdvisedProxy.Invoke(Object proxy, Object target, Type targetType, MethodInfo targetMethod, MethodInfo proxyMethod, Object[] args, IList interceptors) at CompositionAopProxy_53ef54187ee54ce398832382e2df47d8.GetLocationByIP(String ip) at MCMS.Portal.Controllers.UserCommonLocationsController.GetLocationByIP(String ip) at lambda_method399(Closure , Object , Object[] ) at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync() at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync() --- End of stack trace from previous location --- at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync() --- End of stack trace from previous location --- at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) 2025-10-09 11:43:08,703 [385] ERROR MCMS.Portal.Common.GlobalExceptionFilter - URL:http://portal.10g10g.com/UserCommonLocations/IsCommonLocations?userId=1&ip=::1;UserIP:127.0.0.1,ServerVariables:Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestHeaders;Exception:Input string was not in a correct format. at System.Number.ThrowOverflowOrFormatException(ParsingStatus status, TypeCode type) at System.Int64.Parse(String s) at MCMS.Portal.DAO.Hibernate.IpLocationDAO.<>c.b__3_0(String s) in E:\agent\_work\8\s\MCMS.Portal.DAO.Hibernate\IpLocationDAO.cs:line 83 at System.Linq.Enumerable.SelectArrayIterator`2.ToList() at MCMS.Portal.DAO.Hibernate.IpLocationDAO.IpToInt(String ip) in E:\agent\_work\8\s\MCMS.Portal.DAO.Hibernate\IpLocationDAO.cs:line 83 at MCMS.Portal.DAO.Hibernate.IpLocationDAO.GetByIp(String ip) in E:\agent\_work\8\s\MCMS.Portal.DAO.Hibernate\IpLocationDAO.cs:line 26 at MCMS.Portal.Service.Impl.IpLocationService.GetByIp(String ip) in E:\agent\_work\8\s\MCMS.Portal.Service.Impl\IpLocationService.cs:line 131 at _dynamic_MCMS.Portal.Service.Impl.IpLocationService.GetByIp(Object , Object[] ) at Spring.Reflection.Dynamic.SafeMethod.Invoke(Object target, Object[] arguments) at Spring.Aop.Framework.DynamicMethodInvocation.InvokeJoinpoint() at Spring.Aop.Framework.AbstractMethodInvocation.Proceed() at Spring.Transaction.Interceptor.TransactionInterceptor.Invoke(IMethodInvocation invocation) at Spring.Aop.Framework.AbstractMethodInvocation.Proceed() at Spring.Aop.Framework.DynamicProxy.AdvisedProxy.Invoke(Object proxy, Object target, Type targetType, MethodInfo targetMethod, MethodInfo proxyMethod, Object[] args, IList interceptors) at CompositionAopProxy_5639c95a424c408387b61ea027b16411.GetByIp(String ip) at MCMS.Portal.Service.Impl.UserCommonLocationsService.GetLocationByIP(String ip) at MCMS.Portal.Service.Impl.UserCommonLocationsService.IsCommonLocations(Int32 userId, String ip) at _dynamic_MCMS.Portal.Service.Impl.UserCommonLocationsService.IsCommonLocations(Object , Object[] ) at Spring.Reflection.Dynamic.SafeMethod.Invoke(Object target, Object[] arguments) at Spring.Aop.Framework.DynamicMethodInvocation.InvokeJoinpoint() at Spring.Aop.Framework.AbstractMethodInvocation.Proceed() at Spring.Transaction.Interceptor.TransactionInterceptor.Invoke(IMethodInvocation invocation) at Spring.Aop.Framework.AbstractMethodInvocation.Proceed() at Spring.Transaction.Interceptor.TransactionInterceptor.Invoke(IMethodInvocation invocation) at Spring.Aop.Framework.AbstractMethodInvocation.Proceed() at Spring.Aop.Framework.DynamicProxy.AdvisedProxy.Invoke(Object proxy, Object target, Type targetType, MethodInfo targetMethod, MethodInfo proxyMethod, Object[] args, IList interceptors) at CompositionAopProxy_53ef54187ee54ce398832382e2df47d8.IsCommonLocations(Int32 userId, String ip) at MCMS.Portal.Controllers.UserCommonLocationsController.IsCommonLocations(Int32 userId, String ip) at lambda_method408(Closure , Object , Object[] ) at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync() at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync() --- End of stack trace from previous location --- at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync() --- End of stack trace from previous location --- at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)