redis避坑指南

场景1：使用 StackExchange.Redis 客户端出现 timeout 问题

• 症状：开发反馈，使用StackExchange.Redis遇到如下timeout问题：

Unhandled exception. StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=0KiB, 6894ms elapsed, timeout is 5000ms), command=SET, next: SET lms:api:assement:920850957-94280756, inst: 0, qu: 0, qs: 1, aw: False, rs: ReadAsync, ws: Idle, in: 5, in-pipe: 0, out-pipe: 0, serverEndpoint: r-uf6zi628isi0e76pb7.redis.rds.aliyuncs.com:6379, mc: 1/1/0, mgr: 10 of 10 available, clientName: aiservernew-api, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=16,Free=32751,Min=1,Max=32767), v: 2.2.4.27433 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

• 分析1：查看redis(1G读写分离版)连接数只有1.2，应该看代理节点(而非数据节点)的连接数(300+)
• 分析2：修改同步过期时间 SyncTimeout=5000，连接过期时间 ConnectTimeout=15000，响应过期时间 ResponseTimeout=15000，无效

var config = new ConfigurationOptions
    {
        AbortOnConnectFail = false,
        AllowAdmin = true,
        ConnectTimeout = 15000,
        SyncTimeout = 5000,
        ResponseTimeout = 15000,
        Password = "***", 
        EndPoints = { "192.168.2.110:6379" }
    };
    var connect = ConnectionMultiplexer.Connect(config);

• 解决：StackExchange.Redis组件有超时的bug，更换为CSRedisCore解决；

参考：StackExchange.Redis issues 及 使用StackExchange.Redis客户端进行Redis访问出现的Timeout异常排查

场景2：开发反馈 db0 问题

• 症状：Redis 5.0 社区版 1G读写分离版（1节点1只读），开发反馈有报错 Multiple databases are not supported，详细日志如下：

"exception": "StackExchange.Redis.RedisConnectionException: InternalFailure on GET appserver:api:uatupcServer:token\n 
---> StackExchange.Redis.RedisCommandException: Multiple databases are not supported on this server; cannot switch to database: 2\n 
at StackExchange.Redis.PhysicalConnection.GetSelectDatabaseCommand(Int32 targetDatabase, Message message) in /_/src/StackExchange.Redis/PhysicalConnection.cs:line 550\n   
at StackExchange.Redis.PhysicalBridge.SelectDatabaseInsideWriteLock(PhysicalConnection connection, Message message) in /_/src/StackExchange.Redis/PhysicalBridge.cs:line 1185\n   
at StackExchange.Redis.PhysicalBridge.WriteMessageToServerInsideWriteLock(PhysicalConnection connection, Message message) in /_/src/StackExchange.Redis/PhysicalBridge.cs:line 1303\n
--- End of inner exception stack trace ---\n

• 解决：将db2换成db0则解决，这也是为什么建议只使用db0的原因

场景3：maxmemory配置不一致而导致数据丢失

• 例如master的maxmemory配置为4G，而slave配置2G，这个时候主从复制可以成功，但如果在进行某一次全量复制的时候，slave拿到master的RDB加载数据时发现自身的2G内存不够用，这时就会触发slave的maxmemory策略，将数据进行淘汰。更可怕的是，在高可用的集群环境下，如果将这台slave升级成master的时候，就会发现数据已经丢失了。

场景4：Redis command timed out

• 症状：收到开发反馈，某Java业务收到 Redis command timed out 的报错，详细日志如下：

2023-06-24 09:32:47.470 [http-nio-8097-exec-42] ERROR c.k.m.c.BaseControllerAdvice - 内部错误11: 
org.springframework.dao.QueryTimeoutException: Redis command timed out; nested exception is io.lettuce.core.RedisCommandTimeoutException: Command timed out
    at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:70)
    at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:41)
    at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44)
    at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:42)
    at org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException(LettuceConnection.java:257)
    at org.springframework.data.redis.connection.lettuce.LettuceStringCommands.convertLettuceAccessException(LettuceStringCommands.java:718)
    at org.springframework.data.redis.connection.lettuce.LettuceStringCommands.get(LettuceStringCommands.java:63)
    at org.springframework.data.redis.connection.DefaultedRedisConnection.get(DefaultedRedisConnection.java:210)
    at org.springframework.data.redis.core.DefaultValueOperations$1.inRedis(DefaultValueOperations.java:52)
    at org.springframework.data.redis.core.AbstractOperations$ValueDeserializingRedisCallback.doInRedis(AbstractOperations.java:59)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:224)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:184)
    at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:95)
    at org.springframework.data.redis.core.DefaultValueOperations.get(DefaultValueOperations.java:48)
    at com.kidstools.minhang.utils.RedisStringUtil.get(RedisStringUtil.java:106)
    at com.kidstools.minhang.service.GrowthBookOrderService.getPrintData(GrowthBookOrderService.java:1005)
    at com.kidstools.m

• 分析：查看监控，确认 redis 服务器的内存、cpu 均无压力
• 解决：登录部署java业务的服务器，找到 application.properties 调整如下配置（主要是 spring.redis.timeout）：

spring.redis.database=6
spring.redis.host=127.0.0.1
spring.redis.port=6379
spring.redis.password=Admin_147
spring.redis.pool.max-active=200
spring.redis.pool.max-wait=-1
spring.redis.pool.max-idle=8
spring.redis.pool.min-idle=0
spring.redis.timeout=10000
#spring.redis.expiration=3600