redis哨兵搭建
安装 redis 单节点
- [所有节点] 快速安装 Redis 7.0.11,示例 数据目录
/data/redis_${port},端口7100
cd /opt/
wget -c http://iso.sqlfans.cn/redis/redis-7.0.11.tar.gz
wget -c http://iso.sqlfans.cn/redis/install_redis_7011.sh
bash install_redis_7011.sh /data 7100
echo "set dba kevin" | /usr/local/bin/redis-cli -a RbY9k2_NBf1QWy8I -c -p 7100 2>/dev/null
echo "get dba" | /usr/local/bin/redis-cli -a RbY9k2_NBf1QWy8I -c -p 7100 2>/dev/null
echo "save" | /usr/local/bin/redis-cli -a RbY9k2_NBf1QWy8I -c -p 7100 2>/dev/null
echo "info server" | /usr/local/bin/redis-cli -a RbY9k2_NBf1QWy8I -c -p 7100 2>/dev/null | grep redis_version
搭建 redis 哨兵
| IP地址 |
角色 |
redis端口 |
sentinel端口 |
操作系统 |
数据目录 |
| 192.168.31.101 |
主库1 |
7100 |
7200 |
CentOS 7.9 x64 |
/data/redis_7100 |
| 192.168.31.102 |
从库2 |
7100 |
7200 |
CentOS 7.9 x64 |
/data/redis_7100 |
| 192.168.31.103 |
从库3 |
7100 |
7200 |
CentOS 7.9 x64 |
/data/redis_7100 |
1.安装 redis 单节点
- 所有节点:关于 redis 的单节点安装,可参考上面
- 所有节点:关于 redis 的配置,可参考
cat /data/redis_7100/redis_7100.conf
2.搭建 redis 主从
- 2.1.主节点:添加 masterauth 密码(与
requirepass保持一致),主库不用配置replicaof
cat /data/redis_7100/redis_7100.conf | grep "^masterauth" || echo "masterauth RbY9k2_NBf1QWy8I" >> /data/redis_7100/redis_7100.conf
- 2.2.从节点:添加 masterauth 及 replicaof 参数,指定master的ip(示例
192.168.31.101)和port(示例7100)
cat /data/redis_7100/redis_7100.conf | grep "^masterauth" || echo "masterauth RbY9k2_NBf1QWy8I" >> /data/redis_7100/redis_7100.conf
cat /data/redis_7100/redis_7100.conf | grep "^replicaof" || echo "replicaof 192.168.31.101 7100" >> /data/redis_7100/redis_7100.conf
- 2.3.所有节点:重启redis服务,并确认主从状态
#.不可禁用CONFIG命令否则会导致哨兵无法故障转移
sed -i "s/^rename-command CONFIG/#rename-command CONFIG/" /data/redis_7100/redis_7100.conf
/usr/local/bin/redis-cli -a RbY9k2_NBf1QWy8I -h 127.0.0.1 -p 7100 shutdown
sudo -u redis /usr/local/bin/redis-server /data/redis_7100/redis_7100.conf
echo "info replication" | /usr/local/bin/redis-cli -a RbY9k2_NBf1QWy8I -c -p 7100 2>/dev/null | egrep "(role|slave0|slave1)"
3.搭建 reidis 哨兵
- 3.1.所有节点:配置哨兵,
sentinel monitor指定主节点信息,mymaster指定主节点的名称(自定义),192.168.31.101 7100指定主节点的ip和端口,2指定选举master时的quorum值
cat > /data/redis_7100/sentinel_7200.conf <<EOF
bind 0.0.0.0
port 7200
daemonize yes
supervised systemd
logfile /data/redis_7100/log/sentinel.log
pidfile /data/redis_7100/pid/sentinel.pid
dir /data/redis_7100/dump
sentinel monitor mymaster 192.168.31.101 7100 2
sentinel auth-pass mymaster RbY9k2_NBf1QWy8I
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 10000
EOF
chown -R redis.redis /data/redis_7100
chmod 600 /data/redis_7100/*.conf
sudo -u redis /usr/local/bin/redis-sentinel /data/redis_7100/sentinel_7200.conf
netstat -lntp | egrep "(7100|7200)"
cat /etc/rc.local | grep sentinel_7200 || echo "sudo -u redis /usr/local/bin/redis-sentinel /data/redis_7100/sentinel_7200.conf" >> /etc/rc.local
4.验证集群
echo "info replication" | /usr/local/bin/redis-cli -a RbY9k2_NBf1QWy8I -c -p 7100 2>/dev/null | egrep "(role|slave0|slave1)"
echo "set dev sam" | /usr/local/bin/redis-cli -a RbY9k2_NBf1QWy8I -c -p 7100 2>/dev/null
echo "get dev" | /usr/local/bin/redis-cli -a RbY9k2_NBf1QWy8I -c -p 7100 2>/dev/null
- 4.2.从节点:确认集群状态 + 只读状态 + 数据同步
echo "info replication" | /usr/local/bin/redis-cli -a RbY9k2_NBf1QWy8I -c -p 7100 2>/dev/null | egrep "(role|slave0|slave1)"
echo "get dev" | /usr/local/bin/redis-cli -a RbY9k2_NBf1QWy8I -c -p 7100 2>/dev/null
echo "set dev sam" | /usr/local/bin/redis-cli -a RbY9k2_NBf1QWy8I -c -p 7100 2>/dev/null
/usr/local/bin/redis-cli -c -h 192.168.31.101 -p 7200 info sentinel | grep master
/usr/local/bin/redis-cli -c -h 192.168.31.102 -p 7200 info sentinel | grep master
/usr/local/bin/redis-cli -c -h 192.168.31.103 -p 7200 info sentinel | grep master
5.灾难演练 - 模拟主库宕机
- 5.1.主节点:将redis进程杀掉,模拟主库宕机
ps -ef | grep redis | grep -v grep | awk '{print $2}' | xargs kill -9 2> /dev/null
echo "info replication" | /usr/local/bin/redis-cli -a RbY9k2_NBf1QWy8I -c -h 192.168.31.101 -p 7100 2>/dev/null | egrep "(role|slave0|slave1)"
echo "info replication" | /usr/local/bin/redis-cli -a RbY9k2_NBf1QWy8I -c -h 192.168.31.102 -p 7100 2>/dev/null | egrep "(role|slave0|slave1)"
echo "info replication" | /usr/local/bin/redis-cli -a RbY9k2_NBf1QWy8I -c -h 192.168.31.103 -p 7100 2>/dev/null | egrep "(role|slave0|slave1)"
- 5.3.任意从节点:查看哨兵的故障转移过程,总结:主节点宕机会触发选举,并自动故障转移
cat /data/redis_7100/log/sentinel.log | egrep "(+monitor|+sentinel|+sdown|+vote-for-leader|+odown|+config-update-from|+switch-master|+slave)" | tail -n20
注:附上一个完整的故障转移的demo日志:curl http://iso.sqlfans.cn/redis/switch_master_log.txt
- 5.4.故障节点:测试完毕,再将redis及哨兵启动
sudo -u redis /usr/local/bin/redis-server /data/redis_7100/redis_7100.conf
sudo -u redis /usr/local/bin/redis-sentinel /data/redis_7100/sentinel_7200.conf
netstat -lntp | egrep "(7100|7200)"
6.灾难演练 - 模拟从库宕机
- 6.1.任意从节点(假设
reids03):将redis进程杀掉,模拟从库宕机
ps -ef | grep redis | grep -v grep | awk '{print $2}' | xargs kill -9 2> /dev/null
echo "info replication" | /usr/local/bin/redis-cli -a RbY9k2_NBf1QWy8I -c -h 192.168.31.101 -p 7100 2>/dev/null | egrep "(role|slave0|slave1)"
echo "info replication" | /usr/local/bin/redis-cli -a RbY9k2_NBf1QWy8I -c -h 192.168.31.102 -p 7100 2>/dev/null | egrep "(role|slave0|slave1)"
echo "info replication" | /usr/local/bin/redis-cli -a RbY9k2_NBf1QWy8I -c -h 192.168.31.103 -p 7100 2>/dev/null | egrep "(role|slave0|slave1)"
- 6.3.主节点(假设
reids01):主库写入数据,其他从库(假设reids02)正常同步,总结:单个从库宕机不影响哨兵同步
echo "set dev2 sam2" | /usr/local/bin/redis-cli -a RbY9k2_NBf1QWy8I -c -h 192.168.31.101 -p 7100 2>/dev/null
echo "get dev2" | /usr/local/bin/redis-cli -a RbY9k2_NBf1QWy8I -c -h 192.168.31.102 -p 7100 2>/dev/null
- 6.4.故障节点:测试完毕,再将redis及哨兵启动
sudo -u redis /usr/local/bin/redis-server /data/redis_7100/redis_7100.conf
sudo -u redis /usr/local/bin/redis-sentinel /data/redis_7100/sentinel_7200.conf
netstat -lntp | egrep "(7100|7200)"
遇到的问题
场景1:如何彻底卸载redis
- 参考redis的安装过程,可按照如下步骤彻底卸载redis
cd /opt/
ps -ef | grep redis | grep -v grep | awk '{print $2}' | xargs kill -9 2> /dev/null
userdel -r redis 2> /dev/null
rm -f /opt/install_redis_*.sh
rm -rf /data/redis*
rm -rf /usr/local/redis*
rm -f /usr/local/bin/redis*
sed -i '/redis/d' /etc/rc.local
netstat -lnpt | grep redis
场景2:nacos如何配置redis连接串
- 如果redis为单机、主备、读写分离、Proxy集群实例,那么在nacos配置的时候,请使用redis服务的ip和端口
spring.redis.host=192.168.31.101
spring.redis.port=7100
spring.redis.password=<password>
- 如果redis为哨兵模式,那么在nacos配置的时候,请使用哨兵服务的ip和端口
spring.redis.sentinel.master=mymaster
spring.redis.sentinel.nodes=192.168.31.101:7200,192.168.31.102:7200,192.168.31.103:7200
spring.redis.password=<password>
spring.redis.cluster.nodes=<ip1:port1>,<ip2:port2>,<ip3:port3>,<ip4:port4>,<ip5:port5>,<ip6:port6>
spring.redis.password=<password>
场景3:主节点宕机不会发生故障转移
- 症状:2023.07.21,将主节点的redis进程杀掉,模拟主库宕机,但redis哨兵不会发生故障转移,执行
info replication 看到2个从节点仍然是slave
- 原因:由于最佳化安装的时候,配置了
rename-command CONFIG "" 禁止修改配置项,从而导致主机点宕机的时候无法发生故障转移
- 解决:将禁用CONFIG的这一行注释掉并重启redis服务即可,即
#rename-command CONFIG ""
场景4:不要混淆哨兵密码与业务密码
- 很多人混淆:哨兵程序自身密码 vs 哨兵访问 Redis 主从节点的密码。下面是各种密码的对照表:
| 配置示例 |
配置文件 |
作用 |
密码设置 |
| requirepass 123 |
redis.conf (主 / 从) |
客户端访问 redis 数据节点(业务密码) |
全局认证密码 |
| masterauth 123 |
redis.conf (从库) |
从节点连接主库进行复制 |
必须等于 redis requirepass |
| sentinel auth‑pass 123 |
sentinel.conf (所有节点) |
哨兵去访问 redis 主从节点 |
必须等于 redis requirepass |
| requirepass 456 |
sentinel.conf (所有节点) |
访问哨兵端口、哨兵间通信(哨兵密码) |
可不一样,但所有哨兵要统一 |
- 总结:连 redis 数据端口,用 redis 业务密码;连哨兵端口,用哨兵密码
- 只改 redis 的 requirepass,忘记改sentinel auth‑pass,哨兵报NOAUTH,判定主节点主观下线
- 只改一个哨兵的 auth‑pass,其余哨兵没改,集群分裂
- 混淆哨兵的 requirepass 和 sentinel auth‑pass,连接哨兵用 redis 业务密码报认证失败
- Redis 6.2+ ACL 模式:不再使用全局 requirepass,由于哨兵本身也是一个 Redis 客户端,可以指定 ACL 用户名密码,代替全局密码方式
# 创建哨兵用户,赋予必要的命令权限
ACL SETUSER dba_sentinel on >Admin_147 -@all +subscribe +publish +client|setname +client|kill +config|rewrite +role +exec +slaveof +info +multi +ping +script|kill
# 在哨兵配置文件中启用认证:告诉哨兵,连接名为 mymaster 的主节点时,使用账号 dba_sentinel 及其密码
sentinel auth-user mymaster dba_sentinel
sentinel auth-pass mymaster Admin_147
- 注意,由于哨兵配置文件中自动生成的配置部分,显式 ACL 语句 user default on nopass 优先级高于 requirepass,会把 default 用户设置成无密码,导致上面写的 requirepass Admin_147 失效
root@localhost:~# cat /data/redis_7100/sentinel_7200.conf
...
requirepass Admin_147
# Generated by CONFIG REWRITE
...
user default on nopass ~* &* +@all