基于NFS存储与StatefulSet在K8s上构建高可用Redis集群（3主3从）

1. 为什么选择NFSStatefulSet部署Redis集群在Kubernetes中部署有状态服务时StatefulSet是官方推荐的标准方案。相比Deployment它提供了三个关键特性稳定的网络标识、有序的部署扩缩容和持久化存储绑定。当这些特性与NFS存储结合时就构成了生产级Redis集群的理想组合。我曾在多个项目中尝试过不同的存储方案最终发现NFS有三大不可替代的优势跨节点数据共享当Pod发生漂移时新调度的节点能立即访问原有数据运维成本低不需要维护复杂的分布式存储系统性能可预期通过内核级缓存优化实测读写吞吐量可达800MB/sRedis集群的3主3从架构则是经过验证的黄金比例。这种配置下每个主节点管理约5461个哈希槽任意单节点故障时自动触发故障转移读写分离使整体QPS提升2-3倍2. 环境准备与NFS配置2.1 NFS服务器搭建以Ubuntu 20.04为例执行以下命令安装NFS服务端sudo apt update sudo apt install nfs-kernel-server -y创建专用于Redis的共享目录建议使用独立磁盘分区sudo mkdir -p /nfs/redis/pv{1..6} sudo chmod -R 777 /nfs/redis # 生产环境应配置更精细的权限编辑/etc/exports配置文件添加以下内容/nfs/redis/pv1 *(rw,sync,no_subtree_check,no_root_squash) /nfs/redis/pv2 *(rw,sync,no_subtree_check,no_root_squash) /nfs/redis/pv3 *(rw,sync,no_subtree_check,no_root_squash) /nfs/redis/pv4 *(rw,sync,no_subtree_check,no_root_squash) /nfs/redis/pv5 *(rw,sync,no_subtree_check,no_root_squash) /nfs/redis/pv6 *(rw,sync,no_root_squash)应用配置并验证sudo exportfs -arv showmount -e localhost2.2 Kubernetes节点配置所有Worker节点需要安装NFS客户端工具sudo apt install nfs-common -y测试挂载可用性以node1为例mkdir -p /mnt/test mount -t nfs NFS_SERVER_IP:/nfs/redis/pv1 /mnt/test dd if/dev/zero of/mnt/test/testfile bs1M count1024 umount /mnt/test3. 创建持久化存储资源3.1 定义StorageClass创建redis-storageclass.yamlapiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: redis-nfs provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer3.2 创建PV资源编写redis-pv.yaml注意每个PV要对应独立的NFS路径apiVersion: v1 kind: PersistentVolume metadata: name: redis-pv1 spec: capacity: storage: 10Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: redis-nfs nfs: path: /nfs/redis/pv1 server: NFS_SERVER_IP --- # 重复创建pv2-pv6...应用配置kubectl apply -f redis-storageclass.yaml kubectl apply -f redis-pv.yaml验证PV状态kubectl get pv -o wide4. 配置Redis集群参数4.1 创建ConfigMapredis-configmap.yaml包含两个关键文件apiVersion: v1 kind: ConfigMap metadata: name: redis-cluster-config data: redis.conf: | cluster-enabled yes cluster-config-file /data/nodes.conf cluster-node-timeout 5000 appendonly yes dir /data protected-mode no update-node-ip.sh: | #!/bin/sh sed -i -e /myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/ /data/nodes.conf exec $这个配置有几个精妙之处cluster-node-timeout设为5秒平衡了故障检测速度和误报概率update-node-ip.sh脚本解决Pod重建后IP变化的问题禁用protected-mode允许集群内部通信4.2 创建Headless Serviceredis-service.yaml定义无头服务apiVersion: v1 kind: Service metadata: name: redis-cluster spec: clusterIP: None ports: - name: client port: 6379 - name: gossip port: 16379 selector: app: redis-cluster5. 部署StatefulSet5.1 编写StatefulSet配置redis-statefulset.yaml核心配置要点apiVersion: apps/v1 kind: StatefulSet metadata: name: redis-cluster spec: serviceName: redis-cluster replicas: 6 selector: matchLabels: app: redis-cluster template: metadata: labels: app: redis-cluster spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: [redis-cluster] topologyKey: kubernetes.io/hostname containers: - name: redis image: redis:7.2-alpine command: [/conf/update-node-ip.sh, redis-server, /conf/redis.conf] ports: - containerPort: 6379 name: client - containerPort: 16379 name: gossip env: - name: POD_IP valueFrom: fieldRef: fieldPath: status.podIP volumeMounts: - name: conf mountPath: /conf - name: data mountPath: /data volumes: - name: conf configMap: name: redis-cluster-config defaultMode: 0755 volumeClaimTemplates: - metadata: name: data spec: accessModes: [ ReadWriteOnce ] storageClassName: redis-nfs resources: requests: storage: 10Gi关键优化点使用PodAntiAffinity确保Pod分散在不同节点Alpine镜像体积仅30MB启动速度更快通过volumeClaimTemplate自动绑定PV5.2 部署与验证执行部署命令kubectl apply -f redis-configmap.yaml kubectl apply -f redis-service.yaml kubectl apply -f redis-statefulset.yaml检查Pod状态等待所有Pod变为Runningwatch kubectl get pods -l appredis-cluster6. 初始化Redis集群6.1 手动初始化方案进入第一个Pod执行集群初始化kubectl exec -it redis-cluster-0 -- redis-cli --cluster create \ $(kubectl get pods -l appredis-cluster -o jsonpath{range.items[*]}{.status.podIP}:6379 ) \ --cluster-replicas 1当提示确认配置时输入yes。这个过程实际上完成了自动分配哈希槽0-5460, 5461-10922, 10923-16383为每个主节点分配从节点生成集群配置文件6.2 自动化初始化方案对于生产环境建议使用Job资源实现自动化初始化。创建redis-cluster-init.yamlapiVersion: batch/v1 kind: Job metadata: name: redis-cluster-init spec: template: spec: containers: - name: cluster-init image: redis:7.2-alpine command: [/bin/sh, -c] args: - until nc -z redis-cluster-0.redis-cluster 6379; do echo Waiting for Redis...; sleep 2; done; redis-cli --cluster create redis-cluster-0.redis-cluster:6379 redis-cluster-1.redis-cluster:6379 redis-cluster-2.redis-cluster:6379 redis-cluster-3.redis-cluster:6379 redis-cluster-4.redis-cluster:6379 redis-cluster-5.redis-cluster:6379 --cluster-replicas 1 --cluster-yes restartPolicy: OnFailure7. 验证与监控7.1 基础验证检查集群状态kubectl exec -it redis-cluster-0 -- redis-cli cluster info查看节点角色分布for i in {0..5}; do echo redis-cluster-$i: $(kubectl exec redis-cluster-$i -- redis-cli role) done7.2 故障转移测试模拟主节点故障kubectl delete pod redis-cluster-0 # 假设这是主节点 watch kubectl exec redis-cluster-1 -- redis-cli cluster nodes你会观察到约5秒后检测到节点失效从节点自动晋升为主节点原主节点恢复后变为从节点7.3 性能监控建议配置Prometheus监控指标- job_name: redis-cluster metrics_path: /metrics static_configs: - targets: - redis-cluster-0.redis-cluster:9121 - redis-cluster-1.redis-cluster:9121 # ...其他节点关键监控指标包括redis_connected_clients客户端连接数redis_memory_used_bytes内存使用量redis_commands_processed_totalQPSredis_cluster_state集群健康状态8. 常见问题排查问题1Pod启动后立即崩溃检查点NFS挂载权限是否正确redis.conf是否有语法错误存储空间是否已满问题2集群节点无法发现彼此解决方案kubectl exec -it redis-cluster-0 -- redis-cli --cluster fix $(hostname -i):6379问题3数据不同步修复命令kubectl exec -it redis-cluster-0 -- redis-cli --cluster check $(hostname -i):6379我在实际运维中发现90%的问题都源于网络策略或存储权限配置错误。建议在正式上线前用chaos-mesh等工具模拟网络分区和节点故障场景。