redis集群搭建

发表于： 2020年2月3日 2020年2月3日
分类： Redis

一、准备工具

安装docker（来自官网）

(1) 安装所需的软件包

$ sudo yum install -y yum-utils \ device-mapper-persistent-data \ lvm2
复制代码

(2) 使用以下命令来设置稳定的存储库

$ sudo yum-config-manager \ --add-repo \ https://download.docker.com/linux/centos/docker-ce.repo
复制代码

(3) 安装docker ce(docker社区版)

$ sudo yum install docker-ce
复制代码

(4) 启动docker

$ sudo systemctl start docker
复制代码

在docker库获取redis镜像（截至2019-03-27，最新版为5.0.4）

$ docker pull redis
复制代码

至此，docker上redis cluster所有工具准备完毕，我们在命令行上输入docker images，就可以查看到已经安装的镜像

二、集群搭建

创建Redis 容器

#####（1）创建redis配置文件（redis-cluster.tmpl）

我在路径/home下创建一个文件夹redis-cluster,在路径/home/redis-cluster下创建一个文件redis-cluster.tmpl，并把以下内容复制过去。（注：路径可自定义，我用的是/home/redis-cluster）

port ${PORT}                                       ##节点端口
protected-mode no                                  ##开启集群模式
cluster-enabled yes                                ##cluster集群模式
cluster-config-file nodes-${PORT}.conf 
                     ##集群配置名
cluster-node-timeout 5000                          ##超时时间
cluster-announce-ip 192.168.XX.XX                  ##实际为各节点网卡分配ip  先用上网关ip代替
cluster-announce-port ${PORT}                      ##节点映射端口
cluster-announce-bus-port 1${PORT}                 ##节点总线端口
appendonly yes                                     ##持久化模式
复制代码

通过命令，可查看

[root@node-01 redis-cluster]# cat /home/redis-cluster/redis-cluster.tmpl
port ${PORT}
protected-mode no
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
cluster-announce-ip 192.168.1.221
cluster-announce-port ${PORT}
cluster-announce-bus-port 1${PORT}
appendonly yes
复制代码

备注：此模版文件为集群节点通用文件其中${PORT} 将读取命令行变量 ip则根据网卡分配ip进行替换以保证节点配置文件除端口以及ip 全部一致。

(2) 创建自定义network

$ docker network create redis-net
复制代码

docker network ls ##查看网卡信息

[root@node-01 conf]# docker network ls 
NETWORK ID NAME DRIVER SCOPE 7f804fa5fbd2 redis-net bridge local
复制代码

备注：创建redis-net虚拟网卡目的是让docker容器能与宿主（centos7）桥接网络并间接与外界连接

(3) 查看redis-net虚拟网卡网关ip

[root@node-01 conf]# docker network inspect redis-net | grep "Gateway" | grep --color=auto -P '(\d{1,3}.){3}\d{1,3}' -o 172.21.0.1
复制代码

备注：docker network inspect network-name 显示 network-name对应配置信息（gerp 过滤网关配置行并筛选ip ）可人工

(4) 在/home/redis-cluster下生成conf和data目标，并生成配置信息

 $ for port in `seq 7000 7005`; do \
  mkdir -p ./${port}/conf \
  && PORT=${port} envsubst < ./redis-cluster.tmpl > ./${port}/conf/redis.conf \
  && mkdir -p ./${port}/data; \
  done
复制代码

共生成6个文件夹，从7000到7005，每个文件夹下包含data和conf文件夹，同时conf里面有redis.conf配置文件

(5) 创建6个redis容器

$ ip=2
for port in `seq 7000 7005`; do \
docker run -d -p ${port}:${port} -p 1${port}:1${port} \
-v /Users/kevin/Docker/redis-cluster/${port}/conf/redis.conf:/usr/local/etc/redis/redis.conf \
-v /Users/kevin/Docker/redis-cluster/${port}/data:/data \
--restart always --name redis-${port} --net redis-net --ip 172.20.0.${ip} \
 redis redis-server /usr/local/etc/redis/redis.conf; \
 let ip++;
done

复制代码

备注：命令译为  循环7010 - 7015  运行redis 容器
docker  run            运行
-d                          守护进程模式
--restart always     保持容器启动
--name redis-710* 容器起名
--net redis-net    容器使用虚拟网卡
-p                        指定宿主机器与容器端口映射 701*:701*
-P                        指定宿主机与容器redis总线端口映射 1701*:1701*
--privileged=true -v /home/redis-cluster/701*/conf/redis.conf:/usr/local/etc/redis/redis.conf
付权将宿主701*节点文件挂载到容器/usr/local/etc/redis/redis.conf 文件中
--privileged=true -v /home/redis-cluster/${port}/data:/data \
付权将宿主701*/data目录挂载到容器/data目录中
--sysctl net.core.somaxconn=1024 redis redis-server /usr/local/etc/redis/redis.conf;
容器根据挂载的配置文件启动 redis服务端
复制代码

6、通过命令docker ps可查看刚刚生成的6个容器信息

[root@node-01 redis-cluster]# docker ps
CONTAINER ID        IMAGE                                         COMMAND                  CREATED             STATUS                PORTS                                                              NAMES
15c479074b87        redis                                         "docker-entrypoint..."   10 seconds ago      Up 9 seconds          0.0.0.0:7005->7005/tcp, 6379/tcp, 0.0.0.0:17005->17005/tcp         redis-7005
45bec33c2c35        redis                                         "docker-entrypoint..."   11 seconds ago      Up 10 seconds         0.0.0.0:7004->7004/tcp, 6379/tcp, 0.0.0.0:17004->17004/tcp         redis-7004
45482d9e5bb8        redis                                         "docker-entrypoint..."   11 seconds ago      Up 10 seconds         0.0.0.0:7003->7003/tcp, 6379/tcp, 0.0.0.0:17003->17003/tcp         redis-7003
f633d5c767c9        redis                                         "docker-entrypoint..."   11 seconds ago      Up 10 seconds         0.0.0.0:7002->7002/tcp, 6379/tcp, 0.0.0.0:17002->17002/tcp         redis-7002
eefc0d49fedf        redis                                         "docker-entrypoint..."   11 seconds ago      Up 10 seconds         0.0.0.0:7001->7001/tcp, 6379/tcp, 0.0.0.0:17001->17001/tcp         redis-7001
58b311e5dbcb        redis                                         "docker-entrypoint..."   12 seconds ago      Up 11 seconds         0.0.0.0:7000->7000/tcp, 6379/tcp, 0.0.0.0:17000->17000/tcp         redis-7000  
复制代码

7、查看容器分配ip

[root@node-01 conf]# docker network inspect redis-net
[
{
    "Name": "redis-net",
    "Id": "7f804fa5fbd2473f1e6e9783ac52aa980c355e3253b15a28e10f8793c46fc39b",
    "Created": "2019-03-27T12:59:14.45569686+08:00",
    "Scope": "local",
    "Driver": "bridge",
    "EnableIPv6": false,
    "IPAM": {
        "Driver": "default",
        "Options": {},
        "Config": [
            {
                "Subnet": "172.21.0.0/16",
                "Gateway": "172.21.0.1"
            }
        ]
    },
    "Internal": false,
    "Attachable": false,
    "Containers": {
        "449c38916b4ef919d1b1828da3f69e0f5256eb2023204f25f69fbb8c449f2163": {
            "Name": "redis-7005",
            "EndpointID": "5c89e6fdd0e983d6d2b950c9bc41616920265dda93f2852650ca5e9da1da22b1",
            "MacAddress": "02:42:ac:15:00:07",
            "IPv4Address": "172.21.0.7/16",
            "IPv6Address": ""
        },
        "600aa2f84dfbe72a19caf5dc684cb7691e89a43753f70d74355468b7659caa23": {
            "Name": "redis-7000",
            "EndpointID": "75873081222c7a2eedc5f3ee639df5cc7645790012dd72e34c97fd97dcb5cf47",
            "MacAddress": "02:42:ac:15:00:02",
            "IPv4Address": "172.21.0.2/16",
            "IPv6Address": ""
        },
        "78aeae8b2d381d3789848f45f766dd01d7c0d5443505920decf41fce751f765d": {
            "Name": "redis-7002",
            "EndpointID": "a70d816cd9e3650b1d10507a85fa85c0c2e143ba99c0974fb000ec87ad5b173d",
            "MacAddress": "02:42:ac:15:00:04",
            "IPv4Address": "172.21.0.4/16",
            "IPv6Address": ""
        },
        "7ecd2e1372a2bc15cb89fad856ab410757b49936f662ce170990c04e1861037d": {
            "Name": "redis-7004",
            "EndpointID": "f8cdb522601e9667ec06dc0dc0d39b46331b3d24cbea1155b00b73e600eca8c7",
            "MacAddress": "02:42:ac:15:00:06",
            "IPv4Address": "172.21.0.6/16",
            "IPv6Address": ""
        },
        "bd8254045c66bc6569c9a585b58cb028e864b6c7f16d836009debd3a3350913c": {
            "Name": "redis-7003",
            "EndpointID": "6a25ff0046e6d6575579cdf08bccbb2a62b29460e3932de3f8e476bb908fdc9c",
            "MacAddress": "02:42:ac:15:00:05",
            "IPv4Address": "172.21.0.5/16",
            "IPv6Address": ""
        },
        "cfef6ca3aeea9d3b4fa527538fbe2a2f79aac25d52921b4edc31839b6f330887": {
            "Name": "redis-7001",
            "EndpointID": "98a36b61bc6f50f78444a6cefa5e9999f5366e72425602511bc95f602ef0dc93",
            "MacAddress": "02:42:ac:15:00:03",
            "IPv4Address": "172.21.0.3/16",
            "IPv6Address": ""
        }
    },
    "Options": {},
    "Labels": {}
}
]
复制代码

备注：每个容器对应分配了ip 需要修改节点配置文件redis.conf与其对应并重启容器。

(8) 修改宿主挂载目录配置文件

vim /home/redis-cluster/700*/conf/redis.conf ##编辑ip为查询分配的ip 并esc+：wq退出

port 7000
protected-mode no
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
cluster-announce-ip 172.21.0.2
cluster-announce-port 7000
cluster-announce-bus-port 17000
appendonly yes
复制代码

启动集群

进入一个节点

docker exec -it redis-7000 bash
复制代码

使用Redis 5创建集群，redis-cli只需键入：

redis-cli --cluster create 172.20.0.2:7000 172.20.0.3:7001 172.20.0.4:7002 172.20.0.5:7003 172.20.0.6:7004 172.20.0.7:7005 --cluster-replicas 1
复制代码

至此，集群已经创建好了。

[root@node-01 data]# docker exec -it redis-7000 bash
root@600aa2f84dfb:/data# redis-cli --cluster create 172.21.0.2:7000  172.21.0.3:7001  172.21.0.4:7002  172.21.0.5:7003  172.21.0.6:7004  172.21.0.7:7005 --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 172.21.0.6:7004 to 172.21.0.2:7000
Adding replica 172.21.0.7:7005 to 172.21.0.3:7001
Adding replica 172.21.0.5:7003 to 172.21.0.4:7002
M: 0d403d866c4bdfa90f3a1e21b2a33356474014ee 172.21.0.2:7000
   slots:[0-5460] (5461 slots) master
M: ab4bf4833926dd7ffca5f2d34ef754466730e7d7 172.21.0.3:7001
   slots:[5461-10922] (5462 slots) master
M: 846f087ee5885f17d5a5c05d2eb2ba470441f700 172.21.0.4:7002
   slots:[10923-16383] (5461 slots) master
S: e44c2b96f1446b1860fe582547634b12a461407d 172.21.0.5:7003
   replicates 846f087ee5885f17d5a5c05d2eb2ba470441f700
S: 4efa0a9b7b4b159807900efa5f86410e55c1d2b3 172.21.0.6:7004
   replicates 0d403d866c4bdfa90f3a1e21b2a33356474014ee
S: 8a616ee30a7104bcd341d57af825c6b5b30176ff 172.21.0.7:7005
   replicates ab4bf4833926dd7ffca5f2d34ef754466730e7d7
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
....
>>> Performing Cluster Check (using node 172.21.0.2:7000)
M: 0d403d866c4bdfa90f3a1e21b2a33356474014ee 172.21.0.2:7000
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: e44c2b96f1446b1860fe582547634b12a461407d 172.21.0.5:7003
   slots: (0 slots) slave
   replicates 846f087ee5885f17d5a5c05d2eb2ba470441f700
M: ab4bf4833926dd7ffca5f2d34ef754466730e7d7 172.21.0.3:7001
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 8a616ee30a7104bcd341d57af825c6b5b30176ff 172.21.0.7:7005
   slots: (0 slots) slave
   replicates ab4bf4833926dd7ffca5f2d34ef754466730e7d7
M: 846f087ee5885f17d5a5c05d2eb2ba470441f700 172.21.0.4:7002
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 4efa0a9b7b4b159807900efa5f86410e55c1d2b3 172.21.0.6:7004
   slots: (0 slots) slave
   replicates 0d403d866c4bdfa90f3a1e21b2a33356474014ee
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
复制代码

连接客户端，查看主从信息

root@600aa2f84dfb:/data# redis-cli -c -p 7000     
127.0.0.1:7000> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=172.21.0.6,port=7004,state=online,offset=3556,lag=1
master_replid:ef97265d34769b68096786c3e802fb3a13968fe9
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:3556
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:3556
复制代码

其它注意事项

宿主主机对外开放ip

for port in `seq 7000 7005`; do \  
firewall-cmd --zone=public --add-port=${port}/tcp --permanent  
done
#重新载入
firewall-cmd --reload
复制代码

#####需要暂停容器并删除容器以便重复6步骤

for port in `seq 7000 7005`; do \
  docker stop redis-${port};
  docker rm redis-${port};
done