«

keepalived+Lvs-DR集群

指尖二进制 • 1 年前 • 896 次点击 • LB

[TOC]

一：Keepalived概述

1.1：Keepalived概述
keepalived 是一个类似于 layer3 & 4 & 5交换机制的软件，也就是我们平时说的第3层、第4层、第5层交换。Keepalived的作用是检测web服务器的状态，如果有一台web服务器死机或工作出现故障，Keepalived将检测到，并将有故障的web服务器从系统中剔除，当web服务器工作正常后Keepalived自动将web服务器加入到服务器群中，这些工作全部自动完成，不需要人工干涉，需要人工做的只是修复故障的web服务器。

Layer3&4&5工作在IP/TCP协议栈的IP层，TCP层及应用层。
Layer3：Keepalived使用Layer3的方式工作式时，Keepalived会定期向服务器群中的服务器发送一个ICMP的数据包（既我们平时用的Ping程序），如果发现某台服务的IP地址没有激活，Keepalived便报告这台服务器失效，并将它从服务器群中剔除，这种情况的典型例子是某台服务器被非法关机。Layer3的方式是以服务器的IP地址是否有效作为服务器工作正常与否的标准。
Layer4: 主要以TCP端口的状态来决定服务器工作正常与否。如web server的服务端口一般是80，如果Keepalived检测到80端口没有启动，则Keepalived将把这台服务器从服务器群中删除。
Layer5：Layer5就是工作在具体的应用层了，比Layer3,Layer4要复杂一点，在网络上占用的带宽也要大一些。Keepalived将根据用户的设定检查服务器程序的运行是否正常，如果与用户的设定不相符，则Keepalived将把服务器从服务器群中剔除。

1.2：Keepalived的作用与构建
1.管理VIP，VIP会在LVS之间漂移
2.监控LVS分发器
运行在主分发的Keepalived会以组播的形式向网络中宣告自己，即主分发器还活着，备用节点能收到。当备用节点，在一个时间单位中收不到组播，备用节点会认为主LVS挂了，开始接手主分发器工作，把VIP配给自己。切换的时候对外无感知。
3.管理RS，Keepalived会每隔一个时间段去做一次类似于访问的操作如
探针：elinks http://192.168.19.50 –dump
经典高可用web架构: LVS+keepalived+nginx+apache+php+eaccelerator（+nfs可选）
官网：http://www.keepalived.org/
下载：http://www.keepalived.org/download.html

使用Keepalived构建LVS-DR模式的高可用集群

主机名IP用途
client192.168.19.10客户端
keepalived-master192.168.19.17主Lvs
keepalived-slave192.168.19.18备Lvs
realserver1192.168.19.200RS1 web1
realserver2192.168.19.220RS2 web2

网络拓扑

二：使用keepalived实现LVS-DR模式高用

2.1：主节点和备节点安装ipvsadm和keepalived

[root@keepalived-master ~]# yum install -y ipvsadm keepalived
[root@keepalived-slave ~]# yum install -y ipvsadm keepalived

2.2：拓展：VRRP/HSRP
虚拟路由冗余协议(Virtual Router Redundancy Protocol，简称VRRP)是由IETF提出的解决局域网中配置静态网关出现单点失效现象的路由协议。使用组播方式通信。
VRRP是一种路由容错协议，也可以叫做备份路由协议。一个局域网络内的所有主机都设置缺省路由（默认网关），当网内主机发出的目的地址不在本网段时，报文将被通过缺省路由发往外部路由器，从而实现了主机与外部网络的通信。当缺省路由器down掉（即端口关闭）之后，内部主机将无法与外部通信，如果路由器设置了VRRP时，那么这时，虚拟路由将启用备份路由器，从而实现全网通信。

HSRP：热备份路由器协议（HSRP）的设计目标是支持特定情况下 IP 流量失败转移不会引起混乱、并允许主机使用单路由器，以及即使在实际第一跳路由器使用失败的情形下仍能维护路由器间的连通性。换句话说，当源主机不能动态知道第一跳路由器的 IP 地址时，HSRP 协议能够保护第一跳路由器不出故障，是CISCO的私有协议！该协议中含有多台路由器，对应一个HSRP组。该组中只有一个路由器承担转发用户流量的职责，这就是活动路由器。当活动路由器失效后，备份路由器将承担该职责，成为新的活动路由器。这就是热备份的原理。
HSRP和VRRP的区别：HSRP是cisco的专有协议.在Cisco的HSRP之后,internet工程任务小组(internet engineering task force,IETF)也制定一种路由冗余协议:虚拟路由冗余协议(Virtual Router Redundancy Protocol,VRRP),目前包括Csico在内的主流厂商均在其产品中支持VRRP协议！VRRP和HSRP也有很多不同。VRRP和HSRP 的一个主要的区别在安全方面：它允许参与VRRP组的设备间建立认证机制 。另一个主要区别 ：VRRP中只有三种状态----初始状态（Initialize）、主状态（Master）、备份状态（Backup），而HSRP 有六种状态。其余在报文类型 、报文格式和通过TCP而非UDP发送的报文方面也有所不同

配置Keepalived+LVS-DR模式
在这种模式下，虚拟IP在某时刻只能属于某一个节点，另一个节点作为备用节点存在。当主节点不可用时，备用节点接管虚拟IP，提供正常服务。
配置参数： 节点keepalived-master(主节点)； 节点keepalived-slave(备用节点) ； 虚拟IP：192.168.19.50对外提供服务的IP。 要求默认情况下由keepalived-master节点提供服务，当节点keepalived-master不可用时，由节点keepalived-slave提供服务(即虚拟IP漂移至节点keepalived-slave)。

2.3：主节点keepalived-master配置

[root@keepalived-master ~]# rpm -ql keepalived-1.3.5
[root@keepalived-master ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
[root@keepalived-master ~]# vim /etc/keepalived/keepalived.conf
[root@keepalived-master ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
    18810779260@163.com
   }
   notification_email_from 18810779260@163.com
   smtp_server localhost
   smtp_connect_timeout 30
   router_id keepalived-master
#   vrrp_skip_check_adv_addr
#   vrrp_strict
#   vrrp_garp_interval 0
#   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.19.50
    }
}

virtual_server 192.168.19.50 80 {
    delay_loop 6
    lb_algo rr
    lb_kind DR
   #persistence_timeout 50
    protocol TCP

    real_server 192.168.19.200 80 {
        weight 1
    TCP_CHECK {
        connect_timeout 3
        nb_get_retry 3
        delay_before_retry 3
        connect_pirt 80
        }
    }

    real_server 192.168.19.220 80 {
        weight 1
        TCP_CHECK {
                connect_timeout 3
                nb_get_retry 3
                delay_before_retry 3
                connect_pirt 80
        }
    }
}

启动（再打开一个窗口实时监测日志，查看发生了什么）

[root@keepalived-master ~]# tail -f /var/log/messages 
[root@keepalived-master ~]# systemctl restart keepalived
[root@keepalived-master ~]# systemctl enable keepalived
[root@keepalived-master ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.19.50:80 rr

注：没有看到 realserver，是因为两台 realserver 还没有开启 httpd 服务。所以只能看到vip。

查看是否有vip。这个vip是keepalived自动配置的。

[root@keepalived-master ~]# ip a|tail -6
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:09:78:fb brd ff:ff:ff:ff:ff:ff
    inet 192.168.19.17/24 brd 192.168.19.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.19.50/32 scope global ens33
       valid_lft forever preferred_lft forever
[root@keepalived-master ~]# ping 192.168.19.50

2.4：从节点keepalived-slave配置（把主节点的配置文件拷贝过去只需要修改这三个地方。[state BACKUP][priority 90][router_id keepalived-slave]）

[root@keepalived-master ~]# scp /etc/keepalived/keepalived.conf 192.168.19.18:/etc/keepalived/keepalived.conf
[root@keepalived-slave ~]# vim /etc/keepalived/keepalived.conf 
[root@keepalived-slave ~]# cat /etc/keepalived/keepalived.conf 
! Configuration File for keepalived

global_defs {
   notification_email {
    18810779260@163.com
   }
   notification_email_from 18810779260@163.com
   smtp_server localhost
   smtp_connect_timeout 30
   router_id keepalived-slave
#   vrrp_skip_check_adv_addr
#   vrrp_strict
#   vrrp_garp_interval 0
#   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.19.50
    }
}

virtual_server 192.168.19.50 80 {
    delay_loop 6
    lb_algo rr
    lb_kind DR
   #persistence_timeout 50
    protocol TCP

    real_server 192.168.19.200 80 {
        weight 1
    TCP_CHECK {
        connect_timeout 3
        nb_get_retry 3
        delay_before_retry 3
        connect_pirt 80
        }
    }

    real_server 192.168.19.220 80 {
        weight 1
        TCP_CHECK {
                connect_timeout 3
                nb_get_retry 3
                delay_before_retry 3
                connect_pirt 80
        }
    }
}

启动

[root@keepalived-slave ~]# systemctl restart keepalived
[root@keepalived-slave ~]# systemctl enable keepalived
[root@keepalived-slave ~]# tail -f /var/log/messages
[root@keepalived-slave ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.19.50:80 rr

从上面的回显为什么看不到realserver。因为两台的realserver的web端还没启动。所以只能看到vip。

查看是否有vip。没有:因为主节点的keepalived正常启动着呢，只有当主节点的down掉之后从节点才会有vip。

[root@keepalived-slave ~]# ip a|tail -6
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:d0:2f:38 brd ff:ff:ff:ff:ff:ff
    inet 192.168.19.18/24 brd 192.168.19.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fed0:2f38/64 scope link 
       valid_lft forever preferred_lft forever

2.5：测试vip是否漂移：把主节点的down掉。实时监控keepalived-slave的/var/log/messages日志

[root@keepalived-slave ~]# tail -f /var/log/messages 
[root@keepalived-master ~]# systemctl stop keepalived

如果日志分辨不出来，请在两台主备节点查看ip是否漂移成功

[root@keepalived-master ~]# ip a|tail -6
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:09:78:fb brd ff:ff:ff:ff:ff:ff
    inet 192.168.19.17/24 brd 192.168.19.255 scope global ens33
       valid_lft forever preferred_lft forever

slave查看

[root@keepalived-slave ~]# ip a|tail -8
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:d0:2f:38 brd ff:ff:ff:ff:ff:ff
    inet 192.168.19.18/24 brd 192.168.19.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.19.50/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fed0:2f38/64 scope link 
       valid_lft forever preferred_lft forever

看到192.168.19.50飘移过来了，就证明一切正常了

2.6：两台realserver的配置，关闭arp转播并配置虚ip和httpd。
realserver1配置
临时生效：

[root@realserver1 ~]# vim /etc/init.d/lvsrsdr 
#!/bin/bash
#description:start relserver
VIP=192.168.19.50
source /etc/init.d/functions
case $1 in
start)
        echo 'start LVS of Realserver DR'
        /sbin/ifconfig lo:1 $VIP broadcast $VIP netmask 255.255.255.255 up
        /sbin/route add -host $VIP dev lo:1
        echo '1' > /proc/sys/net/ipv4/conf/lo/arp_ignore
        echo '2' > /proc/sys/net/ipv4/conf/lo/arp_announce
        echo '1' > /proc/sys/net/ipv4/conf/all/arp_ignore
        echo '2' > /proc/sys/net/ipv4/conf/all/arp_announce
        ;;
stop)
        /sbin/ifconfig lo:1 down
        echo 'Close LVS of Realserver DR'
        echo '0' > /proc/sys/net/ipv4/conf/lo/arp_ignore
        echo '0' > /proc/sys/net/ipv4/conf/lo/arp_announce
        echo '0' > /proc/sys/net/ipv4/conf/all/arp_ignore
        echo '0' > /proc/sys/net/ipv4/conf/all/arp_announce
        ;;
*)
        echo "Usage:$0 (start|stop)"
exit 1
esac
[root@realserver1 ~]# chmod +x /etc/init.d/lvsrsdr
[root@realserver1 ~]# /etc/init.d/lvsrsdr start
[root@realserver1 ~]# echo "/etc/init.d/lvsrsdr start" >> /etc/rc.local
[root@realserver1 ~]# ip a|head -6
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 192.168.19.50/32 brd 192.168.19.50 scope global lo:1
       valid_lft forever preferred_lft forever

realserver2配置
临时生效：

[root@realserver2 ~]# vim /etc/init.d/lvsrsdr 
#!/bin/bash
#description:start relserver
VIP=192.168.19.50
source /etc/init.d/functions
case $1 in
start)
        echo 'start LVS of Realserver DR'
        /sbin/ifconfig lo:1 $VIP broadcast $VIP netmask 255.255.255.255 up
        /sbin/route add -host $VIP dev lo:1
        echo '1' > /proc/sys/net/ipv4/conf/lo/arp_ignore
        echo '2' > /proc/sys/net/ipv4/conf/lo/arp_announce
        echo '1' > /proc/sys/net/ipv4/conf/all/arp_ignore
        echo '2' > /proc/sys/net/ipv4/conf/all/arp_announce
        ;;
stop)
        /sbin/ifconfig lo:1 down
        echo 'Close LVS of Realserver DR'
        echo '0' > /proc/sys/net/ipv4/conf/lo/arp_ignore
        echo '0' > /proc/sys/net/ipv4/conf/lo/arp_announce
        echo '0' > /proc/sys/net/ipv4/conf/all/arp_ignore
        echo '0' > /proc/sys/net/ipv4/conf/all/arp_announce
        ;;
*)
        echo "Usage:$0 (start|stop)"
exit 1
esac
[root@realserver2 ~]# chmod +x /etc/init.d/lvsrsdr
[root@realserver2 ~]# /etc/init.d/lvsrsdr start
[root@realserver2 ~]# echo "/etc/init.d/lvsrsdr start" >> /etc/rc.local
[root@realserver2 ~]# ip a|head -6
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 192.168.19.50/32 brd 192.168.19.50 scope global lo:1
       valid_lft forever preferred_lft forever

永久生效：

[root@realserver1 ~]# vim /etc/sysctl.conf
net.ipv4.conf.ens33.arp_ignore = 1
net.ipv4.conf.ens33.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
[root@realserver1 ~]# sysctl -p
[root@realserver1 ~]# ifconfig lo:1 192.168.19.50 netmask 255.255.255.255

[root@realserver2 ~]# vim /etc/sysctl.conf
net.ipv4.conf.ens33.arp_ignore = 1
net.ipv4.conf.ens33.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
[root@realserver2 ~]# sysctl -p
[root@realserver2 ~]# ifconfig lo:1 192.168.19.50 netmask 255.255.255.255

http配置

[root@realserver1 ~]# yum install -y httpd
[root@realserver2 ~]# yum install -y httpd
[root@realserver1 ~]# echo 192.168.19.200 >/var/www/html/index.html
[root@realserver1 ~]# systemctl start httpd
[root@realserver1 ~]# systemctl enable httpd
[root@realserver2 ~]# echo 192.168.19.220 >/var/www/html/index.html
[root@realserver2 ~]# systemctl start httpd
[root@realserver2 ~]# systemctl enable httpd

2.7：客户端测试 

[root@client ~]# yum install -y elinks
[root@client ~]# elinks 192.168.19.50 --dump
   192.168.19.200
[root@client ~]# elinks 192.168.19.50 --dump
   192.168.19.220

三：测试keepalived

测试主备切换，首选在主上keepalived-master上查看状态 

[root@keepalived-master ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.19.50:80 rr
  -> 192.168.19.200:80            Route   1      0          2
  -> 192.168.19.220:80            Route   1      0          2

测试主备切换，在主上keepalived-slave上查看状态

[root@keepalived-master ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.19.50:80 rr
  -> 192.168.19.200:80            Route   1      0          0
  -> 192.168.19.220:80            Route   1      0          0

在keepalived-master上停掉keepalived，模拟故障，在查看keepalived-slave

[root@keepalived-master ~]# systemctl stop keepalived
[root@keepalived-slave ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.19.50:80 rr
  -> 192.168.19.200:80            Route   1      0          0
  -> 192.168.19.220:80            Route   1      0          0

[root@keepalived-slave ~]# ip a|tail -8
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:d0:2f:38 brd ff:ff:ff:ff:ff:ff
    inet 192.168.19.18/24 brd 192.168.19.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.19.50/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fed0:2f38/64 scope link
       valid_lft forever preferred_lft forever

当我们重启了主上的keepalived，自动从备分发器转到主分发器上。因为备的优先级低。

[root@keepalived-master ~]# systemctl start keepalived
[root@keepalived-master ~]# ip a|tail -8
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:09:78:fb brd ff:ff:ff:ff:ff:ff
    inet 192.168.19.17/24 brd 192.168.19.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.19.50/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe09:78fb/64 scope link 
       valid_lft forever preferred_lft forever

测试realserver容错（停掉realsever1的http并在keepalived-master上查看刚刚关闭的那台机器不在列表中）

[root@keepalived-master ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.19.50:80 rr
  -> 192.168.19.200:80            Route   1      0          0
  -> 192.168.19.220:80            Route   1      0          0
[root@realserver1 ~]# systemctl stop httpd
[root@keepalived-master ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.19.50:80 rr
  -> 192.168.19.220:80            Route   1      0          0

当keepalived主从优先级一样时，当主恢复后，还是要回切资源的。第一次建立主从关系时，需要10s左右的认证时间。

还没收到回复