歡迎您光臨本站 註冊首頁

Heartbeat/corosync+pacemaker+ldirectord 實現Nginx負載均衡

←手機掃碼閱讀     火星人 @ 2014-03-04 , reply:0

Heartbeat/corosync+pacemaker+ldirectord 實現Nginx負載均衡

Heartbeat/corosync+pacemaker+ldirectord 實現Nginx負載均衡

歡迎訪問yoyotown.com一起討論IT技術。


系統:CentOS 5.4
IP分配:
查看源代碼
列印幫助
HA1     eth0:192.168.0.66   eth1:192.168.10.1
HA2     eth0:192.168.0.69   eth1:192.168.10.2
VIP     192.168.0.120

1. 安裝pacemaker和heartbeat
# wget -O /etc/yum.repos.d/pacemaker.repo http://clusterlabs.org/rpm/epel-5/clusterlabs.repo
# wget ftp://ftp.pbone.net/mirror/centos.karan.org/el5/extras/testing/i386/RPMS/libesmtp-1.0.4-6.el5.kb.i386.rpm
# rpm -ivh libesmtp-1.0.4-6.el5.kb.i386.rpm
# yum install -y pacemaker heartbeat

2. 安裝ldirectord
# yum install -y ldirectord

3. 配置
3.1 配置Heartbeat
# cp /usr/share/doc/heartbeat-3.0.1/{ha.cf,authkeys} /etc/ha.d/

# cat /etc/ha.d/authkeys
查看源代碼
列印幫助
1        auth 1
2        1 crc

# cat /etc/ha.d/ha.cf |grep -v 「#」
查看源代碼
列印幫助
01        logfile /var/log/ha-log
02        logfacility local0
03        keepalive 2
04        deadtime 30
05        warntime 10
06        initdead 120
07        udpport 695
08        ucast eth1 192.168.10.2     # 注意此處HA2改為:ucast eth1 192.168.10.1
09        auto_failback on
10        watchdog /dev/watchdog
11        node    HA1
12        node    HA2
13        ping 192.168.0.1
14        respawn hacluster /usr/lib/heartbeat/ipfail
15        apiauth ipfail gid=haclient uid=hacluster
16        crm on

3.2 用corosync替換heartbeat(可選)
corosync是基於OpenAIS構建的集群引擎,可替代heartbeat進行心跳檢測。
The Corosync Cluster Engine is an open source project Licensed under the BSD License derived from the OpenAIS project. OpenAIS uses a UDP multicast based communication protocol to periodically check for node availability.

# yum install -y corosync
# corosync-keygen
Corosync Cluster Engine Authentication key generator.
Gathering 1024 bits for key from /dev/random.
Press keys on your keyboard to generate entropy.
Writing corosync key to /etc/corosync/authkey.

# scp /etc/corosync/authkey HA2:/etc/corosync/
# cp /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf
# vi !$
查看源代碼
列印幫助
01        # Please read the corosync.conf.5 manual page
02        compatibility: whitetank
03         
04        totem {
05        version: 2
06        secauth: off
07        threads: 0
08        interface {
09        ringnumber: 0
10        bindnetaddr: 192.168.10.0
11        mcastaddr: 226.94.1.1
12        mcastport: 5405
13        }
14        }
15         
16        logging {
17        fileline: off
18        to_stderr: yes
19        to_logfile: yes
20        to_syslog: yes
21        logfile: /var/log/corosync.log
22        debug: off
23        timestamp: on
24        logger_subsys {
25        subsys: AMF
26        debug: off
27        }
28        }
29         
30        amf {
31        mode: disabled
32        }
33         
34        service {
35                # Load the Pacemaker Cluster Resource Manager
36                ver:       0
37                name:      pacemaker
38                use_mgmtd: yes
39        }

# scp /etc/corosync/corosync.conf HA2:/etc/corosync/corosync.conf
# service corosync start
Starting Corosync Cluster Engine (corosync):               [  OK  ]
# chkconfig –level 2345 corosync on
# chkconfig –level 2345 heartbeat off

在HA2上執行:
# chown root:root /etc/corosync/authkey
# chmod 400 /etc/corosync/authkey
# service corosync start
Starting Corosync Cluster Engine (corosync):               [  OK  ]
# chkconfig –level 2345 corosync on
# chkconfig –level 2345 heartbeat off

3.3 安裝配置ldirectord
# cat /etc/ha.d/ldirectord.cf
查看源代碼
列印幫助
01        checktimeout=5
02        checkinterval=7
03        autoreload=yes
04        logfile="/var/log/ldirectord.log"
05        quiescent=yes
06        emailalert=shidl@baihe.com
07        # A server with a page at the main root of the site that displays "Nginx"
08        virtual=192.168.0.120:80
09        real=192.168.0.66:80 gate
10        real=192.168.0.69:80 gate
11        service=http
12        request="/lb.html"    # 在根目錄下編寫lb.html,內容為live
13        receive="live"
14        scheduler=wlc
15        protocol=tcp
16        checktype=negotiate

3.4 配置hosts
# cat /etc/hosts
查看源代碼
列印幫助
1        # Do not remove the following line, or various programs
2        # that require network functionality will fail.
3        127.0.0.1       vpc localhost.localdomain localhost
4        ::1     localhost6.localdomain6 localhost6
5        192.168.10.1    HA1
6        192.168.10.2    HA2

3.5 配置lo:0設備
查看源代碼
列印幫助
1        # cat >>/etc/sysconfig/network-scripts/ifcfg-lo:0<<EOF
2        <pre>DEVICE=lo:0
3        IPADDR=192.168.0.120
4        NETMASK="255.255.255.255"
5        ONBOOT=yes
6        NAME=loopback
7         
8        EOF

3.6 啟用轉發,禁用arp
# vi /etc/sysctl.conf
修改net.ipv4.ip_forward = 0為net.ipv4.ip_forward = 1
添加下面行:
查看源代碼
列印幫助
1        net.ipv4.conf.all.arp_ignore = 1
2        net.ipv4.conf.eth0.arp_ignore = 1
3        net.ipv4.conf.all.arp_announce = 2
4        net.ipv4.conf.eth0.arp_announce = 2

# sysctl -p

# 將配置拷貝到HA2
# scp /etc/ha.d/{ha.cf,authkeys,ldirectord.cf} HA2:/etc/ha.d/
# scp /etc/{hosts,sysctl.conf} HA2:/etc/
# scp /etc/sysconfig/network-scripts/ifcfg-lo:0 HA2:/etc/sysconfig/network-scripts/

在HA2上修改/etc/ha.d/ha.cf
將ucast eth1 192.168.10.2 改為:ucast eth1 192.168.10.1
並使sysctl.conf配置生效:
# sysctl -p

3.7 在HA1和HA2上配置並安裝好nginx
編寫nginx lsb資源代理腳本(注意nginx安裝路徑):
# cat /etc/init.d/nginxd
查看源代碼
列印幫助
01        #!/bin/sh
02         
03        # source function library
04        . /etc/rc.d/init.d/functions
05         
06        # Source networking configuration.
07        . /etc/sysconfig/network
08         
09        # Check that networking is up.
10        [ ${NETWORKING} = "no" ] && exit 0
11         
12        RETVAL=0
13        prog="nginx"
14         
15        nginxDir=/usr/local/nginx
16        nginxd=$nginxDir/sbin/nginx
17        nginxConf=$nginxDir/conf/nginx.conf
18        nginxPid=$nginxDir/nginx.pid
19         
20        nginx_check()
21        {
22            if [[ -e $nginxPid ]]; then
23                ps aux |grep -v grep |grep -q nginx
24                if (( $? == 0 )); then
25                    echo "$prog already running..."
26                    exit 1
27                else
28                    rm -rf $nginxPid &> /dev/null
29                fi
30            fi
31        }
32         
33        start()
34        {
35            nginx_check
36            if (( $? != 0 )); then
37                true
38            else
39                echo -n $"Starting $prog:"
40                daemon $nginxd -c $nginxConf
41                RETVAL=$?
42                echo
43                [ $RETVAL = 0 ] && touch /var/lock/subsys/nginx
44                return $RETVAL
45            fi
46        }
47         
48        stop()
49        {
50            echo -n $"Stopping $prog:"
51            killproc $nginxd
52            RETVAL=$?
53            echo
54            [ $RETVAL = 0 ] && rm -f /var/lock/subsys/nginx $nginxPid
55        }
56         
57        reload()
58        {
59            echo -n $"Reloading $prog:"
60            killproc $nginxd -HUP
61            RETVAL=$?
62            echo
63        }
64         
65        case "$1" in
66                start)
67                        start
68                        ;;
69                stop)
70                        stop
71                        ;;
72                restart)
73                        stop
74                        start
75                        ;;
76                reload)
77                        reload
78                        ;;
79                status)
80                        status $prog
81                        RETVAL=$?
82                        ;;
83                *)
84                        echo $"Usage: $0 {start|stop|restart|reload|status}"
85                        RETVAL=1
86        esac
87        exit $RETVAL

# chmod +x  /etc/init.d/nginxd
# scp  /etc/init.d/nginxd HA2: /etc/init.d/nginxd

# service network restart
# service heartbeat start

# service network restart
# service heartbeat start

4. 配置集群資源:

Heartbeat和其他應用提供的ocf代理腳本或許會有錯誤,我們可以用下面方法排錯:
要檢查 OCF 腳本,請首先設置所需的環境變數。例如,當測試IPaddr OCF 腳本時,您必須通過設置一個變數名稱前綴為 OCF_RESKEY_的環境變數來設置變數 ip 的值。對於此示例,可運行以下命令:
查看源代碼
列印幫助
1        export OCF_RESKEY_ip=
2        /usr/lib/ocf/resource.d/heartbeat/IPaddr validate-all
3        /usr/lib/ocf/resource.d/heartbeat/IPaddr start
4        /usr/lib/ocf/resource.d/heartbeat/IPaddr stop

如果此操作不成功,很可能是您缺少某個必需變數或者只是輸錯了參數。

排錯ldirectord ocf代理腳本:
export OCF_ROOT=/usr/lib/ocf
根據自己的環境設置修改下面兩行:
# vi /usr/lib/ocf/resource.d/heartbeat/ldirectord
查看源代碼
列印幫助
1        LDIRCONF=${OCF_RESKEY_configfile:-/etc/ha.d/ldirectord.cf}
2        LDIRECTORD=${OCF_RESKEY_ldirectord:-/usr/sbin/ldirectord}

# /usr/lib/ocf/resource.d/heartbeat/ldirectord monitor
# echo $?
7     # ldirectord未運行返回7,運行正常返回0

# crm
crm(live)# configure
crm(live)configure# node HA1
crm(live)configure# node HA2
crm(live)configure# primitive ldirectord ocf:heartbeat:ldirectord \
> params configfile=」/etc/ha.d/ldirectord.cf」 \
> op monitor interval=」30s」 timeout=」20s」 \
> meta migration-threshold=」10″ target-role=」Started」
crm(live)configure# primitive vip ocf:heartbeat:IPaddr2 \
> params lvs_support=」true」 ip=」192.168.0.120″ cidr_netmask=」24″ broadcast=」192.168.0.255″ \
> op monitor interval=」1m」 timeout=」20s」 \
> meta migration-threshold=」10″
crm(live)configure# primitive nginxd lsb:nginxd \
> op monitor interval=」30s」 timeout=」20s」 \
> meta migration-threshold=」10″ target-role=」Started」
crm(live)configure# group load-balancing vip ldirectord
crm(live)configure# clone cl-nginxd nginxd
crm(live)configure# location perfer-ha1 load-balancing \
> rule $id=」prefer-ha1-rule」 100: #uname eq HA1
crm(live)configure# property stonith-enabled=」false」 \
> no-quorum-policy=」ignore」 \
> start-failure-is-fatal=」false」 \
> expected-quorum-votes=」2″
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# end
crm(live)# status

============
Last updated: Thu Nov 12 01:00:13 2009
Stack: Heartbeat
Current DC: HA2 – partition with quorum
Version: 1.0.6-f709c638237cdff7556cb6ab615f32826c0f8c06
2 Nodes configured, 2 expected votes
2 Resources configured.
============

Online: [ HA2 HA1 ]

Clone Set: cl-nginxd
Started: [ HA2 HA1 ]
Resource Group: load-balancing
vip    (ocf::heartbeat:IPaddr2):    Started HA1
ldirectord    (ocf::heartbeat:ldirectord):    Started HA1

crm(live)# quit
bye

5. 驗證
# ipvsadm -l
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.0.120:http wlc
-> 192.168.0.69:http            Route   1      0          0
-> 192.168.0.66:http            Local   1      0          0

用瀏覽器訪問網站看是否正常。

禁用HA1的eth1網卡,在HA2上看故障轉移情況。
# crm
crm(live)# status

============
Last updated: Thu Nov 12 18:40:54 2009
Stack: Heartbeat
Current DC: HA2 – partition WITHOUT quorum
Version: 1.0.6-f709c638237cdff7556cb6ab615f32826c0f8c06
2 Nodes configured, 2 expected votes
2 Resources configured.
============

Online: [ HA2 ]
OFFLINE: [ HA1 ]

Clone Set: cl-nginxd
Started: [ HA2 ]
Stopped: [ nginxd:0 ]
Resource Group: load-balancing
vip    (ocf::heartbeat:IPaddr2):    Started HA2
ldirectord    (ocf::heartbeat:ldirectord):    Started HA2

啟用HA1的eth1網卡,在HA1上看故障轉移情況。

# crm status

============
Last updated: Thu Nov 12 18:42:55 2009
Stack: Heartbeat
Current DC: HA1 – partition with quorum
Version: 1.0.6-f709c638237cdff7556cb6ab615f32826c0f8c06
2 Nodes configured, 2 expected votes
2 Resources configured.
============

Online: [ HA2 HA1 ]

Clone Set: cl-nginxd
Started: [ HA1 HA2 ]
Resource Group: load-balancing
vip    (ocf::heartbeat:IPaddr2):    Started HA1
ldirectord    (ocf::heartbeat:ldirectord):    Started HA1

6. 參考:

Load Balanced MySQL Replicated Cluster
Debian Lenny HowTo

相關文章:

    * 2009/12/11 -- MySQL-Nginx-Pacemaker-corosync(openais)-drbd active/passive cluster
    * 2009/12/09 -- Heartbeat實現Nginx高可用性(style 2.x)
    * 2009/12/08 -- Heartbeat實現Nginx高可用性(style 1.x)
    * 2009/12/10 -- 用CRM命令行工具配置集群資源
    * 2009/12/08 -- HA體系架構及內部處理流程

[ 本帖最後由 toakee 於 2009-12-12 11:05 編輯 ]

[火星人 ] Heartbeat/corosync+pacemaker+ldirectord 實現Nginx負載均衡已經有851次圍觀

http://coctec.com/docs/service/show-post-5961.html