lvs簡介: LVS是一個開源的軟體,由畢業於國防科技大學的章文嵩博士於1998年5月創立,可以實現LINUX平台下的簡單負載均衡.LVS是Linux Virtual Server的縮寫,意思是Linux虛擬伺服器.
Lvs 的集群工作模式有3種:VS/NAT,vs/tun,vs/dr.
Lvs的調度演算法:
LVS的演算法分為兩大類:
靜態演算法:只是根據演算法進行調度並不考慮後端REALSERVER的實際連接情況
rr-論調演算法,假如有兩台伺服器A,B,第一個請求給A,第二個給B,第三個給A依次往複
wrr-加權論調,假如有兩台伺服器A,B,A的性能是B的兩倍,則在論調的同時給A上面分配的請求也大致會是B上面的兩倍
dh-假如調度器的後面是兩台緩存伺服器A,B而不是真正的REALSERVER,則會儘可能的把相同請求或者把同一用戶的請求轉發到同一個緩存伺服器上面以提高緩存命中率
sh-假如公司有兩台防火牆讓員工上網,則會把某個員工往外的訪問及向內返回的請求結果定向到同一台防火牆上面,方便防火牆做established的狀態檢測
動態演算法:前端的調度器會根據後端REALSERVER的實際連接情況來分配請求
活動鏈接:當前有數據包傳輸
非活動鏈接:當前連接出於建立狀態但是沒有數據傳輸
lc-同時檢查後端REALSERVER上面活動狀態和非活動狀態的連接數使用(活動連接數*256 非活動連接數)數字小的將接收下次訪問請求
wlc-加權的lc,使用(活動連接數*256 非活動連接數)/權重,數字小的將接收下次訪問請求,是最常用的演算法
sed-不考慮非活動狀態,使用(活動狀態 1)*256,數字小的將接收下次訪問請求, 1主要是為了提高權重大的伺服器的響應能力
nq-假設有兩台伺服器A,B,權重比為10:1,按照sed演算法,只有當A伺服器已經響應了10個請求之時兩者的計算數值才相同,為了避免權重小的伺服器過於空閑,nq沿用sed演算法但是確保讓每個伺服器都不空閑,只有在不考慮非活動連接的情況下nq才能取代wlc演算法
Lvs的dr模型
一:Web1的搭建 192.168.0.101
[root@zzu ~]# ifconfig lo:0 192.168.0.100 netmask 255.255.255.255
配置一個vip地址
[root@zzu ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:4B:61:16
inet addr:192.168.0.101 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe4b:6116/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3707 errors:0 dropped:0 overruns:0 frame:0
TX packets:915 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:354625 (346.3 KiB) TX bytes:127781 (124.7 KiB)
Interrupt:67 Base address:0x2000
lo:0 Link encap:Local Loopback
inet addr:192.168.0.100 Mask:255.255.255.255
UP LOOPBACK RUNNING MTU:16436 Metric:1
設置real伺服器的arp選項保證在進行arp解析的時候只有director的vip進行響應
[root@zzu ~]# echo "net.ipv4.conf.all.arp_announce = 2" >>/etc/sysctl.conf
[root@zzu ~]# echo "net.ipv4.conf.lo.arp_announce = 2" >>/etc/sysctl.conf
[root@zzu ~]# echo "net.ipv4.conf.all.arp_ignore = 1" >>/etc/sysctl.conf
[root@zzu ~]# echo "net.ipv4.conf.lo.arp_ignore = 1" >>/etc/sysctl.conf
設置一條特殊的路由,保證在回復客戶端時使用的是vip的地址
[root@zzu ~]# route add -host 192.168.0.100 dev lo:0
[root@zzu ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.0.100 0.0.0.0 255.255.255.255 UH 0 0 0 lo
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
[root@zzu Server]# rpm -ivh httpd-2.2.3-31.el5.i386.rpm
[root@zzu Server]# cd /var/www/html/
[root@zzu html]# vim index.html
web1
[root@zzu html]# links http://192.168.0.101
二:web2的搭建 192.168.0.102
[root@zzu ~]# ifconfig lo:0 192.168.0.100 netmask 255.255.255.255
配置一個vip地址
[root@zzu ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:4B:61:16
inet addr:192.168.0.102 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe4b:6116/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3707 errors:0 dropped:0 overruns:0 frame:0
TX packets:915 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:354625 (346.3 KiB) TX bytes:127781 (124.7 KiB)
Interrupt:67 Base address:0x2000
lo:0 Link encap:Local Loopback
inet addr:192.168.0.100 Mask:255.255.255.255
UP LOOPBACK RUNNING MTU:16436 Metric:1
設置real伺服器的arp選項保證在進行arp解析的時候只有director的vip進行響應
[root@zzu ~]# echo "net.ipv4.conf.all.arp_announce = 2" >>/etc/sysctl.conf
[root@zzu ~]# echo "net.ipv4.conf.lo.arp_announce = 2" >>/etc/sysctl.conf
[root@zzu ~]# echo "net.ipv4.conf.all.arp_ignore = 1" >>/etc/sysctl.conf
[root@zzu ~]# echo "net.ipv4.conf.lo.arp_ignore = 1" >>/etc/sysctl.conf
[root@zzu ~]# route add -host 192.168.0.100 dev lo:0
設置一條特殊的路由,保證在恢復客戶端使使用的是vip的地址
[root@zzu ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.0.100 0.0.0.0 255.255.255.255 UH 0 0 0 lo
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
[root@zzu Server]# rpm -ivh httpd-2.2.3-31.el5.i386.rpm
[root@zzu Server]# cd /var/www/html/
[root@zzu html]# vim index.html
Web2
[root@zzu html]# links http://192.168.0.102
三:director伺服器的搭建
[root@zzu ~]# yum install ipvsadm*
四:lvs-dr模型下rr的測試
[root@zzu ~]# ipvsadm -A 192.168.0.100:80 -s rr
unexpected argument 192.168.0.100:80
[root@zzu ~]# ipvsadm -A -t 192.168.0.100:80 -s rr
[root@zzu ~]# ipvsadm -a -t 192.168.0.100:80 -r 192.168.0.101 -g
[root@zzu ~]# ipvsadm -a -t 192.168.0.100:80 -r 192.168.0.102 -g
[root@zzu ~]# ipvsdm –ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 rr
-> 192.168.0.102:80 Route 1 0 0
-> 192.168.0.101:80 Route 1 0 0
[root@zzu ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 rr
-> 192.168.0.102:80 Route 1 0 6
-> 192.168.0.101:80 Route 1 0 6
五:lvs-dr模型下rr的測試 (ppc)
[root@zzu ~]# ipvsadm -A -t 192.168.0.100:80 -s rr
-p 300[root@zzu ~]# ipvsadm -a -t 192.168.0.100:80 -r 192.168.0.102 -g
[root@zzu ~]# ipvsadm -a -t 192.168.0.100:80 -r 192.168.0.101 –g
[root@zzu ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 rr persistent 300
-> 192.168.0.101:80 Route 1 0 0
-> 192.168.0.102:80 Route 1 0 0
[root@zzu ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 rr persistent 300
-> 192.168.0.101:80 Route 1 0 5
-> 192.168.0.102:80 Route 1 0 0
[root@zzu ~]# ipvsadm -A -t 192.168.0.100:22 -s rr -p 300
[root@zzu ~]# ipvsadm -a -t 192.168.0.100:22 -r 192.168.0.101 -g
[root@zzu ~]# ipvsadm -a -t 192.168.0.100:22 -r 192.168.0.102 -g
[root@zzu ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 rr persistent 300
-> 192.168.0.101:80 Route 1 0 5
-> 192.168.0.102:80 Route 1 0 0
TCP 192.168.0.100:22 rr persistent 300
-> 192.168.0.102:22 Route 1 0 0
-> 192.168.0.101:22 Route 1 0 0
[root@zzu ~]#
[root@zzu ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 rr persistent 300
-> 192.168.0.101:80 Route 1 0 5
-> 192.168.0.102:80 Route 1 0 0
TCP 192.168.0.100:22 rr persistent 300
-> 192.168.0.102:22 Route 1 0 10
-> 192.168.0.101:22 Route 1 0 0
六:lvs-dr模型下rr的測試 (pcc)
[root@zzu ~]# ipvsadm -A -t 192.168.0.100:0 -s rr -p 300
[root@zzu ~]# ipvsadm -a -t 192.168.0.100:0 -r 192.168.0.102 -g
[root@zzu ~]# ipvsadm -a -t 192.168.0.100:0 -r 192.168.0.101 -g
[root@zzu ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:0 rr persistent 300
-> 192.168.0.101:0 Route 1 0 0
-> 192.168.0.102:0 Route 1 0 0
[root@zzu ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:0 rr persistent 300
-> 192.168.0.101:0 Route 1 0 9
-> 192.168.0.102:0 Route 1 0 0
七:lvs-dr模型下帶防火牆標記的持續連接(80和443)
將http和https打上標籤
1:搭建https伺服器web1
[root@zzu ~]# yum install openssl*
[root@zzu ~]# cd /etc/pki/
[root@zzu pki]# ll
drwx------ 3 root root 4096 2012-02-08 CA
drwxr-xr-x 2 root root 4096 2012-02-08 nssdb
drwxr-xr-x 2 root root 4096 2012-02-08 rpm-gpg
drwxr-xr-x 5 root root 4096 2012-02-08 tls
[root@zzu pki]# vim tls/openssl.cnf
45 dir = /etc/pki/CA
88 countryName = optional
89 stateOrProvinceName = optional
90 organizationName = optional
136 countryName_default = CN
141 stateOrProvinceName_default = beijing
144 localityName_default = Beijing
2:創建3個目錄和兩個文件
[root@zzu pki]# cd CA
[root@zzu CA]# mkdir certs newcerts crl
[root@zzu CA]# touch index.txt serial
[root@zzu CA]# echo "01" >>serial
[root@zzu CA]# openssl genrsa 1024 >private/cakey.pem
Generating RSA private key, 1024 bit long modulus
..............
.....................................................................
e is 65537 (0x10001)
[root@zzu CA]# openssl req -new -key private/cakey.pem -days 3650 -x509 -out cacert.pem
Country Name (2 letter code) [CN]:
State or Province Name (full name) [beijing]:
Locality Name (eg, city) [beijing]:
Organization Name (eg, company) [My Company Ltd]:qinghua
Organizational Unit Name (eg, section) []:qinghua
Common Name (eg, your name or your server's hostname) []:www.qinghua.com
2:為http辦法證書
[root@zzu ~]# mkdir -pv /etc/httpd/certs
[root@zzu ~]# cd /etc/httpd/certs/
[root@zzu certs]# openssl genrsa 1024 > httpd.key
[root@zzu certs]# openssl req -new -key httpd.key -out httpd.csr
Country Name (2 letter code) [CN]:
State or Province Name (full name) [beijing]:
Locality Name (eg, city) [beijing]:
Organization Name (eg, company) [My Company Ltd]:bjdx
Organizational Unit Name (eg, section) []:sec
Common Name (eg, your name or your server's hostname) []:www.bj.com
[root@zzu certs]# openssl ca -in httpd.csr -out httpd.cert
Using configuration from /etc/pki/tls/openssl.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 1 (0x1)
Validity
Not Before: Feb 7 13:28:38 2012 GMT
Not After : Feb 6 13:28:38 2013 GMT
Subject:
countryName = CN
stateOrProvinceName = beijing
organizationName = bjdx
organizationalUnitName = sec
commonName = www.bj.com
3:綁緊證書文件
[root@zzu Server]# rpm -ivh distcache-1.4.5-14.1.i386.rpm
Preparing... ########################################### [100%]
1:distcache ########################################### [100%]
[root@zzu Server]# rpm -ivh mod_ssl-2.2.3-31.el5.i386.rpm
Preparing... ########################################### [100%]
1:mod_ssl ########################################### [100%]
[root@zzu ~]# cd /etc/httpd/certs/
[root@zzu certs]# cp /etc/pki/CA/cacert.pem ./
[root@zzu certs]# ll
-rw-r--r-- 1 root root 1168 02-07 21:34 cacert.pem
-rw-r--r-- 1 root root 0 02-07 21:28 httpd.cert
-rw-r--r-- 1 root root 643 02-07 21:27 httpd.csr
-rw-r--r-- 1 root root 887 02-07 21:26 httpd.key
[root@zzu ~]# vim /etc/httpd/conf.d/ssl.conf
112 SSLCertificateFile /etc/http/certs/httpd.cert
119 SSLCertificateKeyFile /etc/http/certs/httpd.key
128 SSLCertificateChainFile /etc/http/certs/cacert.pem
[root@zzu certs]# service httpd restart 重新啟動www服務
Stopping httpd: [FAILED]
Starting httpd: [ OK ]
4.搭建https 伺服器Web2
[root@server2 ~]# mkdir -pv /etc/httpd/certs
mkdir: created directory `/etc/httpd/certs'
[root@server2 ~]# cd /etc/httpd/certs
[root@server2 certs]# ll
total 0
[root@server2 certs]# scp 192.168.0.101:/etc/httpd/certs/* ./
The authenticity of host '192.168.0.101 (192.168.0.101)' can't be established.
RSA key fingerprint is 91:71:d8:d9:f2:63:a6:78:2f:0c:1e:e8:32:aa:55:3c.
Are you sure you want to continue connecting (yes/no)? y
Please type 'yes' or 'no': yes
Warning: Permanently added '192.168.0.101' (RSA) to the list of known hosts.
root@192.168.0.101's password:
cacert.pem 100% 1168 1.1KB/s 00:00
httpd.cert 100% 3082 3.0KB/s 00:00
httpd.csr 100% 643 0.6KB/s 00:00
httpd.key 100% 887 0.9KB/s 00:00
[root@server2 certs]# ll
-rw-r--r-- 1 root root 1168 Apr 30 17:33 cacert.pem
-rw-r--r-- 1 root root 3082 Apr 30 17:33 httpd.cert
-rw-r--r-- 1 root root 643 Apr 30 17:33 httpd.csr
-rw-r--r-- 1 root root 887 Apr 30 17:33 httpd.key
[root@server2 Server]# rpm -ivh distcache-1.4.5-14.1.i386.rpm
[root@server2 Server]# rpm -ivh mod_ssl-2.2.3-31.el5.i386.rpm
[root@server2~]#scp192.168.0.101:/etc/httpd/conf.d/ssl.conf /etc/httpd/conf.d/ssl.conf
root@192.168.0.101's password:
ssl.conf 100% 9655 9.4KB/s 00:00
[root@server2 ~]# service httpd restart
Stopping httpd: [ OK ]
Starting httpd: httpd: apr_sockaddr_info_get() failed for server2
httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
[ OK ]
5:設置director伺服器
[root@zzu ~]# iptables -t mangle -A PREROUTING -d 192.168.0.100 -p tcp --dport 80 -j MARK --set-mark 1
[root@zzu ~]# iptables -t mangle -A PREROUTING -d 192.168.0.100 -p tcp --dport 443 -j MARK --set-mark 1
[root@zzu ~]# iptables -t mangle -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
MARK tcp -- anywhere 192.168.0.100 tcp dpt:http MARK set 0x1
MARK tcp -- anywhere 192.168.0.100 tcp dpt:https MARK set 0x1
[root@zzu ~]# ipvsadm -A -f 1 -s rr -p 1800
[root@zzu ~]# ipvsadm -a -f 1 -r 192.168.0.101 -g
[root@zzu ~]# ipvsadm -a -f 1 -r 192.168.0.102 -g
[root@zzu ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 1 rr persistent 1800
-> 192.168.0.102:0 Route 1 0 0
-> 192.168.0.101:0 Route 1 0 0
訪問測試:
八:lvs-dr模型下帶防火牆標記的持續連接(ftp的被動方式)
ftp1 伺服器
[root@zzu Server]# rpm -ivh vsftpd-2.0.5-16.el5.i386.rpm
Preparing... ########################################### [100%]
1:vsftpd ########################################### [100%]
[root@zzu ~]# cd /var/ftp/
[root@zzu ftp]# mkdir ftp1
[root@zzu ftp]# ll
total 8
drwxr-xr-x 2 root root 4096 Feb 7 22:27 ftp1
drwxr-xr-x 3 root root 4096 Feb 7 22:26 pub
[root@zzu ~]# vim /etc/vsftpd/vsftpd.conf
12 pasv_min_port=10000
13 pasv_max_port=20000
14 pasv_enable=YES
[root@zzu ftp]# service vsftpd restart
Shutting down vsftpd: [ OK ]
Starting vsftpd for vsftpd: [ OK ]
ftp2伺服器同一
director伺服器的設置
[root@zzu ~]# iptables -t mangle -A PREROUTING -p tcp -d 192.168.0.100/32 --dport 10000:20000 -j MARK --set-mark 21
[root@zzu ~]# iptables -t mangle -A PREROUTING -p tcp -d 192.168.0.100/32 --dport 21 -j MARK --set-mark 21
[root@zzu ~]# iptables -t mangle -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
MARK tcp -- anywhere 192.168.0.100 tcp dpts:ndmp:dnp MARK set 0x15
MARK tcp -- anywhere 192.168.0.100 tcp dpt:ftp MARK set 0x15
[root@zzu ~]# ipvsadm -A -f 21 -s rr -p 1800
[root@zzu ~]# ipvsadm -a -f 21 -r 192.168.0.101 -g
[root@zzu ~]# ipvsadm -a -f 21 -r 192.168.0.102 -g
[root@zzu ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 21 rr persistent 1800
-> 192.168.0.102:0 Route 1 0 0
-> 192.168.0.101:0 Route 1 0 0
[root@zzu ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 21 rr persistent 1800
-> 192.168.0.102:0 Route 1 0 8
-> 192.168.0.101:0 Route 1 0 0
[root@zzu ~]# ipvsadm –lcn 我們在director上查看鏈接的狀態
IPVS connection entries
pro expire state source virtual destination
TCP 00:19 FIN_WAIT 192.168.0.5:1309 192.168.0.100:21 192.168.0.102:21
TCP 00:19 FIN_WAIT 192.168.0.5:1310 192.168.0.100:10499 192.168.0.102:10499
TCP 00:14 FIN_WAIT 192.168.0.5:1306 192.168.0.100:14859 192.168.0.102:14859
TCP 00:14 FIN_WAIT 192.168.0.5:1305 192.168.0.100:21 192.168.0.102:21
IP 28:19 ERR! 192.168.0.5:0 0.0.0.21:0 192.168.0.102:0
[火星人 ] LB群集--lvs-dr模型已經有939次圍觀