Openstack高可用部署指導手冊

1、0.簡介21.需求21.1 四類節(jié)點：21.2 整體架構(gòu)31.3 IP 地址規(guī)劃42.網(wǎng)絡節(jié)點52.1 準備工作52.2網(wǎng)絡規(guī)劃52.3 OpenvSWtich（部分1）62.4 Quantum62.5. OpenVSwitch (部分2)82.6. HAProxy82.7. Corosync and Pacemaker132.8. Ceph (on R610-5 node only)153.控制節(jié)點153.1準備工作153.2. Networking163.3. MySQL163.4. RabbitMQ173.5. DRBD183.6. Pacemaker and Corosync193.7

2、. Create Databases223.8. Ceph233.9. Keystone253.10. Glance263.11. Quantum283.12. Nova293.13. Cinder313.14. Horizon334. Compute Nodes334.1. Preparing the Node334.2. Networking344.3. KVM344.4. OpenVSwitch354.5. Quantum364.6. Ceph364.7. Nova375. Swift Node395.1. Preparing the Node395.2. Networking395.3

3、. Swift Storage405.4. Swift Proxy500.簡介本手冊指導大家如何一步一步構(gòu)建一套多節(jié)點的高可用性（High Availability）Openstack 云平臺，該套平臺同時利用Ceph作為Glance和Cinder的后端存儲，Swift作為對象存儲，Openvswitch作為Quantum 組件1.需求1.1 四類節(jié)點：Controller，Network，Compute and Swift1.2 整體架構(gòu)1.3 IP 地址規(guī)劃HostnameHW modelRoleeth0(external)eth1(mgmt)eth2(vm traffic)eth3(st

4、orage)R710-1R710SwiftR710-2R710Controller_bakR710-3R710ControllerR710-4R710NetworkR710-5R710Network_bakR710-7R710ComputeR710-8R710ComputeVIP-APIVIP-MysqlVIP-Rabbitmq2.網(wǎng)絡節(jié)點2.1 準備工作·安裝Uubuntu 13.04·添加ceph節(jié)點條目到 /etc/hosts文件

5、 R710-3 R710- R610-5更新系統(tǒng)apt-get update -yapt-get upgrade -yapt-get dist-upgrade -y·安裝ntp服務 apt-get install -y ntp添加控制節(jié)點作為NTP服務器，然后重啟服務echo "server " >> /etc/ntp.confecho "server " >> /etc/ntp.confservice ntp rest

6、art·安裝其他服務apt-get install -y vlan bridge-utils·開啟 IP_Forwardingsed -i 's/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/' /etc/sysctl.confsysctl -p2.2網(wǎng)絡規(guī)劃 編輯/etc/network/interfaces, 下面的示例是R610-5 節(jié)點, R610-4 節(jié)點根據(jù)實際作出相關(guān)修改. 同時R610-4 不需要配置eth3 storage IP ，因為其上并沒有ceph組件運行auto eth0iface eth

7、0 inet static0dns-nameservers #Openstack managementauto eth1iface eth1 inet staticaddress netmask #VM trafficauto eth2iface eth2 inet staticaddress netmask #Storage network for cephauto eth3iface eth3 inet staticaddress netmask 255.255.2

8、55.0重啟網(wǎng)絡service networking restart2.3 OpenvSWtich（部分1） 安裝openVSwitchapt-get install -y openvswitch-switch openvswitch-datapath-dkms 創(chuàng)建網(wǎng)橋#br-int will be used for VM integrationovs-vsctl add-br br-int#br-ex is used to make to VM accessible from the external networkovs-vsctl add-br br-ex#br-eth2 is use

9、d to establish VM internal trafficovs-vsctl add-br br-eth2ovs-vsctl add-port br-eth2 eth22.4 Quantum 安裝Quantum openvswitch agent, l3 agent, dhcp agent and metadata-agent組件apt-get -y install quantum-plugin-openvswitch-agent quantum-dhcp-agent quantum-l3-agent quantum-metadata-agent 編輯/etc/quantum/qua

10、ntum.confDEFAULTauth_strategy = keystonerabbit_host = 01rabbit_password=yourpasswordkeystone_authtokenauth_host = 00auth_port = 35357auth_protocol = httpadmin_tenant_name = serviceadmin_user = quantumadmin_password = yourpasswordsigning_dir = /var/lib/quantum/keystone-signing 編輯

11、/etc/quantum/api-paste.inifilter:authtokenpaste.filter_factory = keystoneclient.middleware.auth_token:filter_factoryauth_host = 00auth_port = 35357auth_protocol = httpadmin_tenant_name = serviceadmin_user = quantumadmin_password = yourpassword 編輯 the OVS plugin configuration file /etc/quan

12、tum/plugins/openvswitch/ovs_quantum_plugin.ini with:DATABASEsql_connection = mysql:/quantum:yourpassword00/quantumOVStenant_network_type = vlannetwork_vlan_ranges = physnet1:1000:1100integration_bridge = br-intbridge_mappings = physnet1:br-eth2SECURITYGROUPfirewall_driver = quantum.agent.l

13、inux.iptables_firewall.OVSHybridIptablesFirewallDriver Update /etc/quantum/metadata_agent.ini with:DEFAULTauth_url = 00:35357/v2.0auth_region = RegionOneadmin_tenant_name = serviceadmin_user = quantumadmin_password = yourpasswordnova_metadata_ip = 00nova_metadata_port = 877

14、5metadata_proxy_shared_secret = demo 編輯 /etc/sudoers to give quantum user full access like:Defaults:quantum !requirettyquantum ALL=NOPASSWD: ALL 重啟所有服務cd /etc/init.d/; for i in $( ls quantum-* ); do sudo service $i restart; done2.5. OpenVSwitch (部分2) 編輯 the /etc/network/interfaces to become like thi

15、s:auto eth0iface eth0 inet manualup ifconfig $IFACE upup ip link set $IFACE promisc ondown ip link set $IFACE promisc offdown ifconfig $IFACE down 添加eth0 到網(wǎng)橋br-ex#此步驟執(zhí)行后，會失去internet連接，但不會影響openstack工作ovs-vsctl add-port br-ex eth0 添加external IP 到br-ex 使其恢復internet連接, 添加如下內(nèi)容到 /etc/network/inte

16、rfacesauto br-exiface br-ex inet static 重啟網(wǎng)絡以及quantum服務service networking restartcd /etc/init.d/; for i in $( ls quantum-* ); do sudo service $i restart; done2.6. HAProxy 安裝包到兩個網(wǎng)絡節(jié)點apt-get install -y haproxy Disable auto-start by 編輯ing /etc/default/haproxyENABLED=0 編輯 兩個節(jié)點的/etc/haproxy/haproxy.cfg配置

17、文件，內(nèi)容一致。global log local0 log local1 notice #log loghost local0 info maxconn 4096 #chroot /usr/share/haproxy user haproxy group haproxy daemon #debug #quietdefaultslog globalmaxconn 8000option redispatchretries 3timeout http-request 10stimeout queue 1mtimeout connect 10stimeout c

18、lient 1mtimeout server 1mtimeout check 10slisten dashboard_clusterbind 00:80balance sourceoption tcpkaoption httpchkoption tcplogserver R710-3 :80 check inter 2000 rise 2 fall 5server R710-2 :80 check inter 2000 rise 2 fall 5listen dashboard_cluster_internetbind 192.168

19、.1.43:80balance sourceoption tcpkaoption httpchkoption tcplogserver R710-3 :80 check inter 2000 rise 2 fall 5server R710-2 :80 check inter 2000 rise 2 fall 5listen glance_api_clusterbind 00:9292balance sourceoption tcpkaoption httpchkoption tcplogserver R710-3 10.10.10.

20、3:9292 check inter 2000 rise 2 fall 5server R710-2 :9292 check inter 2000 rise 2 fall 5listen glance_api_internet_clusterbind 3:9292balance sourceoption tcpkaoption httpchkoption tcplogserver R710-3 :9292 check inter 2000 rise 2 fall 5server R710-2 :9292 chec

21、k inter 2000 rise 2 fall 5listen glance_registry_clusterbind 00:9191balance sourceoption tcpkaoption tcplogserver R710-3 :9191 check inter 2000 rise 2 fall 5server R710-2 :9191 check inter 2000 rise 2 fall 5listen glance_registry_internet_clusterbind 3:9191ba

22、lance sourceoption tcpkaoption tcplogserver R710-3 :9191 check inter 2000 rise 2 fall 5server R710-2 :9191 check inter 2000 rise 2 fall 5listen keystone_admin_clusterbind 00:35357balance sourceoption tcpkaoption httpchkoption tcplogserver R710-3 :35357 check i

23、nter 2000 rise 2 fall 5server R710-2 :35357 check inter 2000 rise 2 fall 5server control03 3:35357 check inter 2000 rise 2 fall 5listen keystone_internal_clusterbind 00:5000balance sourceoption tcpkaoption httpchkoption tcplogserver R710-3 :5000 check inter

24、 2000 rise 2 fall 5server R710-2 :5000 check inter 2000 rise 2 fall 5listen keystone_public_clusterbind 3:5000balance sourceoption tcpkaoption httpchkoption tcplogserver R710-3 :5000 check inter 2000 rise 2 fall 5server R710-2 :5000 check inter 2000 rise 2 fa

25、ll 5listen memcached_clusterbind 00:11211balance sourceoption tcpkaoption tcplogserver R710-3 :11211 check inter 2000 rise 2 fall 5server R710-2 :11211 check inter 2000 rise 2 fall 5listen nova_compute_api1_clusterbind 00:8773balance sourceoption tcpkaoption t

26、cplogserver R710-3 :8773 check inter 2000 rise 2 fall 5server R710-2 :8773 check inter 2000 rise 2 fall 5listen nova_compute_api1_internet_clusterbind 3:8773balance sourceoption tcpkaoption tcplogserver R710-3 :8773 check inter 2000 rise 2 fall 5server R710-2

27、 :8773 check inter 2000 rise 2 fall 5listen nova_compute_api2_clusterbind 00:8774balance sourceoption tcpkaoption httpchkoption tcplogserver R710-3 :8774 check inter 2000 rise 2 fall 5server R710-2 :8774 check inter 2000 rise 2 fall 5listen nova_compute_api2_i

28、nternet_clusterbind 3:8774balance sourceoption tcpkaoption httpchkoption tcplogserver R710-3 :8774 check inter 2000 rise 2 fall 5server R710-2 :8774 check inter 2000 rise 2 fall 5listen nova_compute_api3_clusterbind 00:8775balance sourceoption tcpkaoption tcp

29、logserver R710-3 :8775 check inter 2000 rise 2 fall 5server R710-2 :8775 check inter 2000 rise 2 fall 5listen nova_compute_api3_internet_clusterbind 3:8775balance sourceoption tcpkaoption tcplogserver R710-3 :8775 check inter 2000 rise 2 fall 5server R710-2 1

30、:8775 check inter 2000 rise 2 fall 5listen cinder_api_clusterbind 00:8776balance sourceoption tcpkaoption httpchkoption tcplogserver R710-3 :8776 check inter 2000 rise 2 fall 5server R710-2 :8776 check inter 2000 rise 2 fall 5listen cinder_api_internet_clusterb

31、ind 3:8776balance sourceoption tcpkaoption httpchkoption tcplogserver R710-3 :8776 check inter 2000 rise 2 fall 5server R710-2 :8776 check inter 2000 rise 2 fall 5listen novnc_clusterbind 00:6080balance sourceoption tcpkaoption tcplogserver R710-3 :

32、6080 check inter 2000 rise 2 fall 5server R710-2 :6080 check inter 2000 rise 2 fall 5listen novnc_internet_clusterbind 3:6080balance sourceoption tcpkaoption tcplogserver R710-3 :6080 check inter 2000 rise 2 fall 5server R710-2 :6080 check inter 2000 rise 2 f

33、all 5listen quantum_api_clusterbind 00:9696balance sourceoption tcpkaoption httpchkoption tcplogserver R710-3 :9696 check inter 2000 rise 2 fall 5server R710-2 :9696 check inter 2000 rise 2 fall 5listen quantum_api_internet_clusterbind 3:9696balance sourceopt

34、ion tcpkaoption httpchkoption tcplogserver R710-3 :9696 check inter 2000 rise 2 fall 5server R710-2 :9696 check inter 2000 rise 2 fall 5如果haproxy正在運行，令其停止，隨后用pacemaker來接管它service haproxy stop2.7. Corosync and Pacemaker 安裝包apt-get install pacemaker corosync 在一個節(jié)點生成corosync密鑰文件 (R6

35、10-5)corosync-keygen#拷貝到另外一個節(jié)點scp /etc/corosync/authkey R610-4:/etc/corosync/authkey 編輯兩個節(jié)點的 /etc/corosync/corosync.conf , 用節(jié)點的eth1和eth3的地址替代"bindnetaddr" rrp_mode: active interface # The following values need to be set based on your environment ringnumber: 0 mcastport: 5405 interface # Th

36、e following values need to be set based on your environment ringnumber: 1 mcastport: 5405 開啟Corosync的自動啟動#編輯 /etc/default/corosyncSTART=yes·開啟corosync服務service corosync start 檢查corosync狀態(tài)crm status應該可以看到兩個節(jié)點處于online狀態(tài) Download 下載Haproxy的OCF agent腳本cd /usr/lib/ocf/resource.d/heartbeatwget chmod

37、755 haproxy 配置集群資源crm configureproperty stonith-enabled=falseproperty no-quorum-policy=ignorersc_defaults resource-stickiness=100rsc_defaults failure-timeout=0rsc_defaults migration-threshold=10property pe-warn-series-max="1000"property pe-input-series-max="1000"property pe-error

38、-series-max="1000"property cluster-recheck-interval="5min"primitive vip-mgmt ocf:heartbeat:IPaddr2 params ip=00 cidr_netmask=24 op monitor interval=5sprimitive vip-internet ocf:heartbeat:IPaddr2 params ip=3 cidr_netmask=24 op monitor interval=5sprimitive hapr

39、oxy ocf:heartbeat:haproxy params conffile="/etc/haproxy/haproxy.cfg" op monitor interval="5s"colocation haproxy-with-vips INFINITY: haproxy vip-mgmt vip-internetorder haproxy-after-IP mandatory: vip-mgmt vip-internet haproxyverifycommit#檢查pacemaker資源是否正確運行crm status2.8. Ceph (on

40、R610-5 node only)我們利用R610-5 作為第三方ceph監(jiān)控節(jié)點 安裝Ceph庫文件以及相應包wget -q -O- ' | sudo apt-key add -echo deb apt-get update -yapt-get install ceph 建立ceph-c 監(jiān)控目錄#R610-5mkdir /var/lib/ceph/mon/ceph-c3.控制節(jié)點3.1準備工作 安裝ubuntu 13.04在系統(tǒng)安裝進行分區(qū)過程中，請注意要為MYSQL和RabbitMQ預留一定的空間給DRBD使用，筆者這里直接使用兩個獨立的硬盤，也可以使用同一塊磁盤的不同分區(qū)添加c

41、eph節(jié)點條目到/etc/hosts文件 R710-3 R710- R610-5 更新系統(tǒng)apt-get update -yapt-get upgrade -yapt-get dist-upgrade -y 設(shè)置ntp服務器apt-get install -y ntp 添加另外一個控制節(jié)點作為ntp服務器，然后重啟ntp服務.#use for controller_bak nodeecho "server " >> /etc/ntp.confservice n

42、tp restart安裝其他服務apt-get install -y vlan bridge-utils 開啟ip轉(zhuǎn)發(fā)sed -i 's/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/' /etc/sysctl.confsysctl -p3.2. Networking 編輯 /etc/network/interfaces, R710-3 node, R710-2 node 根據(jù)實際情況進行修改，下面的示例是R70-3的auto eth0iface eth0 inet static#Openstack managementauto et

43、h1iface eth1 inet static#Storage network for cephauto eth2iface eth2 inet static重啟網(wǎng)絡service networking restart3.3. MySQL安裝MySQLapt-get install -y mysql-server python-mysqldb確保兩個節(jié)點的mysql有同樣的UID和GID，如果不同，請修改一致 配置mysql 接受所有請求sed -i 's///g' /etc/mysql/f還要注意修改f文件中的內(nèi)容mysql部分要補充修改/e

44、tc/mysql/f中datadir目錄指向預掛載drbd0的目錄，如我指定的/mnt/mysql,同時要編輯/etc/apparmod.d/usr.sbin.mysqld文件，將/var/lib/mysql的兩行全部注釋，同時添加/mnt/mysql/ r,/mnt/mysql/* rwk, 兩行 保存退出 編輯etc/init/mysql.conf注釋掉此行 #start on runlevel 2345#經(jīng)過實際測試，不應該先關(guān)掉mysql，否則會出現(xiàn)pacemaker中的mysql無法啟動的錯誤 Stop mysql to let pacemaker to manageservice

45、mysql stop#在首次做mysql和rabbitmq的HA配置時，如果出現(xiàn)pacemaker中出現(xiàn)mysql和rabbitmq stopped的錯誤，應該在本地預先啟動mysql和rabbitmq的進程（但二者數(shù)據(jù)文件目錄一定要指向的是DRBD共享目錄），最后用crm resource cleanup resoucename 來重啟指定資源或資源組。資源出現(xiàn)stopped的錯誤，實際上有時候可能是因為某些相關(guān)資源沒有啟動導致的，即資源和資源之間存在必然的聯(lián)系，啟動也因此具有先后順序，比如正常的順序是ms-drbd，fs，ip，service本身，所以排錯的時候要注意這一點3.4. Rab

46、bitMQ安裝RabbitMQapt-get install rabbitmq-server -y 確保兩個節(jié)點的rabbitmq有同樣的UID和GID，如果不同，請修改一致 關(guān)閉rabbitmq-server的開機自啟動 update-rc.d -f rabbitmq-server removeStop mysql to let pacemaker to manageservice rabbitmq-server stop3.5. DRBD安裝包apt-get install drbd8-utils xfsprogs -y關(guān)閉DRBD的開機自動啟動update-rc.d -f drbd re

47、move這個步驟執(zhí)行之前應該用pvcreate和vgcreate在兩臺控制節(jié)點分別創(chuàng)建好了名字為R710-3-vg和R710-2-vg的卷組，才能夠執(zhí)行下面的命令#注意在R710-2節(jié)點要更換VG name為R710-2-vg lvcreate R710-3-vg -n drbd0 -L 10Glvcreate R710-3-vg -n drbd1 -L 10G 加載DRBD模塊，Load DRBD modulemodprobe drbd#添加drbd到/etc/modules 文件echo "drbd" >> /etc/modules創(chuàng)建mysql DRBD

48、資源文件/etc/drbd.d/mysql.resresource drbd-mysql device /dev/drbd0; meta-disk internal; on R710-2 address :7788; disk /dev/mapper/R710-2-vg-drbd0; on R710-3 address :7788; disk /dev/mapper/R710-3-vg-drbd0; syncer rate 40M; net after-sb-0pri discard-zero-changes; after-sb-1pri discard

49、-secondary; Create rabbitmq DRBD resource file /etc/drbd.d/rabbitmq.resresource drbd-rabbitmq device /dev/drbd1; meta-disk internal; on R710-2 address :7789; disk /dev/mapper/R710-2-vg-drbd1; on R710-3 address :7789; disk /dev/mapper/R710-3-vg-drbd1; syncer rate 40M; net after-sb

50、-0pri discard-zero-changes; after-sb-1pri discard-secondary; After did configuration above on both nodes, bring up DRBD resources 在兩個節(jié)點都做好了上述配置之后，開啟DRBD resources#Both nodedrbdadm dump drbd-mysqldrbdadm dump drbd-rabbitmq #Both nodeCreate the metadata as follows:drbdadm create-md drbd-mysqldrbdadm c

51、reate-md drbd-rabbitmq #Both nodeBring resources up:drbdadm up drbd-mysqldrbdadm up drbd-rabbitmq#Both nodeCheck that both DRBD nodes have made communication and we'll see that the data is inconsistent as no initial synchronization has been made. For this we do the following:drbd-overview#And th

52、e result will be similar to: 0:drbd-mysql Connected Secondary/Secondary Inconsistent/Inconsistent C r- Initial DRBD Synchronization#Do this on 1st node only:drbdadm - -overwrite-data-of-peer primary drbd-mysqldrbdadm - -overwrite-data-of-peer primary drbd-rabbitmq #And this should show something similar to: 0:drbd-mysql Connected Secondary/Secondary UpToDate/UpToDate C r- Create filesystem#Do this on 1st node only:mkfs -t xfs /dev/drbd0mkfs -t xfs /dev/drbd1 Move/Copy mysql and rabbitmq files to DRBD resources#Do following on 1st node only 拷貝mysql and rabbitmq 文件到DRBD 資源#下面的配置兩個