Suse 11下多路径及udev配置

最近给客户基于SuSe 11 SP3下多路径部署Oracle 10g RAC。SuSe 11下用10g，也算一朵奇葩，连篇文档都比较难找，谁叫Oracle太贵呢。下面主要是描述了在该环境下如何去配置多路径。由于10g下的ocr与votingdisk不能直接存放到asm磁盘，所以依旧要使用raw设备方式来保存。下文供大家参考。

一、查看当前scsi设备及获取设备wwid
#使用 cat /proc/partitions或者/sbin/sfdisk命令查看当前系统中的scsi设备
suse11a:~ # more /proc/partitions
major minor #blocks name

8 0 877264896 sda
8 1 200812 sda1
8 2 104856255 sda2
8 3 104856255 sda3
8 4 667340100 sda4
8 16 2097152000 sdb
8 17 2097150976 sdb1
.....
8 240 1048576 sdp
65 0 1048576 sdq
.....
65 48 2097152000 sdt
65 49 2097150976 sdt1
253 0 2097152000 dm-0
253 1 2097152000 dm-1
253 2 1751121920 dm-2
253 3 1048576 dm-3
253 4 1048576 dm-4
253 5 1751120896 dm-5
253 6 2097150976 dm-6
253 7 2097150976 dm-7

#获取多路径设备的wwid，使用下面的命令来获取当前系统中多路径设备的wwid
for i in `cat /proc/partitions | awk {'print $4'} |grep sd`; do echo "### $i: `/lib/udev/scsi_id --whitelist --replace-whitespace /dev/$i`"; done

### sda: 3600605b007030d301a192f2d13c017a0
### sda1: 3600605b007030d301a192f2d13c017a0
.....................
### sdt: 360022a11000cb6b10a902db200000000
### sdt1: 360022a11000cb6b10a902db200000000
#获取后，过滤本地磁盘及重复的wwid

二、配置mulitpatch.conf文件
#配置mulitpatch.conf文件的主要目的是用于使用有意义，用户自定义的设备名称，持久化多路径设备
#这个步骤是与配置单路径设备最大的区别
#下面是已经配置过后的multipath.conf文件
suse11a:~ # cp /etc/multipath.conf /etc/multipath.conf.bak
suse11a:~ # vi /etc/multipath.conf
suse11a:~ # grep -v ^# /etc/multipath.conf
defaults {
user_friendly_names yes
}
blacklist {
devnode ^hd[a-z]
}
multipaths {
multipath {
wwid 36200bc710099b9190ba6fc350000000a
alias mpatha
}

multipath {
wwid 36200bc710099b9190ba7109b0000000b
alias mpathb
}
multipath {
wwid 36200bc710099b9190ab1155300000009
alias mpathc
}
multipath {
wwid 360022a11000cb6b10a902db200000000
alias mpathd
}
multipath {
wwid 360022a11000cb6b10a9116d000000002
alias mpathe
}
}

三、校验多路径设备
#使用下面的命令用于多路径配置生效
suse11a:~ # service multipathd stop
suse11a:~ # service multipathd start

#下面校验多路径设备
suse11a:~ # multipath -ll | grep mpath
mpathe (360022a11000cb6b10a9116d000000002) dm-1 HUAWEI ,S5500T
mpathd (360022a11000cb6b10a902db200000000) dm-0 HUAWEI ,S5500T
mpathc (36200bc710099b9190ab1155300000009) dm-2 HUAWEI ,S2600T
mpathb (36200bc710099b9190ba7109b0000000b) dm-4 HUAWEI ,S2600T
mpatha (36200bc710099b9190ba6fc350000000a) dm-3 HUAWEI ,S2600T

#下面是设备的详细信息
suse11a:~ # multipath -ll
mpathe (360022a11000cb6b10a9116d000000002) dm-1 HUAWEI ,S5500T
size=2.0T features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 5:0:0:1 sdc 8:32 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 5:0:3:1 sdk 8:160 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 6:0:2:1 sds 65:32 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
`- 6:0:3:1 sdu 65:64 active ready running
mpathd (360022a11000cb6b10a902db200000000) dm-0 HUAWEI ,S5500T
size=2.0T features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 5:0:0:0 sdb 8:16 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 5:0:3:0 sdj 8:144 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 6:0:2:0 sdr 65:16 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
`- 6:0:3:0 sdt 65:48 active ready running
.........

suse11a:~ # dmsetup ls |sort
mpatha (253:3)
mpathb (253:4)
mpathc (253:2)
mpathc_part1 (253:5)
mpathd (253:0)
mpathd_part1 (253:6)
mpathe (253:1)
mpathe_part1 (253:7)

#以下命令均可以用于校验多路径设备
# ll /dev/disk/by-id/
# ls -l /dev/dm-*
# ll /dev/mapper/

#/dev/dm-*下面的为动态名，即系统重启后会发生变化，此部分用于os 内部使用。
#/dev/mapper/下面查看到的内容为持久化名称，也即是我们在mulitpatch.conf文件中定义的。

四、配置RAW
#在Oracle 10g R2 rac安装的过程中，OUI不能够校验共享的快涉笔，因此需要使用raw。Oracle 11g R1后可以直接使用多路径设备。
#使用udev来管理raw，编辑raw文件，如下
suse11a:~ # cat /etc/raw | grep -v \#
raw1:mapper/mpatha
raw2:mapper/mpathb
raw3:mapper/mpathc
raw4:mapper/mpathd
raw5:mapper/mpathe

#重启raw以及配置开机启动raw
suse11a:~ # rcraw start
suse11a:~ # chkconfig raw on
suse11a:~ # raw -qa
/dev/raw/raw1: bound to major 253, minor 3
/dev/raw/raw2: bound to major 253, minor 4
/dev/raw/raw3: bound to major 253, minor 2
/dev/raw/raw4: bound to major 253, minor 0
/dev/raw/raw5: bound to major 253, minor 1

#测试raw的可访问性
# dd if=/dev/zero of=/dev/raw/raw1 bs=1024 count=100
100+0 records in
100+0 records out
102400 bytes (102 kB) copied, 0.762352 seconds, 134 kB/s

#配置权限，该权限用于oracle用户访问裸设备
# Author : Leshami
# Blog : http://blog.csdn.net/leshami
suse11a:~ # more /etc/udev/rules.d/99-raw.rules
KERNEL=="raw[1-2]*", GROUP="oinstall", MODE="664"
KERNEL=="raw[3-5]*", OWNER="oracle", GROUP="oinstall", MODE="664"

五、安装clusterware
#安装clusterware时，关于ocr和votingdisk，选择/dev/raw/raw1，/dev/raw/raw2，如果有配置冗余，此处应该为多个裸设备。
#校验安转好clusterware后裸设备的使用情况
suse11a:oracle:ora10g1 > ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 1048296
Used space (kbytes) : 4592
Available space (kbytes) : 1043704
ID : 892902908
Device/File Name : /dev/raw/raw1
Device/File integrity check succeeded

Device/File not configured

Cluster registry integrity check succeeded

suse11a:oracle:ora10g1 > crsctl query css votedisk
0. 0 /dev/raw/raw2

located 1 votedisk(s).
#对于ASM磁盘可以通过dbca来创建，直接使用/dev/raw/raw*以及/dev/mapper/*。当然suse 11支持asmlib，也可以通过asmlib方式来访问asm磁盘。