IBM GPFS (General Parallel File System ,GPFS)是一款并行的文件系统，它保证在资源组内的所有节点可以并行访问整个文件系统，而且针对此文件系统的服务操作，可以同时安全地在此文件系统的多个节点上实现。GPFS 允许客户共享文件，而这些文件可能分布在不同节点的不同硬盘上，保证了数据的一致性和完整性。GPFS支持多种平台的部署，如Windows、Linux、AIX，每种环境部署方式相同，降低了软件部署的复杂度。
Extracting License Acceptance Process Tool to /usr/lpp/mmfs/5.1.8.1 ... tail -n +660 ./Storage_Scale_Data_Access-5.1.8.1-x86_64-Linux-install | tar -C /usr/lpp/mmfs/5.1.8.1 -xvz --exclude=installer --exclude=*_rpms --exclude=*_debs --exclude=*rpm --exclude=*tgz --exclude=*deb --exclude=*tools* 1> /dev/null
Installing JRE ...
If directory /usr/lpp/mmfs/5.1.8.1 has been created or was previously created during another extraction, .rpm, .deb, and repository related files in it (if there were) will be removed to avoid conflicts with the ones being extracted.
tail -n +660 ./Storage_Scale_Data_Access-5.1.8.1-x86_64-Linux-install | tar -C /usr/lpp/mmfs/5.1.8.1 --wildcards -xvz ibm-java*tgz 1> /dev/null tar -C /usr/lpp/mmfs/5.1.8.1/ -xzf /usr/lpp/mmfs/5.1.8.1/ibm-java*tgz
The Programs listed below are licensed under the following License Information terms and conditions in addition to the Program license terms previously agreed to by Client and IBM. If Client does not have previously agreed to license terms in effect for the Program, the International Program License Agreement (i125-3301-15) applies.
Program Name (Program Number): IBM Storage Scale Erasure Code Edition 5.1.8.1 (5737-J34) IBM Storage Scale Data Management Edition 5.1.8.1 (5737-F34) IBM Storage Scale Data Management Edition 5.1.8.1 (5641-DM1) IBM Storage Scale Data Management Edition 5.1.8.1 (5641-DM3)
Press Enter to continue viewing the license agreement, or enter "1" to accept the agreement, "2" to decline it, "3" to print it, "4" to read non-IBM terms, or "99" to go back to the previous screen. 1
Removing License Acceptance Process Tool from /usr/lpp/mmfs/5.1.8.1 ... rm -rf /usr/lpp/mmfs/5.1.8.1/LAP_HOME /usr/lpp/mmfs/5.1.8.1/LA_HOME
Removing JRE from /usr/lpp/mmfs/5.1.8.1 ... rm -rf /usr/lpp/mmfs/5.1.8.1/ibm-java*tgz
================================================================== Product packages successfully extracted to /usr/lpp/mmfs/5.1.8.1
Cluster installation and protocol deployment To install a cluster or deploy protocols with the IBM Storage Scale Installation Toolkit: /usr/lpp/mmfs/5.1.8.1/ansible-toolkit/spectrumscale -h
To install a cluster manually: Use the GPFS packages located within /usr/lpp/mmfs/5.1.8.1/gpfs_<rpms/debs>
To upgrade an existing cluster using the IBM Storage Scale Installation Toolkit: 1) Review and update the config: /usr/lpp/mmfs/5.1.8.1/ansible-toolkit/spectrumscale config update 2) Update the cluster configuration to reflect the current cluster config: /usr/lpp/mmfs/5.1.8.1/ansible-toolkit/spectrumscale config populate -N <node> 3) Use online or offline upgrade depending on your requirements: - Run the online rolling upgrade: /usr/lpp/mmfs/5.1.8.1/ansible-toolkit/spectrumscale upgrade -h - Run the offline upgrade: /usr/lpp/mmfs/5.1.8.1/ansible-toolkit/spectrumscale upgrade config offline -N; /usr/lpp/mmfs/5.1.8.1/ansible-toolkit/spectrumscale upgrade run You can also run the parallel offline upgrade to upgrade all nodes parallely after shutting down GPFS and stopping protocol services on all nodes. You can run the parallel offline upgrade on all nodes in the cluster, not on a subset of nodes.
To add nodes to an existing cluster using the IBM Storage Scale Installation Toolkit: 1) Add nodes to the cluster definition file: /usr/lpp/mmfs/5.1.8.1/ansible-toolkit/spectrumscale node add -h 2) Install IBM Storage Scale on the new nodes: /usr/lpp/mmfs/5.1.8.1/ansible-toolkit/spectrumscale install -h 3) Deploy protocols on the new nodes: /usr/lpp/mmfs/5.1.8.1/ansible-toolkit/spectrumscale deploy -h
To add NSDs or file systems to an existing cluster using the IBM Storage Scale Installation Toolkit: 1) Add NSDs or file systems to the cluster definition: /usr/lpp/mmfs/5.1.8.1/ansible-toolkit/spectrumscale nsd add -h 2) Install the NSDs or file systems: /usr/lpp/mmfs/5.1.8.1/ansible-toolkit/spectrumscale install -h
To update the cluster definition to reflect the current cluster config examples: /usr/lpp/mmfs/5.1.8.1/ansible-toolkit/spectrumscale config populate -N <node> 1) Manual updates outside of the installation toolkit 2) Sync the current cluster state to the installation toolkit prior to upgrade 3) Switching from a manually managed cluster to the installation toolkit
=================================================================================== To get up and running quickly, consult the IBM Storage Scale Protocols Quick Overview: https://www.ibm.com/docs/en/STXKQY_5.1.8/pdf/scale_povr.pdf ===================================================================================
构建完成后，终端会输出生成的包的位置，然后，我们可以将生成的包复制到其他机器进行部署。默认情况下，生成的包只能部署到架构、分发级别、Linux 内核和 IBM Spectrum Scale 维护级别与构建 gpfs.gplbin 包的机器相同的机器上。不过仍然建议在每个 server 节点上执行构建操作生成本机的 GPFS 可移植层。
相关命令: （以下操作需要在每个 server 节点上执行）
# 构建 gpfs 可移植层， mmbuildgpl --build-package
相关操作日志:
[root@node01 data]# mmbuildgpl --build-package -------------------------------------------------------- mmbuildgpl: Building GPL (5.1.8.1) module begins at Thu Jun 19 21:40:45 CST 2024. -------------------------------------------------------- Verifying Kernel Header... kernel version = 41800348 (418000348000000, 4.18.0-348.el8.x86_64, 4.18.0-348) module include dir = /lib/modules/4.18.0-348.el8.x86_64/build/include module build dir = /lib/modules/4.18.0-348.el8.x86_64/build kernel sourcedir = /usr/src/linux-4.18.0-348.el8.x86_64/include Found valid kernel header file under /usr/src/kernels/4.18.0-348.el8.x86_64/include Getting Kernel Cipher mode... Will use skcipher routines Verifying Compiler... make is present at /bin/make cpp is present at /bin/cpp gcc is present at /bin/gcc g++ is present at /bin/g++ ld is present at /bin/ld Verifying rpmbuild... Verifying libelf devel package... Verifying elfutils-libelf-devel is installed ... Command: /bin/rpm -q elfutils-libelf-devel The required package elfutils-libelf-devel is installed Verifying Additional System Headers... Verifying kernel-headers is installed ... Command: /bin/rpm -q kernel-headers The required package kernel-headers is installed make World ... make InstallImages ... make rpm ... Wrote: /root/rpmbuild/RPMS/x86_64/gpfs.gplbin-4.18.0-348.el8.x86_64-5.1.8-1.x86_64.rpm -------------------------------------------------------- mmbuildgpl: Building GPL module completed successfully at Thu Jun 19 21:41:13 CST 2024. --------------------------------------------------------
3.4、创建集群
GPFS 的仲裁机制和 ZooKeeper 的仲裁机制类似，当有一半以上的节点是 quorum 时，集群才可以启动，即： quorum >= 1 + sizeof(all nodes) / 2 。
[root@node01 data]# mmcrcluster -N /etc/mmfs/nodefile -C gpfscluster -r /usr/bin/ssh -R /usr/bin/scp -A mmcrcluster: Performing preliminary node verification ... mmcrcluster: Processing quorum and other critical nodes ... mmcrcluster: Finalizing the cluster data structures ... mmcrcluster: Command successfully completed mmcrcluster: Warning: Not all nodes have proper GPFS license designations. Use the mmchlicense command to designate licenses as needed. mmcrcluster: [I] The cluster was created with the tscCmdAllowRemoteConnections configuration parameter set to "no". If a remote cluster is established with another cluster whose release level (minReleaseLevel) is less than 5.1.3.0, change the value of tscCmdAllowRemoteConnections in this cluster to "yes". mmcrcluster: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.
[root@node01 data]# mmchlicense server --accept -N all
The following nodes will be designated as possessing server licenses: node01 node02 node03 mmchlicense: Command successfully completed mmchlicense: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.
auto : 磁盘类型为 nvme 或 scsi。IBM Storage Scale 将尝试自动检测实际磁盘类型。
相关命令: （以下操作仅在 node01 上执行即可）
# 创建 nsd mmcrnsd -F /etc/mmfs/nsdfile
# 查看 nsd mmlsnsd -m
# 启动集群 mmstartup -a
# 查看集群状态 mmgetstate -Las
相关操作日志:
[root@node01 data]# mmcrnsd -F /etc/mmfs/nsdfile mmcrnsd: Processing disk sdc mmcrnsd: Processing disk sdd mmcrnsd: Processing disk sdc mmcrnsd: Processing disk sdd mmcrnsd: Processing disk sdc mmcrnsd: Processing disk sdd mmcrnsd: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.
[root@node01 data]# mmlsnsd -m
Disk name NSD volume ID Device Node name or Class Remarks ------------------------------------------------------------------------------------------- data01 0A321B396854D5C1 /dev/sdc node01 server node data02 0A321B396854D5C2 /dev/sdd node01 server node data03 0A321B3A6854D5C3 /dev/sdc node02 server node data04 0A321B3A6854D5C4 /dev/sdd node02 server node data05 0A321B3B6854D5C5 /dev/sdc node03 server node data06 0A321B3B6854D5C6 /dev/sdd node03 server node
[root@node01 data]# mmstartup -a Fri Jun 20 10:34:58 CST 2024: mmstartup: Starting GPFS ... node01: The GPFS subsystem is already active. node02: The GPFS subsystem is already active. node03: The GPFS subsystem is already active.
[root@node01 data]# mmgetstate -Las
Node number Node name Quorum Nodes up Total nodes GPFS state Remarks --------------------------------------------------------------------------------- 1 node01 2 3 3 active quorum node 2 node02 2 3 3 active quorum node 3 node03 2 3 3 active quorum node
Summary information --------------------- Number of nodes defined in the cluster: 3 Number of local nodes active in the cluster: 3 Number of remote nodes joined in this cluster: 0 Number of quorum nodes defined in the cluster: 3 Number of quorum nodes active in the cluster: 3 Quorum = 2, Quorum achieved
The following disks of defaultfs will be formatted on node node01: data01: size 102400 MB data02: size 102400 MB data03: size 102400 MB data04: size 102400 MB data05: size 102400 MB data06: size 102400 MB Formatting file system ... Disks up to size 1.56 TB can be added to storage pool system. Creating Inode File 31 % complete on Fri Jun 20 10:49:07 2024 46 % complete on Fri Jun 20 10:49:12 2024 74 % complete on Fri Jun 20 10:49:17 2024 100 % complete on Fri Jun 20 10:49:21 2024 Creating Allocation Maps Creating Log Files Clearing Inode Allocation Map Clearing Block Allocation Map Formatting Allocation Map for storage pool system Completed creation of file system /dev/defaultfs. mmcrfs: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.
[root@node01 data]# mmlsfs all
File system attributes for /dev/defaultfs: ========================================== flag value description ------------------- ------------------------ ----------------------------------- -f 8192 Minimum fragment (subblock) size in bytes -i 4096 Inode size in bytes -I 32768 Indirect block size in bytes -m 2 Default number of metadata replicas -M 2 Maximum number of metadata replicas -r 2 Default number of data replicas -R 2 Maximum number of data replicas -j scatter Block allocation type -D nfs4 File locking semantics in effect -k all ACL semantics in effect -n 32 Estimated number of nodes that will mount file system -B 4194304 Block size -Q none Quotas accounting enabled none Quotas enforced none Default quotas enabled --perfileset-quota no Per-fileset quota enforcement --filesetdf no Fileset df enabled? -V 31.00 (5.1.7.0) File system version --create-time Fri Jun 20 11:32:26 2024 File system creation time -z no Is DMAPI enabled? -L 33554432 Logfile size -E yes Exact mtime mount option -S relatime Suppress atime mount option -K whenpossible Strict replica allocation option --fastea yes Fast external attributes enabled? --encryption no Encryption enabled? --inode-limit 615424 Maximum number of inodes --uid 3B1B320A:6854D64A File system UID --log-replicas 0 Number of log replicas --is4KAligned yes is4KAligned? --rapid-repair yes rapidRepair enabled? --write-cache-threshold 0 HAWC Threshold (max 65536) --subblocks-per-full-block 512 Number of subblocks per full block -P system Disk storage pools in file system --file-audit-log no File Audit Logging enabled? --maintenance-mode no Maintenance Mode enabled? --flush-on-close no flush cache on file close enabled? --auto-inode-limit no Increase maximum number of inodes per inode space automatically? --nfs4-owner-write-acl yes NFSv4 implicit owner WRITE_ACL permission enabled? -d data01;data02;data03;data04;data05;data06 Disks in file system -A yes Automatic mount option -o none Additional mount options -T /gpfsdata Default mount point --mount-priority 0 Mount priority
3.7、挂载文件系统
该方式用于在 server 节点上挂载测试文件系统。如果需要在其他客户端上挂载测试文件系统，建议查看第四栏目中的集群运维操作。
相关命令:
# 挂载文件系统 # 所有 server 上均挂载，执行时间可能会长一些 mmmount defaultfs /gpfsdata -N all
# 查看挂载点信息 df -hT
# 访问文件系统目录 ls -al /gpfsdata
# 取消挂载 mmumount /gpfsdata
相关操作日志:
[root@node01 data]# mmmount defaultfs /gpfsdata -N all Fri Jun 20 11:34:12 CST 2024: mmmount: Mounting file systems ...
[root@node01 data]# df -hT /gpfsdata Filesystem Type Size Used Avail Use% Mounted on defaultfs gpfs 600G 6.5G 594G 2% /gpfsdata
[root@node01 data]# mmperfmon config generate --collectors node01,node02 mmperfmon: Node node02 is not a perfmon node. mmperfmon: Node node01 is not a perfmon node. mmperfmon: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.
[root@node01 data]# mmchnode --perfmon -N node01,node02,node03 Fri Jun 20 13:59:26 CST 2024: mmchnode: Processing node node03 Fri Jun 20 13:59:26 CST 2024: mmchnode: Processing node node02 Fri Jun 20 13:59:26 CST 2024: mmchnode: Processing node node01 mmchnode: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.
# 客户端挂载 nfs export # 可以使用 ces vip 的两个任意一个 vip 连接 mount -t nfs -o vers=4,ro 10.10.0.102:/gpfsdata/nfsexport01 /mnt/share
相关操作日志:
[root@node02 data]# mmchconfig cesSharedRoot=/gpfsdata mmchconfig: Command successfully completed mmchconfig: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.
[root@node02 data]# mmchnode --ces-enable -N node02,node03 Fri Jun 20 15:49:12 CST 2024: mmchnode: Processing node node03 Fri Jun 20 15:49:19 CST 2024: mmchnode: Processing node node02 mmchnode: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.
Cluster Export Services global parameters ----------------------------------------- Shared root directory: /gpfsdata Enabled Services: None Log level: 0 Address distribution policy: even-coverage
Node Daemon node name IP address CES IP address list ----------------------------------------------------------------------- 2 node02 10.10.0.2 10.10.0.102 3 node03 10.10.0.3 10.10.0.103
[root@node02 data]# mmces service enable nfs mmchconfig: Command successfully completed mmchconfig: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process. node02: NFS: service already running. node03: NFS: service already running.
