Introduction of LUSTRE file storage

LUSTRE is an open-source, extensible, high-performance and object-oriented parallel file system, which is widely used in super computing platforms. More than 70% of super computers in the world use LUSTRE file system. A LUSTRE file system consists of 3 parts, meta data servers(MDS), object storage servers(OSS) and clients. All the storage devices on object storage servers make up a unified storage space managed by meta data servers. The space can be mounted by LUSTRE clients and completely compatible with POSIX semantics. The LUSTRE file system is extremely extensible and have extremely high performance. Tens of petabytes storage space can be easily and well managed by it with aggregate bandwidth reaching tens or even hundreds of gigabytes per second. Therefore, LUSTRE file system is widely used in the field of high performance computing. However, due to the limit of meta data servers and no support on erasure codes and copies, LUSTRE file system has no advantage in maintaining mass small files. LUSTRE file system is mainly used to store mass experiment data and it is the most primary disk file system in Computing Center.

LUSTRE file storage overview

LUSTRE is the most widely used file system in high performance computing platforms. It provides mass storage service for experiments including BES, DYB and JUNO. The information of curret lustre file system is shown in Table 6.

Table 6 information about luster file system

• Lustre file system fully supports Linux POSIX semantics; users can use Linux file system access commands and APIs to access Lustre.
• If user ACL and extended attributes features are needed, please submit the request to ics@ihep.ac.cn

• /workfs is dedicated to user interactivities. It can be read and write on login nodes, but users on compute ndoes can not create or modify files under /workfs. That is, user jobs can not wirte or modify files under /workfs.

• Commands

Check disk quota for a user：

$ lfs quota -u username dirname For example：

$ lfs quota -u zhangsan /publicfsDisk

quotas for user zhangsan (uid XXXX): Filesystem kbytes quota limit grace files quota limit grace /publicfs 338626804 524288000 524288000 - 232010 300000 300100

​ (Used) (soft quota of capacity) (hard quota of capacity) (used inode) (soft quota of inode) (hard quota of inode)

Check usage of diskpool：

$ lfs df –h –p poolname dirname

-p follows with diskpool name，For example, under /publicfs, ATLAS group has disk pool named: acfs.atlaspool， CMS group has disk pool named acfs.cmspool, lhcb group has acfs.lhcbpool; under /sharefs, there is sharefs.hbkgpool,sharefs.mbhpool,sharefs.hepspool,sharefs.alipool

$ lfs df -h -p pubfs.cmspool /publicfs

$ lfs df -p acfs.atlaspool /publicfs |tail -n 3

filesystem summary: 934854246652 267931989532 619787497356 30% /publicfs

Set multi-stripes for a very large files , to improve I/O performance：

$lfs setstripe dirname –c n

n is number of stripe 。

$ lfs setstripe yourdirname –c 6

Set stripe rule for a directory, once set, new files created under "yourdirname" and its subdirectories will have 6 stripes. If you want to set stripe for a single file, touch an empty file first with its name and set stripe number on that empty file.

Set ACL for a directory or file：**

For example ：owner of /besfs4/wanglutest is root:root, and its perfmission is 750， normal user as wanglu(60007:600) does not have access to this direcotry

-bash-4.2$ ll /besfs4 |grep wanglutest

drwxr-x---+ 2 root  root   4096 May 11 15:24 wanglutest

-bash-4.2$ cd /besfs4/wanglutest

-bash: cd: /besfs4/wanglutest: Permission denied

use setfcal command to set ACL permisson for wanglu as (rwx))

setfacl -m user:wanglu:rwx /besfs4/wanglutest

check ACL permission of wanglu

-bash-4.2$ getfacl /besfs4/wanglutest
getfacl: Removing leading '/' from absolute path names
# file: besfs4/wanglutest
# owner: root
# group: root
user::rwx
user:wanglu:rwx
group::r-x
mask::rwx
other::---

Now user wanglu has access(rwx) to this directory

-bash-4.2$ pwd
/afs/ihep.ac.cn/users/w/wanglu
-bash-4.2$ cd /besfs4/wanglutest
-bash-4.2$ cp /etc/hosts .
-bash-4.2$ ls -lh
total 3.1M
-rw-r--r-- 1 wanglu u07 3.0M May 11 15:22 f1
-rw-r--r-- 1 wanglu u07  292 May 11 15:38 hosts

delete wanglu's permisson on directory /besfs4/wanglutest

setfacl -x user:wanglu /besfs4/wanglutest

Notice：

1. Only the owner of a directory/file has authority to set acl for a directory/file
2. After setting of ACL, there will be a “+” at the end of "ls -l " result , and the native Linux permisson rules will disfunced
3. To set acl for a user group, use command
   setfacl -m group:xxx /xxx/xxx

To clear ACL function on a direcotry or file, used command setfacl -b

#setfacl -b /besfs4/wanglutest

To get complete instructions on ACL setting, please refer to:

http://wiki.lustre.org/images/5/57/PosixAccessControlInLinux.pdf

or Linux manpage

# man getfacl 
# man setfacl