8.3 Software Release
Software Releaserefers to the process in which experiment software administrators publish experiment-related software required by the computing environment, allowing jobs submitted by experiment users to load the necessary software packages.
The computing center uses CVMFS to store software related to each experiment group.
cvmfspublisher.ihep.ac.cn is the software publishing server. It only supports login by system users using private keys. To ensure that the experiment software compiled by administrators depends solely on /cvmfs in a system environment with only /cvmfs mounted—so that the software can run normally without relying on any other network storage systems—the server's environment is identical to the login nodes, but without any additional network storage systems. This guarantees the independence of the build environment.
The specific release process and architecture are shown below:
Experiment Software Release Procedure:
Submit a Request
The experiment software administrator contacts the Computing Center to request a customized software repository name $repo, and provides their personal computing cluster account: useraccount.
The CVMFS administrator creates a software publishing account repopubaccount and a software repository /cvmfs/$repo, and synchronizes it to the cvmfssoftware.ihep.ac.cn server.
The administrator also configures editing permissions for useraccount on the /cvmfs/$repo path.
Log in to the Publishing Server (Two Methods)：
2.1 Method 1: Kerberos Ticket，
The experiment software administrator uses their Kerberos ticket (viewable via klist) to log in to cvmfssoftware.ihep.ac.cn from lxlogin.ihep.ac.cn:
[user@localhost ~] -bash-4.2$ klist Ticket cache: FILE:/tmp/krb5cc_10517 Default principal: huqb@IHEPKRB5 Valid starting Expires Service principal 28/08/24 19:04:38 30/08/24 19:04:35 krbtgt/IHEPKRB5@IHEPKRB5 renew until 04/09/24 19:04:35 [user@localhost ~] ssh repopubaccount@cvmfspublisher.ihep.ac.cn2.2 Method 2: Private Key
[user@localhost ~] ssh repopubaccount@cvmfspublisher.ihep.ac.cn -i yourprivatekey
Switch to Release Mode 3.1 Check Current Repository Status
[user@cvmfspublisher ~]$ cvmfs_server list | grep repositoryname | grep transactionIf the repository is in a transaction state, it means another administrator is editing it. You must coordinate, wait until they finish, and avoid releasing their test versions to production.
3.2 Start a Transaction If no transaction is active:
[user@cvmfspublisher ~]$ cvmfs_server transaction repositorynameMaximum editing time is 24 hours. If it exceeds 24h, publish will fail with a tokens expire error.
Compile Software
There are two main approaches for building software in a CVMFS repository：
a.Build on cvmfssoftware.ihep.ac.cn and sync via rsync (Recommended) The software administrator logs in using their cluster account, compiles the software, and edits content under /cvmfs/$repo. After compilation:
[user@cvmfspublisher ~]$ rsync -avz --force useraccount@cvmfssoftware:/cvmfs/$repo.ihep.ac.cn/ /cvmfs/$repo.ihep.ac.cn/Important Notes: CVMFS admin will sync /cvmfs/hunt.ihep.ac.cn/ only once initially.
After that, experiment software administrators must maintain consistency themselves.
Before compiling, always check that: cvmfssoftware.ihep.ac.cn:/cvmfs/$repo.ihep.ac.cn/ is consistent with the cluster environment, to prevent unexpected content from being published.
b.Build Using OS Environment with hep_container On cvmfspublisher.ihep.ac.cn, use the container environment:
[user@cvmfspublisher ~]$ /cvmfs/container.ihep.ac.cn/bin/hep_container shell SL5/SL6/SL7 Singularity yourimage.sif:~> (Make changes to files in the R/W mount: /cvmfs/<repo>/dirA & All /cvmfs/*.ihep.ac.cn can be accessed ) Singularity yourimage.sif:~> exitIf an error occurs (build failure, timeout, etc.):
[user@cvmfspublisher ~]$ cvmfs_server abort repositorynameThen redo:
[user@cvmfspublisher ~]$ cvmfs_server transaction repositorynamePublish Software
After syncing or building:
[user@cvmfspublisher ~]$ cd /cvmfs [user@cvmfspublisher ~]$ cvmfs_server publish -a "tagname" -m "tag description" yourrepositorynameView Recent Release Versions
[user@cvmfspublisher ~]$$ cvmfs_server tag -l hxmt.ihep.ac.cn Name │ Revision │ Channel │ Timestamp │ Branch │ Description ─────────────────────────────┼──────────┼─────────┼──────────────────────┼────────┼───────────── trunk-previous │ 1 │ trunk │ 13 Jul 2020 14:19:08 │ │ default undo target generic-2020-07-13T06:19:07Z │ 1 │ trunk │ 13 Jul 2020 14:19:08 │ │ trunk │ 2 │ trunk │ 13 Jul 2020 15:40:53 │ │ current HEAD origin │ 2 │ trunk │ 13 Jul 2020 15:40:53 │ │ create ─────────────────────────────┴──────────┴─────────┴──────────────────────┴────────┴───────────── listing contains 4 tagsNew software becomes accessible roughly 10 minutes after successful publishing.
Common Issues & Solutions
8.1 Repository Files Being Accessed → Publish Fails
[bes3pub@cvmfspublisher ~]$ cvmfs_server publish -m "boss665p01" bes3.ihep.ac.cn Using auto tag 'generic-2020-07-16T09:34:18Z' WARNING! There are open read-only file descriptors in /cvmfs/bes3.ihep.ac.cn --> This is potentially harmful and might cause problems later on. We can anyway perform the requested operation, but this will most likely break other processes with open file descriptors on /cvmfs/bes3.ihep.ac.cn! The following lsof report might show the processes with open file handles ##Find open file handles [bes3pub@cvmfspublisher ~]$ lsof /cvmfs/bes3.ihep.ac.cn COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rm 95788 bes3pub cwd DIR 0,68 23 9237356 /cvmfs/bes3.ihep.ac.cn/bes3sw/Boss rm 95788 bes3pub 4r DIR 0,68 10 9255952 /cvmfs/bes3.ihep.ac.cn/bes3sw/Boss/6.6.5.p01/Generator/Bhlumi/Bhlumi-00-00-12/x86_64-slc6-gcc46-opt (deleted) rm 95788 bes3pub 5r DIR 0,68 10 9255873 /cvmfs/bes3.ihep.ac.cn/bes3sw/Boss/6.6.5.p01/Generator/Bhlumi (deleted) rm 95788 bes3pub 6r DIR 0,68 10 9255874 /cvmfs/bes3.ihep.ac.cn/bes3sw/Boss/6.6.5.p01/Generator/Bhlumi/Bhlumi-00-00-12 (deleted) ##Investigate via [bes3pub@cvmfspublisher ~]$ strace -p 95788 strace: Process 95788 attached --- stopped by SIGTSTP --- ^Cstrace: Process 95788 detached ##Kill the stuck process [bes3pub@cvmfspublisher ~]$ kill -9 95788 ##Re-run publish [bes3pub@cvmfspublisher ~]$ cvmfs_server publish yourrepository8.2 Incremental changes exceed 2GB → publish hangs，contact: huqb@ihep.ac.cn
Experiment Software Access Procedure
Users can access experiment software after installing the CVMFS client.
install cvmfs client：
a. install cvmfs and cvmfs-ihep-config
[user@localhost ~]# yum -y install https://ecsft.cern.ch/dist/cvmfs/cvmfs-release/cvmfs-release-latest.noarch.rpm [user@localhost ~]# yum -y install cvmfs cvmfs-config-default [user@localhost ~]# cvmfs_config setupb. Configure default.local
[user@localhost ~]# cat /etc/cvmfs/default.local CVMFS_REPOSITORIES=bes3.ihep.ac.cn,bes.ihep.ac.cn,cvmfs-config.cern.ch CVMFS_CACHE_BASE=/scratch/cvmfs CVMFS_QUOTA_LIMIT=40960 CVMFS_HTTP_PROXY=DIRECT OR CVMFS_HTTP_PROXY="http://proxyIP:3128"IHEP-site proxy: 202.122.33.52 SDU-site proxy: cvmfs.hepg.sdu.edu.cn
c. Disable SELinux enforcing
[user@localhost ~]# setenforce 0d. Configure default stratum server
[user@localhost ~]# curl -o /etc/cvmfs/domain.d/cern.ch.local http://cvmfs-stratum-one.ihep.ac.cn/cvmfs/software/client_configure/cern.ch.local [user@localhost ~]# ln -s /etc/cvmfs/domain.d/cern.ch.local /etc/cvmfs/domain.d/egi.eu.local [user@localhost ~]# ln -s /etc/cvmfs/domain.d/cern.ch.local /etc/cvmfs/domain.d/opensciencegrid.org.locale. Configure ihep.ac.cn domain and import key
[user@localhost ~]# mkdir /etc/cvmfs/keys/ihep.ac.cn [user@localhost ~]# curl -o /etc/cvmfs/keys/ihep.ac.cn/ihep.ac.cn.pub http://cvmfs-stratum-one.ihep.ac.cn/cvmfs/software/client_configure/ihep.ac.cn/ihep.ac.cn.pub [user@localhost ~]# curl -o /etc/cvmfs/domain.d/ihep.ac.cn.conf http://cvmfs-stratum-one.ihep.ac.cn/cvmfs/software/client_configure/ihep.ac.cn.conff. Verify CVMFS client
[user@localhost ~]# cvmfs_config probeInstallation complete.