Analysis and Optimization of CVMFS caches on LxPlus

CERN openlab report 2024 Analysis and Optimization of CVMFS caches on LxPlus Author: Jakub Ogrodowczyk Supervisors: Valentin Volkl, [email protected] Steve Traylen, steve[email protected] August 15th, 2024 I. Introduction At CERN, Lxplus (Linux Public Login User Service) is a general-purpose, shared pool of Linux machines widely used by researchers and engineers for tasks such as data processing, simple computations, and interaction with other CERN services (e.g. EOS, AFS, OpenStack). On average, about 1,000 people use it each day. Each Lxplus box comes with a pre-configured Alma 9 system and provides users with access to a host featuring 9 cores and 30 GB of RAM. The configuration of each of the boxes is deployed using the puppet tool. Software distribution on Lxplus is handled through CVMFS (CernVM File System), a POSIX, read-only file system that operates in user space as a FUSE module developed by the CVMFS team from the SFT group. The service allows users to access the latest versions of experiment software without needing to install or update packages locally. Fig.1: CVMFS interaction diagram CVMFS works by mounting images of remote repositories under the /cvmfs directory (ex. /cvmfs/sft.cern.ch/ when needed. After the user tries to read a file from one of the repositories, the FUSE kernel module forwards the request to the CVMFS2 process running in the user space. The file is then fetched from the remote repository and stored 1 CERN openlab report 2024 in the local cache. Subsequent accesses to the file will bypass the fetching process and retrieve it directly from the cache (local storage) which greatly decreases the access time. The CVMFS configuration on Lxplus is, same as other tools, deployed using puppet. By default, the cache size is set to 25GB, shared between all the repositories. When the cache size exceeds this limit, half of the files are removed based on how long it has been since they were last accessed. The aim of this project was to gather data on the usage patterns of CVMFS on Lxplus, explore this data, test hypotheses proposed by the development team, and enhance the existing monitoring infrastructure. II. Data gathering A key objective of the project was to collect data on CVMFS usage across the Lxplus cluster. To achieve this, I used wassh - a parallel SSH tool that enables command execution across all hosts in the cluster. The approach I’ve chosen was running a bash script on each host, which executed a Python program and saved the output to a file in the eos service directory. There are a couple of tools designed to help probe the usage data out of the local CVMFS service. Most notably, the cvmfs_talk tool allows us to communicate with the running CVMFS2 process through the usage of a few internal commands. The ones I’ve utilized in this project are: > cvmfs_talk <command> * cache size - gets current size of files in the cache * cache list - gets the list of files in the cache * internal affairs - gets the list of internal counters and statistics * ... Listing1: Examples of cvmfs_talk subcommands Another method for collecting usage data and internal counters is through extended file attributes. Each mounted repository stores metadata in its extended attributes, which can be retrieved using Linux utilities. For example: > getfattr -d /cvmfs/sft.cern.ch # file: cvmfs/sft.cern.ch/ ... user.revision="30496" user.timestamp_last_ioerr="0" user.uptime="463" user.useddirp="0" user.usedfd="1" ... Listing2: Example of interaction with extended attributes For the purposes of the project, I’ve developed a python library that allows for easy communication with the CVMFS2 process. The library provides a wrapper around the cvmfs_talk tool that allows for easy execution of commands and parsing of the output. The bulk of library’s functionality relies on the run_cvmfs_talk_command function: 1def run_cvmfs_talk_command(repo_name, *commands): 2 command = ["cvmfs_talk", "-i", repo_name] + [c for c in commands] 3 result = subprocess.run(command, stdout=subprocess.PIPE) 4 5 return result Listing3: The run_cvmfs_talk_command function The rest of the library consists of functions that, through wrappers built around the previously mentioned tools, collect and parse the data into serializable objects. The specific data I focused on included: 2 CERN openlab report 2024 1. The list of currently mounted repositories 2. Size of the cache (local storage) 3. Cache revisions – using extended attributes 4. List of files in the cache 5. File to repository mapping 6. Size of files in the cache 7. Startup of the ROOT program Gathering the list of mounted repositories The list of mounted repositories can be easily obtained using the Linux mount -l command. The output can then be parsed using a regular expression ("/cvmfs/([^ ]+)"). Below is the python code: 1def get_cvmfs_mounted_repos(): 2 command = "mount -l | grep cvmfs2" 3 result = subprocess.run(command, shell=True, stdout=subprocess.PIPE) 4 5 lines = result.stdout.decode("utf-8").splitlines() 6 repos = [] 7 for line in lines: 8 m = re.search("/cvmfs/([^ ]+)", line) 9 repos.append(m.group(1)) 10 11 return repos Listing4: The function for getting the list of mounted repositories Gathering the size of the cache Size of the cache can be obtained directly from the cvmfs_talk tool. We only need to parse the output of the command which, as previously, can be done using a regular expression ("\\(([0-9]+)"). 1def get_cvmfs_cache_size(repo_name): 2 result = run_cvmfs_talk_command(repo_name, "cache", "size") 3 m = re.findall("\\(([0-9]+)", str(result.stdout)) 4 if not m: 5 throw_err("Could not find pattern for cache and pinned size.") 6 7 return (m[0], m[1]) Listing5: The function for getting the current size of the cache The function returns two values as a tuple. That is because the cvmfs_talk cache size command returns two separate numbers - the overall size of the cache and the size taken up by the the pinned files. Gathering cache revisions In order to get the local cache revision for each of the mounted repositories, we need to access the extended attributes. To accomplish that, I’ve decided to use the xattr python library. The list of attributes is returned as a dict data structure. The name of the attribute containing the revision number is “user.revision”. 3 CERN openlab report 2024 1def get_extended_attributes(repo_name): 2 full_path = "/cvmfs/" + repo_name 3 attribute_names = xattr.listxattr(full_path) 4 attributes = {} 5 for attribute_name in attribute_names: 6 name = ( 7 attribute_name 8 if isinstance(attribute_name, str) 9 else attribute_name.decode("utf-8") 10 ) 11 attributes[name] = xattr.getxattr(full_path, name).decode("utf-8") 12 13 return attributes Listing6: The function that gets extended attributes of a file as a dict Gathering the list of files in the cache Similarly to the size of the cache, this information is provided by the cvmfs_talk tool using the cache list sub-command. In this case, the return type is a set since the list we are getting out of cvmfs_talk contains paths relative to the repositories, without the repository name. For the simplicity of other operations, we won’t to have no duplicates in the final list, which could be the case if two repositories contained a file with the same relative path. 1def get_cvmfs_cached_files(repo_name): 2 result = run_cvmfs_talk_command(repo_name, "cache", "list") 3 lines = result.stdout.decode("utf-8").splitlines() 4 files = set() 5 6 for line in lines: 7 if line.startswith("/"): 8 files.add(line) 9 continue 10 prefix = "Part of " 11 if line.startswith(prefix): 12 files.add(line[len(prefix) :]) 13 14 return files Listing7: Function that returns the list of files in the cache Mapping file names to repositories As discussed in the previous point, there is no trivial way to find out which remote repository did a file in the cache come from, since 1. The cache is shared between all of the repositories 2. The cvmfs_talk cache list command returns paths relative to the repository Because of that, the easiest solution I came up with was to use a brute force algorithm that one by one prefixes the relative file path with the name of a repository and checks whether such a path exists using the os.path.exists(- path) function. 4 CERN openlab report 2024 1def find_repo_for_file_brute_force(file, mounted_repos): 2 for repo in mounted_repos: 3 path = f"/cvmfs/{repo}/{file}" 4 if os.path.exists(path): 5 return repo 6 7 return None Listing8: Function that by brute force, finds which repository does a file belong to There a couple of things to note about this approach. I : The way I gather the list of files removes duplicate entries. This means that if two or more repositories have a file at the same relative path, the code will only detect one of them, depending on the order of the list returned by the mount -l command. This obviously leads to discrepancies between the actual state of the system and the data we collect. However, since repositories are frequently unmounted (in the current configuration, a repository gets dismounted after one minute of not being used) and the remote image can be updated with a new revision at any time, it is impossible to capture the true state of the system. By the time we take a snapshot, both the remote and local environments have likely changed since the time the file was saved in the cache. Additionally, instances where this occurs are quite rare and account for a marginal fraction of the total files in the cache (there are on average 35,143 files in the cache). As a result, this has little impact on the overall accuracy of the measurement, and attempting to account for this would unnecessarily increase the code complexity without a significant gain in precision. II : For most of the data I’ve gathered I was checking the files against the list of currently mounted repositories which I got from the get_cvmfs_mounted_repos() function. That means that some of the files will not be matched against any repository in the list. In that case, the algorithm returns None. Those files make up around 23.5% of the overall cache. There are a few reason why I haven’t been checking against the list of all repositories: 1. No such list exists 2. The time would grow linearly with the number of checked repos 3. With time, the list could quite possibly go out of date After two months of gathering data, I’ve ended up with a list of repositories which have been mounted over that period and the number of unique ones turned out to be 66. Because I didn’t have access to that list at the beginning, I was unable to check against it. III : This approach is quite slow, as previously mentioned - there are on average 35,143 files in the cache and 10 repositories mounted at each time. In the worst case we will have a time complexity of 𝑂(𝑛𝑚) where 𝑛 - number of files, 𝑚 - number of repositories. In practice, this process has usually taken around 2 minutes for the whole cluster depending on the load of the system and other factors. This amount of time may seem like a lot, but because we do not want to run the data gathering too often, it falls within reasonable bounds. Final version of the data gathering tool checks the full list for files which were not matched to the list of mounted repositories. Due to linear complexity of the algorithm, checking against the list of all known repositories is around 6.6 times slower (𝑚 = 66 vs 𝑚 = 10) but because only around 25% of files come from unmounted repositories, that cost is not as great. Gathering the file sizes Another piece of data I was interested in was the size of files in the cache. Now that we have the full paths of the files in the cache it seems to be trivial and boils down to os.path.getsize(file_path). However, as noted previously, the algorithm checks each file against the list of currently mounted repositories and, because the repositories get unmounted, almost a quarter of the files end up without a match. Later versions of the algorithm had mostly 5 CERN openlab report 2024 eliminated that issue by checking the unmounted files against the list of all found repositories but the data gathered before that had a lot of file which were unmapped. Another thing to note is that some of the files end up being deleted in a new revision of a repository. In that case, this algorithm is unable to find the size of a file. An alternative, and possibly better approach, that solves that issue, is checking the sizes straight from the physical files existing on the disk without accessing them through the CVMFS file system. The problem with that solution, is that the names of those files are their content hashes and in order to map them to the actual file names, I would have to first read the database that CVMFS stored locally which contains that information. In the end, I decided to stick with the first approach as I already have implemented it and the drawbacks weren’t big enough to migrate to the second one. 1def get_file_size(file_name, repo, all_repos): 2 if not file_name.startswith('/'): 3 file_name = '/' + file_name 4 5 file_path = f"/cvmfs/{repo}{file_name}" 6 7 if repo is not None and os.path.exists(file_path): 8 return os.path.getsize(file_path) 9 10 repo = find_repo_for_file_brute_force(file_name, all_repos) 11 file_path = f"/cvmfs/{repo}{file_name}" 12 if os.path.exists(file_path): 13 return os.path.getsize(file_path) 14 15 return None Listing9: Function that finds a file size for a given file and repository Measuring ROOT startup Lastly, I wanted to measure the startup of the ROOT program. ROOT is an open-source data analysis framework developed by the SFT group (the group to which the CVMFS belongs to). It is widely used by researchers and data analysts for things like statistical analysis and data visualization. Because it’s such a powerful and versatile tool, it’s installation grows to quite a size. On Lxplus, ROOT is distributed using CVMFS and has a reputation of taking quite a bit of time on it’s first startup. Measuring the state of the environment when ROOT is run would allow to possibly find some optimizations in the future. The methodology I’ve ended up employing was as such: 1. Save the state of CVMFS before running ROOT 2. Run root across cluster and measure startup time 3. Save the CVMFS state again 4. Run root for the second time and measure the startup again The program I’ve used for the measurements creates a data frame with a 100 entries, defines a new column containing random numbers from a uniform distribution, creates a 1D histogram, and finally draws the histogram using DrawClone(). 1ROOT::RDataFrame rdf(100); 2auto rdf_x = rdf.Define("x", []() { 3 return gRandom->Rndm(); 4}); 5auto h = rdf_x.Histo1D("x"); 6h->DrawClone(); Listing10: An example ROOT program used for startup measurements 6 CERN openlab report 2024 The final bash command was root -q -b -e 'ROOT::RDataFrame rdf(100); auto rdf_x = rdf.Define("x", [](){ return gRandom->Rndm(); }); auto h = rdf_x.Histo1D("x"); h->DrawClone();' III. Expanding the monitoring infrastructure With a system as complex as Lxplus and over a thousand people using it each day, it’s very important to carefully monitor the state of the system at all times. The monitoring infrastructure consists of four parts: 1. data gathering 2. data storage 3. data visualization 4. alerting For the purposes of data visualization and alerting, the IT department uses a tool called Grafana. Grafana allows for an easy way to create dashboards with different visualizations of the collected data and provides an easy way to setup alarms if the newest data doesn’t fit within the safe parameters. Long term storage is done using InfluxDB. The data gathering is done using a plethora of different of tools. The one I was mostly interested in was collectd which is used for the collection of data on CVMFS. The existing infrastructure Previously, the monitored data consisted of the average mount time of a repository and the number of cache cleanups over the past 24 hours. If the average mount time spiked for an extended period, it would indicate that the connection between the cluster and the remote server was likely overloaded. The number of cleanups also had an alarm set and was typically triggered by suspicious user activity or misuse of the provided tools. A recent example involved a user running the following command: grep --color=auto -r POG /cvmfs/cms.cern.ch/alma8_aarch64_gcc11 /cvmfs/cms.cern.ch/alma8_amd64_gcc1 Which resulted in all the recursively greped files being fetched one by one, overloading the cache multiple times and finally setting the alarm off. Fig.2: A screenshot picturing an average mount time plot in grafana 7 CERN openlab report 2024 Fig.3: A screenshot picturing a number of clean ups plot in grafana Adding the monitoring of cache revisions One potential scenario, which occurred recently, is when one of the nodes contains a repository that significantly lags behind the newest revision on the remote server. This can happen if the CVMFS process, for some reason, is unable to connect to the remote server and therefore is unable to update. Monitoring cache revisions helps detect such cases notify the admins of the issue. Adding the revision monitoring to the existing pipeline turned out to be quite straightforward. There already exists a CMVFS plugin for the collectd system, created and maintained by my supervisor Steve. The plugin adds functionality needed to scrape selected extended attributes of mounted repositories. Adding cache revisions was as simple as adding another attribute name to the collectd configuration in puppet. Fig.4: A screenshot from grafana of the newly added max diff from latest revision 8 CERN openlab report 2024 Fig.5: A screenshot from grafana of the newly added max diff from latest revision without unpacked and lhcbdev One thing of note is that the repositories unpacked.cern.ch and lhcbdev.cern.ch tend to quite often be left behind the newest revision. That is due to the nature of those repositories, their purpose is distributing container images through CVMFS and therefore are updated very frequently. 9