ZIPsFS [ZIPsFS-options] path-of-branch1/ branch2/ branch3/ branch4/ : [fuse-options] mount-point
ZIPsFS -h
ZIPsFS functions as a union or overlay file system, merging multiple file structures into a unified directory.
This directory presents the underlying files and sub-directories from the specified source trees (branches) as a single, cohesive structure.
Any newly created or modified files are stored in the first file location, while all other sources remain read-only, ensuring that their files are never altered.
ZIPsFS treats ZIP files as expandable folders, typically naming them by appending .Contents/ to the original ZIP file name.
However, this behavior can be customized using filename-based rules. Extensive configuration options allow adjustments. Changes can be applied without disrupting the file system.
With a trailing slash, the folder name is not part of the virtual path, in accordance with the trailing slash semantics of many UNIX tools.
More flexibility provides the property @path-prefix=....
ZIPsFS includes specialized features like automatic file conversions and performance optimizations tailored for efficiently storing and accessing large-scale data. It manages non-sequential file reading from varying positions (so-called file seek) efficiently to improve reading of remote and zipped files. With file-preload-to-disk, it manages files in remote FTP repositories as if they were normal local files. Compressed (gz,xz,bz2,lrz,Z) can be transparently decompressed. It has a programming interface to create synthetic file content programmatically.
Please Install ZIPsFS.
Without trailing slash, the folder name will be retained in the virtual path. This is the case for branch3.
Virtual file paths in that branch will start with mount point/branch3/.
b1=~/test/ZIPsFS/writable/
b2=~/test/ZIPsFS/branch1/
b3=~/test/ZIPsFS/branch2/
b4=~/test/ZIPsFS/branch3
mkdir -p $b1 $b2 $b3 $b4 ~/test/ZIPsFS/mnt
for c in a b c d e f; do echo hello world $c >$b2/$c.txt; done
for ((i=0;i<10;i++)); do echo hello world $i >$b3/$i.txt; done
zip --fifo $b2/zipfile1.zip <(date) <(echo $RANDOM)
zip --fifo $b3/zipfile2.zip <(hostname) <(ls /)
zip --fifo $b3/20250131_this_is_a_mass_spectrometry_folder.d.Zip <(seq 10)
Run:
ZIPsFS $b1 $b2 $b3 $b4 : -o allow_other ~/test/ZIPsFS/mnt
Note that Tmux is very robust and ZIPsFS can be run in a Tmux window for several month without interruption. The advantage of running ZIPsFS in the foreground in Tmux is that the log output can be observed easily.
Open a file browser or another terminal and browse the files in
~/test/ZIPsFS/mnt/
The first file tree stores files. All others are read-only.
echo "Hello" > ~/test/ZIPsFS/mnt/my_file.txt
cat ~/test/ZIPsFS/mnt/my_file.txt
To get the real storage place of the file, append @SOURCE.TXT
cat ~/test/ZIPsFS/mnt/my_file.txt@SOURCE.TXT
More information is available with suffix @PROPERTIES.TXT.
cat ~/test/ZIPsFS/mnt/my_file.txt@PROPERTIES.TXT
Make sure the UNIX tool curl is installed.
curl --version
Take an URL and replace "//:" and all slashes by commas.
less ~/test/ZIPsFS/mnt/zipsfs/n/ftp,,,ftp.uniprot.org,pub,databases,uniprot,LICENSE
The first time, downloading takes a while. But subsequently, the file is available without delay.
gunzip -c ~/test/ZIPsFS/mnt/zipsfs/n/ftp,,,ftp.ebi.ac.uk,pub,databases,uniprot,current_release,knowledgebase,complete,docs,keywlist.xml.gz
gunzip -c ftp,,,ftp.expasy.org,databases,uniprot,current_release,knowledgebase,reference_proteomes,Viruses,UP001733680,UP001733680_2006137.fasta.gz
Even if a file is stored as a compressed file, you can directly access the decompressed file. Omit the .gz extension and display the file with cat.
Please note, that the size of the decompressed file is estimated. It becomes exactly known after reading the file. While gz decompression works without external programs, bz2, xz, Z and lrz decompression require respective UNIX tools to be installed.
Examples of files remotely stored as bz2 and xz:
head ~/test/ZIPsFS/mnt/zipsfs/n/http,,,ftp.gnu.org,gnu,binutils,binutils-2.23.1.tar # Remote file is .tar.bz2
head ~/test/ZIPsFS/mnt/zipsfs/n/http,,,ftp.gnu.org,gnu,gawk,gawk-4.0.2.tar # Remote file is .tar.xz
Please install curl and rclone (or curlftpfs).
There are FUSE file systems to access remote FTP sites: Curlftpfs, Rclone and Avfs.
They can be nested into ZIPsFS.
All of them are great for file browsing in FTP repositories. However, downloading very large files seems to be problematic.
As a workaround, ZIPsFS can use the robust and fast utility curl.
ZIPsFS uses curl if there is a file <mount-point>.URL containing the FTP address. Please study the
script ZIPsFS_prepare_branch_for_ftp.sh. It sets things up for a few example databases and can be used as a template.
It creates folders like ~/.ZIPsFS/db/pride
The standard output of the script can be used as command line parameters for ZIPsFS.
The expression $(...) denotes Command substitution.
ZIPsFS $b1 $b2 $b3 $b4 $(./ZIPsFS_prepare_branch_for_ftp.sh) : -o allow_other ~/test/ZIPsFS/mnt
The script will use rclone or Curlftpfs to mount some FTP sites.
The mount points will act as source trees in ZIPsFS (nesting). In case of success, the FTP sites can be browsed:
ls ~/test/ZIPsFS/mnt/db
Files with the ending .gz also appear in the file listing without gz suffix because the script adds the property decompression=gz,bz2,Z.
But wait, there is a major problem. Before decompression, ZIPsFS can only communicate an estimated file size of the decompressed file.
Consequently, programs that depend on exact file sizes will not work. Most UNIX tools do not need the file size. However,
/usr/bin/tail does. Lets see what happens if ZIPsFS provides a wrong size. Please try
tail ~/test/ZIPsFS/mnt/mnt/db/pride/2005/08/PRD000004/PRIDE_Exp_Complete_Ac_369.xml
The first time the tail command is run, there will be no output.
Subsequently, tail works nicely, because then the file size is known.
To repeat the test, the preloaded file can be removed
rm ~/test/ZIPsFS/mnt/mnt/db/pride/2005/08/PRD000004/PRIDE_Exp_Complete_Ac_369.xml
ZIPsFS functions as a union (overlay) file system. When files are created or modified, they are stored in the first file tree. If a file exists in multiple source trees, the version from the leftmost source (the first one listed) takes precedence. With an empty string as the first source, the ZIPsFS file system read-only and file creation and modification is disabled.
The physical file path, i.e., the actual storage location of a file, can be retrieved from a
file formed by appending @SOURCE.TXT to the filename.
For example, to determine the real location of:
~/test/ZIPsFS/mnt/1.txt
Run the following command:
cat ~/test/ZIPsFS/mnt/1.txt@SOURCE.TXT
If a remote source tree stops responding, the current file access is blocked (Unless the options WITH_ASYNC_READDIR ... are activated).
After some time, unresponsive file trees will be skipped to prevent that file accesses get blocked.
By default, ZIP files are displayed as folders with the suffix .Content.
This behavior can be customized.
The default rules are tailored to specific use cases in Mass Spectrometry:
-
For ZIP files whose names end with .d.Zip, the virtual folder will end with .d.
-
Flat file list (Try ZIPsFS_testfiles_zipflat.sh): For Sciex instruments, each mass spectrometry record is stored in a set of files which are not organized in sub-folders. For example the record
20231122_MA_HEK_QC_hiFlow_2ug_SWATH_rep01stored in20231122_MA_HEK_QC_hiFlow_2ug_SWATH_rep01.wiff2.Zipmay consist of the following files- 20231122_MA_HEK_QC_hiFlow_2ug_SWATH_rep01.timeseries.data
- 20231122_MA_HEK_QC_hiFlow_2ug_SWATH_rep01.wiff
- 20231122_MA_HEK_QC_hiFlow_2ug_SWATH_rep01.wiff2
- 20231122_MA_HEK_QC_hiFlow_2ug_SWATH_rep01.wiff.scan
These are contained with those of other entries in one long file listing. For ZIPsFS, it is time consuming to inspect all ZIP files to generate the full listing. This work is performed in a separate thread to not block the user thread. Initially, the file listing may still contain
.Contentfolders from ZIP files that are not yet expanded.
A major advantage of using ZIP archives for storing files is that
Zip files contain a record of the CRC32 checksum of each zip entry. This can be used for integrity checks to detect data decay or problems with data transfer.
This checksum can be obtained from ZIPsFS by appending @ARCHIVECRC32.TXT to a file path.
The folder /zipsfs/ provides alternative access paths to alter the view of the file tree and file content preload
/zipsfs/<view-option>/<file preload-option>[/<preload-selector>]
These three folders contain single letter directives. For example a file
<mountpoint>/foo/bar.txt
can also be accessed from
<mountpoint>/zipsfs/z/l/r/foo/bar.txt
│ │ │
│ │ └── Selector: Apply to files from remote sites
│ └──── Preload option: Preload to disk
└────── View option: Do not expand ZIP files. Show ZIP files as ZIP files.
Path encoded options take precedence over source tree settings. Directives are negated when preceded by dash. Detailed descriptions are found in the README files in these folders. here are all READMEs concatenated.
To chose the optimal access path for a particular software and data storage features need to be considered.
- Are the files remote? Does file transfer take long?
- For long file transfers: Is the application opening each file several times or just once?
- Are the files
- Regular
- Compressed or uncompressed ZIP entries
- Compressed files (gz, bz2, xz, Z, lrz) which the software does not support
- Does the software read files sequentially or from varying positions
- Does the software repeatedly request file attributes and folder listings at a high frequency - so-called stat-spamming? This has been observed in software using SQLite databases as file format. Unless SQLite databases are preloaded, Severe NAS performance problems may occur.
The following suggest access paths for specific applications in mass-spectrometry:
| Entry point | Feature | Used for |
|---|---|---|
/ |
Default options. analysis.tdf and analysis.tdf_bin files are preloaded to RAM |
Diann |
/zipsfs/d/lz/r/ |
Decompression. Preload to disk for remote files. ZIP files are preloaded entirely | Fragpipe, Maxquant, Diann on far remote files (Pride-DB, Massive-DB, cloud storage) |
/zipsfs/z/l/r/ |
ZIP files are not expanded. Preload to disk for remote files | Upload to Pride. |
/zipsfs/z/-/ |
ZIP files are not expanded. | Upload to Massive, fast file browsing (Windows explorer) |
The flags in the first subfolder specify
- Whether the 1st source tree or the other source trees are included or excluded.
- Whether ZIP files are expanded.
- Whether converted files are presented along with the original files. Implemented examples: OCR generated texts from image files and mzML files from proprietary mass-spectrometry files.
- Whether decompressed files are generated for gz, bz2, Z, xz and lrz compressed files.
The folder /zipsfs/n/ allows access to internet files. File names are formed from URLs by replacing the colon and slashes by comma.
The 2nd subfolder contains preload options.
| 2nd subfolder | Type of preload |
|---|---|
/-/ |
Apply default preload rules specified in ZIPsFS_configuration.c, ZIPsFS_configuration.h and source tree properties |
/m/ |
Prefetch to RAM |
/l/ |
Prefetch to the local file system |
/lz/ |
Prefetch to the local file system. Instead of single entries, load entire ZIP files to local disk. |
If the 3rd subfolder declares file preload to RAM or disk, then the 3rd subfolder gives the condition. Combinations are combined with logic or.
| 3rd subfolder | Selector |
|---|---|
a |
all |
r |
remote |
z |
ZIP-entry |
In the command line, source trees are specified by their root paths. They can be followed by expressions like @immutable=1 to set specific properties.
Alternatively, properties can be given in a file <root-path>.ZIPsFS.properties. This is demonstrated in ZIPsFS_prepare_branch_for_ftp.sh.
For a list of properties run
ZIPsFS -h
Preload files
Non-linear file loading in a random-access manner is very common for Windows style software were pipes and process substitution are rarely used. This is also the case for the widely used mass-spectrometry programs Diann and FragPipe (Fragger). Vendor specific mass-spectrometry files are loaded from varying file positions. Apparently, this mode of file access is inefficient for remote or ZIP-compressed data.
Data preloading can help to provide file data out-of-order. Remote or compressed files can be preloaded in two different ways: (I) into RAM or (II) to the local disk.
File names to be cached in RAM are specified in the configurable method config_advise_cache_in_ram(). The default
setting includes Brukertimstof mass-spectrometry files. Preloading to the RAM is appropriate for these files because
each file is loaded only once per analysis.
The -l option sets an upper limit on memory usage
for the ZIP RAM cache. When available memory runs low, ZIPsFS can either pause, proceed without
caching file data or just ignore the memory restriction depending on the configuration.
File reading of remote or compressed files can be improved by caching the file content on the local disk. This is for example necessary for Thermo mass-spectrometry raw files analyzed in FragPipe. Above method of preloading into RAM is here inappropriate because each raw file is opened and closed multiple times during computation such that the large files would be transferred over the net several times.
When loading Bruker mass spectrometry files with the Bruker DLLs, The file system is asked dozens of times per second to provide one and the same file listing and the same file attributes. While this so-called stat-spamming is fine for local files, remote file systems and decompressing file reading get overloaded. ZIPsFS implements a transient cache, that lives as long as the specific ZIP file is used.
This cache is distinct from a persistent cache of directory listing.
Project status
Author: Christoph Gille
Current status: Testing and Bug fixing. Occasionally improving concepts. Already running very busy for several weeks without interruption.
If ZIPsFS crashes, please send the stack-trace together with the source code you were using.
ZIPsFS can be killed like any UNIX command by typing Ctrl-C. Sometimes this does not work. Assuming mnt is your mount-point, the following might work:
fusermount -u mnt
sudo umount mnt
If ZIPsFS still hangs, the following will kill all FUSE file systems of the current user. Those of other users won't be affected:
echo 1 | sudo tee /sys/fs/fuse/connections/*/abort
To kill only the specific fuse mount you need to put the minor device number instead of the asterisk. Look at the third column of the command output containing the major:minor device number, e.g., 0:42. The minor number (42) matches the folder name in /sys/fs/fuse/connections/42/.
grep fuse /proc/self/mountinfo
- 202601
- Changing ZIPsFS to lower-case in /zipsfs. This was necessary to export ZIPsFS as a Samba Share to Windows
- 202610
- Re-organized the /zipsfs file branch.
- Solved problem of file truncation due to underestimated file size
Configuration
ZIPsFS can be customized:- optional features can be (de)-activated with preprocessor macros "WITH_SOME_FEATURE" which take the values 0 or 1.
- Rules can be given
- which files are cached in the main memory
- which ZIP entries are inlined
- Timeout values for accessing (remote) files
- Automatic file conversions
Configuration files of ZIPsFS, are files written in the programming language C. They have the prefix ZIPsFS_configuration and the suffix .h of .c.
ZIPsFS is customized for our needs - accessing archived high throughput data such that it can be directly used for mass spectrometry software. These settings can be used as a sample to customize it for other needs.
Changes require recompilation and take effect after restart of ZIPsFS.
With the -s option, the updated ZIPsFS can seamlessly replace running instances without disrupting the virtual file system.
To illustrate how this works, let MNT represent the apparent mount point of the FUSE file system. Suppose we are in the parent directory of MNT, enabling the use of relative paths. Users access files through this apparent mount point, but in reality, MNT is a symbolic link to the actual mount point. The real mount point is not directly accessed by users, as it changes each time a new instance of ZIPsFS is launched.
For example, assume the obsolete ZIPsFS instance is mounted at ./.mountpoints/MNT/1. When a new instance replaces it, it may use any empty directory as mount point. ZIPsFS must be started with the following command line option:
-s MNT
Once the new instance is running, the symbolic link is updated to point to the new mount location. From the user's perspective, nothing changes - the apparent mount point remains MNT. To ensure uninterrupted access, the obsolete instance should remain active for a short period to allow ongoing file operations to complete.
If MNT is within an exported SAMBA or NFS path the real mount points should be in the exported file tree as well. Include into /etc/samba/smb.conf:
follow symlinks = yes
Generated (synthetic) files: Automatic file conversions, Accessing web resources as regular files
Computations often require files from public repositories. Files from the internet (http, ftp, https) can be accessed as files using the URL as file name. ZIPsFS takes care of downloading and updating. They are immutable and cannot be modified unintentionally. In DOS, a trailing colon is a signature for device names. Therefore, the colon and all slashes in the URL need to be replaced by comma. Comma has been chosen as a replacement because it normally does not occur in URLs. Furthermore, it does not require quoting in UNIX shells.Example with mnt/ denoting the mountpoint of the ZIPsFS file system:
sudo apt-get install curl
ls -l mnt/ZIPsFS/n/https,,,ftp.uniprot.org,pub,databases,uniprot,README
more mnt/ZIPsFS/n/https,,,ftp.uniprot.org,pub,databases,uniprot,README
head mnt/ZIPsFS/n/https,,,ftp.uniprot.org,pub,databases,uniprot,README@SOURCE.TXT
To see the real local file path append @SOURCE.TXT to the file path.
The http-header is updated according to a time-out rule in ZIPsFS_configuration.c. Whether the file itself needs updating is decided upon the Last-Modified attribute in the http or ftp header.
Additionally, the file is accessible with a file-name containing the time reported in the header. This feature can be conditionally deactivated.
By modifying the file ZIPsFS_configuration_c.c, users can easily implement file generation using the programming language C.
This path gives the output of the included minimal example.
<mount point>/example_generated_file/example_generated_file.txt
ZIPsFS can generate virtual files by file conversion. This feature is enabled by setting the preprocessor macro WITH_FILECONVERSION to 1 in ZIPsFS_configuration.h. Generated files are stored in the first file branch, allowing them to be served instantly upon repeated requests. The default rules, defined in ZIPsFS_configuration_fileconversion.c, include:
- Image files (JPG, JPEG, PNG, GIF): Smaller versions at 25% and 50% scaling.
- Image files (OCR): Extracted text using Optical Character Recognition (OCR).
- PDF files: Extracted ASCII text.
- ZIP files: Consistency check reports, including checksums.
- Mass spectrometry files: mgf (Mascot) and mzML formats.
- wiff files: Extract ASCII text.
- Apache Parquet files: TSV and TSV.BZ2 formats.
For testing, you can copy an image file (png,jpg) and find the downscaled version in the branch
/ZIPsFS/c/
Note that some of the conversions may require Docker support and membership of the docker group.
The system cannot determine the size of files whose content has not yet been generated. In kernel-managed virtual file systems such as /proc and /sys, virtual files typically report a size of zero via stat(). Despite this, they are not empty and contain dynamically generated content when read.
However, this behavior does not translate well to FUSE-based file systems.
For FUSE, returning a file size of zero to represent an unknown or dynamic size is not recommended. Many programs interpret a size of 0 as an empty file and will not attempt to read from it at all. In ZIPsFS, a placeholder or estimated size is returned if the file content has not been generated. The estimate should be large enough to allow reading the full content. If the size is underestimated, data may be read incompletely, leading to truncated output or application errors. This workaround allows programs to read the file as if it had content, even though the size isn’t known in advance. However, it may still break software that relies on accurate size reporting for buffering or memory allocation.
Example Diann: The mass-spectrometry program Diann does not rely on correct file sizes when reading fasta files. Since Diann does not support fasta.gz files directly, it can benefit from ZIPsFS capability to decompress on download.
Example Fragpipe: Fragpipe is a software to process mass-spectrometry files. Processing Thermo-Fisher mass-spectrometry files with the suffix raw, those are converted by Fragpipe into the free file format mzML. Since ZIPsFS can also convert raw files to mzML, we tried to give the virtual mzML files as input. Initially, their reported file size is 99,999,999,999 Bytes. This large number was chosen to make sure that the estimated file size is larger than the real yet unknown size. Initially Fragpipe attempts to read some bytes from the end of the file. To determine the reading position, it uses the overestimated file size. In this specific case it tried to read at file position 99,999,997,952. ZIPsFS will perform the conversion when serving the first read request. Since the converted mzML file is much smaller than the read position, there will be no data and Fragpipe will fail. When however, at least one byte of the mzML files is read to initiate the conversion process before Fragpipe is started, computation will succeeds.
Logging
ZIPsFS typically runs as a foreground process. To keep it active and monitor its output, it is recommended to use a persistent terminal multiplexer such as tmux. This enables continuous observation of all messages and facilitates long-running sessions. Additional log files are stored in:~/.ZIPsFS
For each mount point there are files specifying more logs.
log_flags.conf
See readme for details:
log_flags.conf.readme
ZIPsFS dynamically generates a HTML status file in /ZIPsFS/ with real-time information.
Fault management. Timouts for remote upstream file systems. Duplicated remote trees
Accessing remote files inherently carries a higher risk of failure. Requests may either:-
Fail immediately with an error code, or
-
Block indefinitely, causing potential hangs.
In many FUSE file systems, a blocking access can render the entire virtual file system unresponsive. ZIPsFS addresses this with built-in fault management for remote branches.
Remote roots in ZIPsFS are specified using a double-slash prefix, similar to UNC paths (//server/share/...). Each remote branch is isolated in terms of fault handling and threading and has its own thread pool, ensuring faults in one do not affect others. To avoid blocking the main file system thread, remote file operations are executed asynchronously in dedicated worker threads.
Timeouts are activiated with the respective macro definitions WITH_ASYNC_XXXXXX. The work for root-paths starting with tripple slash. The fuse thread delegates the file operation to another thread and waits for its completion. After the configurable timeout it gives up.
Timeout is not fully tested yet. It should be considered only for remote paths when blocks are observed.
The worker thread may block permanently. In this case it may be killed automatically and restarted. However killing this thread sometimes does not work.
If the stalled thread cannot be terminated, ZIPsFS will not create a new thread. To check whether all threads are responding, activate logging. For details see
~/.ZIPsFS/.../log_flags.conf.readme
This problem is best resolved by replacing the current by a new ZIPsFS session using the -s command line option.
For redundantly stored files (i.e., available on multiple branches), another branch may take over transparently if one fails or becomes unresponsive.
Checks whether ZIPsFS can generate and print a backtrace in case of errors or crashes. This feature elies on external tools to translate memory addresses into source code locations: On Linux and FreeBSD, it uses addr2line, typically located in /usr/bin/. On macOS, it uses the atos tool instead. Ensure these tools are installed and accessible in your system's PATH for backtraces to work correctly.
See ZIPsFS.compile.sh for activation of sanitizers.
File integrity checks
For ZIP entries loaded entirely into RAM: ZIPsFS performs CRC checksum validation. Any detected inconsistencies are logged, helping to detect corruption or transmission errors.Use case - Archive of high throughput mass spectrometry files
We use closed-source proprietary Windows software to read large experimental data from various types of mass spectrometry machines. The data is immediatly copied into an intermediate storage on the processing PC and eventually archived in a read-only WORM file system.
To reduce the number of individual files and disk usage and to allow for data integrity checks, all files from a single mass spectrometry measurement are bundled into one ZIP archive. With fewer individual files, searching through the entire directory hierarchy takes less than 1 hour.
We initially hoped that files inside ZIP archives would be accessed using
- Pipes
- Named pipes
- Process substitution
- FUSE file systems with which transparently expand multiple ZIP files
- Unzipping and storing extracted files on disk
Unfortunately, these techniques did not work for our use case. Mounting individual ZIP files was initially the only solution. But when sample size of large experiments got large, even this was not feasable.
ZIPsFS was developed to solve the following problems:
-
Growing Number of ZIP Files: Recently, the size of our experiments - and therefore the number of ZIP files - has increased enormously. Mounting thousands of individual ZIP files results in a very long /etc/mtab file and puts a significant strain on the operating system.
-
Write Access Requirements: Some proprietary software requires write access to both files and their parent directories.
-
Inefficiency in Random File Access: Some mass spectrometry files are read from varying positions. Random access is particularly inefficient for compressed ZIP entries, in particular with backward seeks. Buffering of file content is required.
-
Multiple Storage Locations: Experimental records are initially stored in an intermediate storage location and, after verification, are moved to the final archive. Consequently, we need a union file system.
-
Resilience of storage systems: Sometimes access to the archive gets blocked. Otherwise there are several alternative entry points which will continue to work. This adds requirement for fault management.
-
Redundant File System Requests: Some proprietary software generates millions of redundant requests to the file system, which is problematic for both remote files and mounted ZIP files. File attributes need to be cached.
├── src
│ ├── InstallablePrograms
│ │ └── some_software.zip
│ │ └── my_manuscript.zip
└── read-write
├── my_manuscript.zip.Content
├── my-modified-text.txt
├── InstallablePrograms
│ ├── some_software.zip
│ └── some_software.zip.Content
│ ├── help.html
│ ├── program.dll
│ └── program.exe
│ ├── my_manuscript.zip
│ └── my_manuscript.zip.Content
│ ├── my_text.tex
│ ├── my_lit.bib
│ ├── fig1.png
│ └── fig2.png
├── brukertimstof │ └── 202302 │ ├── 20230209_hsapiens_Sample_001.d.Zip │ ├── 20230209_hsapiens_Sample_002.d.Zip │ └── 20230209_hsapiens_Sample_003.d.Zip...
│ └── 20230209_hsapiens_Sample_099.d.Zip └── zenotof └── 202304 ├── 20230402_hsapiens_Sample_001.wiff2.Zip ├── 20230402_hsapiens_Sample_002.wiff2.Zip └── 270230402_hsapiens_Sample_003.wiff2.Zip ... └── 270230402_hsapiens_Sample_099.wiff2.Zip
├── brukertimstof │ └── 202302 │ ├── 20230209_hsapiens_Sample_001.d │ │ ├── analysis.tdf │ │ └── analysis.tdf_bin │ ├── 20230209_hsapiens_Sample_002.d │ │ ├── analysis.tdf │ │ └── analysis.tdf_bin │ └── 20230209_hsapiens_Sample_003.d │ ├── analysis.tdf │ └── analysis.tdf_bin...
│ └── 20230209_hsapiens_Sample_099.d │ ├── analysis.tdf │ └── analysis.tdf_bin └── zenotof └── 202304 ├── 20230402_hsapiens_Sample_001.timeseries.data ├── 20230402_hsapiens_Sample_001.wiff ├── 20230402_hsapiens_Sample_001.wiff2 ├── 20230402_hsapiens_Sample_001.wiff.scan ├── 20230402_hsapiens_Sample_002.timeseries.data ├── 20230402_hsapiens_Sample_002.wiff ├── 20230402_hsapiens_Sample_002.wiff2 ├── 20230402_hsapiens_Sample_002.wiff.scan ├── 20230402_hsapiens_Sample_003.timeseries.data ├── 20230402_hsapiens_Sample_003.wiff ├── 20230402_hsapiens_Sample_003.wiff2 └── 20230402_hsapiens_Sample_003.wiff.scan
...
├── 20230402_hsapiens_Sample_099.timeseries.data ├── 20230402_hsapiens_Sample_099.wiff ├── 20230402_hsapiens_Sample_099.wiff2 └── 20230402_hsapiens_Sample_099.wiff.scan
Implementation
- libfuse
- libzip
- Linux or UNIX
- Gnu-C or clang
For optimal performance, ZIPsFS is written in stanmdard C.
When a client program requests a file from the virtual FUSE file system, the virtual file path needs to be translated into an existing real file or ZIP entry.
All given roots are iterated until the file or ZIP entry is found. The first root is the only that allows file modification and file creation. All others are read only.
There are several caches for file attributes and file contents to improve performance. They can be deactivate using conditional compilation. The respective macro definitions and detailed description are found in ZIPsFS_configuration.h.
The caches are necessary because
- Some mass spectrometry software reads files not sequentially but from varying file positions.
- Mass spectrometry software excerts thousands and millions of redundand file requests
- One type of mass spectrometry instruments requires the ZIP entries to be displayed flat in the folder. To list this directory sufficiently fast requires a file entry cache.
After closing a file or disposing the memcache of a ZIP entry, a call to posix_fadvise(fd,0,0,POSIX_FADV_DONTNEED); may remove the file from the file cache of the OS when it is unlikely that the same file will be used in near future. This can be customized in ZIPsFS_configuration.c.
See field struct fuse_file_info->keep_cache in function xmp_open().
- infloop_memcache(): Loading entire ZIP entries into RAM asynchronously. One thread per root.
- infloop_async(): Calling stat asynchronously. Reading file attributes and directory listings and ZIP file entry listings asynchronously. One thread per root.
- infloop_misc(): Periodically check whether roots are responsive. Unblock blocked threads by calling pthread_cancel(). They re-start automatically with a pthread_cleanup_push() hook. One global thread.
Typically, overlay or union file systems are affected when one of the source file systems is not responding. Remote sources are at risk of failure when for example network problems occur.
Those paths of roots can be indicated by a leading double slash // (according to a convention from MS Windows). Those roots are used only when they have successfully returned from periodical calls to fsstat().
The worst case is that a call to the file API does not return. Optionally, ZIPsFS allows such calls to be run in a separate thread. This is implemented with a queue. Periodically, an infinite loop in a separate thread processes the requests in the queue.
With three slashes, asynchronous calls to stat(), readdir() and open(). This requires that the respective WITH_XXX macro is set to 1.
Only read(int fd, void *buf, size_t count) and the equivalent for ZIP entries are called directly suche that some risk of blocking remains.
Limitations and Bugs
There are some Limitations:Hard links are not supported, though symlinks are fully functional.
Files can only be deleted if their physical location resides in the first source. Files located in other branches are accessed in a read-only mode, and deletion of these files would require a mechanism to remove them from the system, which is currently not implemented.
If you require this functionality, please submit a feature request.
Simultaneous reading and writing of a file using the same file descriptor will only function correctly for files stored in the writable source.
When file sizes are not known, an over-estimate will be reported. If the estimate is too low, the read file content will be truncated.