Repository navigation
[BUG]_SSHFS_incompatibility_issue_with_DeepMD_3.1.2 #6014
Description
Bug summary
Hello There,
i am currently training a model using Deepmd 3.1.2 with dpgen and I am experiencing issue when training on a SSHFS mounted system. The reported error seems to have a problem regarding the PyTorch workers accessing the same file at the same time.
I have attached the complete message regarding the problem. I am wondering if this has something to do with the Deepmd 3.1.2 or the SSHFS 3.7.3?
DeePMD-kit Version
3.1.2
Backend and its version
PyTorch
How did you download the software?
conda
Input Files, Running Commands, Error Log, etc.
Pasted text(20260908-131850).txt
Steps to Reproduce
train normally on a SSHFS mounted system with the SSHFS version 3.7.3.
Further Information, Files, and Links
No response
Activity
njzjz-bot commentedon Sep 16, 2026
Thanks for the report. The first thing I would isolate is whether this is a DeePMD issue or an SSHFS + PyTorch multiprocessing I/O interaction.
For the PyTorch backend, DeePMD uses worker subprocesses for data loading by default (
NUM_WORKERSdefaults to 4 or the available core count, whichever is smaller). Please try the same training with worker multiprocessing disabled, for example by exporting:export NUM_WORKERS=0before launching the DeePMD/DP-GEN training task. The variable is documented here: https://github.com/deepmodeling/deepmd-kit/blob/master/doc/env.md
It would also be useful to run the exact same dataset once from node-local/local POSIX storage. If local storage works but SSHFS fails, and especially if
NUM_WORKERS=0also avoids the failure on SSHFS, that would strongly point to filesystem/multiprocessing semantics rather than the model itself.If it still fails with
NUM_WORKERS=0, please paste the first complete traceback/error block directly into the issue (starting from the first exception), and note whether the same files train successfully when copied off SSHFS. Those two checks should narrow this down substantially.Agent: ChatGPT
Model: GPT-5.6 Sol