Skip to content

Another idea for reproducibility on Sherlock #444

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
thalassemia merged 4 commits into from
Sep 21, 2026
Merged

Another idea for reproducibility on Sherlock #444

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b958871
Sherlock: only CPUs with AVX-512
thalassemia Sep 10, 2026
fd0d335
Merge branch 'master' into sherlock-avx512
thalassemia Sep 19, 2026
055ce0a
Docs: Sherlock CPU generation constraint
thalassemia Sep 19, 2026
0f0dbe3
Link to PR
thalassemia Sep 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 31 additions & 3 deletions doc/hpc.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -286,6 +286,21 @@ than a few minutes from their workflow configuration. Instead, either run these
manually following :ref:`sherlock-interactive` or create a
SLURM batch script to run these analyses following :ref:`sherlock-noninteractive`.

.. _reproducibility:

Reproducibility
===============

To ensure reproducibility, workflow steps on Sherlock are intentionally constrained to nodes
with CPUs that have been tested to produce numerically identical output for a 4-seed,
4-generation baseline glucose workflow. This is necessary because older CPUs lack support
for features like AVX-512 instructions that NumPy and OpenBLAS can utilize on newer CPUs.
As such, older CPUs tend to use different code paths in NumPy and OpenBLAS, leading to
unpredictable floating point differences. See ``sherlock --> nf_config_overrides --> CLUSTER_OPTIONS``
under the source code for :py:data:`runscripts.workflow.CLUSTER_PRESETS` for the current
list of supported CPU generations. As new CPU models are introduced, they will need to be
tested for consistency with existing ones before being added to this list.

.. _sherlock-interactive:

Interactive Container
Expand All @@ -294,12 +309,13 @@ Interactive Container
.. warning::
The following steps should be run on a compute node. See the
`Sherlock documentation <https://www.sherlock.stanford.edu/docs/user-guide/running-jobs/?h=interactive#interactive-jobs>`_
for details.
for details. Be sure to use the ``--constraint`` flag to request nodes with
supported CPU generations as described in :ref:`reproducibility`.

The maximum resource request for an interactive compute
node is 2 hours, 4 CPU cores, and 8GB RAM/core. Scripts that require more
resources should be submitted as SLURM batch scripts to the ``mcovert``
or ``owners`` partition (see :ref:`sherlock-noninteractive`).
resources should be submitted as SLURM batch scripts to the ``owners``
partition (see :ref:`sherlock-noninteractive`).

To run scripts on Sherlock, you must have either:

Expand Down Expand Up @@ -404,11 +420,22 @@ to include one of the following directives at the top of your script:
Importantly, if all 32 cores are in use by ``mcovert`` users, not only will your
script have to wait for resources to free up, so will any workflows. As such,
treat this partition as a limited resource reserved for high priority jobs.

.. warning::
As of Sep 19, 2026, the ``mcovert`` partition does not contain any nodes with
a CPU generation supported by ``workflow.py`` (see :ref:`reproducibility`).
Scripts manually run on this partition will diverge from results generated by
``workflow.py``.

- ``#SBATCH --partition=normal``: Potentially longer queue time than either of the
two options above but no risk of preemption.
- ``#SBATCH --partition=owners,normal``: Uses either the ``owners`` or ``normal``
partition. This is the recommended option for the vast majority of scripts.

If not using the deprecated ``mcovert`` partition, also include the
``#SBATCH --constraint=<supported_cpu_generation>`` directive to request
nodes with a supported CPU generation as described in :ref:`reproducibility`.

Following is a sample of sbatch scripts for requiring more resources to analysis simulation results:

.. code-block:: bash
Expand All @@ -420,6 +447,7 @@ Following is a sample of sbatch scripts for requiring more resources to analysis
#SBATCH --time=20:00
#SBATCH --ntasks=1
#SBATCH --partition=owners,normal
#SBATCH --constraint="CPU_GEN:BGM|CPU_GEN:SIE|CPU_GEN:GEN|CPU_GEN:SPR"
#SBATCH --cpus-per-task=4
#SBATCH --mem=64GB

Expand Down
5 changes: 4 additions & 1 deletion runscripts/workflow.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,10 @@ def hyperqueue_snippets(outdir: str) -> tuple[str, str]:
# See https://github.com/CovertLab/vEcoli/pull/331
"CLUSTER_OPTIONS": {
"prefer": '"CPU_GEN:GEN|CPU_GEN:SPR"',
"constraint": '"CPU_GEN:RME|CPU_GEN:MLN|CPU_GEN:BGM|CPU_GEN:SIE|CPU_GEN:GEN|CPU_GEN:SPR"',
# Removed EPYC Rome and Milan as they do not support AVX-512,
# causing unpredictable floating point differences

@heenasaqib heenasaqib Sep 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we link to PR 444 here, like how you link to PR 331 above?

# See https://github.com/CovertLab/vEcoli/pull/444
"constraint": '"CPU_GEN:BGM|CPU_GEN:SIE|CPU_GEN:GEN|CPU_GEN:SPR"',
},
"QUEUE": "owners,normal",
},
Expand Down
Loading