USIS: A Unified Semantic Interface Specification for Parallel Programming Models — paper artifact

This repository is the artifact for the paper. It contains everything needed to regenerate the specification documents the paper describes, the documents themselves, the paper's running broadcast example as a standalone spec instance, and a browser-based explorer for that example.

---

The idea, in short

Tools for parallel programming models (PPMs) — correctness checkers, performance analyzers, compilers — have to know what each API call means: that a call is collective, which argument is a rank rather than a count, which buffer is read and which is written, and when a buffer may be touched again. That knowledge is stated only in each PPM's specification, in prose, so every tool re-encodes it inside its own source. MPI_Bcast, ncclBroadcast, shmem_broadcastmem and nvshmem_int_broadcast all move one participant's buffer to every other participant of a process group, and a tool supporting the four states that fact four times, in four places, in a form no other tool can reuse or check.

This work replaces those per-model, per-tool encodings with one specification format instantiated for four PPMs. Every entry — one API procedure — has the same eight sections whatever model it belongs to, draws on a small closed vocabulary rather than an embedded expression language, and carries, beyond the classification a tool needs to recognize a call, an operational layer that states which participants a call involves, which memory each reads or writes, and how call instances pair up — enough to act on the call. The documents are not hand-written: a deterministic workflow generates them from each vendor's own artifacts (the LaTeX standards of MPI and OpenSHMEM, the vendor HTML documentation of NCCL and NVSHMEM, the C headers all four ship, and MPI's apis.json), and records for every value how it was derived and whether a human has approved it.

---

Layout

Path	What it is	Written by
external-inputs/	The vendors' own material, vendored verbatim: standards, headers, documentation mirrors. Read-only input.	third parties
curated/	The hand-authored and model-assisted inputs: the JSON Schema, the shape catalog, the per-PPM supplement, the field-provenance table, and the review records.	this work
workflow/	The generator. Pure Python 3, standard library only.	this work
spec/complete/	Instance 1 — the full corpus: 2893 entries across the four PPMs, one document per PPM, with the provenance report beside each.	generated
spec/example-set/	Instance 2 — the paper's broadcast example: the same four documents restricted to the four broadcast entries.	generated
explorer/	A single self-contained HTML file exploring the example set. Open explorer/index.html in a browser; no server, no network, no second file to load.	generated + hand-written
tools/	The two scripts that derive the example set and the explorer's data from the complete corpus.	this work
instances/	Scratch output directory a build writes into. Not tracked; delete freely.	generated

---

Running the generator

Requires Python 3 and nothing else — standard library only, no third-party packages, no network access, no compilation. Tested on CPython 3.11. A full run takes about ten seconds.

python3 workflow/build.py            # the whole chain, into instances/
python3 workflow/build.py --help     # the reference for every option
python3 workflow/build.py --test     # the eight test suites (run a build first)

--test checks the corpus a build produced as well as the generator itself, so run it after build.py, not before — on a clone with no instances/ yet it stops at the first suite that has nothing to read.

The run writes instances/final-<ppm>-api.json and provenance-<ppm>.json for each of mpi, nccl, nvshmem, shmem, plus two whole-corpus reports. Those files are what spec/complete/ holds a copy of, so a fresh run can be diffed against the shipped documents directly:

python3 workflow/build.py --out /tmp/rebuild
python3 - <<'PY'
import json
for m in ("mpi", "nccl", "nvshmem", "shmem"):
    a = json.load(open("/tmp/rebuild/final-%s-api.json" % m))
    b = json.load(open("spec/complete/final-%s-api.json" % m))
    a.pop("generated_at"), b.pop("generated_at")
    print(m, "identical" if a == b else "DIFFERS")
PY

Re-running on unchanged inputs reproduces byte-identical output apart from the generated_at timestamp — no step of the generator makes a model call. Useful variations:

python3 workflow/build.py --ppm mpi,nccl        # restrict to some models
python3 workflow/build.py --list-stages         # the chain, with artifact sizes
python3 workflow/build.py --only classify       # one stage
python3 workflow/build.py --check               # validate without rebuilding

To rebuild the two derived artifacts under spec/example-set/ and explorer/ after a corpus change:

cp instances/final-*-api.json instances/provenance-{mpi,nccl,nvshmem,shmem}.json spec/complete/
python3 tools/make_example_set.py
python3 tools/build_explorer.py

Both accept --check, which verifies they are in sync and writes nothing.

---

Components

The format — curated/schemas/api-schema.json

One JSON Schema, currently v3.12, 29 definitions. Each entry has eight sections — identity, bindings, execution, semantics, parameters, return, tool_integration, relationships — all present in every entry, with a field that does not apply carrying an explicit null rather than being omitted. semantics.formal is the operational layer: participants, data_flow, sync, handle_lifecycle. A consumer of that layer implements against four access operations, five provenance kinds, three lifecycle roles and five matching kinds, plus one reference grammar and one closed set of scope tokens.

The other curated inputs:

• curated/shapes/ — the cross-PPM shape catalog (27 shapes, 58 golden examples). This is what makes the four broadcasts one description rather than four: a shape is a template for semantics.formal, expanded deterministically against per-function bindings.
• curated/supplement/supplement-<ppm>.json — the facts no vendor artifact states machine-readably, per PPM.
• curated/formal-assignments.json — which procedure gets which shape.
• curated/field-provenance.json — the field-by-field table declaring, once, which workflow step owns each field. It drives the staging schema every step validates its own output against.
• curated/review/ — one review record per model-assisted input, grouped by decision pattern rather than by individual value.

The generator — workflow/

The chain, in order, each step a pure function of its inputs writing a complete corpus rather than a fragment:

Step	Script	Does
extract	workflow/extract/cli.py	parses external-inputs/ into entries carrying only what a declaration fixes
classify	workflow/classify/cli.py	applies naming and structural conventions to the fields those conventions determine
assign	workflow/formal/assign.py	proposes a shape per procedure family (curated: not re-run by default, so hand corrections survive)
assemble	workflow/pilot/assemble.py	expands shapes, applies the supplement, concretizes type-generic families, fills remaining absences with explicit null
validate	workflow/validate/consistency.py	cross-entry consistency plus strict-schema conformance
provenance	workflow/provenance/report.py	one record per value: how derived, by which step, approved or not
split	workflow/split/by_model.py	partitions the corpus into one document per PPM

A staged workflow has to distinguish "not decided yet" from "decided, and the answer is nothing". A field is therefore omitted until the step that owns it has run, and only then written as a value or an explicit null — so a null that appears at any stage is an answer, never a placeholder.

Model-assisted content never enters as a step. It enters as input: the shape catalog, the shape assignments and the supplement, each a reviewable file under version control. That is why re-running the generator is deterministic.

The specification documents — spec/

spec/complete/ — 980 authored procedure families expanded to 2893 shipped entries: OpenSHMEM 1608, NVSHMEM 651, MPI 571, NCCL 63. Each document declares the format version it was validated against and is self-contained: loadable by any standard JSON library at tool startup, no bespoke parser, no build-time code generation. Entry keys are model-qualified (mpi:mpi_bcast), so loading one document and loading all four resolve cross-references identically. provenance-<ppm>.json sits beside each, keyed by the same entry keys, and validate-report.json covers the whole corpus.

spec/example-set/ — the same four documents restricted to the paper's running example: mpi:mpi_bcast, nccl:ncclBroadcast, nvshmem:nvshmem_int_broadcast, shmem:shmem_broadcastmem. Every value is copied unchanged from the complete corpus by tools/make_example_set.py; only the document envelope is rebuilt.

The explorer — explorer/index.html

Open it directly in a browser. Five views over the example set:

• Compare — every scalar field of the four entries side by side, with the differing ones highlighted and per-section counts. Clicking a field name shows its definition and its closed vocabulary, read out of the schema rather than restated.
• Parameters & return — kind, direction, memory_space, length and the stated constraints per argument.
• Formal layer — participants, data-flow nodes and the source edge that states the written value is the value read; identical across the four models.
• Provenance — the per-value derivation and approval records.
• Raw JSON — the shipped entries verbatim.

The page is one file: tools/build_explorer.py writes the entries, the provenance records and the schema's field annotations into a marked block inside index.html itself, so there is no sibling data file to go missing and nothing is fetched at load time. That block is also the only place the page gets data from, so it cannot drift from the corpus or the schema. Re-run the tool after a corpus change; --check verifies the page is current without writing.

---

State of the corpus, stated rather than hidden

Two things a reader should know before drawing conclusions from the numbers:

• Supplement coverage is partial. 194 of 2893 entries validate against the strict schema; the rest are missing at least one required field, because the ~24 pure-prose semantic fields are only populated for the 43 authored procedure families the curated supplement covers. validate-report.json says exactly which field of which entry, and each document's unfillable_required_fields lists the three non-nullable booleans (execution.blocking, execute_once, collective) that the final null-fill leaves absent rather than inventing a value for. The four broadcast entries of the example set are all fully populated; a null that remains in them is an answer, not an omission. A recorded gap is visibly absent; a forced fit would be plausible and wrong.
• Review is recorded, not yet performed. Of 60822 provenance records, 53528 are mechanical, 4293 come from a passage of a standard, and 3001 are model-assisted. The review records in curated/review/ group those into decision patterns, and every group currently carries status: "unreviewed" — so the provenance report states 0 of 2243 model-assisted values approved. Nothing in the chain is gated on that by design; the point of the mechanism is that the state is visible rather than assumed.