Skip to content

Share record, enumeration, and array models within a process - #240

Open
cbrisebois-lasp wants to merge 2 commits into
mainfrom
perf/share-type-models
Open

cbrisebois-lasp wants to merge 2 commits into
mainfrom
perf/share-type-models

Conversation

@cbrisebois-lasp

@cbrisebois-lasp cbrisebois-lasp commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Every reference to a type loads its own copy of the model. A component
builds a fresh Packed_U16 for each field that uses it, and that copy
loads fresh copies of its own field types. One cold load of the EMA flight
assembly makes 2,399 model loads for 475 distinct files (packed_u16
alone 205 times), holds a tree of duplicates where a shared graph would
do, and writes 1.8 GB to the model cache on the way. That load is what
every assembly-level generator pays, so it sets the cost of a clean build.

Change

Record, enumeration, and array models are marked shareable, and
model_loader.load_model returns the one object it already built for a
shareable file (and load arguments) within the process. These types are
only completed after they load (a record fills in its fields' type ranges
on first request), never altered. Components stay unshared because the
assembly writes instance data into each one it loads, and submodels
because their parent writes itself into them. A load that passes
ignore_cache=True still builds a fresh object, which then becomes the
shared one, so the flag keeps its meaning.

Measured on one cold flight assembly load

Before After
Load time 101 s 59 s
Models built 2,399 475
Retained Python memory 267 MB 62 MB
Process peak RSS 1,015 MB 291 MB
Model cache written 1.8 GB 640 MB

Verification

  • All 779 generated sources of a clean flight rebuild are byte for byte
    identical to those generated without sharing, apart from the generation
    timestamp line.
  • redo/test/shared_models: a record, enumeration, and array are shared;
    ignore_cache builds fresh and replaces; a component is built anew on
    every load.
  • gen/test/records suites pass; flake8 clean.

Limitation

Sharing applies on the load path. A component restored from the on-disk
model cache is unpickled with its own embedded copies of its types, so a
warm build keeps those duplicates. Fixing that means pickling with
persistent ids and is a separate change.

Comment thread gen/models/base.py Resolved
Comment thread redo/util/model_loader.py Resolved
Comment thread redo/util/model_loader.py Resolved
@cbrisebois-lasp
cbrisebois-lasp force-pushed the perf/share-type-models branch 2 times, most recently from 9603bb9 to fe6e1eb Compare October 5, 2026 19:40
@cbrisebois-lasp cbrisebois-lasp changed the title Perf/share type models Share record, enumeration, and array models within a process Oct 5, 2026
@cbrisebois-lasp
cbrisebois-lasp marked this pull request as ready for review October 5, 2026 19:48
@cbrisebois-lasp
cbrisebois-lasp requested a review from Jbsco October 5, 2026 19:50
Comment thread redo/util/model_loader.py

# Shareable models already loaded by this process, by file and load
# arguments. A model class opts in with its shareable attribute; see base.
_shared_models = {}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like an in-memory cache should live with the model cache database so it is transparent. I tried a similar change a few years ago, but rejected it because it did not affect performance much. But if it is more memory efficient, maybe we should reconsider.

I really don't like the "sharable" flag here. Ideally we could remove need for something like that. The tracking overhead on all model types seems brittle and error prone to manage.

@cbrisebois-lasp
cbrisebois-lasp force-pushed the perf/share-type-models branch 2 times, most recently from 7836b9c to 2619811 Compare October 26, 2026 18:28
Pregeneration ran one pool worker per core, and a worker generating from
a large model, such as a flight assembly with every component loaded,
reaches several gigabytes, so a clean build of a big project asked for
many times the memory of the machine. The kernel then killed workers,
and because a multiprocessing pool waits forever for a result a killed
worker never returns, the build sat idle with the memory still held
rather than failing. Each target now runs in its own short-lived forked
process, admitted only while the memory it is expected to need is free
after accounting for the growth still ahead of the running workers. The
expected need is kept per kind of target, the input model's type, since
an assembly-level generator costs orders of magnitude more than a
record's: it starts as an estimate with one worker of the kind run at a
time as a probe, becomes the peak the finished workers reached, is
raised by any running worker that outgrows it, and persists across the
batches of one build step. A killed worker is reported as a failure
whose target falls back to the ordinary redo path.
Every reference to a type loaded its own copy of the model: a component
built a fresh Packed_U16 for each field that used it, and that copy
loaded fresh copies of its own field types, so one load of a large
assembly made some 2,400 model loads for fewer than 500 distinct files
and held a tree of duplicates where a shared graph would do. The types
are never altered after they load, unlike a component, which the
assembly fills in with its instance data, so a model class now marks
itself shareable and the loader hands back the one object it already
built for that file and those arguments. Sharing applies on the load
path; a component restored from the on-disk model cache still carries
its own embedded copies of its types, so a warm build keeps those
duplicates. Measured on a flight assembly, a cold load drops from 101
to 59 seconds and its retained memory from 267 to 62 MiB, with the
generated sources byte for byte unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants