Skip to content

X and layers round-trip enhancement - #362

Merged
eroell merged 2 commits into
mainfrom
fix/exact-layer-roundtrip
Nov 12, 2026
Merged

eroell merged 2 commits into
mainfrom
fix/exact-layer-roundtrip

Conversation

@eroell

@eroell eroell commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #361

Writing and reading X and layers changed data according to heuristics that we want to drop due to recurrence in our usecases. Codes like "007" became 7.0, the text "nan" became missing, and the zeros of a zero-filled sparse COO, such as OMOP counts, became NaN. The writer encoded object arrays lossily, and the readers guessed their way back, and the guessing did not prove alltogether very helpful. The columns of .obs/.var/.tem were already exact (#304); now X and layers are too.

Encoding (on-disk version 0.3.0)

  • All numbers or missing: an object array whose values are all numbers or missing is written as float64, as before. Bools and Decimal count as numbers.
  • Anything else: written as strings, with a uint8 array of each value's kind (text / number / missing) in .obsm["_ed_ondisk_kinds_X"] or .obsm["_ed_ondisk_kinds_layers_<name>"]. This uses the same mechanism as the reserved 3D keys, for 2D and 3D, .h5ed, .zarr and write_elem. Slight overhead, but the compression keeps extra disk usage at bay and the userfriendlyness is quite an improvement.
  • Never parse strings: strings are never parsed into numbers.
  • Big integers: integers that float64 can't hold are kept as text, with a warning, as Fix: Robustness for dates and missing values in aligned dataframes which are commonly produced from OMOP #304 does for columns.
  • Reserved keys: user .obsm keys starting with _ed_ondisk_ are rejected.
  • What Fix: Robustness for dates and missing values in aligned dataframes which are commonly produced from OMOP #304 deliberately chose stays: each array is cast on its own, the EHRData isn't copied, so a backed EHRData can be written again, and the caller's object is unchanged.

Reading

  • 0.3.0 data decodes exactly: text stays text ("007", "nan"), numbers come back as float and missing values as NaN. String arrays come back as object from both formats.
  • The flags: harmonize_missing_values and cast_variables_to_float are now bool | None = None.
    • None applies them only to files written before 0.3.0 and to plain anndata files, which therefore read as before.
    • True applies them to every file.
    • False applies them to none.
  • COO zeros are kept: the read-time harmonizing no longer applies Add sparse.COO support to harmonize_missing_values and infer_feature_types #306's "a COO's 0 means missing" rule to any file. Pass harmonize_missing_values=True for it.
  • Unknown versions: an unknown on-disk version raises with "please upgrade". Kind arrays that don't match their array are skipped with a warning.
  • Backed reading works with the defaults. Layers are decoded, and a text X raises NotImplementedError.

Compatibility

  • Files written before (0.2.0): read as before by default, except that COO zeros stay zeros. Committed fixtures written by the previous release are tested.
  • Older releases reading 0.3.0 files: 3D arrays and the kind arrays stay in .obsm, backed reading of 3D data fails, and their heuristics apply to the string-encoded arrays.

Tests

  • tests/io/test_ondisk.py runs over .h5ed, .zarr and write_elem/read_elem. It covers:
    • codes, the text "nan", missing values, numbers next to text, Decimal, bools, big integers, 3D, X as object vs None;
    • zero-filled COO;
    • 0.2.0 fixtures read with default and explicit flags;
    • an unknown version, mismatched kind arrays and reserved keys;
    • AnnData input.
  • Existing tests: updated where they relied on the read-time heuristics, plus a layout test and backed-mode tests.

eroell and others added 2 commits November 12, 2026 15:27
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@eroell
eroell marked this pull request as ready for review November 12, 2026 14:56
@eroell
eroell merged commit 3e172cf into main Nov 12, 2026
17 checks passed
@eroell
eroell deleted the fix/exact-layer-roundtrip branch November 12, 2026 15:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reading X and layers changes data that ehrdata wrote correctly

1 participant