Skip to content

write_elem and read_elem: an EHRData as part of a larger HDF5 file or Zarr store - #360

Merged
eroell merged 3 commits into
mainfrom
feature/357-write-read-elem
Nov 11, 2026
Merged

eroell merged 3 commits into
mainfrom
feature/357-write-read-elem

Conversation

@eroell

@eroell eroell commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #357.

Packages that bundle an EHRData with other objects in one file, such as a DonorData in a .dd.h5/.dd.zarr, couldn't write it into a group or read it from there. write_h5ed, read_h5ed and write_zarr only took paths. The workarounds were temporary files or a copy of ehrdata's private on-disk layout.

New

ed.io.write_elem(parent, "edata", edata)  # parent: h5py.Group/File or zarr.Group
edata = ed.io.read_elem(parent["edata"])
  • Same layout as the file writers: parent[key] gets exactly what write_h5ed (HDF5) or write_zarr (Zarr) write to a file. That includes 3D arrays in the reserved obsm keys, sparse 3D COO, tem, the encoding attributes, the column casting and convert_strings_to_categoricals.
  • One implementation per format: write_h5ed/read_h5ed/write_zarr/read_zarr now call the same code. Backed read_h5ed is unchanged.
  • Options per format:
    • compression/compression_opts apply only to HDF5, and chunks only to Zarr. Passing an option for the other format raises.
    • Zarr v2 parents work. chunks="ehrdata_auto" needs v3 and raises before writing anything.
  • Keys:
    • An existing key is replaced, as by anndata.io.write_elem.
    • The key must be a non-empty name without /; pass the nested group as parent instead.
    • Whole files go through write_h5ed/write_zarr.
  • Errors:
    • The input is encoded before anything on disk is touched, so input ehrdata can't write leaves an existing parent[key] intact.
    • Writing into a Zarr store with consolidated metadata raises, as in anndata.
    • read_elem on a group that holds no EHRData names the group in its error.
  • Other children: read_elem ignores group children other than the AnnData slots and tem, as anndata does. A package bundling an EHRData can keep its own metadata in the group.

On-disk format

Unchanged. Files and stores written with main and with this branch were compared member by member: tree, attributes, dtypes, shapes, compression, chunks/shards and values. The comparison covered dense and COO 3D, CSR, object columns and layers, gzip and "ehrdata_auto", with conversion on and off, and found no differences. Each version reads the other's files to the same EHRData.

One detail: ad.io.write_elem writes a null raw that write_h5ad never did, so .h5ed groups drop it after writing.

Also

  • The chunks type hint was fixed: Literal["auto", "ehrdata_auto"].
  • The argument order in the write_zarr docstring example was fixed.
  • API docs list read_elem and write_elem.

Tests

tests/io/test_elem.py runs on an HDF5 file, a Zarr v3 group and a Zarr v2 group, and reuses the existing fixtures. It covers:

  • round trips at the root and in nested groups;
  • dense/COO/CSR data, tem and the column casting;
  • convert_strings_to_categoricals;
  • that the layout on disk is the same as write_h5ed/write_zarr produce;
  • that a failing write keeps an existing key;
  • consolidated metadata, key validation and format-specific options.

🤖 Generated with Claude Code

eroell and others added 3 commits November 11, 2026 20:34
… stores

write_elem writes an EHRData into a group in the same layout as write_h5ed and
write_zarr; read_elem reads it back. The file-level functions now use the same code,
and the on-disk format is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
As anndata does, so a package bundling an EHRData can keep its own metadata in the group.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@eroell
eroell merged commit 415202d into main Nov 11, 2026
17 checks passed
@eroell
eroell deleted the feature/357-write-read-elem branch November 11, 2026 21:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Write and read an EHRData as part of a larger HDF5 file or Zarr store

1 participant