Skip to content

write_h5ed and write_zarr convert the same string columns to categoricals - #359

Merged
eroell merged 3 commits into
mainfrom
fix/358-string-categoricals
Nov 11, 2026
Merged

eroell merged 3 commits into
mainfrom
fix/358-string-categoricals

Conversation

@eroell

@eroell eroell commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator

Fixes #358.

The same EHRData read back with different dtypes depending on the format:

written with obs.sex var.unit tem.kind
write_h5ed category category str
write_zarr (default) category category category
write_zarr(convert_strings_to_categoricals=False) str str str

write_h5ed always converted .obs/.var through AnnData.write_h5ad's default, never converted .tem, and had no option to turn this off.

Changes

  • write_h5ed takes convert_strings_to_categoricals: bool = True, as write_zarr does.
  • Both writers convert with one shared helper (strings_to_categoricals in io/_ondisk.py), so they always convert the same columns. Those are the string columns of .obs, .var and .tem in which a value repeats or is missing, including datetimes and timedeltas, which are written as ISO 8601 strings.
  • write_h5ed now calls AnnData.write_h5ad(..., convert_strings_to_categoricals=False), because the conversion has already happened (or been skipped) on the internal copy.
  • Neither writer modifies the EHRData passed in, views and backed objects included.
  • The deprecated write_h5ad alias keeps its signature.

Behaviour change

  • Default: write_h5ed now also converts the string columns of .tem, so .h5ed and .zarr read back with the same dtypes. .obs/.var behave as before.
  • With convert_strings_to_categoricals=False: both writers keep all string columns as strings. Before, write_h5ed could not leave .obs/.var unconverted.
  • Reading is unchanged, so files written by older versions read as before.

Tests

New tests in tests/io/test_h5ed.py, parametrized over both formats and both flag values:

  • dtypes after the round trip for .obs/.var/.tem: repeated strings, unique strings, nullable string with a missing value, an existing categorical, datetimes and timedeltas;
  • the values that come back;
  • that the caller's .obs/.var/.tem are unchanged;
  • the same frames from .h5ed (normal and backed read) and .zarr.

The .h5ed cases fail on main.

Not part of this PR

With False, an object-dtype string column without missing values reads back as object from .zarr but as str from .h5ed. This seems to come from anndata's Zarr v3 string reader, not from the conversion. Columns of pandas' default str dtype read back the same from both formats.

🤖 Generated with Claude Code

eroell and others added 3 commits November 11, 2026 16:22
write_h5ed and write_zarr default to convert_strings_to_categoricals=False, so an
EHRData reads back with the dtypes it was written with. Files written before keep
their categorical columns. Also fix the argument order in read_zarr's example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@eroell
eroell merged commit 7b40bb5 into main Nov 11, 2026
17 checks passed
@eroell
eroell deleted the fix/358-string-categoricals branch November 11, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

String columns come back differently from .h5ed and .zarr

1 participant