Repository navigation
write_h5ed and write_zarr convert the same string columns to categoricals - #359
Merged
Merged
Conversation
write_h5ed and write_zarr default to convert_strings_to_categoricals=False, so an EHRData reads back with the dtypes it was written with. Files written before keep their categorical columns. Also fix the argument order in read_zarr's example. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #358.
The same
EHRDataread back with different dtypes depending on the format:obs.sexvar.unittem.kindwrite_h5edwrite_zarr(default)write_zarr(convert_strings_to_categoricals=False)write_h5edalways converted.obs/.varthroughAnnData.write_h5ad's default, never converted.tem, and had no option to turn this off.Changes
write_h5edtakesconvert_strings_to_categoricals: bool = True, aswrite_zarrdoes.strings_to_categoricalsinio/_ondisk.py), so they always convert the same columns. Those are the string columns of.obs,.varand.temin which a value repeats or is missing, including datetimes and timedeltas, which are written as ISO 8601 strings.write_h5ednow callsAnnData.write_h5ad(..., convert_strings_to_categoricals=False), because the conversion has already happened (or been skipped) on the internal copy.EHRDatapassed in, views and backed objects included.write_h5adalias keeps its signature.Behaviour change
write_h5ednow also converts the string columns of.tem, so.h5edand.zarrread back with the same dtypes..obs/.varbehave as before.convert_strings_to_categoricals=False: both writers keep all string columns as strings. Before,write_h5edcould not leave.obs/.varunconverted.Tests
New tests in
tests/io/test_h5ed.py, parametrized over both formats and both flag values:.obs/.var/.tem: repeated strings, unique strings, nullablestringwith a missing value, an existing categorical, datetimes and timedeltas;.obs/.var/.temare unchanged;.h5ed(normal and backed read) and.zarr.The
.h5edcases fail onmain.Not part of this PR
With
False, an object-dtype string column without missing values reads back asobjectfrom.zarrbut asstrfrom.h5ed. This seems to come from anndata's Zarr v3 string reader, not from the conversion. Columns of pandas' defaultstrdtype read back the same from both formats.🤖 Generated with Claude Code