diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d719d09..272f6a53 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,6 +29,8 @@ and this project adheres to [Semantic Versioning][]. - {func}`~ehrdata.io.omop.setup_interval_variables` aggregates only the intervals that hold data points, as {func}`~ehrdata.io.omop.setup_variables` does, so memory and time follow the number of data points. ([#349](https://github.com/theislab/ehrdata/pull/349)) @eroell - {func}`~ehrdata.io.omop.setup_variables` and {func}`~ehrdata.io.omop.setup_interval_variables` with `aggregation_strategy="mode"` keep the smallest of several equally frequent values, instead of an arbitrary one. ([#341](https://github.com/theislab/ehrdata/pull/341)) @eroell - {func}`~ehrdata.io.omop.setup_variables` and {func}`~ehrdata.io.omop.setup_interval_variables` with `observation_table="person_visit_occurrence"` attribute a data point whose `visit_occurrence_id` names a visit of its person to that visit only. Before, a data point belonged to every visit of its person whose window held it, so overlapping visits of a person shared their data points, and with `clip_to_end_date=False` a visit's intervals after its end took the data points of the person's next visit. Data points without a `visit_occurrence_id`, or with one that names no visit of their person, still belong to every visit whose window holds them. ([#344](https://github.com/theislab/ehrdata/pull/344)) @eroell +- {func}`~ehrdata.io.write_h5ed`, {func}`~ehrdata.io.write_zarr` and {func}`~ehrdata.io.write_elem` write an `object` `X` or layer whose values are not all numbers or missing as strings, with the kind of each value, text, number or missing, in a reserved `.obsm` key next to it, and no longer parse strings into numbers. An `object` array of only numbers, including booleans and `Decimal`, and missing values is written as float, as before. Integers that float64 does not hold exactly are written as text, with a warning. `.obsm` keys starting with `_ed_ondisk_` are reserved and raise. This changes ehrdata's on-disk version to 0.3.0. ehrdata reads the versions 0.2.0 and 0.3.0, and raises for newer ones. Earlier releases of ehrdata read a file of version 0.3.0 wrongly: they leave its 3D `X` and layers and the kinds in `.obsm` under the reserved `_ed_ondisk_*` keys, fail in backed mode on 3D arrays, and apply their heuristics to the arrays written as strings, so that `"007"` becomes 7.0 and `"nan"` becomes `np.nan`. Writing such data again with an earlier release loses the exact encoding; upgrade ehrdata to read files of version 0.3.0. ([#362](https://github.com/theislab/ehrdata/pull/362)) @eroell +- {func}`~ehrdata.io.read_h5ed`, {func}`~ehrdata.io.read_zarr` and {func}`~ehrdata.io.read_elem` apply `harmonize_missing_values` and `cast_variables_to_float` by default (`None`) only to files written before on-disk version 0.3.0 and to plain anndata files, which therefore read as before; `True` applies them to every file, `False` to none. Backed reading with {func}`~ehrdata.io.read_h5ed` no longer needs `harmonize_missing_values=False` and `cast_variables_to_float=False`, and decodes the layers written as strings with kinds; an `X` written so raises. ([#362](https://github.com/theislab/ehrdata/pull/362)) @eroell - {func}`~ehrdata.io.write_h5ed` and {func}`~ehrdata.io.write_zarr` keep string columns as strings by default (`convert_strings_to_categoricals=False`), so an {class}`~ehrdata.EHRData` reads back with the dtypes it was written with, from `.h5ed` and `.zarr` alike. Before, {func}`~ehrdata.io.write_h5ed` converted the string columns of `.obs` and `.var` in which a value repeats or is missing to categoricals, and {func}`~ehrdata.io.write_zarr` also those of `.tem`. Files written before keep their categorical columns; pass `convert_strings_to_categoricals=True` for the previous conversion. ([#358](https://github.com/theislab/ehrdata/issues/358)) @eroell ### Fixed @@ -41,6 +43,8 @@ and this project adheres to [Semantic Versioning][]. - {func}`~ehrdata.io.omop.setup_variables` and {func}`~ehrdata.io.omop.setup_interval_variables` write `.uns['omop_io_variable_setup']`, as their docstring promised. ([#353](https://github.com/theislab/ehrdata/pull/353)) @eroell - {class}`~ehrdata.integrations.torch.OMOPEHRDataset` reads the setup recorded in `.uns['omop_io_variable_setup']`, and raises if a later call has replaced its values. ([#353](https://github.com/theislab/ehrdata/pull/353)) @eroell - {func}`~ehrdata.io.omop.setup_variables` and {func}`~ehrdata.io.omop.setup_interval_variables` no longer change the {class}`~ehrdata.EHRData` passed to them, and keep its earlier layers when they set up another `layer` of the same variables and intervals. ([#354](https://github.com/theislab/ehrdata/pull/354)) @eroell +- {func}`~ehrdata.io.read_h5ed`, {func}`~ehrdata.io.read_zarr` and {func}`~ehrdata.io.read_elem` return `X` and layers exactly as {func}`~ehrdata.io.write_h5ed`, {func}`~ehrdata.io.write_zarr` and {func}`~ehrdata.io.write_elem` wrote them. ([#362](https://github.com/theislab/ehrdata/pull/362)) @eroell +- {func}`~ehrdata.io.read_h5ed`, {func}`~ehrdata.io.read_zarr` and {func}`~ehrdata.io.read_elem` keep the zeros of a numeric `sparse.COO` `X` or layer, such as the counts of {func}`~ehrdata.io.omop.setup_variables` with `aggregation_strategy="count"` and `sparse=True`, also from files written before. Before, they read them as missing values; this rule of {func}`~ehrdata.harmonize_missing_values` now applies on reading only with `harmonize_missing_values=True`. ([#362](https://github.com/theislab/ehrdata/pull/362)) @eroell ## [0.5.0] diff --git a/src/ehrdata/_feature_types.py b/src/ehrdata/_feature_types.py index acafca24..8a87ca0d 100644 --- a/src/ehrdata/_feature_types.py +++ b/src/ehrdata/_feature_types.py @@ -582,13 +582,15 @@ def harmonize_missing_values( return edata if copy else None -def _harmonize_on_read(edata: EHRData) -> None: +def _harmonize_on_read(edata: EHRData, *, skip_coo: bool) -> None: layers = [None] if edata.X is not None else [] # anndata 0.13's unified `.X` slot shows up as key None in edata.layers too; skip the duplicate layers += [key for key in edata.layers if key is not None] for layer in layers: label = "X" if layer is None else f"layer {layer}" + if skip_coo and isinstance(edata.X if layer is None else edata.layers[layer], sparse.COO): + continue try: harmonize_missing_values(edata, layer=layer) logger.info(f"Harmonizing missing values of {label}") diff --git a/src/ehrdata/core/constants.py b/src/ehrdata/core/constants.py index 2c901b84..35a06052 100644 --- a/src/ehrdata/core/constants.py +++ b/src/ehrdata/core/constants.py @@ -14,6 +14,7 @@ # On-disk format (see :mod:`ehrdata.io._ondisk` for the layout). # EHRData stores time-series as 3D arrays in `X`/`layers`, but anndata only guarantees 2D arrays there (https://github.com/scverse/anndata/issues/2430) and enforces this since release 0.13.0 # Since storage version 0.2.0 (ehrdata release 0.3.0), 3D arrays are relocated into the reserved `.obsm` keys below on write and restored on read. +# Since storage version 0.3.0, an `object` X/layer of text next to numbers or missing values is written as strings, with the kind of each value in a reserved `.obsm` key, and read back exactly. # Bump only when the on-disk layout changes. # # The attribute key that stamps a store as ehrdata differs by format, on purpose: @@ -23,12 +24,24 @@ EHRDATA_ENCODING_TYPE_KEY_ZARR = "encoding-type" EHRDATA_ENCODING_TYPE = "ehrdata" EHRDATA_ONDISK_VERSION_KEY = "ehrdata-encoding-version" -EHRDATA_ONDISK_VERSION = "0.2.0" +EHRDATA_ONDISK_VERSION = "0.3.0" +# the versions this ehrdata reads; files of other versions written by ehrdata raise +EHRDATA_ONDISK_VERSIONS_READ = ("0.2.0", "0.3.0") + +# Prefix of the `.obsm` keys reserved for the on-disk layout. +EHRDATA_OBSM_RESERVED_PREFIX = "_ed_ondisk_" # Reserved `.obsm` keys holding a 3D `X` / a 3D layer `` on disk. EHRDATA_OBSM_3D_X_KEY = "_ed_ondisk_X" EHRDATA_OBSM_3D_LAYER_PREFIX = "_ed_ondisk_layers_" +# Reserved `.obsm` keys holding the kind of each value of an `X` / a layer `` written as strings, as uint8. +EHRDATA_OBSM_KINDS_X_KEY = "_ed_ondisk_kinds_X" +EHRDATA_OBSM_KINDS_LAYER_PREFIX = "_ed_ondisk_kinds_layers_" +VALUE_KIND_TEXT = 0 +VALUE_KIND_NUMBER = 1 # read back as float64 +VALUE_KIND_MISSING = 2 # read back as np.nan + # Missing values # -------------- # These values if encountered as strings are considered to represent missing values in the data diff --git a/src/ehrdata/io/_array_casting.py b/src/ehrdata/io/_array_casting.py index d893fb73..5317e1a2 100644 --- a/src/ehrdata/io/_array_casting.py +++ b/src/ehrdata/io/_array_casting.py @@ -2,6 +2,8 @@ import contextlib import datetime as dt +import decimal +import numbers from typing import TYPE_CHECKING, Any import numpy as np @@ -11,6 +13,7 @@ if TYPE_CHECKING: from ehrdata import EHRData from ehrdata._logger import logger +from ehrdata.core.constants import VALUE_KIND_MISSING, VALUE_KIND_NUMBER, VALUE_KIND_TEXT def _is_numeric_or_bool(array: Any) -> bool: @@ -52,19 +55,97 @@ def _cast_variables_to_float(edata: EHRData) -> None: edata.layers[key] = _cast_columns_to_float(array) -def _cast_object_array_to_float_or_str(array: Any, name: str) -> Any: - """Cast an in-memory `object` array to float, else to str; return other arrays unchanged.""" +# `infer_dtype` of object arrays that are cast to float as a whole +_FLOAT_CASTABLE = {"empty", "floating", "integer", "mixed-integer-float", "boolean", "decimal"} + + +def _is_exact_integer(value: Any) -> bool: + try: + return float(int(value)) == int(value) + except OverflowError: + return False + + +def _number_to_string(value: Any) -> str: + # as written if that reads back as the same float, e.g. "3" for 3 + text = str(value) + with contextlib.suppress(ValueError): + if float(text) == float(value): + return text + return repr(float(value)) + + +def _encode_object_array(array: Any, name: str) -> tuple[Any, np.ndarray | None]: + """Encode an in-memory `object` array for writing; return other arrays unchanged. + + If every value is a real number (also a bool or `Decimal`), and an integer only if float64 holds it exactly, or missing, + the values are returned as float, with `None`. + Otherwise they are returned as strings, with the kind of each value: text, number or missing. + Strings are never parsed into numbers. + """ if not isinstance(array, np.ndarray | sparse.COO) or array.dtype != np.object_: # backed and dask arrays are written as they are - return array - if isinstance(array, np.ndarray): - # missing values become "nan" in a str array, which harmonize_missing_values reads back as np.nan - array = np.where(pd.isna(array), np.nan, array) - try: - return array.astype(np.float64) - except (TypeError, ValueError): - logger.warning(f"edata.{name} is of dtype object and not numeric: it is cast to dtype 'str' for writing.") - return array.astype(str) + return array, None + if isinstance(array, sparse.COO): + try: + return array.astype(np.float64), None + except (TypeError, ValueError): + logger.warning(f"edata.{name} is of dtype object and not numeric: it is cast to dtype 'str' for writing.") + return array.astype(str), None + + is_missing = np.asarray( + pd.isna(array), dtype=bool + ) # TODO: find out if scipy sparse can ever enter, as this would be a densification. + if pd.api.types.infer_dtype(array.ravel(), skipna=True) in _FLOAT_CASTABLE: + with contextlib.suppress(OverflowError): + values = np.where(is_missing, np.nan, array).astype(np.float64) + # integers from 2**53 on may not be exact + if not (np.abs(values[~is_missing]) >= 2**53).any(): + return values, None + + flat, flat_missing = array.ravel(), is_missing.ravel() + kinds = np.full(flat.shape, VALUE_KIND_MISSING, dtype=np.uint8) + strings = np.full(flat.shape, "nan", dtype=object) # an object array of str, not fixed width + type_ids = np.fromiter(map(id, map(type, flat)), dtype=np.uintp, count=flat.size) + inexact_integers = False + # by type, to vectorize + for value_type in set(map(type, flat[~flat_missing])): + mask = (type_ids == id(value_type)) & ~flat_missing + values = flat[mask] + if issubclass(value_type, str): + kinds[mask], strings[mask] = VALUE_KIND_TEXT, values + elif issubclass(value_type, int | np.integer) and not issubclass(value_type, bool): + try: + exact = np.abs(values.astype(np.float64)) < 2**53 + except OverflowError: + exact = np.zeros(values.shape, dtype=bool) + exact[~exact] = np.frompyfunc(_is_exact_integer, 1, 1)(values[~exact]).astype(bool) + inexact_integers |= not exact.all() + kinds[mask] = np.where(exact, VALUE_KIND_NUMBER, VALUE_KIND_TEXT) + strings[mask] = np.fromiter(map(str, values), dtype=object, count=values.size) + elif issubclass(value_type, numbers.Real | decimal.Decimal | np.bool_): + to_string = str if value_type in {float, np.float64} else _number_to_string + kinds[mask] = VALUE_KIND_NUMBER + strings[mask] = np.fromiter(map(to_string, values), dtype=object, count=values.size) + else: + kinds[mask] = VALUE_KIND_TEXT + strings[mask] = np.fromiter(map(_to_string, values), dtype=object, count=values.size) + + if not (kinds == VALUE_KIND_TEXT).any(): + return np.where(is_missing, np.nan, array).astype(np.float64), None + if inexact_integers: + logger.warning(f"edata.{name} holds integers that float64 does not hold exactly: they are written as text.") + return strings.reshape(array.shape), kinds.reshape(array.shape) + + +def _decode_object_array(strings: Any, kinds: np.ndarray) -> np.ndarray: + """Restore the values of an array encoded by :func:`_encode_object_array`.""" + strings = np.asarray(strings) + values = strings.astype(object) + is_number = kinds == VALUE_KIND_NUMBER + values[is_number] = strings[is_number].astype(np.float64) + values[kinds == VALUE_KIND_MISSING] = np.nan + return values def _to_string(value: Any) -> str: diff --git a/src/ehrdata/io/_ondisk.py b/src/ehrdata/io/_ondisk.py index 7b0c96e9..967dc4fb 100644 --- a/src/ehrdata/io/_ondisk.py +++ b/src/ehrdata/io/_ondisk.py @@ -2,6 +2,7 @@ EHRData keeps time-series as 3D arrays in ``X``/``layers``, but anndata only guarantees 2D arrays there, and enforces this 0.13.0 onwards EHRData in memory: keeps 3D arrays in ``X``/``layers``. EHRData on disk: moves 3D arrays into ``.obsm``. +An ``object`` ``X``/layer of text next to numbers or missing values is written as strings, with the kind of each value in ``.obsm``. """ from __future__ import annotations @@ -10,22 +11,33 @@ from typing import TYPE_CHECKING, Any import anndata as ad +import numpy as np import sparse +from ehrdata._feature_types import _harmonize_on_read +from ehrdata._logger import logger from ehrdata.core.constants import ( EHRDATA_ENCODING_TYPE, EHRDATA_ENCODING_TYPE_KEY, EHRDATA_ENCODING_TYPE_KEY_ZARR, EHRDATA_OBSM_3D_LAYER_PREFIX, EHRDATA_OBSM_3D_X_KEY, - EHRDATA_ONDISK_VERSION, + EHRDATA_OBSM_KINDS_LAYER_PREFIX, + EHRDATA_OBSM_KINDS_X_KEY, + EHRDATA_OBSM_RESERVED_PREFIX, EHRDATA_ONDISK_VERSION_KEY, + EHRDATA_ONDISK_VERSIONS_READ, +) +from ehrdata.io._array_casting import ( + _cast_dataframe_columns_to_writable_dtype, + _cast_variables_to_float, + _decode_object_array, + _encode_object_array, ) -from ehrdata.io._array_casting import _cast_dataframe_columns_to_writable_dtype, _cast_object_array_to_float_or_str from ehrdata.io._coo_codec import _binsparse_dtype_str, is_coo_group, read_coo if TYPE_CHECKING: - from collections.abc import Callable + from collections.abc import Callable, MutableMapping import pandas as pd @@ -54,6 +66,20 @@ def _layer_for_obsm_key(key: str) -> str | None: return key.removeprefix(EHRDATA_OBSM_3D_LAYER_PREFIX) if key.startswith(EHRDATA_OBSM_3D_LAYER_PREFIX) else None +def _kinds_key(layer: str | None) -> str: + return EHRDATA_OBSM_KINDS_X_KEY if layer is None else f"{EHRDATA_OBSM_KINDS_LAYER_PREFIX}{layer}" + + +def _pop_kinds(obsm: MutableMapping[str, Any]) -> dict[str | None, Any]: + """Pop the kind arrays from `obsm`, keyed by their layer, `None` for `X`.""" + kinds = {} + if EHRDATA_OBSM_KINDS_X_KEY in obsm: + kinds[None] = obsm.pop(EHRDATA_OBSM_KINDS_X_KEY) + for key in [k for k in obsm if k.startswith(EHRDATA_OBSM_KINDS_LAYER_PREFIX)]: + kinds[key.removeprefix(EHRDATA_OBSM_KINDS_LAYER_PREFIX)] = obsm.pop(key) + return kinds + + def _reject_stray_coo(edata: EHRData) -> None: """Reject a :class:`sparse.COO` anywhere ehrdata cannot persist it. @@ -85,15 +111,22 @@ def encode_for_disk(edata: EHRData) -> tuple[ad.AnnData, dict[str, sparse.COO]]: Dense 3D arrays are placed into the AnnData's ``.obsm`` under the reserved keys. Sparse 3D arrays, not writeable with AnnData, are pulled out into the returned ``sparse_3d_data`` mapping (keyed by the same reserved ``.obsm`` keys) - In-memory ``object`` arrays and the ``.obs``/``.var`` columns anndata cannot write are cast, leaving ``edata`` unchanged. + In-memory ``object`` arrays are encoded (see :func:`~ehrdata.io._array_casting._encode_object_array`), with their kind + arrays in reserved ``.obsm`` keys, and the ``.obs``/``.var`` columns anndata cannot write are cast, leaving ``edata`` unchanged. """ _reject_stray_coo(edata) + reserved = [key for key in edata.obsm if key.startswith(EHRDATA_OBSM_RESERVED_PREFIX)] + if reserved: + msg = f"The keys {reserved} of .obsm are reserved for ehrdata's on-disk layout; rename them to write." + raise ValueError(msg) obsm = dict(edata.obsm) sparse_3d_data: dict[str, sparse.COO] = {} layers = {} - X = _cast_object_array_to_float_or_str(edata.X, "X") + X, kinds = _encode_object_array(edata.X, "X") + if kinds is not None: + obsm[_kinds_key(None)] = kinds if _is_3d(X): if isinstance(X, sparse.COO): sparse_3d_data[EHRDATA_OBSM_3D_X_KEY] = X @@ -104,7 +137,9 @@ def encode_for_disk(edata: EHRData) -> tuple[ad.AnnData, dict[str, sparse.COO]]: for key, value in edata.layers.items(): if key is None: # anndata 0.13: the unified `.X` slot, already handled above continue - value = _cast_object_array_to_float_or_str(value, f"layers[{key!r}]") + value, kinds = _encode_object_array(value, f"layers[{key!r}]") + if kinds is not None: + obsm[_kinds_key(key)] = kinds if _is_3d(value): target = _obsm_key_for_layer(key) if isinstance(value, sparse.COO): @@ -157,26 +192,58 @@ def encode_for_writing( return adata, sparse_3d_data, tem -def decode_init_dict(init: dict[str, Any]) -> dict[str, Any]: - """Restore relocated 3D arrays from ``.obsm`` back into ``X``/``layers`` in a dict of init kwargs.""" +def writes_exact_arrays(version: str | None) -> bool: + """Whether data of on-disk `version` holds `X`/layers exactly, with kind arrays, as since 0.3.0.""" + return version not in {None, "0.2.0"} + + +def _strings_to_object(array: Any) -> Any: + # as written; zarr reads strings as StringDType + return array.astype(object) if isinstance(array, np.ndarray) and array.dtype.kind in "UT" else array + + +def _decode_arrays(X: Any, layers: MutableMapping[str, Any], obsm: MutableMapping[str, Any], version: str) -> Any: + """Decode `X` and `layers` with the kind arrays popped from `obsm`, in place for `layers`; return `X`.""" + kinds = _pop_kinds(obsm) + if not writes_exact_arrays(version): + if kinds: + logger.warning(f"Ignoring {[_kinds_key(k) for k in kinds]} in .obsm, of data of on-disk version {version}.") + return X + for layer, layer_kinds in kinds.items(): + array = X if layer is None else layers.get(layer) + if array is None or array.shape != layer_kinds.shape: + logger.warning(f"Ignoring {_kinds_key(layer)!r} in .obsm, which does not match the array it belongs to.") + continue + if layer is None: + X = _decode_object_array(array, layer_kinds) + else: + layers[layer] = _decode_object_array(array, layer_kinds) + for layer in [k for k in layers if k is not None]: + layers[layer] = _strings_to_object(layers[layer]) + return _strings_to_object(X) + + +def decode_init_dict(init: dict[str, Any], version: str) -> dict[str, Any]: + """Restore relocated 3D arrays from ``.obsm`` back into ``X``/``layers``, and decode the arrays with kind arrays, in a dict of init kwargs.""" obsm = dict(init.get("obsm") or {}) layers = dict(init.get("layers") or {}) - relocated = False if EHRDATA_OBSM_3D_X_KEY in obsm: init["X"] = obsm.pop(EHRDATA_OBSM_3D_X_KEY) - relocated = True for key in [k for k in obsm if _layer_for_obsm_key(k) is not None]: layers[_layer_for_obsm_key(key)] = obsm.pop(key) - relocated = True - - if relocated: - init["obsm"] = obsm - init["layers"] = layers + init["X"] = _decode_arrays(init.get("X"), layers, obsm, version) + init["obsm"] = obsm + init["layers"] = layers return init +def decode_backed_layers(adata: ad.AnnData, version: str) -> None: + """Decode the layers of a backed :class:`~anndata.AnnData`, which holds its layers in memory; its `X` stays backed.""" + _decode_arrays(None, adata.layers, adata.obsm, version) + + def read_mapping_with_coo(group: GroupStorageType) -> dict[str, Any]: """Read an ``.obsm``-like group child-by-child, routing ehrdata COO groups to the COO reader. @@ -204,14 +271,34 @@ def read_init_dict(group: GroupStorageType) -> dict[str, Any]: } -def _check_020_ehrdata_on_disk_format(f: Any) -> bool: - """Return True if the file/store carries the ehrdata 0.2.0 on-disk stamp. +def ehrdata_on_disk_version(f: Any) -> str | None: + """Return the on-disk version of an ehrdata file/store, or `None` for AnnData and ehrdata stores before 0.2.0. The type stamp lives under a different key per format (zarr uses ``encoding-type``, h5ed uses the namespaced ``ehrdata-encoding-type``), so either key identifies an ehrdata store here. + Raises for versions this ehrdata cannot read, such as those of a newer ehrdata. """ stamped_as_ehrdata = EHRDATA_ENCODING_TYPE in ( f.attrs.get(EHRDATA_ENCODING_TYPE_KEY_ZARR), f.attrs.get(EHRDATA_ENCODING_TYPE_KEY), ) - return stamped_as_ehrdata and f.attrs.get(EHRDATA_ONDISK_VERSION_KEY) == EHRDATA_ONDISK_VERSION + version = f.attrs.get(EHRDATA_ONDISK_VERSION_KEY) if stamped_as_ehrdata else None + if version is not None and version not in EHRDATA_ONDISK_VERSIONS_READ: + msg = ( + f"The data was written in ehrdata's on-disk version {version!r}, but this ehrdata reads versions " + f"{', '.join(EHRDATA_ONDISK_VERSIONS_READ)}. Please upgrade ehrdata." + ) + raise ValueError(msg) + return version + + +def apply_read_heuristics( + edata: EHRData, version: str | None, *, harmonize_missing_values: bool | None, cast_variables_to_float: bool | None +) -> None: + """Apply the read-time heuristics; by default (`None`) only to data written before on-disk version 0.3.0.""" + written_before_030 = not writes_exact_arrays(version) + if harmonize_missing_values or (harmonize_missing_values is None and written_before_030): + # zeros of a sparse.COO were written as zeros + _harmonize_on_read(edata, skip_coo=harmonize_missing_values is None) + if cast_variables_to_float or (cast_variables_to_float is None and written_before_030): + _cast_variables_to_float(edata) diff --git a/src/ehrdata/io/elem.py b/src/ehrdata/io/elem.py index 07e8c9ce..829d3faa 100644 --- a/src/ehrdata/io/elem.py +++ b/src/ehrdata/io/elem.py @@ -27,7 +27,7 @@ def write_elem( Use this to write an :class:`~ehrdata.EHRData` as one part of a larger file or store, next to other data. `parent[key]` holds the same layout as a file written with :func:`~ehrdata.io.write_h5ed` for an :class:`h5py.Group`, and as a store written with :func:`~ehrdata.io.write_zarr` for a :class:`zarr.Group`, and - the data is cast for writing as these functions do. + the data is cast for writing as these functions do; `X` and `layers` read back exactly. An existing `parent[key]` is replaced, as with :func:`anndata.io.write_elem`. Zarr stores with consolidated metadata cannot be written to. Read it back with :func:`~ehrdata.io.read_elem`. @@ -94,21 +94,26 @@ def write_elem( def read_elem( group: h5py.Group | zarr.Group, *, - harmonize_missing_values: bool = True, - cast_variables_to_float: bool = True, + harmonize_missing_values: bool | None = None, + cast_variables_to_float: bool | None = None, ) -> EHRData: """Read an :class:`~ehrdata.EHRData` from a group of an HDF5 file or a Zarr store. Reads a group written with :func:`~ehrdata.io.write_elem`, :func:`~ehrdata.io.write_h5ed` or :func:`~ehrdata.io.write_zarr`, fully into memory. - Like :func:`~ehrdata.io.read_h5ed` and :func:`~ehrdata.io.read_zarr`, it also reads :class:`~anndata.AnnData` groups. + Like :func:`~ehrdata.io.read_h5ed` and :func:`~ehrdata.io.read_zarr`, it also reads :class:`~anndata.AnnData` groups, + and reads `X` and `layers` back exactly, applying the heuristics below by default only to groups written before + ehrdata's on-disk version 0.3.0 and to :class:`~anndata.AnnData` groups. Args: group: The group to read, such as `f["edata"]` of an open :class:`h5py.File` or a :class:`zarr.Group`. - harmonize_missing_values: Whether to call `ehrdata.harmonize_missing_values` on all detected layers. + harmonize_missing_values: Whether to call :func:`~ehrdata.harmonize_missing_values` on `X` and all layers. + By default (`None`), only for groups written before on-disk version 0.3.0 and :class:`~anndata.AnnData` groups, + and leaving `sparse.COO` arrays untouched. `True` also applies it to `sparse.COO` arrays, treating their zeros as missing. cast_variables_to_float: For non-numeric arrays, try to cast the values for each variable to dtype `np.float64`. If the cast fails for the values of one variable, then the values of these variable remain unaltered. This can be helpful to recover arrays that were of dtype `object` when they were written to disk. + By default (`None`), only for groups written before on-disk version 0.3.0 and :class:`~anndata.AnnData` groups. Examples: >>> import h5py diff --git a/src/ehrdata/io/h5ed.py b/src/ehrdata/io/h5ed.py index 516d4a9f..aa5fb637 100644 --- a/src/ehrdata/io/h5ed.py +++ b/src/ehrdata/io/h5ed.py @@ -6,24 +6,26 @@ import anndata as ad import h5py -from ehrdata._feature_types import _harmonize_on_read from ehrdata.core.constants import ( EHRDATA_ENCODING_TYPE, EHRDATA_ENCODING_TYPE_KEY, EHRDATA_OBSM_3D_LAYER_PREFIX, EHRDATA_OBSM_3D_X_KEY, + EHRDATA_OBSM_KINDS_X_KEY, EHRDATA_ONDISK_VERSION, EHRDATA_ONDISK_VERSION_KEY, ) from ehrdata.core.ehrdata import _silence_anndata_nd_warning -from ehrdata.io._array_casting import _cast_variables_to_float from ehrdata.io._coo_codec import write_coo_h5 from ehrdata.io._ondisk import ( _allow_write_nullable_strings, - _check_020_ehrdata_on_disk_format, + apply_read_heuristics, + decode_backed_layers, decode_init_dict, + ehrdata_on_disk_version, encode_for_writing, read_init_dict, + writes_exact_arrays, ) if TYPE_CHECKING: @@ -36,12 +38,15 @@ def read_h5ed( filename: Path | str, *, backed: Literal["r", "r+"] | bool | None = None, - harmonize_missing_values: bool = True, - cast_variables_to_float: bool = True, + harmonize_missing_values: bool | None = None, + cast_variables_to_float: bool | None = None, ) -> EHRData: """Read an ehrdata hdf5 file (`.h5ed`) into an :class:`~ehrdata.EHRData` object. Also can read plain anndata `.h5ad` files. + `X` and `layers` read back exactly as written by :func:`~ehrdata.io.write_h5ed`. + Files written before ehrdata's on-disk version 0.3.0, and plain anndata files, are read as before: + missing values written as strings are harmonized, and the variables of non-numeric arrays cast to float. Args: filename: Path to the file or directory to read. @@ -51,12 +56,17 @@ def read_h5ed( That means any changes to other slots like obs will not be written to disk in backed mode. If you would like save changes made to these slots of a backed EHRData, write them to a new file (see :func:`~ehrdata.io.write_h5ed`). - harmonize_missing_values: Whether to call `ehrdata.harmonize_missing_values` on all detected layers. - Cannot be called if `backed`. + Backed reading applies neither `harmonize_missing_values` nor `cast_variables_to_float`. + It does not support 3D arrays, nor an `X` of text next to numbers or missing values. + harmonize_missing_values: Whether to call :func:`~ehrdata.harmonize_missing_values` on `X` and all layers. + By default (`None`), only for files written before on-disk version 0.3.0 and plain anndata files, and + leaving `sparse.COO` arrays untouched. `True` also applies it to `sparse.COO` arrays, treating their zeros as missing. + Cannot be `True` if `backed`. cast_variables_to_float: For non-numeric arrays, try to cast the values for each variable to dtype `np.float64`. If the cast fails for the values of one variable, then the values of these variable remain unaltered. This can be helpful to recover arrays that were of dtype `object` when they were written to disk. - Cannot be called if `backed`. + By default (`None`), only for files written before on-disk version 0.3.0 and plain anndata files. + Cannot be `True` if `backed`. Examples: >>> import ehrdata as ed @@ -85,40 +95,48 @@ def read_h5ed( if mode is True: mode = "r+" - with _silence_anndata_nd_warning(): - adata = ad.read_h5ad(filename, backed=mode) - with h5py.File(filename, "r") as f: tem = ad.io.read_elem(f["tem"]) if "tem" in f else None - # A 0.2.0 file relocates 3D `X`/layers into `.obsm`; backed mode can't move them back + # 3D `X`/layers are relocated into `.obsm`; backed mode can't move them back obsm = f.get("obsm", {}) - if _check_020_ehrdata_on_disk_format(f) and ( + version = ehrdata_on_disk_version(f) + if version is not None and ( EHRDATA_OBSM_3D_X_KEY in obsm or any(k.startswith(EHRDATA_OBSM_3D_LAYER_PREFIX) for k in obsm) ): msg_0 = "Backed reading of .h5ed files with 3D arrays is not supported. Please open an issue on GitHub if you need this feature." raise NotImplementedError(msg_0) - edata = EHRData.from_adata(adata, tem=tem) + if writes_exact_arrays(version) and EHRDATA_OBSM_KINDS_X_KEY in obsm: + msg_1 = "Backed reading of an X of text next to numbers or missing values is not supported; read it without `backed`." + raise NotImplementedError(msg_1) + + with _silence_anndata_nd_warning(): + adata = ad.read_h5ad(filename, backed=mode) + if version is not None: + decode_backed_layers(adata, version) + edata = EHRData.from_adata(adata, tem=tem) return edata -def _read_h5ed_elem(group: h5py.Group, *, harmonize_missing_values: bool, cast_variables_to_float: bool) -> EHRData: +def _read_h5ed_elem( + group: h5py.Group, *, harmonize_missing_values: bool | None, cast_variables_to_float: bool | None +) -> EHRData: """Read an :class:`~ehrdata.EHRData` from an HDF5 group in the `.h5ed` layout, fully into memory.""" from ehrdata import EHRData + version = ehrdata_on_disk_version(group) dictionary_for_init = read_init_dict(group) - - if _check_020_ehrdata_on_disk_format(group): - dictionary_for_init = decode_init_dict(dictionary_for_init) + if version is not None: + dictionary_for_init = decode_init_dict(dictionary_for_init, version) edata = EHRData(**dictionary_for_init) - if harmonize_missing_values: - _harmonize_on_read(edata) - - if cast_variables_to_float: - _cast_variables_to_float(edata) - + apply_read_heuristics( + edata, + version, + harmonize_missing_values=harmonize_missing_values, + cast_variables_to_float=cast_variables_to_float, + ) return edata @@ -134,8 +152,12 @@ def write_h5ed( `.h5ed` is the ehrdata on-disk format. To write the file, `X` and `layers` cannot be written as `object` dtype. - If any of these fields is of `object` dtype, this function will attempt to cast it to a numeric dtype; if this fails, the field will be casted to a string dtype. - The same holds for the `object` columns of `.obs`, `.var` and `.tem`; integers and booleans among them stay exact, and missing values stay missing. + An `object` array whose values are all numbers, including booleans and `Decimal`, or missing is written as float. + Otherwise, its values are written as strings, with the kind of each value (text, number or missing) next to it. + :func:`~ehrdata.io.read_h5ed` reads it back exactly: text stays text, including codes like `"007"` and `"nan"`, + numbers, also booleans, become floats and missing values `np.nan`. Integers that float64 does not hold exactly are written as text. + `.obsm` keys starting with `_ed_ondisk_` are reserved. + The `object` columns of `.obs`, `.var` and `.tem` are written as a numeric dtype where their values allow it, and as strings otherwise; integers and booleans among them stay exact, and missing values stay missing. Datetime and timedelta columns are written as ISO 8601 strings, read back as categorical with `convert_strings_to_categoricals=True` if a value repeats or is missing. @@ -203,8 +225,8 @@ def read_h5ad( filename: Path | str, *, backed: Literal["r", "r+"] | bool | None = None, - harmonize_missing_values: bool = True, - cast_variables_to_float: bool = True, + harmonize_missing_values: bool | None = None, + cast_variables_to_float: bool | None = None, ) -> EHRData: """Deprecated alias for :func:`read_h5ed`. diff --git a/src/ehrdata/io/zarr.py b/src/ehrdata/io/zarr.py index 5a9e1488..8b397c91 100644 --- a/src/ehrdata/io/zarr.py +++ b/src/ehrdata/io/zarr.py @@ -7,19 +7,18 @@ import anndata as ad import zarr -from ehrdata._feature_types import _harmonize_on_read from ehrdata.core.constants import ( EHRDATA_ENCODING_TYPE, EHRDATA_ENCODING_TYPE_KEY_ZARR, EHRDATA_ONDISK_VERSION, EHRDATA_ONDISK_VERSION_KEY, ) -from ehrdata.io._array_casting import _cast_variables_to_float from ehrdata.io._coo_codec import write_coo_zarr from ehrdata.io._ondisk import ( _allow_write_nullable_strings, - _check_020_ehrdata_on_disk_format, + apply_read_heuristics, decode_init_dict, + ehrdata_on_disk_version, encode_for_writing, read_init_dict, ) @@ -37,19 +36,25 @@ def read_zarr( filename: PathLike[str] | zarr.Group | str, *, - harmonize_missing_values: bool = True, - cast_variables_to_float: bool = True, + harmonize_missing_values: bool | None = None, + cast_variables_to_float: bool | None = None, ) -> EHRData: """Read a zarr store into an :class:`~ehrdata.EHRData` object. Can also read :class:`~anndata.AnnData` Zarr stores. In this case, a default `.tem` field is created in the `ehrdata object`. + `X` and `layers` read back exactly as written by :func:`~ehrdata.io.write_zarr`. + Stores written before ehrdata's on-disk version 0.3.0, and anndata stores, are read as before: + missing values written as strings are harmonized, and the variables of non-numeric arrays cast to float. Args: filename: The filename, or a Zarr storage class. - harmonize_missing_values: Whether to call `ehrdata.harmonize_missing_values` on all detected layers. + harmonize_missing_values: Whether to call :func:`~ehrdata.harmonize_missing_values` on `X` and all layers. + By default (`None`), only for stores written before on-disk version 0.3.0 and anndata stores, and + leaving `sparse.COO` arrays untouched. `True` also applies it to `sparse.COO` arrays, treating their zeros as missing. cast_variables_to_float: For non-numeric arrays, try to cast the values for each variable to dtype `np.float64`. If the cast fails for the values of one variable, then the values of these variable remain unaltered. This can be helpful to recover arrays that were of dtype `object` when they were written to disk. + By default (`None`), only for stores written before on-disk version 0.3.0 and anndata stores. Examples: >>> import ehrdata as ed @@ -67,7 +72,9 @@ def read_zarr( ) -def _read_zarr_elem(f: zarr.Group, *, harmonize_missing_values: bool, cast_variables_to_float: bool) -> EHRData: +def _read_zarr_elem( + f: zarr.Group, *, harmonize_missing_values: bool | None, cast_variables_to_float: bool | None +) -> EHRData: """Read an :class:`~ehrdata.EHRData` from a Zarr group in the :func:`~ehrdata.io.write_zarr` layout.""" from ehrdata import EHRData @@ -94,16 +101,17 @@ def _read_zarr_elem(f: zarr.Group, *, harmonize_missing_values: bool, cast_varia err = f"Unknown encoding-type '{encoding_type}'." raise ValueError(err) - if _check_020_ehrdata_on_disk_format(f): - dictionary_for_init = decode_init_dict(dictionary_for_init) + version = ehrdata_on_disk_version(f) + if version is not None: + dictionary_for_init = decode_init_dict(dictionary_for_init, version) edata = EHRData(**dictionary_for_init) - if harmonize_missing_values: - _harmonize_on_read(edata) - - if cast_variables_to_float: - _cast_variables_to_float(edata) - + apply_read_heuristics( + edata, + version, + harmonize_missing_values=harmonize_missing_values, + cast_variables_to_float=cast_variables_to_float, + ) return edata @@ -117,8 +125,12 @@ def write_zarr( """Write :class:`~ehrdata.EHRData` objects to disk. To write to a `.zarr` store, `X`, and `layers` cannot be written as `object` dtype. - If any of these fields is of `object` dtype, this function will attempt to cast it to a numeric dtype; if this fails, the field will be casted to a `str` dtype. - The same holds for the `object` columns of `.obs`, `.var` and `.tem`; integers and booleans among them stay exact, and missing values stay missing. + An `object` array whose values are all numbers, including booleans and `Decimal`, or missing is written as float. + Otherwise, its values are written as strings, with the kind of each value (text, number or missing) next to it. + :func:`~ehrdata.io.read_zarr` reads it back exactly: text stays text, including codes like `"007"` and `"nan"`, + numbers, also booleans, become floats and missing values `np.nan`. Integers that float64 does not hold exactly are written as text. + `.obsm` keys starting with `_ed_ondisk_` are reserved. + The `object` columns of `.obs`, `.var` and `.tem` are written as a numeric dtype where their values allow it, and as strings otherwise; integers and booleans among them stay exact, and missing values stay missing. Datetime and timedelta columns are written as ISO 8601 strings, read back as categorical with `convert_strings_to_categoricals=True` if a value repeats or is missing. diff --git a/tests/conftest.py b/tests/conftest.py index 9259296c..ee159cc4 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,18 +1,22 @@ import datetime as dt +from decimal import Decimal from pathlib import Path import anndata as ad import dask.array as da import duckdb +import h5py import numpy as np import pandas as pd import pytest import scipy.sparse as sp import sparse +import zarr from ehrdata import EHRData -from ehrdata.core.constants import DEFAULT_TEM_LAYER_NAME +from ehrdata.core.constants import DEFAULT_TEM_LAYER_NAME, EHRDATA_ONDISK_VERSION_KEY from ehrdata.dt import mimic_iv_omop +from ehrdata.io import read_elem, read_h5ed, read_zarr, write_elem, write_h5ed, write_zarr from ehrdata.io.omop import setup_connection @@ -163,12 +167,17 @@ def X_any_array_type_322(request): def _edata_with_object_layers(X) -> EHRData: - """EHRData with `object` layers of numbers and of strings, with missing values, next to `X` (may be `None`).""" - numbers = np.array([[[1, None], [2, 3]], [[4, 5], [6, 7]], [[8, 9], [10, 11]]], dtype=object) + """EHRData with `object` layers of numbers, of strings, of codes and of both, with missing values, next to `X` (may be `None`).""" + # e.g. Decimal from SQL drivers + numbers = np.array([[[1, None], [Decimal(2), 3]], [[4, 5], [6, 7]], [[8, 9], [10, 11]]], dtype=object) strings = np.array([[["1", None], ["a", "b"]], [["2", "3"], ["c", None]], [["4", "5"], ["d", "e"]]], dtype=object) + codes = np.array( + [[["007", "010"], ["250", "nan"]], [["V45", "042"], ["1", "2"]], [["3", "4"], ["5", "6"]]], dtype=object + ) + mixed = np.array([[[1, "007"], [None, "nan"]], [[Decimal("2.5"), "a"], [3, None]], [["", 4], [5, 6]]], dtype=object) return EHRData( X=X, - layers={"numbers": numbers, "strings": strings}, + layers={"numbers": numbers, "strings": strings, "codes": codes, "mixed": mixed}, obs=pd.DataFrame(index=["0", "1", "2"]), var=pd.DataFrame(index=["0", "1"]), tem=pd.DataFrame(index=["0", "1"]), @@ -176,17 +185,19 @@ def _edata_with_object_layers(X) -> EHRData: def _assert_object_layers_read_back(edata: EHRData, edata_read: EHRData): + assert edata_read.layers["numbers"].dtype == np.float64 assert np.array_equal( - edata_read.layers["numbers"].astype(np.float64), + edata_read.layers["numbers"], np.array([[[1, np.nan], [2, 3]], [[4, 5], [6, 7]], [[8, 9], [10, 11]]]), equal_nan=True, ) - # cast per variable on read; missing values are np.nan again - strings = edata_read.layers["strings"] - assert np.array_equal(strings[:, 0, :].astype(np.float64), np.array([[1, np.nan], [2, 3], [4, 5]]), equal_nan=True) - _assert_dtype_object_array_with_missing_values_equal( - strings[:, 1, :].astype(object), np.array([["a", "b"], ["c", np.nan], ["d", "e"]], dtype=object) - ) + # exact: text stays text, numbers next to text are floats, missing values np.nan + for key in ("strings", "codes", "mixed"): + layer, written = edata_read.layers[key].ravel(), edata.layers[key].ravel() + assert edata_read.layers[key].dtype == object + assert _values(pd.Series(layer)) == list(written) + assert [isinstance(value, str) for value in layer] == [isinstance(value, str) for value in written] + assert all(isinstance(value, float) for value in layer if not isinstance(value, str)) if edata.X is None: assert edata_read.X is None @@ -195,8 +206,41 @@ def _assert_object_layers_read_back(edata: EHRData, edata_read: EHRData): assert np.array_equal(_to_dense(edata_read.X), _to_dense(edata.X), equal_nan=True) # the input is left unchanged - assert edata.layers["numbers"].dtype == object - assert edata.layers["strings"].dtype == object + assert edata.layers["strings"][0, 0, 1] is None + assert edata.layers["mixed"][0, 0, 0] == 1 + assert all(layer.dtype == object for key, layer in edata.layers.items() if key is not None) + + +def _set_version(group, version: str | None): + if version is not None: + group.attrs[EHRDATA_ONDISK_VERSION_KEY] = version + + +@pytest.fixture(params=["h5ed", "zarr", "elem_h5", "elem_zarr"]) +def write_read(request, tmp_path): + """Write an EHRData as a file, a store, or a group of either, and read it back; `version` overwrites its on-disk version.""" + + def write_read(edata: EHRData, *, version: str | None = None, **read_kwargs) -> EHRData: + if request.param == "h5ed": + write_h5ed(edata, tmp_path / "edata.h5ed") + with h5py.File(tmp_path / "edata.h5ed", "a") as f: + _set_version(f, version) + return read_h5ed(tmp_path / "edata.h5ed", **read_kwargs) + if request.param == "zarr": + write_zarr(edata, tmp_path / "edata.zarr") + _set_version(zarr.open_group(tmp_path / "edata.zarr"), version) + return read_zarr(tmp_path / "edata.zarr", **read_kwargs) + if request.param == "elem_h5": + with h5py.File(tmp_path / "bundle.h5", "w") as f: + write_elem(f, "edata", edata) + _set_version(f["edata"], version) + return read_elem(f["edata"], **read_kwargs) + root = zarr.open_group(tmp_path / "bundle.zarr", mode="w") + write_elem(root, "edata", edata) + _set_version(root["edata"], version) + return read_elem(root["edata"], **read_kwargs) + + return write_read @pytest.fixture diff --git a/tests/data/toy_h5ed/edata_object_layers_v0_2_0.h5ed b/tests/data/toy_h5ed/edata_object_layers_v0_2_0.h5ed new file mode 100644 index 00000000..5a84bdce Binary files /dev/null and b/tests/data/toy_h5ed/edata_object_layers_v0_2_0.h5ed differ diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/X/c/0/0 b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/X/c/0/0 new file mode 100644 index 00000000..24a1917c Binary files /dev/null and b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/X/c/0/0 differ diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/X/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/X/zarr.json new file mode 100644 index 00000000..7451c6a9 --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/X/zarr.json @@ -0,0 +1,59 @@ +{ + "shape": [3, 2], + "data_type": "float64", + "chunk_grid": { + "name": "regular", + "configuration": { + "chunk_shape": [3, 2] + } + }, + "chunk_key_encoding": { + "name": "default", + "configuration": { + "separator": "/" + } + }, + "fill_value": 0.0, + "codecs": [ + { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": [3, 2], + "codecs": [ + { + "name": "bytes", + "configuration": { + "endian": "little" + } + }, + { + "name": "zstd", + "configuration": { + "level": 0, + "checksum": false + } + } + ], + "index_codecs": [ + { + "name": "bytes", + "configuration": { + "endian": "little" + } + }, + { + "name": "crc32c" + } + ], + "index_location": "end" + } + } + ], + "attributes": { + "encoding-type": "array", + "encoding-version": "0.2.0" + }, + "zarr_format": 3, + "node_type": "array", + "storage_transformers": [] +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/layers/codes/c/0/0 b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/layers/codes/c/0/0 new file mode 100644 index 00000000..b938c4e6 Binary files /dev/null and b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/layers/codes/c/0/0 differ diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/layers/codes/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/layers/codes/zarr.json new file mode 100644 index 00000000..0d8d15ba --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/layers/codes/zarr.json @@ -0,0 +1,57 @@ +{ + "shape": [3, 2], + "data_type": "string", + "chunk_grid": { + "name": "regular", + "configuration": { + "chunk_shape": [3, 2] + } + }, + "chunk_key_encoding": { + "name": "default", + "configuration": { + "separator": "/" + } + }, + "fill_value": "", + "codecs": [ + { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": [3, 2], + "codecs": [ + { + "name": "vlen-utf8", + "configuration": {} + }, + { + "name": "zstd", + "configuration": { + "level": 0, + "checksum": false + } + } + ], + "index_codecs": [ + { + "name": "bytes", + "configuration": { + "endian": "little" + } + }, + { + "name": "crc32c" + } + ], + "index_location": "end" + } + } + ], + "attributes": { + "encoding-type": "string-array", + "encoding-version": "0.2.0" + }, + "zarr_format": 3, + "node_type": "array", + "storage_transformers": [] +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/layers/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/layers/zarr.json new file mode 100644 index 00000000..b9997d0f --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/layers/zarr.json @@ -0,0 +1,8 @@ +{ + "attributes": { + "encoding-type": "dict", + "encoding-version": "0.1.0" + }, + "zarr_format": 3, + "node_type": "group" +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obs/_index/mask/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obs/_index/mask/zarr.json new file mode 100644 index 00000000..5ea23428 --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obs/_index/mask/zarr.json @@ -0,0 +1,56 @@ +{ + "shape": [3], + "data_type": "bool", + "chunk_grid": { + "name": "regular", + "configuration": { + "chunk_shape": [3] + } + }, + "chunk_key_encoding": { + "name": "default", + "configuration": { + "separator": "/" + } + }, + "fill_value": false, + "codecs": [ + { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": [3], + "codecs": [ + { + "name": "bytes" + }, + { + "name": "zstd", + "configuration": { + "level": 0, + "checksum": false + } + } + ], + "index_codecs": [ + { + "name": "bytes", + "configuration": { + "endian": "little" + } + }, + { + "name": "crc32c" + } + ], + "index_location": "end" + } + } + ], + "attributes": { + "encoding-type": "array", + "encoding-version": "0.2.0" + }, + "zarr_format": 3, + "node_type": "array", + "storage_transformers": [] +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obs/_index/values/c/0 b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obs/_index/values/c/0 new file mode 100644 index 00000000..792d95d1 Binary files /dev/null and b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obs/_index/values/c/0 differ diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obs/_index/values/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obs/_index/values/zarr.json new file mode 100644 index 00000000..3a1a490d --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obs/_index/values/zarr.json @@ -0,0 +1,57 @@ +{ + "shape": [3], + "data_type": "string", + "chunk_grid": { + "name": "regular", + "configuration": { + "chunk_shape": [3] + } + }, + "chunk_key_encoding": { + "name": "default", + "configuration": { + "separator": "/" + } + }, + "fill_value": "", + "codecs": [ + { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": [3], + "codecs": [ + { + "name": "vlen-utf8", + "configuration": {} + }, + { + "name": "zstd", + "configuration": { + "level": 0, + "checksum": false + } + } + ], + "index_codecs": [ + { + "name": "bytes", + "configuration": { + "endian": "little" + } + }, + { + "name": "crc32c" + } + ], + "index_location": "end" + } + } + ], + "attributes": { + "encoding-type": "string-array", + "encoding-version": "0.2.0" + }, + "zarr_format": 3, + "node_type": "array", + "storage_transformers": [] +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obs/_index/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obs/_index/zarr.json new file mode 100644 index 00000000..205ef72f --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obs/_index/zarr.json @@ -0,0 +1,9 @@ +{ + "attributes": { + "na-value": "NaN", + "encoding-type": "nullable-string-array", + "encoding-version": "0.1.0" + }, + "zarr_format": 3, + "node_type": "group" +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obs/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obs/zarr.json new file mode 100644 index 00000000..5809a145 --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obs/zarr.json @@ -0,0 +1,10 @@ +{ + "attributes": { + "column-order": [], + "_index": "_index", + "encoding-type": "dataframe", + "encoding-version": "0.2.0" + }, + "zarr_format": 3, + "node_type": "group" +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/fill_value/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/fill_value/zarr.json new file mode 100644 index 00000000..87517180 --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/fill_value/zarr.json @@ -0,0 +1,36 @@ +{ + "shape": [1], + "data_type": "float64", + "chunk_grid": { + "name": "regular", + "configuration": { + "chunk_shape": [1] + } + }, + "chunk_key_encoding": { + "name": "default", + "configuration": { + "separator": "/" + } + }, + "fill_value": 0.0, + "codecs": [ + { + "name": "bytes", + "configuration": { + "endian": "little" + } + }, + { + "name": "zstd", + "configuration": { + "level": 0, + "checksum": false + } + } + ], + "attributes": {}, + "zarr_format": 3, + "node_type": "array", + "storage_transformers": [] +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/indices_0/c/0 b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/indices_0/c/0 new file mode 100644 index 00000000..c3dc92cc Binary files /dev/null and b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/indices_0/c/0 differ diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/indices_0/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/indices_0/zarr.json new file mode 100644 index 00000000..233ca0fa --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/indices_0/zarr.json @@ -0,0 +1,36 @@ +{ + "shape": [2], + "data_type": "int64", + "chunk_grid": { + "name": "regular", + "configuration": { + "chunk_shape": [2] + } + }, + "chunk_key_encoding": { + "name": "default", + "configuration": { + "separator": "/" + } + }, + "fill_value": 0, + "codecs": [ + { + "name": "bytes", + "configuration": { + "endian": "little" + } + }, + { + "name": "zstd", + "configuration": { + "level": 0, + "checksum": false + } + } + ], + "attributes": {}, + "zarr_format": 3, + "node_type": "array", + "storage_transformers": [] +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/indices_1/c/0 b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/indices_1/c/0 new file mode 100644 index 00000000..b56c64b1 Binary files /dev/null and b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/indices_1/c/0 differ diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/indices_1/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/indices_1/zarr.json new file mode 100644 index 00000000..233ca0fa --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/indices_1/zarr.json @@ -0,0 +1,36 @@ +{ + "shape": [2], + "data_type": "int64", + "chunk_grid": { + "name": "regular", + "configuration": { + "chunk_shape": [2] + } + }, + "chunk_key_encoding": { + "name": "default", + "configuration": { + "separator": "/" + } + }, + "fill_value": 0, + "codecs": [ + { + "name": "bytes", + "configuration": { + "endian": "little" + } + }, + { + "name": "zstd", + "configuration": { + "level": 0, + "checksum": false + } + } + ], + "attributes": {}, + "zarr_format": 3, + "node_type": "array", + "storage_transformers": [] +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/indices_2/c/0 b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/indices_2/c/0 new file mode 100644 index 00000000..1684494b Binary files /dev/null and b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/indices_2/c/0 differ diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/indices_2/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/indices_2/zarr.json new file mode 100644 index 00000000..233ca0fa --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/indices_2/zarr.json @@ -0,0 +1,36 @@ +{ + "shape": [2], + "data_type": "int64", + "chunk_grid": { + "name": "regular", + "configuration": { + "chunk_shape": [2] + } + }, + "chunk_key_encoding": { + "name": "default", + "configuration": { + "separator": "/" + } + }, + "fill_value": 0, + "codecs": [ + { + "name": "bytes", + "configuration": { + "endian": "little" + } + }, + { + "name": "zstd", + "configuration": { + "level": 0, + "checksum": false + } + } + ], + "attributes": {}, + "zarr_format": 3, + "node_type": "array", + "storage_transformers": [] +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/values/c/0 b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/values/c/0 new file mode 100644 index 00000000..9d8a6972 Binary files /dev/null and b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/values/c/0 differ diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/values/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/values/zarr.json new file mode 100644 index 00000000..6a1d5e4d --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/values/zarr.json @@ -0,0 +1,36 @@ +{ + "shape": [2], + "data_type": "float64", + "chunk_grid": { + "name": "regular", + "configuration": { + "chunk_shape": [2] + } + }, + "chunk_key_encoding": { + "name": "default", + "configuration": { + "separator": "/" + } + }, + "fill_value": 0.0, + "codecs": [ + { + "name": "bytes", + "configuration": { + "endian": "little" + } + }, + { + "name": "zstd", + "configuration": { + "level": 0, + "checksum": false + } + } + ], + "attributes": {}, + "zarr_format": 3, + "node_type": "array", + "storage_transformers": [] +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/zarr.json new file mode 100644 index 00000000..6996255c --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/_ed_ondisk_layers_counts/zarr.json @@ -0,0 +1,9 @@ +{ + "attributes": { + "ehrdata-encoding-type": "ehrdata-coo", + "ehrdata-encoding-version": "0.2.0", + "binsparse": "{\"version\": \"0.1\", \"format\": \"COO\", \"shape\": [3, 2, 2], \"number_of_stored_values\": 2, \"data_types\": {\"indices_0\": \"int64\", \"indices_1\": \"int64\", \"indices_2\": \"int64\", \"values\": \"float64\"}, \"fill\": true}" + }, + "zarr_format": 3, + "node_type": "group" +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/zarr.json new file mode 100644 index 00000000..b9997d0f --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsm/zarr.json @@ -0,0 +1,8 @@ +{ + "attributes": { + "encoding-type": "dict", + "encoding-version": "0.1.0" + }, + "zarr_format": 3, + "node_type": "group" +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsp/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsp/zarr.json new file mode 100644 index 00000000..b9997d0f --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/obsp/zarr.json @@ -0,0 +1,8 @@ +{ + "attributes": { + "encoding-type": "dict", + "encoding-version": "0.1.0" + }, + "zarr_format": 3, + "node_type": "group" +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/raw/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/raw/zarr.json new file mode 100644 index 00000000..ce7e4d5e --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/raw/zarr.json @@ -0,0 +1,36 @@ +{ + "shape": [], + "data_type": "bool", + "chunk_grid": { + "name": "regular", + "configuration": { + "chunk_shape": [] + } + }, + "chunk_key_encoding": { + "name": "default", + "configuration": { + "separator": "/" + } + }, + "fill_value": false, + "codecs": [ + { + "name": "bytes" + }, + { + "name": "zstd", + "configuration": { + "level": 0, + "checksum": false + } + } + ], + "attributes": { + "encoding-type": "null", + "encoding-version": "0.1.0" + }, + "zarr_format": 3, + "node_type": "array", + "storage_transformers": [] +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/uns/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/uns/zarr.json new file mode 100644 index 00000000..b9997d0f --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/uns/zarr.json @@ -0,0 +1,8 @@ +{ + "attributes": { + "encoding-type": "dict", + "encoding-version": "0.1.0" + }, + "zarr_format": 3, + "node_type": "group" +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/var/_index/mask/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/var/_index/mask/zarr.json new file mode 100644 index 00000000..3c5106e3 --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/var/_index/mask/zarr.json @@ -0,0 +1,56 @@ +{ + "shape": [2], + "data_type": "bool", + "chunk_grid": { + "name": "regular", + "configuration": { + "chunk_shape": [2] + } + }, + "chunk_key_encoding": { + "name": "default", + "configuration": { + "separator": "/" + } + }, + "fill_value": false, + "codecs": [ + { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": [2], + "codecs": [ + { + "name": "bytes" + }, + { + "name": "zstd", + "configuration": { + "level": 0, + "checksum": false + } + } + ], + "index_codecs": [ + { + "name": "bytes", + "configuration": { + "endian": "little" + } + }, + { + "name": "crc32c" + } + ], + "index_location": "end" + } + } + ], + "attributes": { + "encoding-type": "array", + "encoding-version": "0.2.0" + }, + "zarr_format": 3, + "node_type": "array", + "storage_transformers": [] +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/var/_index/values/c/0 b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/var/_index/values/c/0 new file mode 100644 index 00000000..4f5926dd Binary files /dev/null and b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/var/_index/values/c/0 differ diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/var/_index/values/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/var/_index/values/zarr.json new file mode 100644 index 00000000..37beee4e --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/var/_index/values/zarr.json @@ -0,0 +1,57 @@ +{ + "shape": [2], + "data_type": "string", + "chunk_grid": { + "name": "regular", + "configuration": { + "chunk_shape": [2] + } + }, + "chunk_key_encoding": { + "name": "default", + "configuration": { + "separator": "/" + } + }, + "fill_value": "", + "codecs": [ + { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": [2], + "codecs": [ + { + "name": "vlen-utf8", + "configuration": {} + }, + { + "name": "zstd", + "configuration": { + "level": 0, + "checksum": false + } + } + ], + "index_codecs": [ + { + "name": "bytes", + "configuration": { + "endian": "little" + } + }, + { + "name": "crc32c" + } + ], + "index_location": "end" + } + } + ], + "attributes": { + "encoding-type": "string-array", + "encoding-version": "0.2.0" + }, + "zarr_format": 3, + "node_type": "array", + "storage_transformers": [] +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/var/_index/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/var/_index/zarr.json new file mode 100644 index 00000000..205ef72f --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/var/_index/zarr.json @@ -0,0 +1,9 @@ +{ + "attributes": { + "na-value": "NaN", + "encoding-type": "nullable-string-array", + "encoding-version": "0.1.0" + }, + "zarr_format": 3, + "node_type": "group" +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/var/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/var/zarr.json new file mode 100644 index 00000000..5809a145 --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/var/zarr.json @@ -0,0 +1,10 @@ +{ + "attributes": { + "column-order": [], + "_index": "_index", + "encoding-type": "dataframe", + "encoding-version": "0.2.0" + }, + "zarr_format": 3, + "node_type": "group" +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/varm/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/varm/zarr.json new file mode 100644 index 00000000..b9997d0f --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/varm/zarr.json @@ -0,0 +1,8 @@ +{ + "attributes": { + "encoding-type": "dict", + "encoding-version": "0.1.0" + }, + "zarr_format": 3, + "node_type": "group" +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/varp/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/varp/zarr.json new file mode 100644 index 00000000..b9997d0f --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/varp/zarr.json @@ -0,0 +1,8 @@ +{ + "attributes": { + "encoding-type": "dict", + "encoding-version": "0.1.0" + }, + "zarr_format": 3, + "node_type": "group" +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/zarr.json new file mode 100644 index 00000000..90e83685 --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/anndata/zarr.json @@ -0,0 +1,8 @@ +{ + "attributes": { + "encoding-type": "anndata", + "encoding-version": "0.1.0" + }, + "zarr_format": 3, + "node_type": "group" +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/tem/_index/mask/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/tem/_index/mask/zarr.json new file mode 100644 index 00000000..3c5106e3 --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/tem/_index/mask/zarr.json @@ -0,0 +1,56 @@ +{ + "shape": [2], + "data_type": "bool", + "chunk_grid": { + "name": "regular", + "configuration": { + "chunk_shape": [2] + } + }, + "chunk_key_encoding": { + "name": "default", + "configuration": { + "separator": "/" + } + }, + "fill_value": false, + "codecs": [ + { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": [2], + "codecs": [ + { + "name": "bytes" + }, + { + "name": "zstd", + "configuration": { + "level": 0, + "checksum": false + } + } + ], + "index_codecs": [ + { + "name": "bytes", + "configuration": { + "endian": "little" + } + }, + { + "name": "crc32c" + } + ], + "index_location": "end" + } + } + ], + "attributes": { + "encoding-type": "array", + "encoding-version": "0.2.0" + }, + "zarr_format": 3, + "node_type": "array", + "storage_transformers": [] +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/tem/_index/values/c/0 b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/tem/_index/values/c/0 new file mode 100644 index 00000000..4f5926dd Binary files /dev/null and b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/tem/_index/values/c/0 differ diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/tem/_index/values/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/tem/_index/values/zarr.json new file mode 100644 index 00000000..37beee4e --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/tem/_index/values/zarr.json @@ -0,0 +1,57 @@ +{ + "shape": [2], + "data_type": "string", + "chunk_grid": { + "name": "regular", + "configuration": { + "chunk_shape": [2] + } + }, + "chunk_key_encoding": { + "name": "default", + "configuration": { + "separator": "/" + } + }, + "fill_value": "", + "codecs": [ + { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": [2], + "codecs": [ + { + "name": "vlen-utf8", + "configuration": {} + }, + { + "name": "zstd", + "configuration": { + "level": 0, + "checksum": false + } + } + ], + "index_codecs": [ + { + "name": "bytes", + "configuration": { + "endian": "little" + } + }, + { + "name": "crc32c" + } + ], + "index_location": "end" + } + } + ], + "attributes": { + "encoding-type": "string-array", + "encoding-version": "0.2.0" + }, + "zarr_format": 3, + "node_type": "array", + "storage_transformers": [] +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/tem/_index/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/tem/_index/zarr.json new file mode 100644 index 00000000..205ef72f --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/tem/_index/zarr.json @@ -0,0 +1,9 @@ +{ + "attributes": { + "na-value": "NaN", + "encoding-type": "nullable-string-array", + "encoding-version": "0.1.0" + }, + "zarr_format": 3, + "node_type": "group" +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/tem/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/tem/zarr.json new file mode 100644 index 00000000..5809a145 --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/tem/zarr.json @@ -0,0 +1,10 @@ +{ + "attributes": { + "column-order": [], + "_index": "_index", + "encoding-type": "dataframe", + "encoding-version": "0.2.0" + }, + "zarr_format": 3, + "node_type": "group" +} diff --git a/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/zarr.json b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/zarr.json new file mode 100644 index 00000000..08e0449a --- /dev/null +++ b/tests/data/toy_zarr/edata_object_layers_v0_2_0.ehrdata.zarr/zarr.json @@ -0,0 +1,8 @@ +{ + "attributes": { + "encoding-type": "ehrdata", + "ehrdata-encoding-version": "0.2.0" + }, + "zarr_format": 3, + "node_type": "group" +} diff --git a/tests/data/toy_zarr_h5ad_test_files_creation_reference.py b/tests/data/toy_zarr_h5ad_test_files_creation_reference.py index 062d7c30..e559f1b6 100644 --- a/tests/data/toy_zarr_h5ad_test_files_creation_reference.py +++ b/tests/data/toy_zarr_h5ad_test_files_creation_reference.py @@ -11,6 +11,7 @@ import numpy as np import pandas as pd import scipy as sp +import sparse import ehrdata as ed from ehrdata.core.constants import DEFAULT_TEM_LAYER_NAME @@ -75,7 +76,8 @@ ######################### -# edata_minimal_v0_2_0.h5ed / .ehrdata.zarr: tiny EHRData written in the current 0.2.0 encoding format +# edata_minimal_v0_2_0.h5ed / .ehrdata.zarr: tiny EHRData written in the 0.2.0 encoding format, by ehrdata before +# on-disk version 0.3.0 (running this with a later ehrdata writes its own version) # (3D arrays relocated into .obsm + `ehrdata-encoding-version="0.2.0"` stamp). Together with the # plain-anndata "version 0" adata_basic.* above, these form the minimal read-test corpus # (2x2: {h5, zarr} x {version-0 plain-anndata, 0.2.0 encoding format}). @@ -89,3 +91,23 @@ ) ed.io.write_h5ed(edata_minimal.copy(), TOY_H5ED / "edata_minimal_v0_2_0.h5ed") ed.io.write_zarr(edata_minimal.copy(), TOY_ZARR / "edata_minimal_v0_2_0.ehrdata.zarr") + + +######################### +# edata_object_layers_v0_2_0.h5ed / .ehrdata.zarr: an `object` layer of codes and missing values, and a zero-filled +# sparse.COO layer, written by ehrdata before on-disk version 0.3.0 (strings, missing values as "nan"); run with such an ehrdata +counts = np.zeros((3, 2, 2)) +counts[0, 0, 1] = 2 +counts[2, 1, 0] = 1 +edata_object_layers = ed.EHRData( + X=np.arange(6, dtype=float).reshape(3, 2), + layers={ + "codes": np.array([["007", None], ["250", "V45"], ["042", "nan"]], dtype=object), + "counts": sparse.COO.from_numpy(counts), + }, + obs=pd.DataFrame(index=["0", "1", "2"]), + var=pd.DataFrame(index=["0", "1"]), + tem=pd.DataFrame(index=["0", "1"]), +) +ed.io.write_h5ed(edata_object_layers, TOY_H5ED / "edata_object_layers_v0_2_0.h5ed") +ed.io.write_zarr(edata_object_layers, TOY_ZARR / "edata_object_layers_v0_2_0.ehrdata.zarr") diff --git a/tests/io/test_h5ed.py b/tests/io/test_h5ed.py index d7256028..8bb2a348 100644 --- a/tests/io/test_h5ed.py +++ b/tests/io/test_h5ed.py @@ -13,13 +13,13 @@ _assert_columns_anndata_cannot_write_read_back, _assert_dtype_object_array_with_missing_values_equal, _assert_io_read, - _assert_object_layers_read_back, _assert_shape_matches, _edata_with_columns_anndata_cannot_write, _edata_with_object_layers, ) from ehrdata import EHRData +from ehrdata.core.constants import EHRDATA_ONDISK_VERSION from ehrdata.io import read_h5ad, read_h5ed, read_zarr, write_h5ad, write_h5ed, write_zarr TEST_PATH_H5AD = TEST_DATA_PATH / "toy_h5ad" @@ -191,7 +191,7 @@ def test_write_h5ed_v2_relocates_3d_arrays_to_obsm(edata_333, tmp_path): assert "_ed_ondisk_layers_tem_data" in f["obsm"] assert "tem_data" not in f["layers"] assert f.attrs["ehrdata-encoding-type"] == "ehrdata" - assert f.attrs["ehrdata-encoding-version"] == "0.2.0" + assert f.attrs["ehrdata-encoding-version"] == EHRDATA_ONDISK_VERSION edata_read = read_h5ed(path) _assert_shape_matches(edata_read, (3, 3, 3)) @@ -405,15 +405,19 @@ def test_write_read_h5ed_casts_columns_anndata_cannot_write(X_any_array_type_322 _assert_columns_anndata_cannot_write_read_back(edata, read_h5ed(path, harmonize_missing_values=False)) -@pytest.mark.parametrize("with_X", [True, False]) -def test_write_read_h5ed_casts_object_layers(X_any_array_type_322, with_X, tmp_path): - # each layer is cast on its own, whatever `X` is - edata = _edata_with_object_layers(X_any_array_type_322 if with_X else None) - path = tmp_path / "object_layers.h5ed" - - write_h5ed(edata, path) +def test_write_h5ed_object_arrays_with_kinds(tmp_path): + # strings, with the kind of each value in a reserved .obsm key + edata = _edata_with_object_layers(np.array([["a", 1], [None, "007"], ["b", "c"]], dtype=object)) + write_h5ed(edata, tmp_path / "edata.h5ed") - _assert_object_layers_read_back(edata, read_h5ed(path)) + with h5py.File(tmp_path / "edata.h5ed", "r") as f: + assert f["X"].dtype.kind == "O" + assert f["obsm"]["_ed_ondisk_kinds_X"][()].tolist() == [[0, 1], [2, 0], [0, 0]] + assert f["obsm"]["_ed_ondisk_layers_numbers"].dtype == np.float64 + assert "_ed_ondisk_kinds_layers_numbers" not in f["obsm"] + for key in ("strings", "codes", "mixed"): + assert f["obsm"][f"_ed_ondisk_kinds_layers_{key}"].dtype == np.uint8 + assert f["obsm"][f"_ed_ondisk_kinds_layers_{key}"].shape == (3, 2, 2) @pytest.mark.parametrize("convert_strings_to_categoricals", [True, False]) @@ -499,19 +503,24 @@ def test_write_read_string_columns_same_from_h5ed_and_zarr(edata_string_columns, ("write", "read", "suffix"), [(write_h5ed, read_h5ed, "h5ed"), (write_zarr, read_zarr, "zarr")] ) def test_write_backed_h5ed(write, read, suffix, tmp_path): - # h5py reads the string layer as `object`; a backed EHRData cannot be copied - edata = EHRData(X=np.zeros((2, 2)), layers={"strings": np.array([["a", None], ["c", "d"]], dtype=object)}) + # a backed EHRData cannot be copied + edata = EHRData(X=np.zeros((2, 2)), layers={"strings": np.array([["a", None], ["007", 1]], dtype=object)}) write_h5ed(edata, tmp_path / "edata.h5ed") - edata_backed = read_h5ed( - tmp_path / "edata.h5ed", backed="r", harmonize_missing_values=False, cast_variables_to_float=False - ) + edata_backed = read_h5ed(tmp_path / "edata.h5ed", backed="r") path = tmp_path / f"edata_rewritten.{suffix}" write(edata_backed, path) - _assert_dtype_object_array_with_missing_values_equal( - read(path).layers["strings"].astype(object), np.array([["a", np.nan], ["c", "d"]], dtype=object) - ) + expected = np.array([["a", np.nan], ["007", 1.0]], dtype=object) + for edata_read in (edata_backed, read(path)): + _assert_dtype_object_array_with_missing_values_equal(edata_read.layers["strings"], expected) + + +def test_read_h5ed_backed_object_X_raises(tmp_path): + write_h5ed(EHRData(X=np.array([["a", None], ["c", "d"]], dtype=object)), tmp_path / "edata.h5ed") + + with pytest.raises(NotImplementedError, match="Backed reading of an X"): + read_h5ed(tmp_path / "edata.h5ed", backed="r") def test_write_h5ed_repeated_column_names(tmp_path): diff --git a/tests/io/test_ondisk.py b/tests/io/test_ondisk.py new file mode 100644 index 00000000..38a92981 --- /dev/null +++ b/tests/io/test_ondisk.py @@ -0,0 +1,182 @@ +import anndata as ad +import h5py +import numpy as np +import pandas as pd +import pytest +import sparse +from tests.conftest import TEST_DATA_PATH, _assert_object_layers_read_back, _edata_with_object_layers, _values + +from ehrdata import EHRData +from ehrdata._logger import logger +from ehrdata.io import read_h5ed, read_zarr, write_h5ed + +TOY_H5ED = TEST_DATA_PATH / "toy_h5ed" +TOY_ZARR = TEST_DATA_PATH / "toy_zarr" + + +@pytest.mark.parametrize("with_X", [True, False]) +def test_write_read_object_layers(write_read, X_any_array_type_322, with_X): + # each layer is encoded on its own, whatever `X` is + edata = _edata_with_object_layers(X_any_array_type_322 if with_X else None) + + _assert_object_layers_read_back(edata, write_read(edata)) + + +@pytest.mark.parametrize("shape", [(3, 2), (3, 2, 2)]) +@pytest.mark.parametrize( + ("values", "expected"), + [ + ( + ["007", "nan", None, 1, 2.5, "V45", 2**53 + 1, None, "a", "", True, "010"], + # integers that float64 does not hold exactly stay text; bools become floats + ["007", "nan", None, 1.0, 2.5, "V45", str(2**53 + 1), None, "a", "", 1.0, "010"], + ), + ([1, 2**53 + 1, None, *range(9)], [1.0, str(2**53 + 1), None, *map(float, range(9))]), + ], + ids=["text", "integers"], +) +def test_write_read_object_X(write_read, shape, values, expected): + X = np.array(values, dtype=object)[: np.prod(shape)].reshape(shape) + edata = EHRData(X=X) + + X_read = write_read(edata).X + + assert X_read.dtype == object + assert _values(pd.Series(X_read.ravel())) == expected[: X.size] + assert [type(value) for value in X_read.ravel() if value is not None and not pd.isna(value)] == [ + type(value) for value in expected[: X.size] if value is not None + ] + # the input is left unchanged + assert list(edata.X.ravel()) == values[: X.size] + + +@pytest.mark.parametrize( + ("read_kwargs", "zeros_read_as"), + [({}, 0.0), ({"harmonize_missing_values": True}, np.nan)], + ids=["default", "harmonize"], +) +def test_write_read_zero_filled_coo(write_read, read_kwargs, zeros_read_as): + # e.g. counts of OMOP's setup_variables(..., sparse=True) + dense = np.array([[[0, 1], [2, 0]], [[0, 0], [3, 0]], [[4, 0], [0, 5]]], dtype=np.float64) + edata = EHRData(X=sparse.COO.from_numpy(dense)) + + X_read = write_read(edata, **read_kwargs).X + + assert isinstance(X_read, sparse.COO) + assert np.array_equal(X_read.todense(), np.where(dense == 0, zeros_read_as, dense), equal_nan=True) + + +@pytest.mark.parametrize( + ("read_kwargs", "codes_read_as", "zeros_read_as"), + [ + ({}, [["007", "nan"], ["250", "V45"]], 0.0), + ( + {"harmonize_missing_values": True, "cast_variables_to_float": True}, + [[7.0, None], [250.0, "V45"]], + np.nan, + ), + ], + ids=["default", "heuristics"], +) +def test_write_read_str_layer(write_read, read_kwargs, codes_read_as, zeros_read_as): + codes = np.array([["007", "nan"], ["250", "V45"]]) + counts = np.array([[[0, 1], [2, 0]], [[0, 0], [3, 0]]], dtype=np.float64) + edata = EHRData(layers={"codes": codes, "counts": sparse.COO.from_numpy(counts)}, shape=(2, 2)) + + edata_read = write_read(edata, **read_kwargs) + + # object from every format + assert edata_read.layers["codes"].dtype == object + assert [_values(pd.Series(row)) for row in edata_read.layers["codes"]] == codes_read_as + assert np.array_equal( + edata_read.layers["counts"].todense(), np.where(counts == 0, zeros_read_as, counts), equal_nan=True + ) + + +@pytest.mark.parametrize( + "path", [TOY_H5ED / "edata_object_layers_v0_2_0.h5ed", TOY_ZARR / "edata_object_layers_v0_2_0.ehrdata.zarr"] +) +@pytest.mark.parametrize( + ("read_kwargs", "codes_read_as", "zeros_read_as"), + [ + # strings are harmonized and cast, zeros stay + ({}, [[7.0, None], [250.0, "V45"], [42.0, None]], 0.0), + ({"harmonize_missing_values": True}, [[7.0, None], [250.0, "V45"], [42.0, None]], np.nan), + ( + {"harmonize_missing_values": False, "cast_variables_to_float": False}, + [["007", "nan"], ["250", "V45"], ["042", "nan"]], + 0.0, + ), + ], + ids=["default", "harmonize", "exact"], +) +def test_read_v0_2_0_object_layers(path, read_kwargs, codes_read_as, zeros_read_as): + # written by ehrdata before on-disk version 0.3.0, see the creation reference script + edata = (read_h5ed if path.suffix == ".h5ed" else read_zarr)(path, **read_kwargs) + + assert [_values(pd.Series(row)) for row in edata.layers["codes"]] == codes_read_as + counts = np.zeros((3, 2, 2)) + counts[0, 0, 1], counts[2, 1, 0] = 2, 1 + assert np.array_equal( + edata.layers["counts"].todense(), np.where(counts == 0, zeros_read_as, counts), equal_nan=True + ) + + +def test_read_stale_kinds_ignored(write_read, monkeypatch): + # e.g. a 0.3.0 file read and written again by ehrdata before 0.3.0 + edata = EHRData(X=np.array([["007", None], ["250", 1]], dtype=object)) + messages = [] + monkeypatch.setattr(logger, "warning", lambda msg, **kwargs: messages.append(msg)) + + X_read = write_read(edata, version="0.2.0").X + + assert any("Ignoring ['_ed_ondisk_kinds_X']" in message for message in messages) + assert [_values(pd.Series(row)) for row in X_read] == [[7.0, None], [250.0, 1.0]] + + +@pytest.mark.parametrize("change", ["layer_missing", "other_shape"]) +def test_read_h5ed_mismatched_kinds_ignored(change, tmp_path, monkeypatch): + edata = EHRData(X=np.zeros((2, 2)), layers={"codes": np.array([["007", None], ["250", 1]], dtype=object)}) + write_h5ed(edata, tmp_path / "edata.h5ed") + with h5py.File(tmp_path / "edata.h5ed", "a") as f: + if change == "layer_missing": + f.move("layers/codes", "layers/other") + else: + del f["obsm/_ed_ondisk_kinds_layers_codes"] + ad.io.write_elem(f["obsm"], "_ed_ondisk_kinds_layers_codes", np.zeros((2, 1), dtype=np.uint8)) + + messages = [] + monkeypatch.setattr(logger, "warning", lambda msg, **kwargs: messages.append(msg)) + + edata_read = read_h5ed(tmp_path / "edata.h5ed") + + assert messages == [ + "Ignoring '_ed_ondisk_kinds_layers_codes' in .obsm, which does not match the array it belongs to." + ] + assert not any(key.startswith("_ed_ondisk_") for key in edata_read.obsm) + # the stored strings + layer = edata_read.layers["other" if change == "layer_missing" else "codes"] + assert [_values(pd.Series(row)) for row in layer] == [["007", "nan"], ["250", "1"]] + + +def test_write_reserved_obsm_key_raises(write_read): + edata = EHRData(X=np.zeros((2, 2)), obsm={"_ed_ondisk_kinds_layers_zzz": np.zeros((2, 2), dtype=np.uint8)}) + + with pytest.raises(ValueError, match="reserved for ehrdata's on-disk layout"): + write_read(edata) + + +def test_read_newer_version_raises(write_read): + with pytest.raises(ValueError, match="upgrade ehrdata"): + write_read(EHRData(X=np.zeros((2, 2))), version="99.0.0") + + +@pytest.mark.parametrize(("write", "read"), [("write_h5ad", read_h5ed), ("write_zarr", read_zarr)]) +def test_read_anndata_heuristics(write, read, tmp_path): + # anndata files carry no on-disk version of ehrdata + path = tmp_path / "adata" + getattr(ad.AnnData(X=np.array([["007", "nan"], ["250", "V45"]])), write)(path) + + X_read = read(path).X + + assert [_values(pd.Series(row)) for row in X_read] == [[7.0, None], [250.0, "V45"]] diff --git a/tests/io/test_zarr.py b/tests/io/test_zarr.py index 5cd1b65f..6b6a93cb 100644 --- a/tests/io/test_zarr.py +++ b/tests/io/test_zarr.py @@ -12,11 +12,9 @@ _assert_columns_anndata_cannot_write_read_back, _assert_dtype_object_array_with_missing_values_equal, _assert_io_read, - _assert_object_layers_read_back, _assert_shape_matches, _check_aligned_anndata_parts_equal, _edata_with_columns_anndata_cannot_write, - _edata_with_object_layers, ) from ehrdata.core.constants import EHRDATA_ONDISK_VERSION @@ -179,7 +177,7 @@ def test_write_zarr_v2_relocates_3d_arrays_to_obsm(edata_333, tmp_path): def test_write_read_zarr_X_none_with_3d_layer(edata_330, tmp_path): - # X=None is a first-class state (encode_for_disk drops a None X); reading it with harmonization on (the default) must not crash on the None X. + # X=None is a first-class state (encode_for_disk drops a None X); reading it with harmonization on must not crash on the None X. edata = edata_330.copy() edata.layers["tem_data"] = np.arange(3 * 3 * 3).reshape(3, 3, 3).astype(float) edata.X = None @@ -187,7 +185,7 @@ def test_write_read_zarr_X_none_with_3d_layer(edata_330, tmp_path): path = tmp_path / "edata_X_none.ehrdata.zarr" write_zarr(edata.copy(), path) - edata_read = read_zarr(path) # harmonize_missing_values=True by default + edata_read = read_zarr(path, harmonize_missing_values=True) _assert_shape_matches(edata_read, (3, 3, 3), check_X_None=True) assert edata_read.X is None assert np.array_equal(edata.layers["tem_data"], edata_read.layers["tem_data"]) @@ -313,7 +311,7 @@ def test_read_minimal_corpus_zarr(): store_020 = zarr.open(TEST_PATH_ZARR / "edata_minimal_v0_2_0.ehrdata.zarr") assert store_020.attrs["ehrdata-encoding-type"] == "ehrdata" - assert store_020.attrs["ehrdata-encoding-version"] == str(EHRDATA_ONDISK_VERSION) + assert store_020.attrs["ehrdata-encoding-version"] == "0.2.0" edata_020 = read_zarr(TEST_PATH_ZARR / "edata_minimal_v0_2_0.ehrdata.zarr") _assert_shape_matches(edata_020, (3, 2, 2)) assert np.array_equal(np.asarray(edata_020.layers["tem_data"]), np.arange(3 * 2 * 2, dtype=float).reshape(3, 2, 2)) @@ -327,14 +325,3 @@ def test_write_read_zarr_casts_columns_anndata_cannot_write(X_any_array_type_322 write_zarr(edata, path) _assert_columns_anndata_cannot_write_read_back(edata, read_zarr(path, harmonize_missing_values=False)) - - -@pytest.mark.parametrize("with_X", [True, False]) -def test_write_read_zarr_casts_object_layers(X_any_array_type_322, with_X, tmp_path): - # each layer is cast on its own, whatever `X` is - edata = _edata_with_object_layers(X_any_array_type_322 if with_X else None) - path = tmp_path / "object_layers.zarr" - - write_zarr(edata, path) - - _assert_object_layers_read_back(edata, read_zarr(path))