Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ and this project adheres to [Semantic Versioning][].
- {func}`~ehrdata.io.omop.setup_interval_variables` aggregates only the intervals that hold data points, as {func}`~ehrdata.io.omop.setup_variables` does, so memory and time follow the number of data points. ([#349](https://github.com/theislab/ehrdata/pull/349)) @eroell
- {func}`~ehrdata.io.omop.setup_variables` and {func}`~ehrdata.io.omop.setup_interval_variables` with `aggregation_strategy="mode"` keep the smallest of several equally frequent values, instead of an arbitrary one. ([#341](https://github.com/theislab/ehrdata/pull/341)) @eroell
- {func}`~ehrdata.io.omop.setup_variables` and {func}`~ehrdata.io.omop.setup_interval_variables` with `observation_table="person_visit_occurrence"` attribute a data point whose `visit_occurrence_id` names a visit of its person to that visit only. Before, a data point belonged to every visit of its person whose window held it, so overlapping visits of a person shared their data points, and with `clip_to_end_date=False` a visit's intervals after its end took the data points of the person's next visit. Data points without a `visit_occurrence_id`, or with one that names no visit of their person, still belong to every visit whose window holds them. ([#344](https://github.com/theislab/ehrdata/pull/344)) @eroell
- {func}`~ehrdata.io.write_h5ed`, {func}`~ehrdata.io.write_zarr` and {func}`~ehrdata.io.write_elem` write an `object` `X` or layer whose values are not all numbers or missing as strings, with the kind of each value, text, number or missing, in a reserved `.obsm` key next to it, and no longer parse strings into numbers. An `object` array of only numbers, including booleans and `Decimal`, and missing values is written as float, as before. Integers that float64 does not hold exactly are written as text, with a warning. `.obsm` keys starting with `_ed_ondisk_` are reserved and raise. This changes ehrdata's on-disk version to 0.3.0. ehrdata reads the versions 0.2.0 and 0.3.0, and raises for newer ones. Earlier releases of ehrdata read a file of version 0.3.0 wrongly: they leave its 3D `X` and layers and the kinds in `.obsm` under the reserved `_ed_ondisk_*` keys, fail in backed mode on 3D arrays, and apply their heuristics to the arrays written as strings, so that `"007"` becomes 7.0 and `"nan"` becomes `np.nan`. Writing such data again with an earlier release loses the exact encoding; upgrade ehrdata to read files of version 0.3.0. ([#362](https://github.com/theislab/ehrdata/pull/362)) @eroell
- {func}`~ehrdata.io.read_h5ed`, {func}`~ehrdata.io.read_zarr` and {func}`~ehrdata.io.read_elem` apply `harmonize_missing_values` and `cast_variables_to_float` by default (`None`) only to files written before on-disk version 0.3.0 and to plain anndata files, which therefore read as before; `True` applies them to every file, `False` to none. Backed reading with {func}`~ehrdata.io.read_h5ed` no longer needs `harmonize_missing_values=False` and `cast_variables_to_float=False`, and decodes the layers written as strings with kinds; an `X` written so raises. ([#362](https://github.com/theislab/ehrdata/pull/362)) @eroell
- {func}`~ehrdata.io.write_h5ed` and {func}`~ehrdata.io.write_zarr` keep string columns as strings by default (`convert_strings_to_categoricals=False`), so an {class}`~ehrdata.EHRData` reads back with the dtypes it was written with, from `.h5ed` and `.zarr` alike. Before, {func}`~ehrdata.io.write_h5ed` converted the string columns of `.obs` and `.var` in which a value repeats or is missing to categoricals, and {func}`~ehrdata.io.write_zarr` also those of `.tem`. Files written before keep their categorical columns; pass `convert_strings_to_categoricals=True` for the previous conversion. ([#358](https://github.com/theislab/ehrdata/issues/358)) @eroell

### Fixed
Expand All @@ -41,6 +43,8 @@ and this project adheres to [Semantic Versioning][].
- {func}`~ehrdata.io.omop.setup_variables` and {func}`~ehrdata.io.omop.setup_interval_variables` write `.uns['omop_io_variable_setup']`, as their docstring promised. ([#353](https://github.com/theislab/ehrdata/pull/353)) @eroell
- {class}`~ehrdata.integrations.torch.OMOPEHRDataset` reads the setup recorded in `.uns['omop_io_variable_setup']`, and raises if a later call has replaced its values. ([#353](https://github.com/theislab/ehrdata/pull/353)) @eroell
- {func}`~ehrdata.io.omop.setup_variables` and {func}`~ehrdata.io.omop.setup_interval_variables` no longer change the {class}`~ehrdata.EHRData` passed to them, and keep its earlier layers when they set up another `layer` of the same variables and intervals. ([#354](https://github.com/theislab/ehrdata/pull/354)) @eroell
- {func}`~ehrdata.io.read_h5ed`, {func}`~ehrdata.io.read_zarr` and {func}`~ehrdata.io.read_elem` return `X` and layers exactly as {func}`~ehrdata.io.write_h5ed`, {func}`~ehrdata.io.write_zarr` and {func}`~ehrdata.io.write_elem` wrote them. ([#362](https://github.com/theislab/ehrdata/pull/362)) @eroell
- {func}`~ehrdata.io.read_h5ed`, {func}`~ehrdata.io.read_zarr` and {func}`~ehrdata.io.read_elem` keep the zeros of a numeric `sparse.COO` `X` or layer, such as the counts of {func}`~ehrdata.io.omop.setup_variables` with `aggregation_strategy="count"` and `sparse=True`, also from files written before. Before, they read them as missing values; this rule of {func}`~ehrdata.harmonize_missing_values` now applies on reading only with `harmonize_missing_values=True`. ([#362](https://github.com/theislab/ehrdata/pull/362)) @eroell

## [0.5.0]

Expand Down
4 changes: 3 additions & 1 deletion src/ehrdata/_feature_types.py
Original file line number Diff line number Diff line change
Expand Up @@ -582,13 +582,15 @@ def harmonize_missing_values(
return edata if copy else None


def _harmonize_on_read(edata: EHRData) -> None:
def _harmonize_on_read(edata: EHRData, *, skip_coo: bool) -> None:
layers = [None] if edata.X is not None else []
# anndata 0.13's unified `.X` slot shows up as key None in edata.layers too; skip the duplicate
layers += [key for key in edata.layers if key is not None]

for layer in layers:
label = "X" if layer is None else f"layer {layer}"
if skip_coo and isinstance(edata.X if layer is None else edata.layers[layer], sparse.COO):
continue
try:
harmonize_missing_values(edata, layer=layer)
logger.info(f"Harmonizing missing values of {label}")
Expand Down
15 changes: 14 additions & 1 deletion src/ehrdata/core/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
# On-disk format (see :mod:`ehrdata.io._ondisk` for the layout).
# EHRData stores time-series as 3D arrays in `X`/`layers`, but anndata only guarantees 2D arrays there (https://github.com/scverse/anndata/issues/2430) and enforces this since release 0.13.0
# Since storage version 0.2.0 (ehrdata release 0.3.0), 3D arrays are relocated into the reserved `.obsm` keys below on write and restored on read.
# Since storage version 0.3.0, an `object` X/layer of text next to numbers or missing values is written as strings, with the kind of each value in a reserved `.obsm` key, and read back exactly.
# Bump only when the on-disk layout changes.
#
# The attribute key that stamps a store as ehrdata differs by format, on purpose:
Expand All @@ -23,12 +24,24 @@
EHRDATA_ENCODING_TYPE_KEY_ZARR = "encoding-type"
EHRDATA_ENCODING_TYPE = "ehrdata"
EHRDATA_ONDISK_VERSION_KEY = "ehrdata-encoding-version"
EHRDATA_ONDISK_VERSION = "0.2.0"
EHRDATA_ONDISK_VERSION = "0.3.0"
# the versions this ehrdata reads; files of other versions written by ehrdata raise
EHRDATA_ONDISK_VERSIONS_READ = ("0.2.0", "0.3.0")

# Prefix of the `.obsm` keys reserved for the on-disk layout.
EHRDATA_OBSM_RESERVED_PREFIX = "_ed_ondisk_"

# Reserved `.obsm` keys holding a 3D `X` / a 3D layer `<name>` on disk.
EHRDATA_OBSM_3D_X_KEY = "_ed_ondisk_X"
EHRDATA_OBSM_3D_LAYER_PREFIX = "_ed_ondisk_layers_"

# Reserved `.obsm` keys holding the kind of each value of an `X` / a layer `<name>` written as strings, as uint8.
EHRDATA_OBSM_KINDS_X_KEY = "_ed_ondisk_kinds_X"
EHRDATA_OBSM_KINDS_LAYER_PREFIX = "_ed_ondisk_kinds_layers_"
VALUE_KIND_TEXT = 0
VALUE_KIND_NUMBER = 1 # read back as float64
VALUE_KIND_MISSING = 2 # read back as np.nan

# Missing values
# --------------
# These values if encountered as strings are considered to represent missing values in the data
Expand Down
103 changes: 92 additions & 11 deletions src/ehrdata/io/_array_casting.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

import contextlib
import datetime as dt
import decimal
import numbers
from typing import TYPE_CHECKING, Any

import numpy as np
Expand All @@ -11,6 +13,7 @@
if TYPE_CHECKING:
from ehrdata import EHRData
from ehrdata._logger import logger
from ehrdata.core.constants import VALUE_KIND_MISSING, VALUE_KIND_NUMBER, VALUE_KIND_TEXT


def _is_numeric_or_bool(array: Any) -> bool:
Expand Down Expand Up @@ -52,19 +55,97 @@ def _cast_variables_to_float(edata: EHRData) -> None:
edata.layers[key] = _cast_columns_to_float(array)


def _cast_object_array_to_float_or_str(array: Any, name: str) -> Any:
"""Cast an in-memory `object` array to float, else to str; return other arrays unchanged."""
# `infer_dtype` of object arrays that are cast to float as a whole
_FLOAT_CASTABLE = {"empty", "floating", "integer", "mixed-integer-float", "boolean", "decimal"}


def _is_exact_integer(value: Any) -> bool:
try:
return float(int(value)) == int(value)
except OverflowError:
return False


def _number_to_string(value: Any) -> str:
# as written if that reads back as the same float, e.g. "3" for 3
text = str(value)
with contextlib.suppress(ValueError):
if float(text) == float(value):
return text
return repr(float(value))


def _encode_object_array(array: Any, name: str) -> tuple[Any, np.ndarray | None]:
"""Encode an in-memory `object` array for writing; return other arrays unchanged.

If every value is a real number (also a bool or `Decimal`), and an integer only if float64 holds it exactly, or missing,
the values are returned as float, with `None`.
Otherwise they are returned as strings, with the kind of each value: text, number or missing.
Strings are never parsed into numbers.
"""
if not isinstance(array, np.ndarray | sparse.COO) or array.dtype != np.object_:
# backed and dask arrays are written as they are
return array
if isinstance(array, np.ndarray):
# missing values become "nan" in a str array, which harmonize_missing_values reads back as np.nan
array = np.where(pd.isna(array), np.nan, array)
try:
return array.astype(np.float64)
except (TypeError, ValueError):
logger.warning(f"edata.{name} is of dtype object and not numeric: it is cast to dtype 'str' for writing.")
return array.astype(str)
return array, None
if isinstance(array, sparse.COO):
try:
return array.astype(np.float64), None
except (TypeError, ValueError):
logger.warning(f"edata.{name} is of dtype object and not numeric: it is cast to dtype 'str' for writing.")
return array.astype(str), None

is_missing = np.asarray(
pd.isna(array), dtype=bool
) # TODO: find out if scipy sparse can ever enter, as this would be a densification.
if pd.api.types.infer_dtype(array.ravel(), skipna=True) in _FLOAT_CASTABLE:
with contextlib.suppress(OverflowError):
values = np.where(is_missing, np.nan, array).astype(np.float64)
# integers from 2**53 on may not be exact
if not (np.abs(values[~is_missing]) >= 2**53).any():
return values, None

flat, flat_missing = array.ravel(), is_missing.ravel()
kinds = np.full(flat.shape, VALUE_KIND_MISSING, dtype=np.uint8)
strings = np.full(flat.shape, "nan", dtype=object) # an object array of str, not fixed width
type_ids = np.fromiter(map(id, map(type, flat)), dtype=np.uintp, count=flat.size)
inexact_integers = False
# by type, to vectorize
for value_type in set(map(type, flat[~flat_missing])):
mask = (type_ids == id(value_type)) & ~flat_missing
values = flat[mask]
if issubclass(value_type, str):
kinds[mask], strings[mask] = VALUE_KIND_TEXT, values
elif issubclass(value_type, int | np.integer) and not issubclass(value_type, bool):
try:
exact = np.abs(values.astype(np.float64)) < 2**53
except OverflowError:
exact = np.zeros(values.shape, dtype=bool)
exact[~exact] = np.frompyfunc(_is_exact_integer, 1, 1)(values[~exact]).astype(bool)
inexact_integers |= not exact.all()
kinds[mask] = np.where(exact, VALUE_KIND_NUMBER, VALUE_KIND_TEXT)
strings[mask] = np.fromiter(map(str, values), dtype=object, count=values.size)
elif issubclass(value_type, numbers.Real | decimal.Decimal | np.bool_):
to_string = str if value_type in {float, np.float64} else _number_to_string
kinds[mask] = VALUE_KIND_NUMBER
strings[mask] = np.fromiter(map(to_string, values), dtype=object, count=values.size)
else:
kinds[mask] = VALUE_KIND_TEXT
strings[mask] = np.fromiter(map(_to_string, values), dtype=object, count=values.size)

if not (kinds == VALUE_KIND_TEXT).any():
return np.where(is_missing, np.nan, array).astype(np.float64), None
if inexact_integers:
logger.warning(f"edata.{name} holds integers that float64 does not hold exactly: they are written as text.")
return strings.reshape(array.shape), kinds.reshape(array.shape)


def _decode_object_array(strings: Any, kinds: np.ndarray) -> np.ndarray:
"""Restore the values of an array encoded by :func:`_encode_object_array`."""
strings = np.asarray(strings)
values = strings.astype(object)
is_number = kinds == VALUE_KIND_NUMBER
values[is_number] = strings[is_number].astype(np.float64)
values[kinds == VALUE_KIND_MISSING] = np.nan
return values


def _to_string(value: Any) -> str:
Expand Down
Loading
Loading