Repository navigation
indexing by a list of slices #10479
Description
Is your feature request related to a problem?
In some situations, it is important to be able to select multiple (disconnected) contiguous regions along a dimension, even if the dimension itself is very large.
If there's enough client memory, it is possible to emulate this by materializing the slices into an array of integers, but this becomes infeasible if that array is too large (and while supposedly we should be able to index by a dask integer array, I'm not sure how efficient that would be).
Examples of where this would be useful include:
- The healpix MOC index at healpix moc index xarray-contrib/xdggs#151, where cell ids are represented as a set of disconnected ranges at the smallest possible refinement level. To be able to support
Index.sel, I'd need to return aIndexSelResultwith either a list of slices, or materialize these into an integer array and error out if that wouldn't fit into memory (or try to usedaskas an indexer). - @tomwhite's use-case of selecting disconnected regions in a genome (see Add
bcftools-style filtering sgkit-dev/sgkit#1330 (comment)). I'll let him provide further details.
cc @benbovy, @shoyer, @TomNicholas, @dcherian
Describe the solution you'd like
I'd love to be able to specify this as another kind of indexer:
indexer = SliceSet([slice(20, 5000), slice(12078432, 1850372894)])
ds.isel(cells=indexer)but that will obviously further increase the complexity of the indexing machinery
Describe alternatives you've considered
Manually iterating of the slices, then concatenating the result is possible, but will have an additional overhead if done using the xarray API. However, I don't see a way that can work as part of IndexSelResult.
Activity
benbovy commentedon Jul 1, 2025
Easiest I think would be to do that at the Variable level, i.e., implement a special case in
Variable.isel()that will iterate over each slice and callVariable.concat()as needed. There may be some overhead but I think it would be limited at that level. If necessary it's still possible to introduce some internal fastpath inVariable.concat()or maintain an optimized internal version.For coordinate variables associated with a custom index, there's the possibility of implementing custom logic in
Index.isel()(returning a new index object) andIndex.create_variables()(create variables from the new index object) overridden methods. This will bypassVariable.isel()for those variables.shoyer commentedon Jul 1, 2025
keewis commentedon Jul 1, 2025
Personally, my main motivation for this would be the return value of
Index.sel, which we can't work around in user code.As for the lazy concatenation, I believe there's #10402 as an experiment on how this could be implemented.
TomNicholas commentedon Jul 1, 2025
I am curious if this has any crossover with what flox does to create groups
ilan-gold commentedon Jul 16, 2025
Maybe relevant: zarr-developers/zarr-python#3175
dcherian commentedon Jul 16, 2025
technically we don't need lazy concat here. We can hack it using the fact that
LazilyIndexedArrayacts as a blocker on indexing; i.e. it only passes through indexers at compute time.So you could work with
LazilyIndexedArray(MultiplySlicedArray(large_lazy_array)). This would only need concat at runtime.keewis commentedon Jul 25, 2025
that's a good point! This does work, but will need some thought on how to implement
_updated_key(and_index_indexer_1d)dcherian commentedon Jul 25, 2025
But doesn't that just condense all indexers down to a unified indexer that is then passed to
LazilyIndexedArray.array?keewis commentedon Jul 25, 2025
yes, exactly. I'll have to think a bit more about whether implementing that is actually necessary, but it doesn't currently support lists of slices.
dcherian commentedon Jul 25, 2025
Yes I'm suggesting that you handle the list of slices elsewhere, wrap with LazilyIndexedArray and then assign to Variable
keewis commentedon Jul 25, 2025
where would you create that class? In
LazilyIndexedArray._oindex_get? That may result in something likeLazilyIndexedArray(MultiplySlicedArray(LazilyIndexedArray(data)))if we're not careful, which would reduce the usefulness of_updated_keya bit.