SmallZarrGroups (WIP)

In memory hierarchy of arrays and attributes loaded from disk or to be saved to disk with Zarr.

Warning: SmallZarrGroups is currently a WIP and its API may drastically change at any time.

Limitations

1. If you need to store huge datasets that cannot fit uncompressed in memory consider using https://github.com/JuliaIO/HDF5.jl or https://github.com/JuliaIO/Zarr.jl

2. If you just want to serialize arbitrary Julia data consider using https://github.com/JuliaIO/JLD2.jl or https://github.com/invenia/JLSO.jl

3. Numpy datetime64 (“M”) and timedelta64 (“m”) data types are read as Int64.

4. Zarr filters are not supported.

Overview

1. ZGroup represents a tree with arrays as leaves.
2. ZGroup leaf arrays are uncompressed but store metadata about how they should be compressed when saved to disk.
3. ZGroup can have JSON3 serializable attributes attached to any node or leaf.
4. Data can be quickly accessed and modified in ZGroup.
5. No file open close semantics. Use the Julia garbage collector to clean memory up.
6. Save and load ZGroup in a directory or zip file in Zarr v2 format.
7. ZGroup saved to a directory or zip file can be read in other languages.

Examples

See test/test_simple-usage.jl for examples and documentation.