EncodedArrays.jl

EncodedArrays provides an API for arrays that store their elements in encoded/compressed form. This package is meant to be lightweight and only implements a simple codec VarlenDiffArrayCodec. As codec implementations are often complex and have various dependencies, more advanced codecs should be implemented in separate packages.

Random access on an encoded array will typically be very inefficient, but linear access may be efficient (depending on the codec). Accessing the whole array contents at once, e.g. via collect(A), A[:], or copying/appending/conversion to a regular array, must be efficient.

This package defines two central abstract types, AbstractEncodedArray and AbstractArrayCodec. It also defines a concrete type EncodedArray that implements most of the API and only leaves EncodedArrays.encode_data! and EncodedArrays.decode_data! for a new codec to implement.

Vectors of arrays that are all encoded with the same codec can be stored efficiently using VectorOfEncodedArrays and VectorOfEncodedSimilarArrays; broadcasting a codec over a vector of arrays (A .|> codec) and broadcasting collect over the result construct these directly.

Both types integrate with the split-mode API of ArraysOfArrays: ArraysOfArrays.fused decodes all element arrays into flat data and ArraysOfArrays.getsplitmode describes their layout, so functions like flatview, innersum or mapat work directly on vectors of encoded arrays (decoding all elements in the process). push!, append! and vcat encode and add further arrays.