abstract type AbstractArrayCodec

Abstract type for array codecs.

Most codecs can use EncodedArray as their encoded-array type and only need to implement EncodedArrays.encode_data! and EncodedArrays.decode_data!. Codecs that use a custom subtype of AbstractEncodedArray must implement its full API.

AbstractEncodedArray{T,N} <: AbstractArray{T,N}

Abstract type for arrays that store their elements in encoded/compressed form.

In addition to the standard AbstractArray API, an AbstractEncodedArray must support the functions

• EncodedArrays.getcodec(A::AbstractEncodedArray): Returns the codec.
• Base.codeunits(A::AbstractEncodedArray): Returns the internal encoded data representation.

Encoded arrays will typically be created via

A_enc = A |> codec

Decoding happens via standard array conversion or assignment:

A_dec = Array(A)
A_dec = convert(Array,A)
A_dec = A[:]

A_dec = Array{T,N}(undef, size(A_enc)...)
A_dec[:] = A_enc

EncodedArray{T,N,C,DV} <: AbstractEncodedArray{T,N}

Concrete type for AbstractEncodedArrays.

Constructor:

EncodedArray{T}(
    codec::AbstractArrayCodec,
    size::NTuple{N,Integer},
    encoded::AbstractVector{UInt8}
)

Codecs using EncodedArray only need to implement EncodedArrays.encode_data! and EncodedArrays.decode_data!.

If length of the decoded data can be inferred from the encoded data, a constructor

EncodedArray{T,N}(codec::MyCodec,encoded::AbstractVector{UInt8})

should also be defined. By default, two EncodedArrays that have the same codec and size are assumed to be equal if and only if their code units are equal.

Generic methods for the rest of the AbstractEncodedArray API are already provided for EncodedArray.

VarlenDiffArrayCodec <: AbstractArrayCodec

VectorOfEncodedArrays{T,N,...}

A vector of encoded arrays.

The code units of all entries are stored in contiguous fashion using an ArraysOfArrays.PartsView. All element arrays are encoded using the same codec.

Constructors:

VectorOfEncodedArrays{T,N}(codec::AbstractArrayCodec)
VectorOfEncodedArrays{T}(codec::AbstractArrayCodec, innersizes::AbstractVector{<:Dims{N}}, encoded::PartsView{UInt8})

A .|> codec encodes a vector of arrays A, collect.(A_enc) decodes it into a VectorOfArrays again. push!, append! and vcat encode and add further arrays.

ArraysOfArrays.fused decodes all element arrays into a flat vector and ArraysOfArrays.getsplitmode describes their layout, so splitup(fused(A_enc), getsplitmode(A_enc)) is the decoded vector of arrays. Operations based on these, like flatview, innersum or mapat, decode all elements and so allocate.

VectorOfEncodedSimilarArrays{T,M,C,...}

A vector of encoded arrays that have the same original size.

The code units of all entries are stored in contiguous fashion using an ArraysOfArrays.PartsView. All element arrays are encoded using the same codec.

Constructors:

VectorOfEncodedSimilarArrays{T}(codec::AbstractArrayCodec, innersize::Dims{M})
VectorOfEncodedSimilarArrays{T}(codec::AbstractArrayCodec, innersize::Dims{M}, encoded::PartsView{UInt8})

A .|> codec encodes a vector of similar arrays A, collect.(A_enc) decodes it into a VectorOfSimilarArrays again. push!, append! and vcat encode and add further arrays of the same size.

ArraysOfArrays.fused decodes all element arrays, so operations based on it, like flatview, stack, parent, comparisons, innersum or mapat, allocate.

|>(A::AbstractArray{T}, codec::AbstractArrayCodec)::AbstractEncodedArray

Encode A using codec and return an AbstractEncodedArray. The default implementation returns an EncodedArray.

decode_data!(data::AbstractArray, codec::AbstractArrayCodec, encoded::AbstractVector{UInt8})

Depending on codec, may or may not resize data to fit the size of the decoded data. Codecs may require data to be of correct size (e.g. to improve performance or when the size/shape of the decoded data cannot be easily inferred from the encoded data).

Returns data.

encode_data!(encoded::AbstractVector{UInt8}, codec::AbstractArrayCodec, data::AbstractArray)

Will resize encoded as necessary to fit the encoded data.

Returns encoded.

EncodedArrays.getcodec(A::AbstractEncodedArray)::AbstractArrayCodec

Returns the codec used to encode/compress A.

read_autozz_varlen(io::IO, ::Type{<:Integer})

Read an integer of type T from io, using zig-zag decoding depending on whether T is signed or unsigned.

read_varlen(io::IO, T::Type{<:Unsigned})

Read an unsigned variable-length integer value of type T from io. If the next value encoded in x is too large to be represented by T, an exception is thrown.

See EncodedArrays.write_varlen.

write_varlen(io::IO, x::Unsigned)

Write unsigned integer value x to IO using variable-length coding. Data is written in LSB fashion in units of one byte. The highest bit of each byte indicates if more bytes will need to be read, the 7 lower bits contain the next 7 bits of x.