PropertyFunctions

Provides functionality for easy access to and efficient broadcasting over properties.

PPath{path}

Represents a path of nested property names.

Constructors:

PPath(:a, :b)
PPath{(:a, :b)}()

PPath objects are callable, PPath(:a, :b)(x) returns x.a.b.

The property paths that a PropertyFunction may access are part of its type signature, PropertyFunction{Tuple{PPath{(:a, :b)}, ...}}.

const PPaths = Tuple{Vararg{PPath}}

The supertype of the property-path Tuple types used in the first type parameter of PropertyFunction.

PropSelFunction{Paths<:Tuple,trg_names} <: PropertyFunction

A special kind of PropertyFunction that selects (and possibly renames) properties, but does no other computations.

A PropSelFunction can be constructed via the @pf macro

propsel = @pf (;$c, d = $a)

or directly via

propsel = PropSelFunction(:c, :a => :d)
propsel = PropSelFunction(:c, PPath(:a, :b) => :d)

or

propsel = PropSelFunction{Tuple{PPath{(:c,)}, PPath{(:a,)}}, (:c, :d)}()

or just

PropSelFunction{Tuple{PPath{(:c,)}, PPath{(:a,)}}}()

if no property name mapping is required.

The selected property paths of a PropSelFunction are always pairwise disjoint: @pf generates plain PropertyFunctions for selections with duplicated or overlapping sources, while the PropSelFunction constructors reject such selections.

See also @pf.

struct PropertyFunction{Paths<:Tuple, F} <: Function

Create property functions with the @pf macro, or at runtime via the PropertyFunction(expr, env) constructor. Use the type itself only for dispatch in special cases - user code should not construct instances from a Paths type and a function directly.

The Paths type parameter is a Tuple type of PPath types (Paths <: PPaths), e.g. Tuple{PPath{(:a,)}, PPath{(:b, :c)}}. It is the minimal cover of the property paths the function may access (a path subsumes all paths below it) and is part of the public API, so that specialized implementations (e.g. for tables with expensive column access) can rely on it. The paths in Paths must be pairwise disjoint (enforced during construction), and their order, while part of the type identity, carries no semantic meaning. The type parameter F is internal and subject to change.

PropertyFunction(expr::Union{Expr,Symbol,Number}, env = (;))

Compile the expression expr, which references input properties via $property like @pf, to a PropertyFunction at runtime.

PropertyFunction(:($a + $c^2)) behaves like @pf $a + $c^2. Use it where property functions must be constructed from expressions that only become available at runtime, e.g. from configuration files.

Extended help

In contrast to @pf, expressions must not contain macro calls (evaluate macro-generated values beforehand and pass them via env) and input calls f(_) are not supported.

Free names in expr are resolved via env, a NamedTuple or an AbstractDict{Symbol}. Names not found in env resolve in the PropertyFunctions module scope, so Base functions are available without an env entry. env entries that expr does not reference are ignored. Note that a bare Symbol is a free name, not a property reference: properties are always referenced via $property.

Example:

pf = PropertyFunction(:(f($a) + b), (f = sin, b = 42))
pf((a = 0.0,)) == 42.0

The name #pf_arg# is reserved and must not occur in expr or as an env key.

abstract type PropertyFunctions._PropSelector <: Function

Abstract supertype of pure property selectors and extractors.

PropertyFunctions.unfixed

Singleton value that unfixes a fixed input property again when passed as its value in PropertyFunctions.fix_input_properties.

@pf expression

Generates a function that accesses the properties of its argument referenced via $property in expression.

@pf($a + $c^2) is equivalent to x -> x.a + x.c^2.

Nested properties can be referenced via $a.b.c or $(a.b.c), and broadcasting will read only the required nested columns (see PropertyFunctions.subcolumn).

Examples:

xs = StructArrays.StructArray((
    a = [0.9, 0.1, 0.9, 0.2, 0.7, 0.0, 0.7, 0.5, 0.2, 0.6],
    b = [0.1, 0.5, 0.9, 0.9, 0.9, 0.6, 0.1, 0.9, 0.8, 0.2],
    c = [0.4, 0.1, 0.4, 0.1, 0.9, 0.2, 0.4, 0.8, 0.0, 0.1]
))

@pf($a + $c^2)(xs[1])
xs .|> @pf $a + $c^2

Property values can also be passed as keyword arguments instead of an object, @pf($a + $c^2)(a = 0.9, c = 0.4) is equivalent to @pf($a + $c^2)((a = 0.9, c = 0.4)).

Functions generated by @pf come with broadcasting specializations that try to ensure that only the columns referenced via $colname in expr will be read, reducing memory traffic. So data.b will not be accessed in the example above. If the broadcasted function generates structs (including NamedTuples), broadcasting specialization will try to return a StructArrays.StructArray.

Expressions that purely select properties, like

propsel = @pf (;$c, d = $a)

(or equivalently @pf (c = $c, d = $a)), as well as tuple selections @pf ($c, $a) and single-property extractions @pf $a, have special broadcasting optimizations for table-like arguments. This can make broadcasts of such property selections zero-copy O(1) operations:

new_xs = propsel.(xs)
new_xs.c === xs.c
new_xs.d === xs.a

Functions that support PropertyFunctions.input_property_paths, like property functions themselves, can be called on the whole input via input calls f(_):

f = @pf $mu + $sigma
g = @pf $a * f(_)

The properties accessed by f then count as accessed by g as well, so broadcasting g reads only the columns a, mu and sigma. The callee is evaluated once, when the property function is constructed, and so must not depend on names bound inside the expression.

@pf is also very handy in sortby and filterby:

xs |> sortby(@pf $a + $c^2)
xs |> filterby(@pf $a + $c^2 < 0.5)

fg = f ∘ pf::PropertyFunction

Function composition with a property function as the inner function results in a PropertyFunction again, since fg accesses the same properties as pf.

Compositions of pure property selections and extractions fuse into single selections that may access fewer and more specific properties, e.g. @pf($x) ∘ @pf((; x = $a.b, y = $c)) === @pf $a.b.

filterby([getindex|view,] f)

Generates a function that filters a table-like array by f, returning either a copy (default) or a view (ignored if the object does not support views).

The generated function also accepts unmaterialized broadcasts (Base.Broadcast.Broadcasted), which are materialized only once.

Example:

xs = [0.9, 0.1, 0.9, 0.2, 0.7, 0.0, 0.7, 0.5, 0.2, 0.6]
xs |> filterby(x -> x < 0.5)

PropertyFunctions.fix_input_properties(f::PropertyFunction; kwargs...)

Fix input properties of f to the given values.

The fixed properties no longer count as accessed by the resulting property function and are ignored if present in its input. Fixed values that f does not access have no effect, fixing already-fixed properties replaces their values, and setting a property to PropertyFunctions.unfixed unfixes it again.

Use PropertyFunctions.unfix_input_properties to undo completely.

When PartialFunctions.jl is loaded, f $ (; a = 4.2) is equivalent to fix_input_properties(f, a = 4.2).

Example:

using PartialFunctions

f = @pf $a + $b + $c

g = f $ (; b = 2)
g((a = 1, c = 3)) == 6
g(a = 1, c = 3) == 6
g(a = 1, b = 0, c = 3) == 6 # g ignores fixed input b

# Fixing an already-fixed property replaces its fixed value:
h = g $ (; b = 20)
h(a = 1, c = 3) == 24

# Setting a fixed property to `unfixed` unfixes it again. Since no
# fixed properties remain here, this returns the original function:
h $ (; b = PropertyFunctions.unfixed) === f

innermerge(tbl, tbls...)

Merge the columns of table-like objects, with columns of later tables taking precedence (like merge for NamedTuples).

Tries to preserve the table type of tbl, but returns a StructArrays.StructArray where the result would otherwise be a row-oriented table.

Example:

xs = StructArrays.StructArray((a = [1, 2], b = [3, 4]))
ys = innermerge(xs, (c = [5, 6],))
ys.a === xs.a

innermerge may be specialized for table types with expensive column access (e.g. lazy or on-disk tables).

PropertyFunctions.input_property_paths(f)::Tuple{Vararg{PPath}}

Return the property paths that the function f may access on its input.

Defined for PropertyFunction and PPath. Specialize it to make other callable types usable in f(_) calls inside of @pf, such types must be callable with the input object as their argument.

sortby(
    [getindex|view,]
    f;
    rev::Bool = false,
)

Generates a function that sorts an array by f, returning either a copy (default) or a view (ignored if the object does not support views).

The generated function also accepts unmaterialized broadcasts (Base.Broadcast.Broadcasted), which are materialized only once.

Example:

xs = [0.9, 0.1, 0.9, 0.2, 0.7, 0.0, 0.7, 0.5, 0.2, 0.6]
xs |> sortby(x -> (x - 0.5)^2)

PropertyFunctions.subcolumn(col::AbstractArray, name::Symbol)
PropertyFunctions.subcolumn(col::AbstractArray, ::Val{name})

Get the column name of an array col of structs.

Returns getproperty.(col, name) by default, StructArray columns provide zero-copy access. Specialize for array types that support efficient column access.

The Val variants keep the property name in the type domain, which is required for type inference on columns that are not StructArrays.

subst_prop_refs(expr, argsym::Symbol = gensym(:x))

Replace $-escaped property references in expr by property accesses on the argument name argsym, and input calls f(_) by calls of generated callee names.

Usage:

paths, argsym, callees, has_macro, new_expr = subst_prop_refs(expr)

callees pairs the generated name of each input-call callee with its expression, in order of first use, and has_macro indicates whether the expression contains macro calls outside of quote contexts.

PropertyFunctions.unfix_input_properties(f::PropertyFunction)

Undo PropertyFunctions.fix_input_properties, returning the original property function. Returns f unchanged if no properties are fixed.