API reference¶
Generated from the source.
Cache¶
matview.cache.Cache ¶
Ask for a value; get the stored one if it is current, and compute it if not.
Not global, deliberately. The decorator registry being global will tempt an implementer into making this global too; it must not be.
get ¶
get(recipe: str, *, part: str | None = None, on_miss: OnMiss = 'compute', force: bool = False, force_deps: bool = False, deadline: float | None = None, **params: Any) -> Any
Return the decoded value.
The codec is a property of the recipe, so decoding is a lookup and never an inference.
get_bytes ¶
get_bytes(recipe: str, *, part: str | None = None, on_miss: OnMiss = 'compute', force: bool = False, force_deps: bool = False, deadline: float | None = None, **params: Any) -> bytes | dict[str, bytes]
The stored bytes.
The escape hatch, and the one to use for replication and any future HTTP serving: decode-then-reencode both wastes work and risks failing to round-trip.
get_many ¶
get_many(requests: Sequence[Request | tuple[Any, ...]], *, on_miss: OnMiss = 'compute', force: bool = False, force_deps: bool = False, deadline: float | None = None) -> BatchResult
Resolve many roots in one walk.
This is what makes the spec's story about cold runs true. A recipe blocked on
its first ctx.get cannot continue, so a single root reports one protected
miss and stops — but five hundred independent roots in one walk report
everything they need between them, which is the difference between one run and
five hundred.
Roots resolve in order, sharing one memo, so a dependency common to several of them is computed once. They are not run in parallel: the scheduler is plain recursion by design, and making the memo, the in-flight set and the miss accumulator concurrent would change its character. See DECISIONS.md D33.
A protected miss is a policy outcome and is collected. An exception raised by a recipe is not, and propagates immediately.
plan_many ¶
plan_many(requests: Sequence[Request | tuple[Any, ...]], *, on_miss: OnMiss = 'compute', force_deps: bool = False) -> PlanReport
Walk many roots without computing, and report on all of them together.
plan ¶
plan(recipe: str, *, part: str | None = None, on_miss: OnMiss = 'compute', force_deps: bool = False, **params: Any) -> PlanReport
Walk the graph without computing anything.
Absent and stale are reported separately, because stale work is often near-free while absent work at a protected root means you cannot proceed at all.
The walk stops at an absent Python node, necessarily: its dependencies are only knowable by running it. A foreign recipe declares its inputs, so the walk continues through it — which is why the spec calls the declared graph the privileged case.
drop ¶
Remove one recipe version's artifacts. See DECISIONS.md D19.
Context¶
matview.context.Context ¶
What a recipe body receives as its first argument.
deadline_remaining
property
¶
Wall-clock seconds left in the budget, or None when there is no deadline.
get ¶
Resolve a dependency and return the decoded value.
Decoding is by the dependency's own declared codec, so a recipe body should never contain a manual decode call.
get_bytes ¶
get_bytes(recipe: str, *, part: str | None = None, on_miss: OnMiss | None = None, **params: Any) -> bytes | dict[str, bytes]
The stored bytes, undecoded. For replication and pass-through, not for logic.
Registration¶
matview.registry.recipe ¶
recipe(*, version: int, codec: str, index: tuple[str, ...] | list[str] = (), on_miss: OnMiss = 'compute', deterministic: bool = True, max_age: int | None = None, registry: Registry | None = None) -> Callable[[Callable[..., Any]], Callable[..., Any]]
Register a function as a recipe.
deterministic=False declares that the output varies run to run with identical
inputs (sampling, unseeded RNG, wall-clock, a network fetch). Such a recipe defeats
early cutoff and cascades unconditionally. The flag does not change scheduling; it
exists so the cost is visible and plan() can report it, and nobody spends an
afternoon wondering why one branch always rebuilds. See DECISIONS.md D4.
max_age is a generic freshness ceiling in seconds. Matview decides whether a
stored value is old; callers still decide when to read it. See DECISIONS.md D34.
matview.registry.Recipe
dataclass
¶
Recipe(name: str, version: int, codec: str, index: tuple[str, ...], on_miss: OnMiss, deterministic: bool, max_age: int | None, params: tuple[str, ...], defaults: Mapping[str, Scalar], fn: Callable[..., Any] | None = None, source_hash: str | None = None, foreign: ForeignSpec | None = None)
A named, versioned computation. A kind of computation, not a particular result.
bind ¶
Fill declared defaults, then canonicalize.
Defaults are applied before hashing, so sma(symbol="AAPL", window=200) and
sma(symbol="AAPL", window=200, interval="1d") are the same artifact when
interval defaults to "1d".
index_fields ¶
The subset of the key promoted to real columns for queryability.
Policy and reports¶
matview.policy.Policy
dataclass
¶
Policy(on_miss: OnMiss = 'compute', force: bool = False, force_deps: bool = False, deadline_at: float | None = None)
What a caller asked for, carried down the recursion.
on_miss here is the caller's ask. It is combined with each recipe's own
declaration at each node, and the stricter of the two wins. A recipe's declaration
protects that recipe; it does not propagate to the recipe's own dependencies.
from_call
classmethod
¶
from_call(*, on_miss: OnMiss = 'compute', force: bool = False, force_deps: bool = False, deadline: float | None = None) -> Policy
Convert a wall-clock budget to an absolute monotonic instant, once, at the top.
A deadline is deliberately preferred over a compute budget in abstract cost units, because per-recipe cost estimates are guesses that go stale, whereas elapsed time is a measurement.
narrow ¶
Restrict further. A caller may never loosen, so this only ever tightens.
check ¶
Raise if the budget is already exhausted, before descending any further.
matview.types.PlanReport
dataclass
¶
PlanReport(absent: tuple[MissingNode, ...], stale: tuple[ArtifactRef, ...], valid: tuple[ArtifactRef, ...], nondeterministic: tuple[str, ...])
What a walk found without computing anything.
Absent and stale are reported separately because stale work is often near-free while absent work at a protected root means you cannot proceed at all.
matview.types.MissingNode
dataclass
¶
MissingNode(recipe: str, version: int, key: Mapping[str, Scalar], part: str | None, action: Literal['compute', 'enqueue', 'raise'], path: tuple[str, ...] = ())
A node that was needed but could not be computed under the effective policy.
matview.types.stricter ¶
The less permissive of two policies. See rule 6: never the caller's alone.
Artifacts and edges¶
matview.types.Artifact
dataclass
¶
Artifact(recipe: str, version: int, codec: str, key_hash: str, key_json: str, part: str | None, digest: str, fingerprint: str, n_bytes: int, data: bytes, computed_at: int, refreshed_at: int, compute_ms: int | None = None, valid: bool = True, id: int | None = None, source_hash: str | None = None, index_fields: Mapping[str, str] = dict())
One stored result: (recipe, key_hash, part) -> bytes.
matview.types.ArtifactRef
dataclass
¶
ArtifactRef(id: int, recipe: str, version: int, key_hash: str, key_json: str, part: str | None, digest: str, fingerprint: str, valid: bool, n_bytes: int, computed_at: int, refreshed_at: int, compute_ms: int | None)
An artifact without its payload, for querying and planning.
matview.types.DepEdge
dataclass
¶
DepEdge(recipe: str, version: int, key_hash: str, part: str | None, digest: str, artifact_id: int, key_json: str = '')
One observed edge: a dependent read this artifact, at this digest, at compute time.
version is here because the store needs to resolve the row, and artifact_id
because the cascade walks on it. Neither reaches the fingerprint — see rule 8 and
the note in matview.fingerprint.fingerprint.
matview.types.StoreResult
dataclass
¶
What a write actually did.
written=False is the rule 4 path: a row with the same digest and fingerprint was
already there and its payload was left alone.
Graph queries¶
Lineage, impact and cost over the DAG the cache recorded as it ran. See the Graph queries page for what each one answers and how to read it.
matview.graph.lineage ¶
The input closure of one artifact: everything that went into producing it.
Answers "which raw inputs produced this number?" from what was actually read at
compute time, not from what a recipe declares it might read. depth=1 stops at the
artifact's direct dependencies.
Needs no recipe registry: it reads the recorded DAG.
matview.graph.impact_of_change ¶
impact_of_change(store: GraphStore, artifact_id: int, *, depth: int | None = None, enumerate_all: bool = False, limit: int | None = None) -> ImpactReport
What would go stale if this artifact's bytes changed.
This is the cascade the store already runs inside a write, asked as a question. The walk is the same one; only artifacts currently valid would actually change state, which the summary reports separately, because an already-stale dependent is not additional damage.
matview.graph.impact_of_bump ¶
impact_of_bump(store: GraphStore, recipe: str, *, enumerate_all: bool = False, limit: int | None = None) -> ImpactReport
What happens if this recipe's version is bumped.
The honest answer is counter-intuitive and is most of this function's value: nothing goes stale. Under rule 8 a dependency's recipe version is not part of a dependent's fingerprint, so a bump on its own invalidates nothing anywhere.
What actually happens is that the bump creates a new recipe row. The existing
artifacts are still in the database, intact, but nothing resolves to them any more.
On the next read each dependent re-resolves the dependency at the new version, finds
it absent, recomputes it — and cascades only if the recomputed bytes differ from
what the edge recorded. If the new version produces identical bytes, the whole graph
above it stays valid and nothing is recomputed. That is early cutoff, and it is what
rule 8 is for.
matview.graph.impact_of_drop ¶
impact_of_drop(store: GraphStore, recipe: str, *, version: int | None = None, enumerate_all: bool = False, limit: int | None = None) -> ImpactReport
What cache.drop() would delete, and what it would mark stale.
drop is deliberately not a command-line operation; being able to see what it would
do is the useful half. Per D19 a drop deletes the recipe's artifacts outright — the
one place in the library where bytes go away — and marks every transitive dependent
stale, since their inputs no longer exist to revalidate against.
matview.graph.embodied_cost ¶
embodied_cost(store: GraphStore, artifact_id: int, *, depth: int | None = None, with_critical_path: bool = False) -> CostReport
What this artifact and everything it was built from cost to compute.
Summed over distinct values rather than artifact rows, so a partitioned dependency counts once. The value count is reported alongside the total precisely so that the deduplication is visible rather than something a reader has to take on trust.
A value whose compute_ms was never recorded contributes nothing, and is counted
under unmeasured_values: the total is a floor, not an estimate.
matview.graph.critical_path ¶
critical_path(store: GraphStore, artifact_id: int, *, depth: int | None = None) -> tuple[PathStep, ...]
The most expensive chain through an artifact's input closure.
The chain, not just its length: the point of asking is to see which step dominates, and a number alone does not answer that.
A topological walk in Python, which is the right shape here. Input closures are small — around fifty nodes for an aggregate in the recorded benchmark — and this is the input direction, not the dependent direction where one shared root reaches 255,000 artifacts.
No deduplication is needed along a path: a single chain passes through at most one
part of any value, so each step's own compute_ms is that value's cost, counted
once.
matview.graph.recipe_costs ¶
Per recipe version: artifacts, distinct values, total and mean cost, and bytes.
No graph walk at all — a GROUP BY over the artifact table — which makes it the
cheapest question here and, for anyone about to optimize something, usually the
first one worth asking.
matview.graph.ArtifactGraph
dataclass
¶
ArtifactGraph(nodes: Mapping[int, ArtifactRef], edges: tuple[tuple[int, int], ...], roots: tuple[int, ...], direction: Direction = 'inputs', truncated: bool = False)
A walked piece of the recorded DAG, with no reference to the store it came from.
A plain value object on purpose: build it once, render it four ways. edges is
always (dependent, dependency) whichever direction the walk ran in, because that
is the direction the dep table records and reversing it in storage would make the
two walks disagree about what an edge means. direction says which end of an edge a
renderer should descend from.
children_of ¶
The nodes one step further from the roots, in the walk's own direction.
to_tree ¶
An indented tree, roots first.
The DAG is not a tree, so a node reachable by more than one path is printed again and marked rather than expanded twice — which both keeps the output bounded on a diamond and makes the sharing visible.
to_json ¶
The graph as data: nodes, edges and roots, with the walk's direction.
matview.graph.ImpactReport
dataclass
¶
ImpactReport(question: Literal['changed', 'bump', 'drop'], subject: str, notes: tuple[str, ...], groups: tuple[ImpactGroup, ...] = ())
What would happen, for one of the three questions impact answers.
notes carries the prose. For a version bump that prose is the answer, and is the
part of the fingerprint spec that confuses every new reader, so it is data rather
than something the command-line layer prints on the side.
matview.graph.ImpactGroup
dataclass
¶
ImpactGroup(label: str, total: int, by_recipe: tuple[RecipeCount, ...], refs: tuple[ArtifactRef, ...] = (), truncated: bool = False, ids: tuple[int, ...] = ())
One named set of affected artifacts within an impact answer.
matview.graph.CostReport
dataclass
¶
CostReport(root: ArtifactRef, artifacts: int, values: int, compute_ms: int, own_compute_ms: int | None, unmeasured_values: int, n_bytes: int, depth: int | None = None, path: tuple[PathStep, ...] = ())
What one artifact embodies, and optionally where that cost is concentrated.
matview.graph.GraphStore ¶
Bases: Protocol
The store reads these queries need.
Declared here rather than added to Store so that the persistence seam stays the
set of operations a backend must implement to hold artifacts. A backend that also
wants to answer graph questions satisfies this structurally, as SqliteStore does.
Storage¶
matview.store.base.Store ¶
Bases: Protocol
Persistence for artifacts and the observed DAG.
fetch_all_parts ¶
Every part of one (recipe, version, key_hash), NULL part first.
Not in the spec's sketch of this protocol, and needed by it: a codec may emit a dict of parts, and a whole-value read has to find them all without knowing the labels in advance. See DECISIONS.md D11.
store_parts ¶
Every part of one value, in one transaction.
The partitioned generalization of store(). A codec returning a dict produces
several artifacts from a single compute; they share a dep set, differ only in
part, and a reader merging them must never see half of a rewrite. See
DECISIONS.md D24.
deps_of ¶
The stored edges of one artifact: what it actually read, last time it ran.
The stale read path walks these to recompute a fingerprint without running anything, which is what makes revalidation-without-recompute possible.
enqueue ¶
on_miss="enqueue" writes a task row here. Nothing in v1 consumes it.
matview.store.sqlite.SqliteStore ¶
SqliteStore(path: str | PathLike[str], *, max_artifact_bytes: int = DEFAULT_MAX_ARTIFACT_BYTES, busy_timeout_ms: int = DEFAULT_BUSY_TIMEOUT_MS, read_only: bool = False)
One SQLite file. Connections are per-thread, with WAL on.
Open a cache database, creating it if it does not exist.
read_only=True opens it through SQLite's mode=ro, requires the file to
already exist, and creates nothing — no schema, no WAL sidecar, no journal-mode
change. An inspection tool needs that: a plain open creates a database on a
mistyped path and reports it as empty, and switching someone's journal mode while
they are diagnosing a problem is not a read. See DECISIONS.md D38.
deps_of ¶
The stored edges of one artifact: what it actually read, last time it ran.
deps_of_many ¶
The stored edges of many artifacts, in one query.
Keyed by the artifact asked about, with every requested id present: a root maps to an empty list rather than being absent, so a caller can index the result without deciding what a missing key would mean.
The graph queries need this; the scheduler does not, despite appearances. Its
two stored-subgraph walks loop over the parts of one node, so batching across
that loop saves nothing and measured slower — see DECISIONS.md D52. deps_of
delegates here rather than keeping a second query, so the two cannot drift.
dependents_of ¶
Everything that transitively read these artifacts. Walks dep_reverse.
The roots themselves are excluded. depth=1 stops at direct dependents,
depth=None walks the whole closure.
inputs_of ¶
Everything these artifacts transitively read: the input closure.
The mirror of dependents_of, following dep the other way, and with the same
contract — the roots themselves are excluded from the result, and depth=1 stops
after one level. This is the walk that answers "which raw inputs produced this
number?", which is the audit question and the reproducibility question both.
Nothing here resolves a recipe. It reads the observed DAG as recorded, so it works against a database whose recipes this process has never imported.
list_refs ¶
list_refs(recipe: str | None = None, *, version: int | None = None, part: str | None = None, valid: bool | None = None, fields: Mapping[str, str] | None = None, limit: int | None = None, offset: int = 0) -> list[ArtifactRef]
List artifacts with the filters an inspection tool needs.
Separate from query() rather than an extension of it: query() is the seam the
spec fixes, and its **indexed_fields signature has no room for a limit or a
valid filter without a caller's field name colliding with an option name.
ref_by_id ¶
One artifact's metadata by row id, without needing to know its key.
refs_by_ids ¶
Metadata for many artifacts by row id, in one query, ordered by id.
Ids that name no row are silently absent rather than an error: a caller holding ids from a graph walk wants what is there, and the walk cannot produce an id that was not in the database when it ran.
ids_of_recipe ¶
Every artifact row id belonging to a recipe, or to one version of it.
Ids rather than refs because the callers — drop() and the impact queries —
want a set to walk the graph from, and a recipe with 250,000 artifacts should not
have its metadata materialized to answer that.
fetch_by_id ¶
One artifact including its payload, by row id.
refs_by_key_hash ¶
Every artifact under one key hash, across recipes, versions and parts.
A key hash is not unique on its own: the same parameters can be shared by several recipes, and a version bump leaves the old artifact in place.
counts_by_recipe ¶
Group a set of artifact ids by recipe version, counting in SQL.
The point is not to materialize the set. One shared root's dependent closure in
the recorded 250k profile is 255,000 artifacts, and the useful answer about it is
a handful of counts — which this produces without ever building 255,000
ArtifactRef objects to throw away.
recipe_rows ¶
Every recipe version the database knows about, with its artifact count.
Read from the recipe table, so it answers what is stored rather than what is
registered in this process. A version present here with no registered recipe is
exactly the situation the CLI exists to make visible.
recipe_cost_rows ¶
Per recipe version: artifacts, distinct values, cost and bytes.
The sibling of recipe_rows() rather than an extension of it, because the
deduplication below costs a grouped scan that stats has no reason to pay for.
compute_ms is summed per value, not per artifact. A partitioned value is
several artifact rows that each record the cost of the one compute that produced
them all, so summing the column would multiply that value's cost by its part
count. MAX picks the one shared figure back out of the group. See DECISIONS.md
D51; this is the single most likely place in these queries to be confidently
wrong.
summary ¶
Aggregates for the whole database, computed in SQL rather than in Python.
At the scale the design cites — thousands of keys times dozens of indicators — materializing every row to count it is not an option.
iter_payloads ¶
Yield (artifact_id, digest, payload) in batches.
Batched rather than one query, because verifying a large cache must not hold every payload in memory at once.
recipe_source_hash ¶
The advisory source hash last recorded for a recipe version, if any.
pending_tasks ¶
Rows written by on_miss="enqueue". Nothing in v1 consumes them.
store ¶
Artifact, dep edges and cascade in one transaction, or none of them.
The lookup is part IS ? rather than a reliance on the schema's
UNIQUE (recipe_id, key_hash, part): SQLite treats NULLs as distinct in a UNIQUE
index, so that constraint does not actually cover unpartitioned rows. Writers
serialize on BEGIN IMMEDIATE, so select-then-write is safe across processes.
See DECISIONS.md D23.
store_parts ¶
Every part of one value, in one transaction.
A codec that returns a dict produces several artifacts from a single compute.
They share a dep set and differ only in part, and a reader merging them must
never see half of a rewrite, so they land together. This is the partitioned
generalization of store(), which the spec fixes for the single-artifact case.
See DECISIONS.md D24.
Parts that this computation did not produce are removed. That is not invalidation, which never deletes: it is a part that no longer exists, and leaving it would let a whole-value read merge a phantom.
invalidate ¶
Mark these artifacts and everything downstream of them stale.
Transitive on purpose: an artifact nobody can trust makes its dependents untrustworthy too, and a non-transitive public invalidate is a footgun.
revalidate ¶
The stale-but-current path: the fingerprint matched, so the bytes are good.
drop ¶
Remove a recipe version's artifacts. See DECISIONS.md D19.
gc ¶
Drop superseded versions with no dependents.
A version bump creates a new recipe row, and old artifacts remain in place, unreachable but intact. Dropping the version back finds them again. This is what actually removes them. Iterates to a fixed point, since removing one artifact can orphan another.
enqueue ¶
on_miss="enqueue" writes here. Nothing in v1 consumes it.
Codecs¶
matview.codecs.Codec ¶
matview.codecs.BaseCodec ¶
Shared behaviour. A codec that never partitions inherits the right refusal.
matview.codecs.get_codec ¶
Look up a codec, importing its optional extra on first use.
matview.codecs.register_codec ¶
Register a codec under its own name. Re-registering the same name replaces it.
Foreign recipes¶
matview.foreign.load_manifests ¶
load_manifests(directory: str | Path, *, registry: Registry | None = None, store: Store | None = None) -> list[Recipe]
Register every *.toml manifest under directory, by directory scan.
Pass a store to have source hashes checked against what the cache last saw; a
changed source with an unchanged version is a warning, since the artifacts already
stored under that version were produced by different code.
matview.foreign.load_manifest ¶
Read one manifest and register it exactly as if a decorator had.
matview.foreign.check_declared_cycles ¶
Cycle-check the declared graph before anything executes.
Only foreign recipes appear here, because only they declare their inputs. A cycle that runs through a Python recipe is caught at runtime by the in-flight stack instead.
Canonicalization and fingerprints¶
The interoperability contract these implement is SPEC.md, versioned independently of
the code.
matview.fingerprint ¶
Hashes, per SPEC.md.
The other half of the interoperability contract. SPEC.md is authoritative; this
module implements it and nothing more.
canonical_key_hash ¶
Return (key_json, key_hash), so callers hash and store the same canonical form.
digest_bytes ¶
digest = BLAKE2b(artifact_bytes), over the encoded payload exactly as stored.
source_hash ¶
Advisory only. Recorded on the recipe row, never fingerprinted.
fingerprint ¶
fingerprint(*, recipe: str, version: int, key_hash: str, part: str | None, deps: Sequence[DepEdge]) -> str
The hash of the inputs that produced an artifact.
An artifact is current when its recomputed fingerprint matches the stored one.
Note the asymmetry, which will look like a bug to anyone reading this fresh: A's
own recipe version is in the fingerprint; its dependencies' recipe versions are
not. Dependencies contribute only their content digests, which is why DepEdge
carries a version that this function pointedly ignores.
This is deliberate. It buys early cutoff: if a dependency is recomputed under a new
version and produces identical bytes, nothing downstream is invalidated. It costs
the ability to force a downstream rebuild by bumping an upstream version, which is
occasionally wanted during development and is served instead by force_deps.
Do not "fix" it. See SPEC.md, and rule 8 in the spec's hard rules.
matview.canon.canonicalize_key ¶
Normalize a key dict. Defaults must already have been filled in by the caller.
matview.canon.jcs ¶
Serialize with the JSON Canonicalization Scheme (RFC 8785).
Object keys sorted by UTF-16 code unit, no insignificant whitespace, UTF-8 output, ES6 number rendering. Delegated rather than hand-rolled: the hard part is number formatting, and getting that wrong shows up as a cold cache, not an exception. See DECISIONS.md D3.
Errors¶
matview.errors ¶
Exceptions raised by matview.
All of them descend from MatviewError, so a caller can catch this library without
catching everything.
MatviewError ¶
Bases: Exception
Base for everything this library raises.
InvalidKey ¶
Bases: MatviewError
A key field is not a scalar, is NaN or infinite, or uses a reserved name.
RecipeNotFound ¶
Bases: MatviewError
No recipe is registered under that name.
CycleError ¶
Bases: MatviewError
A recipe asked, transitively, for its own output.
Cycles are detected by the in-flight stack rather than by static analysis, so the message carries the full path that closed the loop.
DeadlineExceeded ¶
MissingArtifacts ¶
Bases: MatviewError
Nodes that were needed but could not be computed under the effective policy.
Carries every such node rather than the first, because the point of the protection is to tell a cold run exactly what needs ingesting in one pass.
ArtifactTooLarge ¶
Bases: MatviewError
A payload exceeded max_artifact_bytes.
Raised loudly on purpose: it surfaces a codec or partitioning problem when it appears rather than as mysterious slowness months later.
CodecError ¶
Bases: MatviewError
A codec could not encode or decode a value.
CodecUnavailable ¶
Bases: CodecError
A codec exists but its optional extra is not installed.
ForeignRecipeError ¶
Bases: MatviewError
A foreign recipe's manifest, invocation or output was unusable.
StoreError ¶
Bases: MatviewError
The store could not satisfy a request.