Notebook Concepts¶

Architecture¶

Strata Notebook is a content-addressed compute graph over Python. Every cell output is an artifact, and every cell execution is a materialize(inputs, transform, environment) → artifact operation.

┌───────────────┬───────────────┬─────────────────────────────┐
│ Web UI (Vue)  │ Terminal TUI  │ Coding agent (CLI / MCP)    │
│ edit and run  │ read-only     │ drives a session remotely   │
└───────────────┴───────────────┴─────────────────────────────┘
        │               │                      │
        └───────────────┴──────────────────────┘
                        │  REST + WebSocket
                        ▼
┌──────────────────────────────────────────────────────────────┐
│ Notebook Backend (FastAPI + WebSocket)                       │
│ (session mgmt, DAG, cascade planner, executor)               │
└──────────────────────────────────────────────────────────────┘
            │                                │
            │ local subprocess               │ HTTP
            ▼                                ▼
   ┌──────────────────┐         ┌──────────────────────────┐
   │ Cell harness     │         │ Remote worker            │
   │ (notebook venv)  │         │ (SSH / Fly / Modal /     │
   │                  │         │  a pool-run machine)     │
   └──────────────────┘         └──────────────────────────┘
            └───────────────┬────────────────┘
                            ▼
┌──────────────────────────────────────────────────────────────┐
│ Strata Artifact Store                                        │
│ metadata: SQLite, or Postgres when several servers share one │
│ blobs: local disk, S3, GCS, Azure                            │
│ provenance dedup, lineage, and an optional team cache that   │
│ answers a local miss from a shared store                     │
└──────────────────────────────────────────────────────────────┘

The notebook is an orchestration layer: it decides what to run next. The cell harness is an executor: it runs Python code. The artifact store decides whether a result already exists and persists it.

The three surfaces are peers on one protocol, not a primary and two add-ons: they speak the same REST and WebSocket API. A cell an agent runs against a live session appears in the web UI and the terminal viewer as it happens - though only when the agent addresses the session rather than the notebook directory, since the local backend edits files and the server learns nothing. Of the three, only the TUI is deliberately read-only.

Why "every output is an artifact"¶

The guarantees fall out of two design choices:

Provenance is identity. An artifact is keyed by a hash of the cell's source (AST-normalized), its resolved inputs, and the environment fingerprint. Identical computations produce identical hashes, so the cache hit check is "have we computed this exact hash before?", not "do two things look similar?"
Artifacts are immutable. Once stored, a version never mutates. A downstream cell reading artifact @v=3 today reads the same bytes it would have read yesterday, even if upstream code has since changed and a new @v=4 exists.

Together: the cache isn't an optimization layer that might be stale, it's the single source of truth for "has this work been done." That's why cache hits are safe to serve in milliseconds, why renaming a cell doesn't invalidate a downstream artifact (the provenance hash depends on semantic source, not whitespace or comments), and why you can fork a loop from iteration 17 without having to re-run iterations 0–16.

Stateful objects and value semantics (ML training)¶

The flip side of "every variable is an independent, immutable artifact" is that Strata is a value-semantics engine: a cell receives copies of its inputs (deserialized from artifacts), not the caller's live objects. A linear Jupyter kernel shares one memory space, so a model trained in one cell is visible everywhere; Strata's cells are isolated, so two patterns that work in Jupyter break here - both silently, because the run still goes green:

In-place mutation that isn't exported. A cell that trains a model with optimizer.step() (a method call, no model = … reassignment) mutates model in place. Strata's data-flow only treats a variable as a cell's output if it's assigned, subscripted, attr-set or passed inplace=True, so the trained model is never re-exported, and downstream cells read the pre-training version.
Shared mutable references split across cells. optimizer = Adam(model.parameters()) makes the optimizer hold references to the same tensors as the model. Stored as two separate artifacts and reloaded into a later cell, they become two different tensor sets - so optimizer.step() updates the optimizer's private copies and the model you evaluate/save never changes. Training is a silent no-op.

Both are the same root: shared, mutable state doesn't survive a boundary where each variable is an independent immutable artifact. Strata deliberately does not try to fake it (preserving cross-artifact object identity, or silently re-exporting a mutated input, would make provenance lie).

The pattern that works¶

Express training as one content-addressed transform - data in, trained model out - by keeping the model, its optimizer, and the training loop in one cell, and making the trained model that cell's output:

# @name train
model = build_model()
optimizer = torch.optim.Adam(model.parameters())
for epoch in range(epochs):
    for xb, yb in train_loader:
        optimizer.zero_grad()
        loss = loss_fn(model(xb), yb)
        loss.backward()
        optimizer.step()
# `model` (trained) is this cell's output - downstream cells get the trained one.
trained_model = model

For a large handoff, export a checkpoint instead of a live object: save a state_dict (or a file path) from the training cell and load it downstream with explicit map_location - see the device/placement guidance in Distributed Workers.

Notebook File Format¶

Each notebook is a directory on disk:

my_notebook/
├── notebook.toml          # Stable config: ID, name, cell list, workers, mounts, env, ai
├── pyproject.toml         # Python dependencies (uv-managed)
├── uv.lock                # Locked dependencies
├── cells/
│   ├── a1b2c3d4.py        # Cell source files
│   └── e5f6g7h8.py
└── .strata/               # Gitignored, runtime state, not committed
    ├── runtime.json           # Display outputs, provenance hashes, env metadata
    ├── console/               # Per-cell stdout/stderr ({cell_id}.json)
    └── artifacts/
        ├── artifacts.sqlite   # Artifact metadata
        └── blobs/             # Serialized cell outputs

notebook.toml holds stable config you'd commit to git; .strata/ holds runtime state that changes on every execution (display outputs, console snapshots, the last uv sync timestamp, per-cell provenance hashes). notebook.toml's updated_at tracks only structural edits (adding/removing cells, changing workers or mounts), so example notebooks don't churn under version control.

notebook.toml defines the notebook identity and cell ordering:

notebook_id = "f7bd9094-..."
name = "my_analysis"

[[cells]]
id = "a1b2c3d4"
file = "a1b2c3d4.py"
language = "python"
order = 0

[[cells]]
id = "e5f6g7h8"
file = "e5f6g7h8.py"
language = "python"
order = 1

Version Control¶

Strata notebooks are designed to live in git. Unlike Jupyter .ipynb files, which co-mingle source, outputs, and execution counts in one JSON blob and produce a multi-kilobyte diff every time a cell is re-run, a Strata notebook is just a directory of plain text:

Cells are plain text files. Python, prompt and SQL cells are .py, R cells .r, markdown cells .md and widget cells .widget. Normal git diff, git blame, code review on a pull request, syntax highlighting in every IDE. Reordering a cell edits one number in notebook.toml, not a giant JSON re-serialize.
notebook.toml is the manifest. Stable config only: cell list, workers, mounts, env, AI defaults, and the active variant per group (see Variant Cells). Reviewers see exactly what changed about the notebook's shape, not its execution history.
.strata/ is gitignored. Display outputs, console snapshots, the uv sync timestamp, per-cell provenance hashes, the artifact store: none of it touches commits. Re-running a cell never changes the tracked tree.
updated_at only bumps on structural edits. Adding/removing a cell, changing a worker, mounting a path, picking a different variant: those bump the timestamp. Editing source or running cells does not.
Secrets stay off disk. Env keys matching KEY/SECRET/TOKEN/ PASSWORD/CREDENTIAL are blanked before persisting, so the writer can't accidentally commit an API key. The name survives only when the block also holds a non-sensitive value; an [env] with nothing but blanked secrets is not written at all. An empty value for such a key is a placeholder, not an override: until you re-enter it, cells see the server's own value for that variable.
uv lockfile in committed config. pyproject.toml + uv.lock pin the Python environment exactly the same way the rest of your repo does; collaborators get a reproducible environment from a fresh clone.

Put together: a Strata notebook commit shows what changed about the work, not what happened during the last run.

DAG and Variable Analysis¶

Each cell's source code is analyzed via Python's AST to extract:

Defines: top-level variable assignments (x = 1, df = pd.read_csv(...))
References: free variables used but not defined in this cell

The DAG builder connects references to producers:

The last cell that defines a variable is its producer (handles shadowing)
Edges flow from producer cells to consumer cells
Cycle detection prevents circular dependencies

The DAG is rebuilt automatically on every cell source change.

Cell Execution Flow¶

When you run a cell, this happens:

Compute provenance hash: sha256(sorted_input_hashes + source_hash + env_hash)
Cache check: Look up the hash in the artifact store → return immediately on hit
Resolve inputs: Load upstream variable artifacts into a temp directory
Execute: Spawn a subprocess running the cell harness in the notebook's venv, or, for a cell annotated with a # @worker, POST the work to that worker over HTTP instead. Everything either side of this step is identical; only where the Python runs changes. In service mode a cell on the server's own host runs as a separate OS user (STRATA_NOTEBOOK_HARNESS_USER) or not at all; see What a cell can read.
Harness: Deserializes inputs → exec(source, namespace) → serializes new variables
Store outputs: Each consumed variable becomes an artifact
Broadcast: WebSocket sends status, output, and console messages to the UI

Caching and Provenance¶

The provenance hash determines cache identity. It includes:

Component	In hash?	Why
Source code	Yes	Different code = different result
Upstream artifact hashes	Yes	Different inputs = different result
Environment lockfile hash	Yes	Different packages = different result
Env vars the cell declares (`@env`) or reads by name	Yes	A different setting can change the result
Mount fingerprints (URI and contents)	Yes	Different mounted data = different result
`@table` snapshot, `@fetch` digest, `@dataset` version	Yes	New data at the source = different result
Cell ID	No	Same code in a different cell = same result
Notebook ID or project name	No	Same cell in another notebook with the same packages = same result
Execution time	No	Same inputs should produce same output

When you change a cell's source, its provenance hash changes, and all downstream cells become stale.

Because the hash describes the computation rather than the machine, a match found somewhere else is just as valid. That is what the optional team cache trades on: with a shared store configured, a local miss asks whether a colleague has already run this exact computation, and a cell that does run offers its outputs back. Nobody has to decide in advance which results are worth publishing - the expensive middle of a pipeline is shared on the same terms as anything else.

This applies even to a leaf cell, one with no downstream consumer, like a cell that only prints a diagnostic. It has no output artifact to cache, but its console output is keyed by the same provenance hash and replayed on an unchanged re-run, so it doesn't re-execute. That is what makes the notebook a usable scratchpad. A cell that must always run, because it has a side effect (writing a file, calling an API) or produces a fresh value (the clock, random), opts out of caching with # @nocache.

Serialization¶

Cell outputs are serialized based on their Python type:

Type	Format	File extension
PyArrow tables, pandas and polars frames and series, numpy arrays and scalars, torch and jax tensors, `datetime` / `Decimal` / `UUID` / `bytes` / `complex` values	Arrow IPC	`.arrow`
Any other type exporting `__arrow_c_stream__` or `__dlpack__`	Arrow IPC	`.arrow`
Dicts, lists, scalars (int, float, str, bool, None)	JSON	`.json`
Everything else	Pickle	`.pickle`

The content type is stored in the artifact metadata so the read side knows how to deserialize.

JAX and 64-bit values¶

JAX defaults to 32 bits: it turns a float64 into a float32 when it builds an array, unless jax_enable_x64 is on. Every other library here keeps the dtype it was given, so this is the one case where a value can come back narrower than it was stored.

Strata will not hand you the narrowed value. Reading a stored 64-bit array turns the setting on and converts again, and if that still does not reproduce the dtype it raises rather than returning something plausible and wrong.

That switch is process-wide, and jax offers no per-array alternative. The warm pool and Run All reuse one process across cells, so once it is on, later cells in that worker get 64-bit defaults too, and no cell's provenance records which width it got, so a cached result can depend on what ran before it. When this happens the cell's console says so.

If a notebook works in 64-bit, say so explicitly:

# @env JAX_ENABLE_X64=1
import jax.numpy as jnp

jax reads that at import, every execution path applies the environment before deserializing anything, and it becomes part of the cell's env hash. The width is then the same on every path and is recorded in what the result is keyed on.

The second row is what keeps a library Strata has never heard of out of the pickle path. DuckDB relations, cuDF frames, Ibis tables and anything else implementing the Arrow PyCapsule interface are stored as real Arrow, so the artifact stays readable by other tools and the notebook can show its schema and row count.

Two things to know about that row. The first is type identity: the libraries named in the first row are handed back as themselves, while a capsule-sourced value comes back as a pyarrow.Table and a DLPack-sourced one as a numpy.ndarray. The second matters for libraries whose values are lazy query expressions rather than data, Ibis among them. Storing one executes it, because an artifact holds data and a handle is not data. A downstream cell therefore receives the result rather than the expression, and cannot keep composing on it. Pin the query where you want it evaluated and store that.

One-shot streams are refused rather than stored. A pyarrow.RecordBatchReader is consumed by reading, so storing one would leave the object empty for whatever reads it next; it is not picklable either, so the cell fails instead. Call .read_all() and store the table. The check looks for an iterator, so a wrapper that presents the Arrow protocol over a one-shot reader without being iterable itself is not caught: it is stored, correctly, but the object is spent afterwards.

Each value a cell produces is serialized once, even when it is both a variable and the cell's last expression. That matters for a lazy handle, where writing is what runs the query: a cell ending in a bare df evaluates it once, not once per artifact.

Cascade Execution¶

When a cell's upstream dependencies aren't ready, the cascade planner generates an execution plan:

BFS backwards from the target cell to find all upstream cells needing execution
Returns cells in topological order with reasons (stale, missing, or target)
The frontend auto-accepts the cascade and executes cells sequentially

This means you can edit an early cell and run a downstream cell, and Strata will re-execute the pipeline up to it. Within one run each cell executes at most once, even when two branches of the DAG both need it.

Run All¶

Run All runs every cell in notebook order. Consecutive plain Python cells that run locally share one harness process (a batch); a cell with a worker, a timeout, a read-write mount, @loop, @fetch or @dataset, a sweep variant or a consumer of one, and every non-Python cell runs on its own.

When a cell fails, Run All carries on, but not through the failure: every cell that reads from a failed cell, directly or further down, is skipped and marked stale with the reason that an upstream failed. Cells the failure does not reach run as normal. Rerun All does the same while bypassing the cache.

Staleness¶

A cell is out of date when no stored result matches its current provenance. This happens when:

Its source code changed
An upstream cell's output changed
The environment (uv.lock, or an @env it declares) changed
A declared outside input moved: a mounted file, an @table snapshot, an @fetch URL's bytes, an @dataset alias

A cell that still has a result and whose upstream moved reads stale · upstream changed. A cell whose own source, environment or declared outside input changed has nothing that matches, so it reads idle until it runs, and the cells downstream of it read stale. The @fetch and @dataset checks run at most once a minute while you edit, and always right before the cell runs; these lookups and the @table catalog read happen outside the lock that serializes the staleness walk, so one slow host does not stall the server.

The causality chain explains why a cell is stale, tracing the change back to its root cause (e.g., "upstream cell X changed its source").

Cell Status Lifecycle¶

idle ──→ running ──→ ready ──→ stale
              └───→ error

idle: no current result: never executed, its own source, environment or a declared outside input changed since it ran, or, after a reopen, a leaf cell that keeps no record of its run (# @nocache, or a read-write mount)
running: currently executing. If an upstream cell is edited while it runs, it stays running and finishes stale, since it read the old value.
ready: last execution succeeded (or was a cache hit), artifact is current
stale: it has a result, but something upstream has moved, or Run All skipped it because a cell it reads from failed
error: last execution failed. The traceback and the console output before it are kept against the source that failed, so strata cell show, the MCP view, a reopen, a snapshot and an export all still report them. The error stands until the cell is edited or runs successfully (a cache hit counts).