Service Mode¶
Service mode is what you run when more than one person uses Strata through a network. It's the right mode when:
- Multiple users are sharing one Strata instance with their own identities (not just "everyone is logged in as me").
- The server is reachable beyond a loopback interface.
- Multi-tenancy matters, separate caches, separate QoS, separate metrics per tenant.
- You want the platform to mediate writes - either routing them
through server-side transforms, or letting authenticated clients
publish directly to a shared store with
service_writes_enabled(the shared research store).
For one person, use personal mode. A team can also give each member a personal server and share results through a service-mode store; see Connecting a notebook to the shared store.
Switching from the default¶
STRATA_DEPLOYMENT_MODE defaults to personal. To run in service
mode you set the mode explicitly and fill in the matching auth /
artifact configuration:
STRATA_DEPLOYMENT_MODE=service
STRATA_AUTH_MODE=trusted_proxy
STRATA_PROXY_TOKEN=<shared-secret>
STRATA_ARTIFACT_DIR=/path/to/dir # required with any artifact store, blob backend or not
A startup coherence check fires clear ValueErrors on boot if anything's missing, a sloppy service-mode deploy refuses to start rather than silently exposing write endpoints.
The trusted-proxy contract¶
Strata does not authenticate users itself. It trusts an upstream proxy that:
- Terminates auth: JWT, OIDC, mTLS, Cloudflare Access, SAML, whatever. Strata doesn't care which.
- Sets every identity header on every request, replacing whatever the client sent:
| Header | Value |
|---|---|
X-Strata-Principal |
Stable user identifier (email, sub claim, etc.) |
X-Tenant-ID |
Tenant the user belongs to. Header name is configurable via tenant_header. Empty when multi-tenancy is off. |
X-Strata-Scopes |
Space-separated capability set (e.g. notebook:read notebook:write artifacts:write admin:cache). Empty for a user with no scopes. |
X-Strata-Proxy-Token |
Shared secret matching STRATA_PROXY_TOKEN; proves the request came through the proxy |
Strata cannot tell a header the proxy set from one the client sent
and the proxy passed along. A header the proxy leaves alone reaches
Strata from the client: a proxy that sets no X-Strata-Scopes for a
user lets that user send X-Strata-Scopes: admin:* and become an
admin. So the proxy sets all four on every request, and sets an
empty value when the user has none (nginx drops a header set to
"", which is what you want). A minimal nginx block:
map $http_upgrade $connection_upgrade { # in the http {} block
default upgrade;
'' close;
}
location / {
proxy_pass http://strata:8765;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Host $http_host;
# Every identity header, on every request, from the auth layer only.
proxy_set_header X-Strata-Proxy-Token "<shared-secret>";
proxy_set_header X-Strata-Principal $authenticated_user;
proxy_set_header X-Tenant-ID $authenticated_tenant; # "" if multi-tenancy is off
proxy_set_header X-Strata-Scopes $authenticated_scopes; # "" for no scopes
}
$authenticated_* stand for whatever your auth layer provides (for
example auth_request_set from an auth_request subrequest). If you
renamed a header with STRATA_PRINCIPAL_HEADER, STRATA_TENANT_HEADER
or STRATA_SCOPES_HEADER, set the renamed one. Check it from outside
the proxy: a request carrying X-Strata-Scopes: admin:* for an
ordinary user must still get 403 from GET /v1/logs. The demo
stack's .docker/service-mode/nginx.conf sets all four.
API keys are a separate auth mode (STRATA_AUTH_MODE=api_key) for a
server no proxy fronts: strata apikey create issues one, and the key
carries its own principal, tenant and scopes. A trusted-proxy server
ignores them, so machine callers (a CI job, an ETL service) of a proxied
server go through the proxy like everyone else. See
Configuration → API keys.
- Is the only path to Strata. Strata is on a private network / VPC / Kubernetes namespace; the proxy is the only ingress. Without that, anything that can reach Strata directly can forge the headers above and impersonate any user.
The proxy-token check is a backstop, not the security boundary: the network-level isolation is. If an attacker can reach Strata's IP directly, they can read the token from any leaked config and forge headers. Treat the token as defense-in-depth.
The proxy must pass the browser's Host header through unchanged,
port included (nginx: proxy_set_header Host $http_host;, not
$host, which drops the port). Strata accepts a browser write or a
notebook WebSocket only when its Origin matches that Host; a proxy
that rewrites Host has to list the public origin in
STRATA_CORS_ALLOW_ORIGINS instead.
Running the demo stack¶
The repo ships a complete service-mode demo: Strata + a notebook executor sidecar + an nginx proxy that injects two pre-baked identities for testing.
The proxy exposes Strata on two ports, same server, different synthesized callers:
| URL | What nginx injects | Use for |
|---|---|---|
http://localhost:8865 |
Principal: demo-user, scopes: notebook:read notebook:write notebook:execute |
Normal-user view |
http://localhost:8866 |
Principal: demo-admin, scopes: admin:* notebook:read notebook:write notebook:execute |
Admin-only operations |
Both URLs route to the same Strata instance. Tenant header
(X-Tenant-ID: demo-team) is injected on both. Open either in a
browser to use the notebook UI; hit /v1/... endpoints with curl
to exercise the REST surface.
Cells run on the demo's executor, not on the Strata container: assign a
cell or notebook the gpu-http worker. A cell left on local is refused,
because a service-mode server does not run cell code on its own host (see
What a cell can read).
The configuration is in .docker/service-mode/:
pyproject.toml, Strata's service-mode config (mounted into the container as/home/strata/pyproject.toml). Includes the proxy token, multi-tenancy on, tenant header name, and a sample worker-catalog entry pointing at the executor sidecar.nginx.conf, the twoserver {}blocks that inject the demo identities.
To experiment with new identities, edit nginx.conf and restart
the proxy container (docker compose -f docker-compose.service.yml
restart proxy).
Production reference architecture¶
┌────────────────┐ ┌─────────────────┐ ┌──────────────┐
│ Your auth │ │ │ │ │
│ provider │◄────┤ Auth proxy │ │ Strata │
│ (OIDC/SAML/ │ │ (nginx, │────►│ (service │
│ Cloudflare │ │ Envoy, │ │ mode) │
│ Access, …) │ │ Cloud Run │ │ │
│ │ │ ingress, …) │ │ │
└────────────────┘ │ │ └──────┬───────┘
│ Adds headers: │ │
│ Principal │ ▼
│ Tenant │ ┌──────────────┐
│ Scopes │ │ Notebook │
│ Proxy-Token │ │ executors │
└─────────────────┘ │ (sidecars or │
│ remote) │
└──────┬───────┘
│
▼
┌─────────────────────────┐
│ Artifact store │
│ │
│ metadata → Postgres │
│ blobs → S3 / GCS / │
│ Azure │
└─────────────────────────┘
Strata sits on a private network, only the auth proxy can reach it.
Notebook execution dispatches to executors; the demo stack runs one
locally, production typically uses HTTP executors on dedicated nodes
or remote backends like Modal / Fly Machines. Cells do not run on the
Strata host itself unless you set STRATA_NOTEBOOK_HARNESS_USER (see
What a cell can read).
The artifact store is two halves, and both have to be shared before
you can run a second replica. Blobs go to S3, GCS or Azure rather
than a local volume, so they survive container churn. Metadata is a
SQLite file under STRATA_ARTIFACT_DIR by default, which is local to
one machine; set STRATA_ARTIFACT_METADATA_DSN to put it on Postgres
instead. A DSN with STRATA_ARTIFACT_BLOB_BACKEND=local is rejected at
startup for the same reason: shared metadata pointing at blobs only one
node can read is worse than either alone. See
Configuration → artifact metadata
for the settings and strata migrate for moving an existing store
across, and Running more than one replica
for what else the nodes must share.
Minimum service-mode env vars¶
# Required
STRATA_DEPLOYMENT_MODE=service
STRATA_AUTH_MODE=trusted_proxy
STRATA_PROXY_TOKEN=<shared-secret-with-proxy>
STRATA_ARTIFACT_DIR=/path/to/dir # required with any artifact store, blob backend or not
STRATA_HOST=0.0.0.0 # default 127.0.0.1; bind wider when the proxy runs on another host
STRATA_TRANSFORM_SIGNING_SECRET=<stable-secret> # unset: a random one per process, so signed URLs die on restart and differ across replicas
# Multi-tenancy (optional but recommended for >1 team)
STRATA_MULTI_TENANT_ENABLED=true
STRATA_REQUIRE_TENANT_HEADER=true
STRATA_TENANT_HEADER=X-Tenant-ID # match what your proxy injects
Run the server normally:
Rotating the proxy token and the signing secret¶
STRATA_PROXY_TOKEN holds one value, with no overlap window: change it on
the proxy and the server together, and requests that arrive between the two
changes get a 401. Changing STRATA_TRANSFORM_SIGNING_SECRET invalidates
every signed URL already issued, so a build in flight on a signed worker
fails; rotate it between runs and give every replica the same value.
What service mode changes¶
Compared to personal mode:
- No default artifact dir. The artifact store exists only when
STRATA_ARTIFACT_DIRis set, even when its metadata and blobs live elsewhere. Service mode refuses to start with a DSN, a non-local blob backend orSTRATA_SERVICE_WRITES_ENABLEDand noSTRATA_ARTIFACT_DIR. Without any of those it runs scan-only, with no artifact store. - What is durable, and where. The metadata store (Postgres with
STRATA_ARTIFACT_METADATA_DSN, otherwiseartifacts.sqlitein the artifact directory) holds artifact metadata, names, aliases, publications, pins, API keys and the server-managed worker registry. The blob store holds artifact bytes: the configured bucket, orblobs/in the artifact directory with the local backend. Notebooks live inSTRATA_NOTEBOOK_STORAGE_DIR. Back up those three. The artifact directory itself is node-local working space, and the SQLite file and local blobs are the only durable things in it, so with a DSN and a bucket it holds nothing that needs a backup. The row-group cache (STRATA_CACHE_DIR) is a cache and can always be thrown away. - Reads work; direct writes are off by default. Clients can read
results - scan/stream a table, fetch an artifact's data
(
GET /v1/artifacts/{id}/v/{n}/data), and resolve a dataset by name (GET /v1/names/{name}) - all tenant-scoped and ACL-gated. Direct write endpoints (put,set_name, …) are disabled at the surface by default; the platform decides what gets materialized via server-side transforms (transforms_config/[tool.strata.transforms]). To let authenticated clients publish directly - the shared-research-store pattern - opt in withservice_writes_enabled(see below). - ACLs apply.
acl_configdeny / allow rules gate every table a request reads or writes: a scan throughPOST /v1/materialize, an artifact built from a table, cache warming, and writing an artifact into a table (POST /v1/artifacts/{id}/v/{n}/export). Deny rules cannot be bypassed by allow rules, deny-first evaluation. A stream (GET /v1/streams/{id}) is readable only by the principal that started it, in the tenant it started it in, and admin endpoints such asPOST /v1/cache/clearneed their scope. - Per-tenant resources when multi-tenancy is on. Each tenant gets its own QoS limiter pool, its own metric labels, and its own cache keying, bulk queries from tenant A can't starve tenant B's dashboards.
Running more than one replica¶
Several service-mode nodes can sit behind one proxy when they share everything durable:
- One metadata store and one blob store. Every node sets the same
STRATA_ARTIFACT_METADATA_DSNand the same bucket (see above). The worker registry lives in the metadata store, so a worker added through/v1/admin/notebook-workerson one node is offered by every node from its next request. Each node can hold up to 41 connections to the database, so size itsmax_connectionsfor every node at once (see Sharing one artifact store). - One notebook storage. Every node mounts the same
STRATA_NOTEBOOK_STORAGE_DIR(NFS, EFS, Filestore or similar), so any node can open any notebook. A notebook's environment is built inside its directory (or inSTRATA_NOTEBOOK_SHARED_ENV_DIR, which then has to be shared too), so run the same image on every node. - The same secrets.
STRATA_PROXY_TOKENandSTRATA_TRANSFORM_SIGNING_SECRETmust match on every node, or a URL signed on one node fails on another. SetSTRATA_NODE_ADVERTISED_URLper node to its own address, so stream fetches and a remote cell's console reach the right node (see Running several nodes). - Session-affine routing. A notebook session, which
POST /v1/notebooks/openor/createreturns, lives in the memory of the node that opened it, together with its WebSocket and running cells. Every later request for it must reach that node; any other node answers404. Route each user to one node, with a sticky cookie or a hash of the principal header, so the open and everything after it land together.
Not supported:
- Moving a live session. When a node goes away its sessions go with it. Opening the notebook again on another node starts a new session; cells whose inputs did not change are cache hits when they run again.
- One notebook open on two nodes at once. Each node keeps its own session and writes the same files, with nothing coordinating them.
- Cluster-wide views of per-node state. Cache warm jobs
(
/v1/cache/warm/jobs*), QoS limits, rate limits and metrics are per node; sum the metrics in your dashboards.
Multi-tenancy¶
STRATA_MULTI_TENANT_ENABLED=true activates per-tenant isolation.
With STRATA_REQUIRE_TENANT_HEADER=true, requests without a tenant
header are rejected. Under api_key auth the tenant is the key's own
and an X-Tenant-ID the caller sends is ignored, so a key cannot pick
another tenant's limiters or log attribution; under trusted_proxy it
is the header the proxy sets. The tenant ID is validated as 1–64
alphanumeric / _ / - characters and hashed into:
- Cache keys: tenant A and tenant B can scan the same Iceberg table and never see each other's row-group cache entries.
- Cache directories: per-tenant subdirs under the row-group cache dir. Artifacts are not split by directory; each row records its tenant, and reads are filtered by it.
- QoS limiters: interactive + bulk semaphores per tenant.
- Notebooks: each tenant's notebooks live under
<notebook_storage_dir>/<tenant>/, and/discover,/open,/createand the imports stay inside it. A live session belongs to the tenant that opened it; to any other tenant its REST routes, WebSocket and MCP tools answer as for an unknown session. A caller the proxy sends without a tenant header is the default tenant (_default) for both: its notebooks live under<notebook_storage_dir>/_default/and its sessions are visible to that tenant only.admin:*sees every tenant's notebooks and sessions. This keeps tenants out of each other's notebooks, but cells still run on one host: it is a convenience for small, trusted tenants, not an isolation boundary (seeSTRATA_NOTEBOOK_HARNESS_USERand remote workers for what a cell can reach). - Cache warm jobs: a background job from
POST /v1/cache/warm/asyncis listed, read and cancelled (/v1/cache/warm/jobs*) only by its own tenant;admin:*sees every tenant's. - Metric labels: under principal auth Prometheus output names no
tenants, tables or principals:
/metrics/prometheusis unauthenticated for scrapers, so the per-tenant series (strata_tenant_*,strata_build_tenant_*) and the per-table series are left out, and the AI usage series (strata_ai_*) are per-model totals with notenantorprincipallabel. Read per-tenant usage fromGET /v1/admin/tenants(admin:tenants) and per-table usage fromGET /metrics/tables(admin:*).
A tenant registry tracks active tenants (LRU-bounded; only a tenant with nothing in flight is evicted). Implementation details are in the source tree if you need to extend the tenant-scoping behavior.
ACLs¶
acl_config is a pyproject.toml block:
[tool.strata.acl_config]
# Action when no rule matches. Defaults to "allow" -- set it to "deny"
# if you want an allowlist, or every table not named below stays readable.
default = "deny"
# Deny rules evaluate first; explicit denies cannot be bypassed.
# "*:" covers every address of a table (see the configuration reference).
[[tool.strata.acl_config.deny]]
principal = "guest@example.com"
tables = ["*:internal.*"]
[[tool.strata.acl_config.allow]]
principal = "analyst@example.com"
tenant = "marketing" # optional; omit to match any tenant
tables = ["file:marketing.*", "file:public.*"]
A rule matches when the principal matches (* matches any, and is the
default), the tenant matches if the rule names one, and at least one table
pattern matches. tables is required: a rule with no patterns can never match
anything, so the server refuses to start rather than loading a rule that does
nothing. A key Strata does not recognize (tenants for tenant, say) is
refused at startup too, rather than dropped from a rule that is then
wider than it reads.
Evaluation: deny rules → allow rules → default. Wildcard and principal matching semantics are documented in source for anyone extending the ACL engine.
Under principal auth a denied table, whether scanned or used as a transform
input, is refused before planning with 404 Table not found, so the answer
does not reveal that the table exists. Set
STRATA_HIDE_FORBIDDEN_AS_NOT_FOUND=false to get 403 instead.
The cache is shared, so the same rules gate reading a stored result: its data,
metadata, lineage and dependents, publishing it, and exporting it into a table
(POST /v1/artifacts/{id}/v/{n}/export). An export copies the result into a
table the caller can then scan, so it is refused both when the caller is denied
the target table and when the caller is denied a table the result was read from.
Rules have no separate write permission: a table a caller may read, it may export
into, unless another tenant owns it.
A result is gated by every table in its lineage, not only the ones it read
directly. A transform over a cached scan of a denied table is denied too, as is
anything computed from that result, and the same holds when such a result is
used as a transform input. The check follows the store's recorded input edges up
to 100 steps back; a result with a deeper lineage is refused with 403, since a
table past that point cannot be checked. An ancestor that retention has already
collected cannot be checked either, so a deny rule only reaches results whose
lineage is still in the store.
Scope-gated endpoints¶
A few operations require a specific scope under trusted-proxy or API-key
auth (admin:* satisfies any of them):
| Scope | Gates |
|---|---|
admin:cache |
POST /v1/cache/clear, GET /v1/cache/entries, GET /v1/cache/histogram, GET /v1/debug/cache/inspect, POST /v1/metadata/cleanup |
admin:tenants |
GET /v1/admin/tenants and GET /v1/admin/tenants/{tenant_id} |
admin:notebook-workers |
The server-managed worker registry, every /v1/admin/notebook-workers* route (list, replace, add, update, delete, refresh, reload); a personal server, which has no principal auth, serves them to whoever can reach it |
admin:notebooks |
Quiescing a notebook or project and releasing it (POST /v1/notebooks/{id}/quiesce and /release, POST /v1/projects/{path}/quiesce and /release) |
admin:* |
Garbage collection (POST /v1/artifacts/gc, which sweeps the whole store), reading another tenant's GET /v1/artifacts/usage / stats, the server-wide log buffer (GET /v1/logs, GET /v1/logs/stream), per-table metrics (GET /metrics/tables*), and the process diagnostics under /v1/debug/* (latency, GC pauses, pools, connections, memory, rate limits) |
admin:registry |
POST /v1/registry/pending/approve and .../reject - deciding protected-alias changes |
artifacts:pin |
Pinning and unpinning a version against garbage collection (POST / DELETE /v1/artifacts/{id}/v/{n}/pin) |
artifacts:publish |
Minting, editing and withdrawing a publication (POST /v1/artifacts/{id}/v/{n}/publish, PATCH / DELETE /v1/publications/{token}) |
artifacts:write |
Publishing in service mode (put / set_name / set_alias / tags, and name on POST /v1/materialize) when service_writes_enabled=true. See below. |
notebook:read |
Every notebook GET over REST, and observing a notebook over its WebSocket (sync, previews, profiling) |
notebook:write |
Changing a notebook without running anything: creating, editing, reordering and deleting cells, and setting mounts, connections, workers, env, timeout, name and variants; also opening, creating, importing, closing and deleting notebooks, a cell's test source, promote, and recents validation. REST and WebSocket alike. |
notebook:execute |
Running code or changing its environment: executing a cell or its tests, run-all, dependency changes and environment sync, requirements imports, the Python version, SSH workers, the inspect REPL and widget updates. REST and WebSocket alike. |
The notebook scopes are checked against one table for both transports
(strata.notebook.scopes), so a viewer holding only notebook:read can't run a
cell over REST any more than over the socket. An operation the table doesn't
classify requires notebook:execute.
Registry approval additionally enforces separation of duty: the
principal who requested a protected-alias move cannot approve it
themselves unless they hold admin:*. The registry audit read
(GET /v1/registry/audit) and the events feed that follows it
(GET /v1/events) are tenant-scoped - a principal sees only its own tenant's
history; admin:* sees the whole store.
Authenticated write-back: the shared research store¶
Preview
Authenticated write-back is a preview feature - it deliberately re-opens writes in service mode, which is security-sensitive. It's off by default, auth-required, scope-gated, tenant-scoped, and audited, but the surface is new and may change. Evaluate it before relying on it in production. The server logs a notice at startup when it's enabled.
By default service mode is read-only to clients, computation goes through server-side transforms. But a common deployment wants the inverse: a team of researchers, each driving their own notebook, who publish processed datasets to one central store so a dataset computed once is available to the whole team.
service_writes_enabled opts into that. It lets authenticated clients write
directly - put, set_name, set_alias, tags - under a strict contract:
- Opt-in and auth-required. Off by default; setting it requires
auth_mode=trusted_proxy(enforced at startup), so every write is attributable. - Scope-gated. Publishing requires the
artifacts:writescope in the proxy-issued token (admin:*also satisfies it). Members without it stay read-only. - Tenant-scoped (team = tenant). A write lands in the caller's tenant and
can't target another, so teammates share a namespace and other teams are
isolated. A name, alias or tag aimed at another tenant's artifact answers
404 Artifact not found, the same as a missing artifact or a read of it. An upload or import that names an artifact id another tenant already holds, at any version, answers409. The publishing principal is recorded in the registry audit.
STRATA_DEPLOYMENT_MODE=service
STRATA_AUTH_MODE=trusted_proxy
STRATA_PROXY_TOKEN=<shared-secret-with-proxy>
STRATA_ARTIFACT_DIR=/path/to/dir # required with any artifact store, blob backend or not
STRATA_MULTI_TENANT_ENABLED=true # team = tenant
STRATA_SERVICE_WRITES_ENABLED=true # the opt-in
The proxy injects X-Strata-Scopes: artifacts:write for principals allowed to
publish. The publish → consume loop then looks like:
# Researcher A (team-a, artifacts:write) publishes a processed dataset:
strata.put(inputs=[], transform={"ref": "clean@v1"}, data=cleaned, name="team/cleaned-events")
# Any teammate (team-a) resolves the name to its current artifact and reads it:
info = strata.resolve_name("team/cleaned-events") # {artifact_uri, version, …}
# Other-team principals (team-b) cannot resolve team-a's name - tenant isolation.
Connecting a notebook to the shared store¶
Each researcher runs their own notebook, which computes locally but points
its ambient strata client at the central store via
STRATA_NOTEBOOK_REMOTE_STORE_URL. The notebook's own cell outputs and
provenance stay local; only what a cell explicitly publishes
(strata.put(name=…)) goes to the shared store.
# On each researcher's notebook server:
STRATA_NOTEBOOK_REMOTE_STORE_URL=https://store.team.example
# Auth the remote store needs (set via env, not committed config):
STRATA_NOTEBOOK_REMOTE_STORE_HEADERS='{"X-Strata-Proxy-Token":"…","X-Strata-Principal":"alice@team","X-Tenant-ID":"team-a","X-Strata-Scopes":"artifacts:write"}'
In a fully proxy-fronted setup the notebook's requests instead flow through the
same auth proxy, which injects identity, and notebook_remote_store_headers
can be omitted.
On a shared notebook server several members run cells, and the static
headers name one identity for all of them. When a request carries a principal,
the server sends that caller's id as X-Strata-Principal to the remote store,
replacing the one in the static headers. This covers results offered to the
team cache, promotions, registry reads and approvals from the Registry tab, and
a cell's own ambient client. The team cache's "computed by" and the registry
audit then name the member. The remote store still has to accept that principal:
the static headers are what authenticate the server to it. Set
STRATA_NOTEBOOK_REMOTE_STORE_FORWARD_PRINCIPAL=false for a store that expects
one fixed service identity. Personal mode has no principal and is unaffected.
The team cache: sharing results nobody named¶
Everything above is explicit publish - a researcher decides a dataset is worth sharing and names it. That leaves out the expensive middle of a pipeline, because nobody names their intermediate results, and that is exactly where the recomputation is.
STRATA_NOTEBOOK_TEAM_CACHE_ENABLED=true closes that gap. On a local cache
miss the notebook asks the shared store whether anyone has already run this
exact computation, and on a hit it fetches the result instead of running the
cell. After a cell does run, its outputs are offered back to the store so the
next teammate hits.
# On each researcher's notebook server, alongside the two settings above:
STRATA_NOTEBOOK_TEAM_CACHE_ENABLED=true
It works because a cell's provenance key -
sha256(sorted_input_hashes + source_hash + env_hash) - contains no notebook id
and no cell id, and env_hash leaves out the notebook's own project name in
uv.lock. Two people running the same source over the same inputs in the
same environment already arrive at the same hash; the store just had no way to
be asked.
Lineage answers the follow-up. Opening a result's lineage (from the cell's
artifact strip, or the registry panel) shows each step with who computed it,
on what, and what it cost - alice@lab · cpython-3.14-linux-x86_64 · 38s.
Each step also carries its environment identity - env:3d731494 (which
package set) alongside the platform (on what). Those two together are what
answers the question a shared cache generates on day one: "you got a hit and I
didn't - why?" The answer is almost always that the identities differ, and
until they were readable there was no way to see it.
On a shared notebook server the author column names the member who ran each cell: every artifact the run stores, variables and displays alike, records the principal of the request that started it (a WebSocket run, the REST execute route, or an MCP tool call), even when another member is watching the same notebook. Promoting a result keeps those authors, so the team store says who computed each step, not who promoted it. In personal mode there is no authenticated identity, so the column fills in only for steps that came from the store and stays blank throughout on a solo notebook. The platform and environment identity are recorded on every run, so what a shared store changes is not that they appear but that they stop being the same value on every row. A graph where one step ran on someone else's machine, in a different environment, is the case these columns exist to make visible.
All of it is best-effort and recent: artifacts produced before these fields existed show blanks, and re-running the cell is what fills them in.
The notebook's profiling panel splits the savings once a team hit happens: "Cache savings ~12m" alongside "From your team ~8m (4 hits, alice, bob)". The team figure is priced by what the publisher's run actually cost, carried on the artifact - whoever gets the hit never ran the cell, so their own history holds no comparable number and a locally-derived estimate would credit zero for exactly the case the shared store exists to create.
What to know before switching it on:
- It is a separate opt-in from the URL, deliberately. Wanting a shared store
to publish to is not the same as wanting one to silently source results
from. Enabling it without
STRATA_NOTEBOOK_REMOTE_STORE_URLis rejected at startup rather than left silently inert. - Publishing needs
artifacts:write. Members without the scope still get team cache hits; they just do not contribute. That is a supported configuration, and the notebook logs a warning when a publish is refused so it does not look like the cache is simply empty. - It never fails a cell. A store that is unreachable, refusing, or missing one of a cell's outputs ends in "run it locally".
- A notebook whose environment does not match its lockfile will not publish.
A failed
uv synckeeps the previous venv and stays usable on purpose, so a transient network failure does not lock you out of your own notebook - but provenance is computed fromuv.lock, which has moved on, so anything produced afterwards would be stamped with an environment it was not built in. Locally that is your own problem; published it is permanent and everyone's, because the store keeps the first writer's result. The cell still runs and its result is still cached locally; only the publish is refused, with a warning naming the cell. Re-run the environment sync to resume publishing. - Point it at an authenticated store. Without auth there is no tenant, so
every team shares one flat namespace, and no principal, so every result
arrives authored by nobody. The server warns at startup if the team cache is
on with no
notebook_remote_store_headers. - The environment key does not include the platform, but the artifact
records it.
uv.lockresolves to different wheels on macOS-arm64 and Linux-x86_64, and the env hash covers the lockfile only. That is deliberate, since hashing the platform would drop cross-machine hit rate to roughly zero and delete the feature in order to protect it. So a hit can legitimately cross machines, and every artifact carries the interpreter and hardware that produced it (cpython-3.14-linux-x86_64), reported by the process that actually ran the cell: the notebook venv locally, or the worker's interpreter for a remote cell. A team hit says which platform it came from rather than crossing one silently. In practice a shared cache handing the whole team one number is the more reproducible outcome, but if your work depends on hardware-identical results, leave the team cache off for now.
Sharing on purpose: promoted and strata artifact promote¶
Offering every downstream-consumed variable of every successful cell is right for a server whose whole purpose is a shared cache. It is wrong for a personal server: there, it means every intermediate a researcher ever computed lands in the team's store, whether or not they meant to share it.
STRATA_NOTEBOOK_TEAM_CACHE_PUBLISH is the setting between "everything" and
"nothing":
| Value | Offers outward | Pulls |
|---|---|---|
all (default) |
every consumed variable of every successful cell | yes |
promoted |
nothing automatically | yes |
off |
nothing | no |
Under promoted, a result reaches the team when someone says so:
strata artifact promote nb_taxi_cell_c2_var_model \
--to https://store.example --name taxi/model --alias champion
--tag KEY=VALUE (repeatable) tags the promoted version, --header 'Name:
value' (repeatable) adds a header for --to beyond the STRATA_STORE_TOKEN
bearer, and --max-depth (default 10) bounds how far the chain is walked.
The chain travels with it, and has to. The cache is keyed by provenance, so
each ancestor that arrives is a hit for the next person whose cell computes the
same thing - promoting the result alone would share the answer and none of the
work. Each row the promotion writes is stamped with its name (an nb_promotion
tag, kept out of the registry's tag lists), so a colleague's hit on any of it
says which promotion it came from: their profiling panel lists it under "Via
promotions". A row the store already held, from a cache publish or an earlier
promotion, keeps whatever it had. A protected alias (registry_protected_aliases) queues for approval
rather than moving, and the command says so.
The Registry tab and the per-cell strip describe that store too. They read the local one until a team store is configured, then forward to it with the server's remote-store headers: the notebook names things there, so describing the local store would show an empty registry. Every registry route forwards the same way (names, aliases, tags, lineage and approvals), so a promotion from the tab lands where the tab reads, and a protected alias filed from one person's server gets approved from another's.
A cell can promote one of its own upstream results without leaving the notebook, naming it the way the cell reads it:
off is the whole feature off without unsetting
STRATA_NOTEBOOK_REMOTE_STORE_URL, which a cell's ambient strata client
still needs.
Promoting into an Iceberg table¶
A tabular result can also become a table in the team's warehouse, so tools
outside Strata read it as a table and a notebook reads it with @table:
strata artifact promote nb_taxi_cell_c1_var_features \
--to https://store.example --name taxi/features --alias champion \
--table "s3://lake/warehouse#taxi.features"
The team store does the writing, with its own catalog settings, since that is
where the bytes and the warehouse credentials are. --table takes the same
forms @table reads: <warehouse>#namespace.table, or namespace.table in the
store's configured catalog.
- The first promotion creates the table and appends. Later promotions overwrite
it, so the current snapshot is always one version. A table Strata did not
write is refused rather than replaced. A notebook's
@tableon it goes stale when the next version is written, as for any other table. - A table belongs to the tenant whose export last wrote it. Another tenant's
export to it is refused like a denied table (
404 Table not found, or403withSTRATA_HIDE_FORBIDDEN_AS_NOT_FOUND=false), even when the ACL lets that tenant read it. Anadmin:*export is not tenant-scoped, and leaves a table only another unscoped export can replace. - Each snapshot's summary names what it holds:
strata.artifact_id,strata.version,strata.provenance_hash,strata.promoted_bywhen the store knows the caller, andstrata.tenantwhen the export was tenant-scoped. (An overwrite commits a delete and then an append; the append is the snapshot that holds the version.) - A new version may add columns or widen a type. One whose schema the table cannot evolve to is refused before anything is written.
- The alias is an Iceberg tag on the snapshot. Moving the alias later, including approving a protected one, moves the tag, as long as that version was written to the table. When two moves overlap, the tag ends on the version the alias ends on.
The same write is strata artifact export --table <table> <ref> against a local
store (--alias tags the new snapshot, --by is recorded as
strata.promoted_by), and POST /v1/artifacts/{id}/v/{version}/export with
{"table": ..., "alias": ...} for a platform that exports once a promotion
lands. Only Arrow tables can be written; a JSON value, a pickle, an array or a
scalar is refused.
What a cell can read¶
A cell is arbitrary Python, spawned by default with the server's whole
environment. On a laptop that is right and there is nothing to protect. On a
shared server it means every member who can run a cell can read
STRATA_NOTEBOOK_REMOTE_STORE_HEADERS, STRATA_PROXY_TOKEN, worker tokens and
every data-source credential the server holds - from os.environ, from
/proc/<pid>/environ, or from any file the server process can open.
Two settings close this, and they only work together.
Who a cell is: STRATA_NOTEBOOK_HARNESS_USER¶
A service-mode server refuses to run cell code on its own host unless you say how that is safe. There are two answers:
- Run cells on another machine. Assign cells a server-managed worker (a pool machine or a remote worker). This is the recommended answer: a different host is isolation that needs nothing arranged on this one.
- Run cells as a separate OS user. Set
STRATA_NOTEBOOK_HARNESS_USERto a user that exists on the server host. Cells then cannot read the server's environment through/proc, nor state the server keeps owner-only (below).
Without either, a cell that would run on the server host fails with a message
naming both. Cache hits are still served, since a hit starts no cell code.
Personal mode is unaffected. An embedded:// executor worker counts as this
host: it runs the harness in place.
Switching users needs the privilege to do it, so the server runs as root and drops to the harness user for every process that runs cell code: the cold and R harnesses, the batch harness behind Run All, the warm pool workers, the inspect REPL, cell tests and the R package restore. It is POSIX only. What the harness user needs:
| Path | Access |
|---|---|
| notebook directories (and, with multi-tenancy, each tenant's directory above them) | read, or traverse for the tenant directory |
each notebook's .venv |
read and execute |
| the Python interpreter behind the venvs | read and execute |
each notebook's .strata/ |
traverse |
The interpreter row is the one that catches people. uv installs managed Pythons
under the installing user's home, which for a root server is /root and
unreadable by anyone else, so every cell fails with Permission denied on the
venv's python. Install them somewhere world-readable with
UV_PYTHON_INSTALL_DIR=/opt/uv-python. Nothing else needs arranging: the
per-run directories a cell writes into are handed to the harness user, the
harness runs the venv's interpreter directly rather than through uv run, and
the artifact store is never read by the cell. The server, in turn, never
unpickles what a cell stored: prompt and SQL cells do not load a pickled value,
and cell tests load their inputs in the test process.
A cell run this way still shares the host's kernel and sees what any local user can. A process a cell starts in the background is not stopped when the cell finishes: it keeps running as the harness user, so the server never writes by path into a directory it has handed over. A notebook that needs more isolation than that wants a worker on another machine.
One harness user serves every tenant, so it is not a tenant isolation
boundary. It keeps cells away from the server; it does not keep one tenant's
cells away from another's. Every cell runs as the same user, and that user has
to read every notebook directory, so a cell can read any tenant's notebook
source and data files, and the bytes another notebook fetched (.strata/fetch)
or mounted (.strata/mount_cache), given the path. Tenants that must not see
each other need separate servers (or containers, each with its own storage),
or every cell on workers that serve one tenant each.
What Strata keeps from the harness user, and from any other account on the host, is its own state. It creates these owner-only, and narrows any an earlier release left wider when it next opens them:
| State | Mode |
|---|---|
STRATA_ARTIFACT_DIR, its artifacts.sqlite (with -wal / -shm) and blobs/ |
directories 0700, files 0600 |
STRATA_CACHE_DIR |
0700 |
each notebook's .strata/ |
0711: the harness user passes through to its per-run directories and inputs, but cannot list it |
in it, artifacts/, console/, runtime.json and environment_jobs.json |
0700 / 0600 |
| each tenant's notebook directory under the storage root | 0711 |
Strata only removes permission bits, so a stricter umask stands. Publication
tokens are stored as their SHA-256, so even a copy of artifacts.sqlite holds no
working link. Not covered, and yours to make owner-only: the server's config file
(a pyproject.toml holding proxy_token), the server's home, and
STRATA_METADATA_DB.
SQL cells are different: their queries run inside the server process, not as
the harness user, so the server checks which database file a SQLite or DuckDB
connection opens. After resolving ../ and following links, its path must be
inside the notebook's own directory, or else pass the mount-root rule below and
not be inside the server's state (the artifact directory, the cache directory,
the metadata database's directory, the notebook storage directory or the
server's home). The server opens the resolved file, checked again just before
opening, so a link changed after the first check is not followed. A SQLite
connection may not use uri, whose parameters can name
any file. Saving such a connection fails with a 400, and one already in
notebook.toml fails the cell and the connection's schema listing, naming it.
The keys the server sets on a connection it opens (confine_to,
mount_sources, catalog_properties) are dropped from a connection block.
A DuckDB SQL cell is also confined. Once its connection is
set up, it can reach only its own database, the roots of the mounts it reads,
and the locations of the catalog tables it reads; file access outside those is
refused, and the cell cannot turn it back on. So read_text of a server file,
COPY ... TO, and ATTACH of another path all fail. A catalog table whose data
files live outside its own location cannot be read this way.
A mount's root is therefore as much of the disk as the cell may read, as the
server, so a SQL cell refuses a local (file://) mount root, after following
links, that is / or has fewer than two path components (/data); that holds
or is inside the server's state (the artifact directory, the cache directory,
the metadata database's directory, the notebook storage directory or the
server's home), other than the notebook's own directory; or that is inside the
server's home, /proc, /sys or /dev. The cell fails
naming the mount. Python cells mount the same roots unchecked: they run as the
harness user, whose own file permissions decide what a root exposes. A SQLite cell also
runs in the server process: in a write cell ATTACH, DETACH and VACUUM are
refused, since they reach other files (a read cell runs only reads already).
Postgres, Snowflake and BigQuery cells send the query to their database server.
The server reads a BigQuery connection's credentials_path and
write_credentials_path key files itself, so each must be a file in the
notebook's own directory, after following links; one elsewhere fails the cell,
the schema listing and saving the connection. A BigQuery connection must name
credentials_path: without a key file the driver would use the server's own
Google credentials (its application default credentials), so one with none is
refused in the same three places.
A ${VAR} in a connection's auth reads the notebook's env, never the server's
environment, which would otherwise go to whatever host the notebook names; a
secret the server holds reaches a connection only as a named credential.
Prompt cells call their model from the server process too, and show the author
what came back. A base_url from the notebook's [ai] section is checked like
an @fetch URL: http(s) only, and a host on a loopback, private or link-local
address (the cloud metadata address among them) is refused unless it is named in
STRATA_NOTEBOOK_FETCH_ALLOWED_HOSTS, the same list @fetch uses. The
connection goes only to an address that passed the check, so it ignores proxy
settings. STRATA_AI_BASE_URL is yours and is not checked, nor is a notebook
naming that same URL or a provider's default one. Your STRATA_AI_API_KEY is
sent only to those URLs; a notebook naming another host must bring its own key.
Installing a notebook's Python packages runs as the server's user too, and building
a package from a source distribution runs that package's build backend, code
from wherever the package came from, with the server's environment. In service mode every uv command
a notebook runs (sync, add, lock, the uv run that starts a cell)
installs wheels only: a dependency with no wheel for the notebook's Python
fails to resolve, and uv's message says a wheel is required. Personal mode
still builds from source.
R has no wheels-only switch: renv::restore() builds CRAN packages from source
on Linux, running their configure scripts, and every Rscript started in a
notebook directory sources the notebook's .Rprofile. So in service mode that
code runs as the harness user, never as the server. The restore on notebook
open runs as the harness user with the filtered environment below, and the
notebook's renv/ directory (and, with the shared environment backend, the
shared R library and package cache) is handed to that user to write. The R
package listing in the environment panel runs as the harness user too. With no
harness user nothing is restored, and the server log says why. Installing an R
package from the notebook (renv::init, adding a package) is refused, since it
writes renv.lock and .Rprofile into the notebook directory: commit the
package in renv.lock where the notebook is authored, and the server restores it
when the notebook opens. Personal mode is unchanged.
What a cell is given: STRATA_NOTEBOOK_HARNESS_ENV_ALLOWLIST¶
The allowlist narrows the environment a cell receives:
# Only what cells actually need; everything else stays with the server.
STRATA_NOTEBOOK_HARNESS_ENV_ALLOWLIST=AWS_*,HF_TOKEN
Entries are exact names or a prefix with a trailing *. The essentials a
subprocess cannot start without are always included: PATH, HOME, the temp
and locale variables, and UV_*, PYTHON*, VIRTUAL_ENV, R_* and
RSTUDIO_*, except names that look like credentials (containing KEY,
SECRET, TOKEN, PASSWORD or CREDENTIAL), UV_PUBLISH_*,
UV_INDEX_*_USERNAME, and the index URLs (UV_INDEX, UV_INDEX_URL,
UV_EXTRA_INDEX_URL, UV_DEFAULT_INDEX, UV_INDEX_*_URL), which can carry a
login as https://user:password@host/simple. So a private index's login, which
the server's uv sync needs, does not reach cells. STRATA_* is dropped
unless named exactly - a prefix rule broad enough to catch a credential by
accident is the failure the setting exists to prevent. A name you list exactly
always passes. It applies to every process that runs cell code, the same list
as above.
The list stays short because a cell's own configuration does not come through
the process environment. [env] in notebook.toml and mount credentials
travel in the cell manifest and are applied inside the harness.
On its own the allowlist is not isolation. A cell running as the server's user
can still read the server's whole environment from /proc/<server pid>/environ,
because it is that user. The allowlist decides what a cell is handed; the
harness user is what stops it taking the rest.
A notebook's secret manager¶
A notebook's [secret_manager] block names where its secrets come from, and the
server fetches them with its own Infisical credentials (INFISICAL_CLIENT_ID /
INFISICAL_CLIENT_SECRET or INFISICAL_TOKEN). Two consequences on a shared
server:
- The host is yours. The server logs in only at
INFISICAL_HOST(or the publichttps://app.infisical.comwhen unset). A notebook whosebase_urlnames another host fails to fetch, with a message namingbase_url, and no login is attempted: otherwise the author could have the server send its credentials anywhere. - The project is the author's.
project_id,environmentandpathcome from the notebook, and Strata has no list of projects a notebook may read. Any notebook author can read every secret the server's machine identity can read. Scope that identity to what every author on this server may see, or run a separate server (with its own identity) per group that may see more.
Migrating from personal mode¶
If you've been running personal mode and want to grow into service:
-
Decide on the auth boundary. Anything from "Cloudflare Access in front of a Fly app" to "OIDC behind an enterprise ingress": the only requirement is that the proxy can inject the four headers above and that Strata is otherwise unreachable.
-
Pick an artifact backend. Local-disk artifacts don't survive container restarts cleanly in multi-replica setups. Configure one of
STRATA_ARTIFACT_BLOB_BACKEND=s3|gcs|azureand the matching credentials, and keepSTRATA_ARTIFACT_DIRset: service mode refuses a non-local blob backend without it. See Artifact Storage. -
Flip the mode in env or
Boot will fail with a clear error if anything's missing, that's the fail-closed property at work.pyproject.toml: -
(Optional) Add multi-tenancy. Once multiple teams are using the same instance and you want isolation, flip
STRATA_MULTI_TENANT_ENABLED=trueand start injectingX-Tenant-IDfrom the proxy. -
(Optional) Add server-side transforms. Configure the
[tool.strata.transforms]block inpyproject.tomlto expose the computations you want the platform to run on the client's behalf. The block is an allowlist:enabled = true(orSTRATA_TRANSFORMS_ENABLED=true) alone refuses every transform, so list each one. The built-in SQL transform runs in the server process:
[tool.strata.transforms]
enabled = true
[[tool.strata.transforms.registry]]
ref = "duckdb_sql@v1"
executor_url = "embedded://local"
The notebook executor in the demo stack is another example.
The demo compose stack is a working starting point you can fork:
swap nginx.conf for your real auth proxy config, move the
artifact blobs to S3, and you have most of what production needs.