Distributed Workers¶

Strata Notebook can dispatch individual cells to remote machines via the executor protocol. A worker is any HTTP endpoint that accepts cell source code and inputs, runs them, and returns the outputs. You bring the compute; Strata handles the routing, serialization, and caching.

How it works¶

┌─────────────────────┐    multipart POST     ┌──────────────────────┐
│  Strata Notebook    │ ──────────────────►  │  Worker (HTTP)        │
│  (orchestrator)     │                       │  remote_executor.py   │
│                     │  ◄──────────────────  │                       │
│  routes cell to     │    gzipped bundle     │  runs harness.py      │
│  @worker annotation │    (outputs + blobs)  │  returns results      │
└─────────────────────┘                       └──────────────────────┘

You annotate a cell with # @worker my-gpu.
Strata looks up my-gpu in the notebook's [[workers]] config.
Cell source + serialized input variables are sent as a multipart POST /v1/execute.
The worker runs the cell in a subprocess and returns outputs as a gzipped bundle.
Strata stores the outputs as artifacts; cache hits work identically to local cells.

Cells run in the worker's Python environment, so install your workload dependencies (torch, datafusion, sentence-transformers, etc.) into the worker image before launching it. Unlike Strata's own server, the worker process does not require a uv-managed env - it can be pip-installed into a plain Docker image.

For the wire-level contract - request envelopes, response bundle format, error codes, the pull-model with signed URLs - see the Executor Protocol reference. This page covers deployment and registration; that one covers the bytes on the wire and is what you'd implement against to write a custom worker that doesn't use strata-worker.

Quick start: run a worker locally¶

Start by getting a worker running on your own machine. This verifies your install before you spend time on a cloud deploy, and the same # @worker name annotation works against both local and cloud workers.

1. Start the worker:

uv run strata-worker --port 9000

You should see uvicorn start up:

INFO:     Started server process [12345]
INFO:     Uvicorn running on http://0.0.0.0:9000

2. Verify it's healthy:

curl http://localhost:9000/health

Expected response:

{
  "status": "healthy",
  "capabilities": {
    "protocol_versions": ["v1"],
    "transform_refs": ["notebook_cell@v1"],
    "features": {
      "notebook_protocol_version": "notebook-cell-v1",
      "output_format": "notebook-output-bundle@v1"
    }
  },
  "uptime_seconds": 5.2,
  "active_executions": 0
}

3. Register it in your notebook. Either through the Workers panel in the sidebar, or by editing notebook.toml:

[[workers]]
name = "local"
backend = "executor"
runtime_id = "local-dev"

[workers.config]
url = "http://127.0.0.1:9000/v1/execute"
transport = "direct"

4. Use it in a cell:

# @worker local
import platform
hostname = platform.node()

When the cell runs, the UI shows a pulsing "dispatching → local" badge during execution. The hostname artifact is what the worker process saw, not your laptop - confirming the cell really ran remotely.

Once this works locally, the cloud deploys below just change config.url from http://127.0.0.1:9000 to the worker's public URL.

Run cells on a machine you can SSH to¶

If you have a box you reach over SSH (a GPU machine, a bigger instance, one closer to the data), Strata can turn it into a worker for you in one step, without you deploying anything or hand-editing notebook.toml. Strata SSHes in, installs strata-worker if it's missing, launches it bound to the box's localhost, opens an ssh -L tunnel back to the notebook server, and registers it as a direct-transport worker. Cells then run on that box, cached by provenance like any other worker.

Driving a coding agent (the common case). When you're driving the notebook with a coding agent (Drive with a Coding Agent), tell it the target and it calls the connect_ssh_worker MCP tool:

You: use ssh user@gpu-box for the training cells
Agent → connect_ssh_worker(session_id, "user@gpu-box")

The agent's working agreement already tells it to reach for this when you hand it an SSH target, and to keep light cells local with # @worker local. To wire the worker up as the session opens instead, pass it to the on-ramp:

strata agent ./my-notebook --worker-ssh user@gpu-box

From the CLI. Against a running server (the tunnel is owned by the server, since that's where cells dispatch from), with the session id the server assigned:

strata worker add-ssh user@gpu-box --server http://localhost:8765 --session <session-id>
# tear it down (add --stop-remote to also stop the worker process on the box):
strata worker rm-ssh gpu-box --server http://localhost:8765 --session <session-id>

add-ssh makes the new worker the notebook default (pass --no-default to skip); --name overrides the name derived from the host, and --no-install fails fast rather than installing strata-worker on the box.

What to know:

Key-based SSH only. Strata runs ssh in batch mode and never handles passwords, so the target must authenticate non-interactively (an agent/key that works when you run ssh user@gpu-box yourself). A user@host, a bare host, or an ~/.ssh/config alias all work.
The first connect can take a minute while it installs strata-worker on the box (via uv tool install); reconnects adopt the already-running worker.
Security. The worker binds the box's 127.0.0.1 (never a public port) and is reachable only through the authenticated SSH tunnel. A per-worker bearer token is generated for defense-in-depth; it's held in the notebook server's memory and never written to notebook.toml.
A remote cell runs on the box's filesystem. Its file:// mounts and absolute paths resolve there, not on your machine, and cloud mounts use the box's own credentials. Results are cached under the remote environment's identity, so they don't collide with local runs.

Deploy to the cloud¶

Strata ships a reference executor as the strata-worker console script. Any platform that can run an HTTP service on a Python image will work. Two walkthroughs follow:

Platform	Best for	Cost model
Fly.io	CPU workloads (DataFusion, pandas-heavy pipelines) that need always-on or fast cold starts	Per-second VM billing; can scale to zero
Modal	GPU workloads (torch, embeddings, fine-tuning) that benefit from scale-to-zero	Per-second VM billing; cold-start ~10–30 s for GPU

You can register many workers per notebook; each cell picks its target independently. Mixing Fly (cheap CPU) and Modal (on-demand GPU) is a common setup.

Fly.io (CPU worker)¶

Prerequisites:

# Install the Fly CLI (macOS; see https://fly.io/docs/flyctl/install/ for others)
brew install flyctl

# Log in (opens a browser)
fly auth login

1. Create a project directory with three files:

my-strata-worker/
├── Dockerfile
├── fly.toml
└── .dockerignore

Dockerfile - installs strata-notebook and your workload deps from a uv-managed venv. Unlike strata-notebook, the worker entry (strata-worker) is not gated by Strata's runtime guard, so a plain pip install would also work - but the uv-python base image keeps tooling consistent across server + worker and drops a few hundred MB of build stage versus a source install.

FROM ghcr.io/astral-sh/uv:python3.12-bookworm-slim

ENV VIRTUAL_ENV=/opt/strata-venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

# Install strata-notebook + your workload deps into a uv-managed
# venv. Pin to an exact strata-notebook version in production so
# workers can't drift relative to the notebook server's expected
# protocol version.
RUN uv venv $VIRTUAL_ENV && \
    uv pip install \
      strata-notebook \
      "datafusion>=42" \
      "pandas>=2" \
      "pyarrow>=18"

EXPOSE 8080
CMD ["strata-worker", "--host", "0.0.0.0", "--port", "8080"]

fly.toml:

app = "my-strata-worker"
primary_region = "iad"  # pick a region close to your Strata server

[http_service]
  internal_port = 8080
  force_https = true
  auto_stop_machines = "stop"
  auto_start_machines = true
  min_machines_running = 0  # set to 1 for always-on; 0 for scale-to-zero

[[vm]]
  cpu_kind = "shared"
  cpus = 1
  memory = "1gb"  # bump for pandas/duckdb workloads

.dockerignore (keeps the build context small):

.git
*.pyc
__pycache__
.venv

2. Deploy:

fly launch --no-deploy   # first time only - creates the app, accepts fly.toml
fly deploy

The first build takes ~30 seconds (wheel download + layer assembly). Subsequent deploys reuse the layer cache and finish in seconds.

3. Verify the deployed worker:

curl https://my-strata-worker.fly.dev/health

Expect the same JSON as the local-worker step. If you see a 404 or timeout, jump to Troubleshooting.

4. Register in notebook.toml:

[[workers]]
name = "fly-cpu"
backend = "executor"
runtime_id = "fly-cpu-v1"

[workers.config]
url = "https://my-strata-worker.fly.dev/v1/execute"
transport = "direct"

Prerequisites:

pip install modal
modal token new   # one-time browser auth

1. Create worker.py:

import modal

# Modal's pip_install pulls wheels from PyPI. strata-notebook ships
# pre-built abi3-py312 wheels so no Rust toolchain is needed, and the
# worker entry isn't gated by the runtime guard (only strata-notebook is)
# so Modal's standard image stack works without going through uv.
gpu_image = (
    modal.Image.debian_slim(python_version="3.12")
    .pip_install(
        "pyarrow>=18.0.0", "pandas>=2.0.0", "numpy>=1.26.0",
        # Your workload dependencies:
        "torch>=2.3",
        "sentence-transformers>=3.0",
        # Pin to an exact version in production so the worker
        # protocol can't drift relative to the notebook server.
        "strata-notebook",
    )
)

app = modal.App("my-gpu-worker", image=gpu_image)


@app.function(gpu="A10G", scaledown_window=60)
@modal.asgi_app()
def gpu_executor():
    from strata.notebook.remote_executor import create_notebook_executor_app
    return create_notebook_executor_app()

2. Deploy:

modal deploy worker.py

Modal prints the deployed URL after the build finishes - something like https://your-username--my-gpu-worker-gpu-executor.modal.run. The first build with torch + sentence-transformers takes ~5 minutes; redeploys with no changes hit the layer cache and finish in seconds.

3. Verify:

curl https://your-username--my-gpu-worker-gpu-executor.modal.run/health

The first request after a scale-down cold-starts the container (~20–30 s for GPU). Health-check requests do not start the GPU function on most Modal plans - if /health returns immediately, the function is warm; if it doesn't respond, send a real cell from the notebook to wake it.

4. Register in notebook.toml:

[[workers]]
name = "modal-gpu"
backend = "executor"
runtime_id = "modal-a10g-v1"

[workers.config]
url = "https://your-username--my-gpu-worker-gpu-executor.modal.run/v1/execute"
transport = "direct"

Registering workers¶

Workers live in notebook.toml under [[workers]]. You can add them through the Workers panel sidebar (which writes the same TOML) or edit the file directly:

Field	Description
`name`	Used in `@worker <name>` annotations and the dropdown UI
`backend`	Always `"executor"` for HTTP workers
`runtime_id`	Stable identifier hashed into cell provenance - see Caching below
`config.url`	The HTTP endpoint for the executor protocol
`config.transport`	`"direct"` for direct push (the default), `"signed"` for the pull-model with signed URLs
`config.token`	Literal bearer token (dev only) - see Authentication
`config.token_env`	Env var name holding the bearer token (preferred for prod)

A typical multi-worker notebook ends up with:

[[workers]]
name = "fly-cpu"
backend = "executor"
runtime_id = "fly-cpu-v1"
[workers.config]
url = "https://my-strata-worker.fly.dev/v1/execute"
transport = "direct"
token_env = "STRATA_FLY_WORKER_TOKEN"

[[workers]]
name = "modal-gpu"
backend = "executor"
runtime_id = "modal-a10g-v1"
[workers.config]
url = "https://...--my-gpu-worker-gpu-executor.modal.run/v1/execute"
transport = "direct"
token_env = "STRATA_MODAL_WORKER_TOKEN"

Authentication¶

By default the worker accepts any caller that can reach its URL. For any worker deployed to a public endpoint, set a bearer token so only your notebook server can dispatch cells.

1. Generate a token (any opaque string; 32+ random bytes is plenty):

python -c "import secrets; print(secrets.token_urlsafe(32))"

2. Set STRATA_WORKER_TOKEN on the worker.

For Fly.io, store it as a secret (encrypted, injected at runtime, not visible in fly.toml):

fly secrets set STRATA_WORKER_TOKEN=<paste-token-here>

For Modal, attach a secret to the function:

@app.function(
    gpu="A10G",
    scaledown_window=60,
    secrets=[modal.Secret.from_name("strata-worker-token")],
)
@modal.asgi_app()
def gpu_executor():
    ...

…then create the Modal secret once: modal secret create strata-worker-token STRATA_WORKER_TOKEN=<paste-token-here>.

3. Tell the notebook server about the token. Export it as an environment variable wherever you run strata-notebook:

export STRATA_FLY_WORKER_TOKEN=<paste-token-here>
uv run strata-notebook

…and reference that env var in notebook.toml:

[workers.config]
url = "https://my-strata-worker.fly.dev/v1/execute"
transport = "direct"
token_env = "STRATA_FLY_WORKER_TOKEN"

token_env is preferred over token because the literal-token form gets committed to your notebook repo. Use token = "..." only for one-off local experiments.

A worker with STRATA_WORKER_TOKEN set rejects unauthenticated requests with 401 Unauthorized. /health stays open so platform health probes work without the secret.

Using workers in cells¶

Annotate any cell with # @worker <name>:

# @name Embed Abstracts
# @worker modal-gpu
# @timeout 300
embeddings = model.encode(abstracts, batch_size=256)

The worker annotation is the only change needed; the cell code itself is identical to local execution. If the worker has the right packages installed, it just works.

Precedence¶

When multiple levels define a worker, the most specific wins:

# @worker X annotation in the cell source (highest)
Cell-level worker override (from the cell's stored config)
Notebook-level worker default (from the Workers panel)

Device and tensor placement (PyTorch / GPU)¶

Each cell runs in an isolated process and exchanges values (pickled artifacts), not a shared Python kernel. That has a consequence specific to device-bound objects like torch.device and CUDA tensors when a DAG mixes a local CPU machine with a remote GPU @worker:

Don't share a resolved device across the boundary. A cell that does device = "cuda" if torch.cuda.is_available() else "cpu" and exports device resolves it where that cell ran. If it runs locally (no CUDA), the value is "cpu", and a GPU @worker cell consuming it will train on CPU on the GPU box. Resolve the device at point of use, inside each cell, rather than passing one device artifact downstream.
Save/load model weights with explicit map_location. A CUDA tensor or a CUDA-resident model pickled on the worker fails to deserialize on a CPU-only host (torch.cuda.is_available() is False there). Move tensors to CPU before they become a cross-boundary artifact (state = model.cpu().state_dict()), or load with torch.load(path, map_location="cpu").
Prefer passing a state_dict / checkpoint path between a training cell and downstream analysis cells, rather than a live CUDA model object.

This is partly notebook design - Strata can't know your placement intent - but the isolation boundary makes it a footgun worth designing around explicitly.

Caching and provenance¶

Remote execution results are cached identically to local cells. The provenance hash includes the worker's runtime_id, so:

Same code + same inputs + same runtime_id = cache hit, no remote call.
Changing runtime_id (e.g., switching from gpu-a10g to gpu-h100) invalidates the cache for cells using that worker.

When to bump runtime_id:

You upgraded the worker's Python dependencies (new torch version, new model weights baked into the image) and want downstream cells to re-run.
You moved a worker to different hardware (CPU type, GPU SKU) and the numerical output may differ.
You explicitly want to bust the cache for a debugging session.

When to leave runtime_id alone:

Redeploying the same image (no dep changes). The cache is correct by construction; re-running is wasted compute.
Scaling the number of worker instances. Output is deterministic given the same inputs.

If you don't set runtime_id, Strata uses the worker name as a fallback. That's fine for solo notebooks but risks cache surprises if two notebooks both have a worker named gpu pointing at different deployments - set runtime_id explicitly in shared notebooks.

Health checks¶

Every worker exposes GET /health. The notebook UI polls this and shows a green/red badge next to cells that use the worker; cells refuse to dispatch to an unhealthy worker.

curl https://my-worker.example.com/health

The /health endpoint is not gated by STRATA_WORKER_TOKEN - platform health probes (Fly, k8s liveness, Cloudflare) don't need the secret.

Troubleshooting¶

401 Unauthorized when running a cell. STRATA_WORKER_TOKEN is set on the worker but the notebook isn't sending it. Confirm token_env (or token) in notebook.toml matches an env var that's exported in the strata-notebook's shell. Restart strata-notebook after exporting; it reads env at startup.

Connection refused or Could not resolve host. config.url doesn't match where the worker is actually listening. From the strata-notebook host, run curl <config.url base>/health - it should respond. For Fly, fly status shows the public hostname; for Modal, modal app list shows deployed URLs.

Worker /health works but cells fail with ModuleNotFoundError: <package>. The worker image is missing the dependency the cell needs. Add it to the Dockerfile's pip install (Fly) or the .pip_install(...) chain (Modal) and redeploy. The worker uses its own Python env; nothing from the notebook server's env transfers.

Cells dispatched to a Modal worker hang for 30+ seconds before output. Cold start. Modal scales the function to zero after scaledown_window seconds idle; the first request after a scale-down has to provision a fresh container. Either bump scaledown_window, set min_containers=1 on the @app.function, or just expect the latency on the first cell after idle.

413 Payload Too Large from the worker. A cell input is larger than the worker's max-input limit. Default is 256 MB; override with STRATA_WORKER_MAX_INPUT_BYTES=<bytes> on the worker. Better: shrink the input by selecting columns / filtering rows in an upstream cell.

Fly build fails with error: failed to fetch wheel from a workload dep. Some Python deps (torch, sentence-transformers) don't ship abi3 wheels and fall back to building from source. If your worker needs one, add build-essential (plus the dep-specific toolchain) to the Dockerfile via RUN apt-get install -y --no-install-recommends build-essential && rm -rf /var/lib/apt/lists/* before the uv pip install step.

Modal redeploy hangs at "Building image". You changed .pip_install(...) - Modal is rebuilding the image layer. With torch + sentence-transformers this takes ~5 minutes the first time on a new image hash. Subsequent deploys with no dep changes hit the layer cache and finish in seconds.

Live status¶

When a cell dispatches to a remote worker, the UI shows a pulsing "dispatching → " badge during execution. After completion, the worker name and transport type appear in the cell metadata.