Batch: Jobs & Sweeps

Two tools for non-interactive work: a job runs one command to completion and gives the box back; a sweep (super map) fans one command out over many argument bindings as a durable server-side batch.

Jobs (super job)

super job run runs like super run, with one difference: what --retries counts. The command runs on a session whose /workspace is saved like any session's, so you can super ls or super download its files after the command ends; a new session gives its box back when the command ends.

job run flags: --sku SKU · --volume NAME[:MOUNT] (repeatable) · --upload PATH (repeatable, also spelled --include; uploaded before the command starts) · --env K=V · --secret KEY[=VALUE] · --retries N · --min-disk-gb N · --detach (print the ids and return; cannot be combined with --retries).

super job run --sku gpu_1x_l4 --upload train.py --retries 2 -- \
  python train.py --out /workspace/model.pt
super download <session-id> /workspace/model.pt ./model.pt

Outputs that other sessions should read belong in a volume:

super volume create results
super job run --sku gpu_1x_l4 --volume results:/workspace/results -- \
  python train.py --out /workspace/results/model.pt
super volume files results               # committed when the job returned
super volume download results model.pt   # pull them back locally

With mounted volumes, an attached job run returns once the command has exited and its volumes are committed, so super volume files right after it shows the job's output.

Retries. --retries N re-runs the command only after a lost box; a non-zero exit is final. (super run --retries N instead re-runs a command that exits non-zero, after a short growing pause — except exit 126/127 and an OOM kill, which a re-run on the same box cannot change.) A retry runs in the same session and /workspace, so save checkpoints there and load them on start. Only when the session itself ended (failed or dead) does the retry start a new session with an empty /workspace (uploads are sent again; stderr says so) — checkpoints that must survive that belong on a volume. A job closed on request (super job kill, super stop) is never retried. After a preemption, follow the same job id with super job wait/logs; do not submit a duplicate run.

The SDK's project.job(...) is a different thing: an ephemeral job whose workspace is discarded at close, so its only durable outputs are its volumes and logs (see SDK mirrors). super job wait and super job kill work on both.

Rule of thumb: a session when you'll iterate interactively; a job for one command you want run to completion and then released.

Sweeps (super map)

A sweep runs your command N times with different arguments as a durable server-side fan-out: submit, disconnect, and the service drives every task through its own session — queueing for capacity, retrying failures, and recording exit codes and logs. No changes to your code; your script's CLI is the interface.

# Hyperparameter sweep (cartesian product: 3 × 2 = 6 tasks)
super map --name lr-sweep --sku gpu_1x_l4 \
  -- python train.py --lr {lr} --bs {bs} ::: lr=1e-4,3e-4,1e-3 ::: bs=32,64

# Datagen over indices (shell brace expansion feeds the group)
super map --name gen -- python gen.py --shard {} --of 500 ::: {0..499}

# One task per line from a pipe (no ::: → stdin binds {})
jq -r '.items[]' work.json | super map --name batch -- python proc.py {}

Supplying arguments (GNU-parallel style)

Everything after the command is parsed for ::: markers: each one starts a group of values (space- or comma-separated), and each task runs the command with one combination substituted in.

Placeholders: {} = the single group's value · {1} {2} = group by position · {lr} = named group (write it ::: lr=1e-4,3e-4) · {i} = task index (0-based) · {n} = task count. A command with no placeholders gets each task's values appended at the end. Every task also gets env vars SUPERCOMPUTER_TASK_ID, SUPERCOMPUTER_WORLD_SIZE, and SUPERCOMPUTER_JOB, so a script can self-shard and skip templating entirely. super help map prints this reference.

Code binding

The sweep pins an immutable snapshot at submit, so task 400 runs the same bytes as task 0:

• Default: the current dir (or --code DIR) is hashed and uploaded once; each task session starts from it.
• --from <session-id>: fork a session you just verified — each task starts from that session's synced workspace (deps installed, weights downloaded) and inherits its image. Stop the session first (super stop <sid> --wait) so the manifest is finalized.

Canary

Task 0 runs first and gates the rest — a broken sweep costs one task, not N. Nothing else launches until the canary passes, including while it waits to be retried. super map blocks until the canary passes, then detaches (Ctrl-C detaches the watch; it does not cancel). Use --canary-smoke 'python -c "import train"' to gate on a cheap check when one task takes minutes; --no-canary/--no-wait opt out entirely.

With --retries left at its default, a canary whose command exits non-zero fails the gate on that first attempt. Pass --retries N and the canary gets that budget like every other task. After fixing the command, super sweep retry <name> re-runs the canary first and launches the rest once it passes.

Fan-out width

--max-parallel N caps how many tasks run at once (default 8; more tasks run in waves). Changeable on a live sweep: super sweep set <name> --max-parallel N.

Outputs

Write task-keyed files (shard-{i}.jsonl) into each task's workspace and pull them all with harvest — no session-id bookkeeping, no filename collisions:

super sweep harvest lr-sweep '*.json' ./out   # → out/task-<index>/<path>

Or write into a shared volume: commits are per-path merges (last writer wins per file), so key every task's output by {i}.

Tracking & recovery

super sweeps                             # all sweeps: state + done/run/fail counts
super sweep status <name>                # counts + failures clustered by error
super sweep status <name> --watch        # repainted until terminal
super sweep logs <name> [--task N]       # durable output (defaults to the exemplar failure)
super sweep retry <name> [--all]         # re-run failed tasks; after a canary failure, the canary first, then the rest
super sweep cancel <name> [--remaining]  # stop; --remaining keeps running tasks
super sweep wait <name>                  # block until terminal; exit 0 on success
super sweep gc <name> [--yes]            # after harvesting: retire the finished sweep's sessions

Every task attempt leaves a cold session behind ($0, hidden from super sessions but listed by super sessions --all). Once you have harvested what you need, super sweep gc <name> retires them: they leave the session list and can no longer be resumed or harvested, while their history and billing stay. It only runs on a finished sweep; super sessions --all --retired still shows them.

A task with no host yet keeps its queued session and is reported as blocked on capacity — that's the fleet scaling up, not your sweep breaking. Spot-preempted tasks requeue without consuming the retry budget.

--webhook URL POSTs the status document on canary pass/fail and completion — agents should submit with --no-wait --json and wake on the webhook instead of polling.

SDK mirrors

# Job
job = project.job("python train.py --out /results/model.pt",
                  sku="gpu_1x_l4",
                  volume_mounts=[{"name": "results", "mount_path": "/results"}],
                  include=["./src"], retries=1)
log = job.wait()                   # ProcessLog: exit_code, output

# Sweep — the *type* of args picks the combinator (no ::: syntax in the SDK)
sweep = client.map("python train.py --lr {lr} --bs {bs}",
                   args={"lr": [1e-4, 3e-4], "bs": [32, 64]},   # 4 tasks
                   name="lr-sweep", sku="gpu_1x_l4", code_dir=".")
sweep.wait_canary()
sweep.wait()