modded-nanogpt

Hardware: gpu_8x_h100. Save the files under Source files into a folder named modded-nanogpt, then follow the steps below.

Runs Keller Jordan's modded-nanogpt GPT-2 speedrun on an 8x H100 host provisioned via supercomputer. Targets 3.28 FineWeb validation cross-entropy. At the pinned upstream commit training takes about 45 seconds; a cold start adds host provisioning, data download and a ~9 minute compile warmup — total wall clock ~15–20 minutes.

What's here

supercomputer.json — image config, mirroring upstream's own Dockerfile and requirements.txt: a CUDA 13.1 devel base with PyTorch 2.10 from the cu128 wheel index (super routes torch there on GPU SKUs). The CUDA 13 base is required because upstream's attention kernel links libcudart.so.13.
run.sh — runs inside the session: checks out a pinned upstream commit, fetches the attention kernel, downloads FineWeb tokens, then runs the speedrun (torchrun --standalone --nproc_per_node=8). It carries one workaround, which upstream's README documents under "Running the current record": the flash-attention kernel devenpzak/flash-attn3-12864 cannot be downloaded anonymously through the kernels library (the Hub answers 401), so run.sh downloads the same revision as a model repo and sets LOCAL_KERNELS to the copy. Upstream still verifies the binary's sha256 on load, and no HF token is needed.

To move to a newer upstream commit, update COMMIT and the FA3_* values in run.sh, and re-sync supercomputer.json with upstream's Dockerfile and requirements.txt.

The compile warmup is paid on every fresh host: supercomputer's gVisor executor defaults TORCHINDUCTOR_CACHE_DIR and TRITON_CACHE_DIR to /dev/shm, which is not part of the synced /workspace. The clone, the kernel and the FineWeb shards are in /workspace and survive super stop.

Usage

From this directory (so super picks up the local supercomputer.json):

# 1. Create a persistent 8x H100 session. It is cold ($0) until first used.
super create --sku gpu_8x_h100 --json
# {"session_id": "s-XXXX", "status": "closed", "sku": "gpu_8x_h100", ...}

# 2. Upload run.sh and kick off training (substitute the session id). The
#    upload brings the session online: provisioning the host took ~5 min.
super upload --on s-XXXX run.sh run.sh
super spawn --on s-XXXX -- bash /workspace/run.sh

# 3. Watch logs, or block until it exits (the spawn printed a process id).
super logs s-XXXX <process_id> --follow
super wait s-XXXX <process_id>

# 4. Stop when done — releases the host, keeps /workspace.
super stop s-XXXX

There is no resume command: using a stopped session again brings it back on a fresh host with /workspace restored.

The host billed $17.99/hr when this was last run (2026-10-01); super sessions shows the live rate. A cold run is roughly $5.

Expected output

Compiling model, warming up kernels and capturing CUDA graphs (~7 minutes on first execution)
...
step:1175/1194 train_time:41530ms step_avg:35.34ms
step:1194/1194 val_loss:3.2763 train_time:43078ms step_avg:36.08ms
peak memory allocated: 48411 MiB reserved: 62120 MiB

Single runs vary around the 3.28 target: two runs on 2026-10-01 gave 3.2763 and 3.2845.

Source files

`run.sh`

#!/usr/bin/env bash
# Clone modded-nanogpt, download FineWeb tokens, and run the 8x H100 speedrun.
# Run this INSIDE the supercomputer session, not on your laptop. See README.md.
set -euo pipefail

WORKDIR="${WORKDIR:-/workspace/modded-nanogpt}"
NUM_SHARDS="${NUM_SHARDS:-8}"   # each shard is 100M tokens; speedrun budget ~400M

# Upstream moves fast and each commit expects its own kernel build and CUDA
# runtime, so an unpinned clone breaks this example without any change here.
# COMMIT and the FA3_* values below move together with supercomputer.json, which
# mirrors upstream's Dockerfile base and requirements.txt at COMMIT.
COMMIT=4ea6b937337a4889b8cfe3f38a93d120048d8f71
# FA3_REPO / FA3_REVISION from track_1_short/model/attention.py at COMMIT.
FA3_REPO=devenpzak/flash-attn3-12864
FA3_REVISION=64c1e6d1f2780e7931839f41426ddcdb564a7cb9
FA3_DIR="${FA3_DIR:-/workspace/fa3-12864}"

# From some egress ranges github.com serves the ref advertisement but answers
# git's POST /git-upload-pack with a spurious 401 — only over HTTP/2. Git reads
# that as "authenticate", asks for a username, and with no writer on `super run`'s
# PTY the clone hangs until the run's whole budget expires. The same clone works
# over HTTP/1.1, and the refusal is deterministic, so pin it rather than retry.
git config --global http.version HTTP/1.1
# If a credential prompt ever does get raised, fail loudly instead of hanging.
export GIT_TERMINAL_PROMPT=0

# Validate the checkout, not just the dir: a run killed mid-fetch (preemption)
# leaves a half-fetched tree that satisfies -d but breaks every rerun.
if [ "$(git -C "$WORKDIR" rev-parse HEAD 2>/dev/null)" != "$COMMIT" ] \
        || [ ! -f "$WORKDIR/data/cached_fineweb10B.py" ]; then
    rm -rf "$WORKDIR"
    git init -q "$WORKDIR"
    git -C "$WORKDIR" fetch --depth 1 https://github.com/KellerJordan/modded-nanogpt "$COMMIT"
    git -C "$WORKDIR" checkout -q FETCH_HEAD
fi
cd "$WORKDIR"

# Anonymous get_kernel of FA3_REPO fails with a 401: the Hub serves it only as a
# model repo and the `kernels` library asks for a kernel repo. Upstream's README
# says to `hf auth login` or pre-seed the kernel; pre-seeding needs no HF token.
# attention.py still checks the binary's sha256.
python -c "
from huggingface_hub import snapshot_download
snapshot_download('$FA3_REPO', revision='$FA3_REVISION', local_dir='$FA3_DIR')
"
export LOCAL_KERNELS="$FA3_REPO=$FA3_DIR"

# Tokenized FineWeb10B shards from HF (kjj0/fineweb10B-gpt2). Each ~200 MB.
python data/cached_fineweb10B.py "$NUM_SHARDS"

# torchrun --standalone --nproc_per_node=8 train_gpt.py
exec bash run.sh

`supercomputer.json`

{
    "sku": "gpu_8x_h100",
    "image": {
        "base": "nvidia/cuda:13.1.1-cudnn-devel-ubuntu24.04",
        "apt": [
            "git", "ca-certificates", "build-essential", "python3-dev",
            "openssh-server", "wget", "curl", "dnsutils"
        ],
        "pip": [
            "numpy",
            "tqdm",
            "torch==2.10",
            "huggingface-hub==1.29.0",
            "kernels==0.16.1",
            "setuptools",
            "typing_extensions==4.15.0",
            "datasets",
            "tiktoken"
        ],
        "env": {"PYTHONUNBUFFERED": "1"}
    }
}