Qwen3.5-9B simple inference

Hardware: gpu_1x_l4. Save the files under Source files into a folder named qwen3.5-9b, then follow the steps below.

Downloads Qwen3.5-9B — the largest dense Qwen3.5 model that fits a single 24 GB L4 in BF16 — into a persistent volume and runs one chat completion through transformers on one L4 GPU provisioned via supercomputer. The first run pulls ~19.3 GB of BF16 weights into the qwen3.5-9b-weights volume; every subsequent run that mounts the volume skips straight to load (measured: 0 bytes re-downloaded).

Measured on an L4: peak 18.0 GB / 24 GB VRAM, ~11 tok/s for single-prompt decoding. The ~6 GB of headroom is your KV-cache + activation budget.

Fit notes

9B BF16 is the dense ceiling for an L4. Weights are ~18 GB; 14B BF16 (~28 GB) will not fit. If you need more capability on the same L4, Qwen3.5-27B-GPTQ-Int4 loads in ~15 GB — but that's a different load path (an INT4 kernel, not the plain BF16 from_pretrained used here).
Context, not weights, is what OOMs an L4. Qwen3.5's native context is 262K. This script uses transformers generate(), which grows the KV cache with the actual sequence length (no 262K pre-allocation), so short prompts stay cheap. Keep prompt + --max-new-tokens to a few thousand tokens and you stay inside the 6 GB headroom.

What's here

supercomputer.json — image config: PyTorch 2.10 + CUDA 12.8 (cuDNN 9) with transformers, accelerate, and hf_transfer. Declares a qwen3.5-9b-weights volume mounted at /workspace/.cache/huggingface (where HF_HOME points), so the weights are cached once and reused by any future session that attaches the volume.
infer.py — runs inside the session. Loads Qwen/Qwen3.5-9B in BF16 on the single GPU, runs one chat completion, and prints the answer, tokens/s, and peak VRAM. --prompt overrides the default question; --no-thinking drops the <think> reasoning blocks.

Usage

From this directory (so supercomputer picks up the local supercomputer.json):

# 0. Sanity check.
super health

# 1. Create the weights volume — one-time per workspace.
super volume create qwen3.5-9b-weights

# 2. One-shot: upload + run in a fresh session. First run downloads ~19.3 GB
#    of weights into the volume; subsequent runs skip the download.
super run infer.py -- --no-thinking --max-new-tokens 256

# --- OR --- keep a persistent session for repeated experiments:
super create --sku gpu_1x_l4 --json
# {"session_id": "s-XXXX", ...}
super upload s-XXXX infer.py infer.py
super exec s-XXXX -- python3 /workspace/infer.py --prompt "Explain attention in one sentence."
super stop s-XXXX

Note: super run … -- --prompt "multi word" currently mis-splits the value (the run path joins argv without shell-quoting). For a multi-word prompt, use a persistent session with super exec as above, or pass a single-token --prompt. The in-script default prompt works fine with super run.

Source files

`infer.py`

#!/usr/bin/env python3
"""Single-prompt inference with Qwen3.5-9B on a single L4.

Loads Qwen/Qwen3.5-9B in BF16 on one GPU, runs one chat completion
through transformers' generate(), and prints the answer, tokens/s, and
peak VRAM. Qwen3.5-9B is the largest *dense* Qwen3.5 that fits a 24 GB
L4 in BF16 (~18 GB of weights, ~20 GB peak). Meant to run INSIDE a supercomputer
session (see README.md). Weights download once into the qwen3.5-9b-weights
volume; later runs that attach the volume skip straight to load.
"""
from __future__ import annotations

import argparse
import time

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "Qwen/Qwen3.5-9B"


def parse_args() -> argparse.Namespace:
    """Command-line knobs for prompt, output length, and decoding mode."""
    p = argparse.ArgumentParser(description="Qwen3.5-9B simple inference")
    p.add_argument("--prompt",
                   default="Give me a short introduction to large language models.")
    p.add_argument("--max-new-tokens", type=int, default=512)
    p.add_argument("--temperature", type=float, default=1.0)
    p.add_argument("--no-thinking", action="store_true",
                   help="disable Qwen3.5 <think> reasoning blocks")
    return p.parse_args()


def main() -> None:
    """Load the model, run one generation, print the answer and tokens/s."""
    args = parse_args()

    tokenizer = AutoTokenizer.from_pretrained(MODEL)
    model = AutoModelForCausalLM.from_pretrained(
        MODEL,
        torch_dtype=torch.bfloat16,
        device_map="auto",
    )

    messages = [{"role": "user", "content": args.prompt}]
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
        enable_thinking=not args.no_thinking,
    )
    inputs = tokenizer([text], return_tensors="pt").to(model.device)

    start = time.perf_counter()
    output_ids = model.generate(
        **inputs,
        max_new_tokens=args.max_new_tokens,
        do_sample=args.temperature > 0,
        temperature=args.temperature,
        top_p=1.0,
    )
    elapsed = time.perf_counter() - start

    new_tokens = output_ids[0][inputs.input_ids.shape[1]:]
    response = tokenizer.decode(new_tokens, skip_special_tokens=True)

    print("--- prompt ---")
    print(args.prompt)
    print("\n--- completion ---")
    print(response.strip())
    print("\n--- stats ---")
    print(f"generated tokens: {len(new_tokens)}")
    print(f"wall time:        {elapsed:.2f}s")
    print(f"throughput:       {len(new_tokens) / elapsed:.1f} tok/s")
    if torch.cuda.is_available():
        peak = torch.cuda.max_memory_allocated() / 1e9
        print(f"peak VRAM:        {peak:.1f} GB / 24 GB (L4)")


if __name__ == "__main__":
    main()

`supercomputer.json`

{
    "sku": "gpu_1x_l4",
    "image": {
        "base": "pytorch/pytorch:2.10.0-cuda12.8-cudnn9-devel",
        "pip": [
            "transformers>=4.51",
            "accelerate",
            "hf_transfer"
        ],
        "env": {
            "PYTHONUNBUFFERED": "1",
            "HF_HOME": "/workspace/.cache/huggingface",
            "HF_HUB_CACHE": "/workspace/.cache/huggingface/hub",
            "TRANSFORMERS_CACHE": "/workspace/.cache/huggingface/hub",
            "HF_XET_CACHE_DIR": "/workspace/.cache/huggingface/xet",
            "HF_HUB_ENABLE_HF_TRANSFER": "1"
        }
    },
    "volumes": [
        {"name": "qwen3.5-9b-weights", "mount": "/workspace/.cache/huggingface"}
    ]
}