Fully containerized, production-ready deployment of Qwen/Qwen3.5-35B-A3B on the NVIDIA GB10 using vLLM. Managed through a single script. No local Python environment required — all tests run inside the container.
The default configuration uses INT4 (AWQ) + triton_attn: 2.18× baseline throughput with −2.01 pp accuracy cost on MMLU-Redux.
Benchmark conditions: 100 requests · concurrency 20 · 512 output tokens/request. Accuracy: MMLU-Redux, zero-shot.
| Configuration | Throughput | vs. Baseline | E2E Latency | Model Size | MMLU-Redux |
|---|---|---|---|---|---|
| BF16 + flash_attn (Baseline) | 156.2 t/s | — | 65.55 s | 65.53 GiB | 92.47% |
| FP8 + flash_attn | 254.9 t/s | +63.2% | 40.16 s | 33.38 GiB | 90.74% |
| INT4 + triton_attn (Default) | 341.0 t/s | +118.3% | 30.03 s | 21.38 GiB | 90.46% |
qwen-deployment/
├── Dockerfile # vllm/vllm-openai:cu130-nightly + evalscope + aiohttp
├── docker-compose.yml # Single service, configured via MODEL_PATH env var
├── requirements.txt # Python deps installed into the image
├── manage.sh # Unified lifecycle script — the only entry point you need
├── scripts/
│ ├── benchmark.py # Async performance benchmark (TTFT, throughput, latency)
│ └── stability_test.py # Long-running stability test (CV and failure tracking)
└── results/ # Stability test logs (auto-saved here by manage.sh)
- NVIDIA GPU (≥ 24 GiB VRAM for FP8/INT4; ≥ 80 GiB for BF16)
- NVIDIA Drivers + NVIDIA Container Toolkit
- Docker v24+
jq:sudo apt-get install jq- Hugging Face CLI:
pip install huggingface_hub— required to download model weights
Download the INT4 (AWQ) model weights from Hugging Face:
hf download QuantTrio/Qwen3.5-35B-A3B-AWQ --local-dir /path/to/your/model
export MODEL_PATH=/path/to/your/modelThe pre-built image (mtk53722/qwen-deployment:v1) is pulled from Docker Hub automatically on first run — no build step needed.
bash manage.sh startBuild from source (optional): If you want to build the image locally instead, run
bash manage.sh buildbeforestart. The local build will be used in place of the Docker Hub image.
bash manage.sh health[SUCCESS] API is UP and responding.
"Qwen/Qwen3.5-35B-A3B"
bash manage.sh benchmark # Performance: TTFT, throughput, latency
bash manage.sh evaluate # Accuracy: MMLU_redux without CoT
bash manage.sh stability # Stability: 1,100 samples, CV trackingAll commands run inside the container. Results from stability are saved to results/.
bash manage.sh stopbash manage.sh <command> [args]
Server:
build Build the image locally (optional — pre-built on Docker Hub)
start Start the server (auto-pulls image if not cached)
stop Stop the server
health Check if the API is responding
logs Follow server logs
Testing (all run inside the container — no local Python required):
benchmark Performance test (100 requests, concurrency 20)
evaluate [tasks] Accuracy evaluation (default: mmlu_redux, zero-shot)
stability [N] Long-running stability test (default: 1100 samples)
The server exposes an OpenAI-compatible API on http://localhost:8000/v1.
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="Qwen/Qwen3.5-35B-A3B",
messages=[{"role": "user", "content": "Hello!"}],
max_tokens=512,
)
print(response.choices[0].message.content)hf download QuantTrio/Qwen3.5-35B-A3B-AWQ --local-dir /path/to/your/model
export MODEL_PATH=/path/to/your/model
bash manage.sh start
bash manage.sh health
bash manage.sh benchmark # expect ~341 t/s, ~30.0s latency
bash manage.sh evaluate # expect MMLU-Redux ~90.5%
bash manage.sh stability # expect 1100 samples, 0 failures, CV < 5%| Backend | Throughput (BF16) | Notes |
|---|---|---|
flash_attn |
156.2 t/s | Default; best on A100/H100 |
triton_attn |
183.4 t/s | Best on GB10 — JIT-compiled for target hardware |
flashinfer |
157.9 t/s | Similar throughput to flash_attn but elevated TTFT variance on GB10 (P95: 0.586 s vs 0.460 s) |
To switch backends, add --attention-backend <name> to the command block in docker-compose.yml.