Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

11 Commits

Folders and files

Repository files navigation

Qwen3.5-35B Inference Server — Optimized Deployment with vLLM

Fully containerized, production-ready deployment of Qwen/Qwen3.5-35B-A3B on the NVIDIA GB10 using vLLM. Managed through a single script. No local Python environment required — all tests run inside the container.

The default configuration uses INT4 (AWQ) + triton_attn: 2.18× baseline throughput with −2.01 pp accuracy cost on MMLU-Redux.

Performance

Benchmark conditions: 100 requests · concurrency 20 · 512 output tokens/request. Accuracy: MMLU-Redux, zero-shot.

Configuration Throughput vs. Baseline E2E Latency Model Size MMLU-Redux
BF16 + flash_attn (Baseline) 156.2 t/s — 65.55 s 65.53 GiB 92.47%
FP8 + flash_attn 254.9 t/s +63.2% 40.16 s 33.38 GiB 90.74%
INT4 + triton_attn (Default) 341.0 t/s +118.3% 30.03 s 21.38 GiB 90.46%

Repository Structure

qwen-deployment/
├── Dockerfile              # vllm/vllm-openai:cu130-nightly + evalscope + aiohttp
├── docker-compose.yml      # Single service, configured via MODEL_PATH env var
├── requirements.txt        # Python deps installed into the image
├── manage.sh               # Unified lifecycle script — the only entry point you need
├── scripts/
│   ├── benchmark.py        # Async performance benchmark (TTFT, throughput, latency)
│   └── stability_test.py   # Long-running stability test (CV and failure tracking)
└── results/                # Stability test logs (auto-saved here by manage.sh)

Prerequisites

  • NVIDIA GPU (≥ 24 GiB VRAM for FP8/INT4; ≥ 80 GiB for BF16)
  • NVIDIA Drivers + NVIDIA Container Toolkit
  • Docker v24+
  • jq: sudo apt-get install jq
  • Hugging Face CLI: pip install huggingface_hub — required to download model weights

Quick Start

1 — Download the model and set your model path

Download the INT4 (AWQ) model weights from Hugging Face:

hf download QuantTrio/Qwen3.5-35B-A3B-AWQ --local-dir /path/to/your/model
export MODEL_PATH=/path/to/your/model

2 — Start the server

The pre-built image (mtk53722/qwen-deployment:v1) is pulled from Docker Hub automatically on first run — no build step needed.

bash manage.sh start

Build from source (optional): If you want to build the image locally instead, run bash manage.sh build before start. The local build will be used in place of the Docker Hub image.

3 — Confirm it's ready

bash manage.sh health
[SUCCESS] API is UP and responding.
"Qwen/Qwen3.5-35B-A3B"

4 — Run tests

bash manage.sh benchmark       # Performance: TTFT, throughput, latency
bash manage.sh evaluate        # Accuracy: MMLU_redux without CoT
bash manage.sh stability       # Stability: 1,100 samples, CV tracking

All commands run inside the container. Results from stability are saved to results/.

5 — Stop

bash manage.sh stop

Command Reference

bash manage.sh <command> [args]

Server:
  build               Build the image locally (optional — pre-built on Docker Hub)
  start               Start the server (auto-pulls image if not cached)
  stop                Stop the server
  health              Check if the API is responding
  logs                Follow server logs

Testing (all run inside the container — no local Python required):
  benchmark           Performance test (100 requests, concurrency 20)
  evaluate [tasks]    Accuracy evaluation (default: mmlu_redux, zero-shot)
  stability [N]       Long-running stability test (default: 1100 samples)

API Usage

The server exposes an OpenAI-compatible API on http://localhost:8000/v1.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="Qwen/Qwen3.5-35B-A3B",
    messages=[{"role": "user", "content": "Hello!"}],
    max_tokens=512,
)
print(response.choices[0].message.content)

Reproducing the Benchmark Results

hf download QuantTrio/Qwen3.5-35B-A3B-AWQ --local-dir /path/to/your/model
export MODEL_PATH=/path/to/your/model
bash manage.sh start
bash manage.sh health
bash manage.sh benchmark    # expect ~341 t/s, ~30.0s latency
bash manage.sh evaluate     # expect MMLU-Redux ~90.5%
bash manage.sh stability    # expect 1100 samples, 0 failures, CV < 5%

Attention Backend Reference (GB10-specific)

Backend Throughput (BF16) Notes
flash_attn 156.2 t/s Default; best on A100/H100
triton_attn 183.4 t/s Best on GB10 — JIT-compiled for target hardware
flashinfer 157.9 t/s Similar throughput to flash_attn but elevated TTFT variance on GB10 (P95: 0.586 s vs 0.460 s)

To switch backends, add --attention-backend <name> to the command block in docker-compose.yml.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages