Qwen3.8-27B V100 Quants

1. 2026-08-14

   W4A16 live, NVFP4 in progress

   Published the W4A16 AWQ card: whole model INT4 at 19.5 GB, vision and MTP intact, think-mode calibrated. NVFP4 was still quantizing (layer 62 of 65 that day). Both got their own cards with V100 numbers once validated.

2. 2026-08-14

   Full V100 validation

   Ran the battery on 2x V100 through 1Cat-vLLM: coherence, think-tag discipline, image understanding, GSM8K, long-form generation, and MTP in both draft modes. All checks passed. The stock prod serve later measured GSM8K 0.99 over 100 items.

3. 2026-08-15

   Tokenizer fix across 15 repos

   A reporter hit an HTTP 400 on a 2500x1406 image: the calibration save had left an active truncation block in tokenizer.json. Restored both tokenizer files byte-for-byte from the base model and pushed the fix to all 15 affected repos.

4. 2026-08-16

   NVFP4 bake-off

   Ran five candidate NVFP4 recipes against the Inferact and unsloth public quants on one rig with one battery. Found the GatedDeltaNet in_proj projections are the sensitive spot and calibration is the real quality lever, not which layers you keep in 16-bit.

5. 2026-08-17

   NVFP4 v2 shipped

   Replaced the NVFP4 checkpoint with recipe C, the bake-off winner: GDN in_proj kept BF16, a 4x larger and more diverse thinking-mode calibration set, act-order and MSE observer. Confabulation dropped 39 to 27 of 75 and GSM8K rose 0.940 to 0.965, at 96% of v1's single-stream V100 speed.

6. 2026-09-19

   Serving moved to 1Cat-vLLM 1.5.0

   Cut the service that runs this checkpoint over from the 1.3.0 line to 1.5.0. Functional validation passed clean; a warm throughput rerun was still pending at cutover, so the card's 1.2.2 tables stand as the published numbers.

7. 2026-10-03

   Upstream 1.5.1

   1Cat-vLLM tagged 1.5.1. Per the always-newest-engine rule the recommended base moved to it; the 1.5.1 bundled profile prefers FP8 E4M3 KV.

1. 1

   Pick a format

   W4A16 AWQ if you have Ampere-or-newer GPUs or a pair of V100s and want the simplest path; it is the format with the downloads behind it (over 64,000). NVFP4 if you are on Blackwell and want native FP4, or if you specifically need the NVFP4 build that loads on Volta. Both carry the vision tower and the MTP head.

2. 2

   Download the weights

   W4A16 is 19.5 GB (a packed body plus a 0.85 GB BF16 MTP head). If you cloned before 2026-08-15, re-fetch tokenizer.json and tokenizer_config.json; an early calibration run shipped a broken truncation block that failed large image inputs.

   shellcopy

   hf download philbert440/Qwen3.8-27B-W4A16-AWQ --local-dir ./Qwen3.8-27B-W4A16-AWQ

   philbert440/Qwen3.8-27B-W4A16-AWQphilbert440/Qwen3.8-27B-NVFP4

3. 3

   Stock vLLM on modern GPUs

   compressed-tensors W4A16 is natively supported, so on Ampere or newer there is nothing special to do. Add the MTP speculative config to get the draft head working. NVFP4 runs the same way on Blackwell; on Ampere/Hopper it falls back to a dequant path.

   shellcopy

   vllm serve philbert440/Qwen3.8-27B-W4A16-AWQ --max-model-len 32768 \
     --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'

   Model card: serving

4. 4

   Install the engine for 2x V100

   Volta has no INT4 tensor-core path in stock vLLM, so the SM70 route is 1Cat-vLLM's TurboMind dequant path, from the stock upstream wheel; there is no custom wheel to build. Clone qwen-3.8-v100 and run its install.sh: it downloads the 1.5.1 release wheel, checks it against the pinned sha256, and sets up a Python 3.12 venv with the wheel and ninja (ninja has to be on PATH for the JIT kernels). The pip line below is the same thing by hand.

   shellcopy

   git clone https://github.com/philbert440/qwen-3.8-v100 && cd qwen-3.8-v100
   ./install.sh
   # by hand: uv pip install https://github.com/1CatAI/1Cat-vLLM/releases/download/v1.5.1/1cat_vllm-1.5.1-cp312-cp312-linux_x86_64.whl ninja

   qwen-3.8-v1001Cat-vLLM v1.5.1 release

5. 5

   Launch on 2x V100 (TP2, MTP)

   The default profile, tp2-card, is the model card's SM70 serve command verbatim: TP2, FP8 E5M2 KV, 32K context, MTP with 2 draft tokens, utilization 0.78, plus the env a V100 pair needs. download.sh fetches the checkpoint at its pinned HF revision. Extra arguments to serve.sh go to vllm serve after the profile, so they override it. The raw command it runs is below the first two lines.

   shellcopy

   ./download.sh w4a16 ./Qwen3.8-27B-W4A16-AWQ
   QWEN38_MODEL=./Qwen3.8-27B-W4A16-AWQ ./serve.sh
   
   # what the tp2-card profile runs:
   VLLM_SM70_QUANT_BACKEND=turbomind VLLM_SM70_FLASH_ATTN_V100=1 \
   VLLM_SM70_MTP_DYNAMIC_DRAFT_VOCAB_DEFAULT=0 NCCL_P2P_DISABLE=1 \
   vllm serve ./Qwen3.8-27B-W4A16-AWQ --dtype half --attention-backend FLASH_ATTN_V100 \
     --tensor-parallel-size 2 --gpu-memory-utilization 0.78 \
     --max-model-len 32768 --kv-cache-dtype fp8_e5m2 \
     --enable-prefix-caching --reasoning-parser qwen3 \
     --speculative-config '{"method":"mtp","num_speculative_tokens":2,"attention_backend":"FLASH_ATTN_V100","draft_sample_method":"probabilistic"}'

   tp2-card profileModel card: 2x V100 / SM70

The recommendation is stock upstream 1Cat-vLLM. These checkpoints need no engine fork and no custom wheel; the SM70 TurboMind and NVFP4 paths they rely on are in the upstream release.

qwen-3.8-v100 is not an engine fork. It holds the 2x V100 launch profile, the serve script, its env and flags, and a smoke test, pinned to a specific upstream 1Cat-vLLM release and HF revision. Its releases tag that pairing; there is no wheel in it.

The published model-card serve command and the throughput table were measured on 1Cat-vLLM 1.2.2. The launch profile pins 1.5.1, the newest upstream release, by wheel sha256, and keeps the cards' command verbatim, FP8 E5M2 KV included; the 1.5.1 bundled profile would prefer E4M3. Its release tag 1cat-v1.5.1 names that pairing.

• The calibration run's save_pretrained wrote an active truncation block into tokenizer.json, and transformers 5 also drifted the pre-tokenizer regex. Image inputs larger than the truncation limit then failed with a token-count mismatch, an HTTP 400 out of vLLM. It hit 15 repos before it was caught and fixed by restoring the two tokenizer files verbatim from the base model.
• NVFP4 has a CUDA-graph sizing trap, not a format flaw. With MTP each sequence schedules K+1 tokens per step, so an explicit cudagraph_capture_sizes below max_num_seqs times (K+1) pushes concurrent decode off CUDA graphs and collapses aggregate throughput about 3x. It reproduces identically on the W4A16 sibling. Engine-default auto sizing is correct; do not hand-set the capture sizes small.
• ModelOpt-exported NVFP4 does not run on Volta: it requires compute capability 7.5+ and its FP8 attention path has no SM70 kernel. The RadixArk checkpoint would not load at all. Building NVFP4 as compressed-tensors nvfp4-pack-quantized instead is what lets capability 7.0 serve it.
• The MTP head crashed generation when it ran in bf16 under an earlier engine build; it was fixed in a later point release, and the bf16-head regression check is the first step of the MTP test sequence now.
• On a V100 pair without NVLink, leaving NCCL peer-to-peer enabled cost roughly 4x throughput. NCCL_P2P_DISABLE=1 is mandatory there.
• VLLM_SM70_MTP_DYNAMIC_DRAFT_VOCAB_DEFAULT=0 is required whenever max_num_seqs is greater than 1 with MTP; the dynamic draft vocab guard otherwise forces max_num_seqs to 1 at boot.
• The model card still names 1Cat-vLLM 1.2.2 as the validated engine. The service that runs this checkpoint has since moved up through 1.5.0 and onto the 1.5.1 line; the throughput tables here are the 1.2.2 numbers and have not been re-measured on 1.5.1 single-stream.