← models

V100 Builds · released 2026-10-08

GLM-5.3-Flash EXL3

A 321B MoE at ~2.1 bpw on four V100s. 91 tok/s single stream with MTP, or 1M context on eight.

Get it running Hugging Face GitHub repo
tok/s decode
91.4–91.9 tok/s decodesingle stream on 4× V100, MTP k=3
context
1M contexton 8× V100, 175.1 tok/s aggregate at 8 streams
mean KL vs FP8
0.132 mean KL vs FP887.8% top-1 agreement
gsm8k-200
0.985 gsm8k-20075/75 factual

GLM-5.3-Flash is a 321B-parameter MoE with about 18B active per token. I quantized it to EXL3 at 2.12 bpw so it fits four 32 GB V100s, and wrote the SM70 kernels to serve it in 1Cat-vLLM. Everything ran on my own hardware, including the conversion.

The checkpoint uses a V100-native tile order, so the tensor-core path decodes the trellis without lane remapping. The engine side is a stack of PRs against 1Cat-vLLM: an EXL3 routed-expert MoE path, an FP32-softmax sparse MLA kernel for decode and prefill, and a handful of decode fast paths. One of them, the MTP wiring, is merged upstream.

On the card config (TP4, MTP with 3 draft tokens) it does 91.4-91.9 tok/s for one user. On all eight GPUs it serves 1M context and 175 tok/s aggregate at eight streams, but loses MTP. The 2-bit quant holds up on retrieval and short tasks. It does not hold up in long multi-turn agent sessions.

runs on
4× V1008× V100Tesla V100 32GB
base model
zai-org/GLM-5.3-Flash (FP8)
engine
1Cat-vLLM (SM70) · 1Cat-vLLM main @ ec60883ad
links
philbert440/GLM-5.3-Flash-EXL3 (Hugging Face)zai-org/GLM-5.3-Flash (base, FP8)1CatAI/1Cat-vLLM1Cat-vLLM #1091 (merged)1Cat-vLLM #10921Cat-vLLM #11021Cat-vLLM #11251Cat-vLLM #11591Cat-vLLM #11611Cat-vLLM #11651Cat-vLLM #1166turboderp-org/exllamav3exllamav3 #475 (colmajor tile order)ai-bond/flash-attention-v100
stack
  • EXL3 (exllamav3, patched for SM70)
  • 1Cat-vLLM (SM70)
  • MTP speculative decoding
  • FP8 E4M3 KV cache
  • CUDA 12.8

01

Research log

  1. Tile order and pilot

    Picked a column-major tile order for the m8n8k4 tensor-core path. Pilot on two PCIe V100s: the exllamav3 encoder runs on SM70 and the order is quality-neutral. First decode microbench came in at 2.1-2.3x an FP16-weight kernel of the same structure.

  2. Converter fixes on SM70

    Fixed the BF16-pinned KDA path, FP8 scales applied twice on reload, and a Triton fp16 tl.dot that silently returned wrong rows on SM70. Downloaded the FP8 source, 306 GB.

  3. Restarted the first build

    The converter's forward pass decoded my tile order as the standard one, so calibration past the first quantized layer was garbage. Fixed and restarted from scratch. Found cross-socket PCIe P2P copies silently corrupting data and staged them through host RAM.

  4. First build serving

    First build done and serving TP4 on the quad through 1Cat-vLLM with my native EXL3 kernel: 48.2 tok/s without speculation, vision working, 101K needle passing. Per-layer KL sensitivity scan pointed at layers 3-17.

  5. Build 2 and long-context looping

    Build 2 used conversation-start-only calibration and measured 5-10% worse in KL. Both builds looped in agent sessions past ~85K context.

  6. Build 3 on real sessions

    Build 3 calibrated on replayed real agent sessions. It improved exact-string copying at ~56K context from 0/6 to 3-4/6, still short of cloud GLM's 3/3.

  7. MTP on SM70, build 4 started

    Got GLM's built-in MTP head working as a speculator on SM70. Started build 4 with the routed/all-token Hessian blend (-14% held-out expert error) and 3 bpw on five sensitive layers.

  8. Release on Hugging Face

    Build 4 finished at 2.12 bpw and went up to Hugging Face. KL vs FP8 dropped to 0.132 (from 0.165 on the first build). Served with MTP k=3 and the new FP32 sparse-MLA kernel at 74.9 tok/s.

  9. Drafter sweeps and first PRs

    MTP k=3 beat DFlash2 at every draft length and kept 131K context. Opened #1091 (MTP), #1092 (sparse MLA) and #1102 (EXL3 MoE) on 1Cat-vLLM, then #1125: prefill 4.4x faster on 29K tokens.

  10. #1091 merged

    The MTP wiring merged into 1Cat-vLLM main. A reported vision regression on dense screenshots turned out to be near-tie flips in both directions, not a kernel bug.

  11. Decode fast paths, 76 to 91

    Strip-major expert kernels and native mHC stages took decode from ~76 to ~86 tok/s. A profile found the eight-kernel KDA norm; #1165 and #1166 took it to ~91.

  12. Card recipe fixed

    Built the card verbatim and it failed at MTP init. Switched the recipe to cherry-picks, re-verified from a clean clone at 91.4-91.9 tok/s, and corrected #1166's gain to 0.10 ms/pass.

  13. Eight GPUs, 1M context

    TP4 × PP2 across both NVLink quads: 1M context, 175.1 tok/s aggregate at 8 streams, 210K needle passing, no MTP. Long agentic coding sessions showed the 2-bit loop and corruption past ~85K.

built on the work of

  • Z.ai for GLM-5.3-Flash and the FP8 release (MIT).
  • turboderp for exllamav3 (MIT), the quantizer under all of this.
  • Frenchy2k1 for the SM70 exllamav3 fork the conversion started from.
  • 1CatAI for 1Cat-vLLM and its SM70 GLM path, and for reviewing and merging #1091.
  • ai-bond for flash-attention-v100, which glm-attn-v100 and the FP32 sparse MLA kernel were forked from.

02

Get it running

you'll need

  • 4× V100 32GB for the card config (TP4 needs all four). Tested on SXM2 with NVLink; the card also lists PCIe.
  • 8× V100 32GB for the 1M-context config (TP4 × PP2, one pipeline stage per NVLink quad).
  • About 25.4 GiB of weights per GPU at TP4; KV pool about 1.3 GiB per GPU at 131K context.
  • To build the engine from source: CUDA 12.8 toolkit, gcc/g++ 14, 40-60 minutes.
  1. 1

    Check the hardware

    You need four V100 32GB cards in one box, all visible to one process. NVLink between the four is what I measured on. If you span GPUs that are only linked over cross-socket PCIe, check that peer-to-peer copies are actually correct: on my box they silently returned garbage between non-NVLinked pairs, and NCCL_P2P_LEVEL=NVL was the fix.

    shell
    nvidia-smi topo -m
  2. 2

    Download the weights

    About 97 GiB. Point the server at the local directory; that is the tested path. Stock exllamav3 cannot load this checkpoint, because the tile order is V100-specific.

    shell
    hf download philbert440/GLM-5.3-Flash-EXL3 --local-dir ./GLM-5.3-Flash-EXL3

    philbert440/GLM-5.3-Flash-EXL3

  3. 3

    Install the engine

    Install the prebuilt SM70 wheel from the glm-5.3-v100 release. It packages exactly the source tree the card numbers were measured on: 1Cat-vLLM main at ec60883ad with #1125, #1159, #1161, #1166 and #1165 cherry-picked and #1102 merged. The numbers themselves came from a source build of that tree; the wheel passes upstream's SM70 artifact check and installs and imports cleanly, but hasn't been benchmarked from the wheel yet. You still need the CUDA 12.8 toolkit, a C++ compiler and ninja at runtime, because the TileLang kernels JIT-compile at startup. To build from source instead, clone the repo at the tag (40-60 minutes); the kernel tests (80 pass) run from a source build.

    shell
    # from https://github.com/philbert440/glm-5.3-v100/releases/tag/glm53-card-20261010
sha256sum -c SHA256SUMS
uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python ./1cat_vllm-1.5.2.dev1486+glm53card-cp312-cp312-linux_x86_64.whl ninja --torch-backend=cu128

    Release glm53-card-20261010 (wheel)glm-5.3-v100: build from sourceModel card: build from source#1102#1125#1159#1161#1166#1165

  4. 4

    Launch (TP4, card config)

    v100/serve_glm53.sh in the repo runs the card's serve command, shown below in full; it reads GLM53_MODEL, GLM53_SEQS, HOST, PORT and CUDA_VISIBLE_DEVICES. Keep --max-num-seqs set: the default of 256 makes the MLA prefill workspace too big and the server will not start at 131K. --max-num-seqs 4 still fits and gives 143-156 tok/s aggregate. The alternative is TP4 × PP2 on eight GPUs with --max-model-len 1048576: 8× the context and more concurrency, but no MTP, so single-stream drops to 39.5 tok/s.

    shell
    GLM53_MODEL=./GLM-5.3-Flash-EXL3 ./v100/serve_glm53.sh

# the same thing by hand:
export CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0,1,2,3
export VLLM_USE_V2_MODEL_RUNNER=1 VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0
python -m vllm.entrypoints.openai.api_server \
  --model ./GLM-5.3-Flash-EXL3 --served-model-name glm-5.3-flash \
  --trust-remote-code --dtype half --tensor-parallel-size 4 \
  --kv-cache-dtype fp8_e4m3 --max-model-len 131072 --gpu-memory-utilization 0.92 \
  --max-num-batched-tokens 4096 --max-num-seqs 1 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3,"draft_sample_method":"greedy"}' \
  --reasoning-parser deepseek_r1 --tool-call-parser glm47 --enable-auto-tool-choice

    serve_glm53.shModel card: serve

  5. 5

    Smoke test

    The startup log should show EXL3 routed experts: mcg, K=2 ... colmajor tile order and the SM70 FP16 sparse MLA with packed E4M3FN KV and FP32-softmax decode/verify route. Healthy on an NVLink quad is about 91 tok/s on a 512-token greedy reply. MTP tok/s moves with the prompt text, so compare builds by ms per pass (32.4 here), not one prompt.

    shell
    curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"glm-5.3-flash","messages":[{"role":"user","content":"What is 17*23?"}],"max_tokens":512,"temperature":0}'

03

Downloads

VariantFormatRuns onSizeDL / 30d
EXL3 ~2.1 bpw, SM70 tile order
philbert440/GLM-5.3-Flash-EXL3
EXL3 ~2.1 bpw (sm70_colmajor tile order)4× V10096.9 GiB437

04

Inference stack

Build = main ec60883ad, cherry-picked #1125 (two commits, contains #1092), #1159, #1161, #1166, #1165, then merged #1102.

Cherry-pick, don't merge: those five PR branches sit on a newer main that breaks MTP draft-model init. #1102 predates it.

#1091 (MTP speculator wiring) is already merged into main.

The PRs are drafts under review. glm-5.3-v100 pins the combination: tag glm53-card-20261010 is the card build's exact source tree (b321701ed88f), with upstream imported unmodified and each PR commit replayed under its original author, and its release carries a prebuilt SM70 wheel.

upstream PRs carried · 1CatAI/1Cat-vLLM

  • #1102[Quantization][SM70] EXL3 routed-expert MoE, loading exllamav3 checkpoints
  • #1125[Perf][SM70] GLM-5.3: FP32 tensor-core sparse MLA for FP8-KV prefill
  • #1159[Perf][SM70] GLM-5.3: native mHC final stage for 2-8 tokens, cuBLAS KDA projection above one token
  • #1161[Bugfix][SM70] Serialize set-valued policy fields in the acceleration report
  • #1166[Perf][SM70] GLM-5.3: fused native mHC post+dot for 1-8 tokens
  • #1165[Perf][SM70] GLM-5.3 KDA: Triton gated RMSNorm on the FP32 recurrent output

05

Recipe

5.1Conversion

Source was the FP8 release, not BF16. 95.3% of the BF16 expert values sit exactly on the FP8 grid, so BF16 is close to an upcast. Conversion ran on V100s with a patched exllamav3: it started on two PCIe V100s and the release build ran across four, then eight, cards on my V100 box.

Getting exllamav3 to convert correctly on SM70 took several fixes. The KDA path was pinned to BF16, Triton 3.2 fp16 tl.dot silently returned wrong rows on SM70, FP8 weight scales were applied twice on reload, and batched global-scale search pinned scales at the grid edge. The worst one: the converter's own forward decoded my new tile order as the standard one, so every layer after the first quantized one was calibrated on scrambled activations. I restarted that run from scratch.

The tile order is column-major within each 16×16 tile, so a lane decodes eight consecutive positions straight into its m8n8k4 B fragment. In a pilot it cost nothing in quality (-0.02% proxy error vs the standard order). It does lock the checkpoint to engines that know the order.

5.2Bits and calibration

Routed experts are 2 bpw (K2, mcg codebook). Layers 4, 5, 7, 8 and 10 get 3 bpw, picked by a per-layer end-to-end KL scan; the converter's own error ranking did not predict which layers mattered. The MTP layer's experts are 4 bpw. Shared experts, attention, dense MLP, eh_proj, lm_head, routers, norms, embeddings and the vision tower stay FP16 or source dtype.

The release used 352 calibration rows of 8,192 tokens. Calibration moved from self-sampled chat and agent traces to replayed real agent sessions across builds. Conversation-start-only rows measured 5-10% worse in KL, so that idea was dropped.

Expert Hessians blend the all-token Hessian with one from the tokens actually routed to that expert, at 0.5. On held-out routed tokens that cut expert output error about 14%, 16 of 16 experts better.

5.3Engine work

#1102 is the EXL3 routed-expert MoE on SM70. It loads the exllamav3 output directly, decodes the trellis inside the matmul and reads both tile orders. The loader reorders tiles in place to a strip-major layout with per-lane window loads and 8-deep prefetch, bitwise identical to the previous kernels.

The sparse MLA kernel started as glm-attn-v100, a fork of ai-bond's flash-attention-v100. The old FP8-KV decode route wrote unscaled scores to FP16; the new kernel keeps scores and softmax in FP32 with tensor-core accumulation. #1092 covers decode and verify, #1125 routes prefill through it too: 4.4x on a 29K prompt.

#1159 runs the mHC final stage natively for 2-8 token verify batches and puts the KDA input projection on cuBLAS above one token. A profile of that build then found the KDA output norm running as eight eager kernels per layer; #1165 moves it to the Triton gated RMSNorm, bit-identical. #1166 fuses the mHC post-mapping and logits dot into one native kernel for 1-8 tokens.

5.4Speculative decoding

I started with DFlash2 as the primary drafter and native MTP as a fallback. Once MTP worked on SM70, I measured both on the same build. MTP with 3 draft tokens did 74.8 tok/s; DFlash2 plateaued around 69 tok/s for 3-7 draft tokens and only won on code prompts.

MTP also fits 131K context where DFlash2 maxed out at 99,072 tokens. Each extra MTP draft token added about 4.8 ms per step for only 0.13-0.18 more accepted tokens, so k=3 is the sweet spot. Quality was the same within noise on the task battery.

06

Results

Decode, card config

One user, greedy, from the model card speed table. Each MTP row adds the named kernels to the one above.

BuildSpeculationDecode tok/s
No speculationnone~48
Previous kernelsMTP k=3~76
+ #1102 strip-major + #1159MTP k=3~86
+ #1166 + #1165 (card build, clean clone)MTP k=391.4-91.9

measured on: 4× V100-SXM2-32GB (NVLink), TP4, FP8 KV; 1Cat-vLLM main ec60883ad + #1125, #1159, #1161, #1166, #1165, #1102

Card build, per pass
MetricValue
Single prompt, 512-token reply91.4-91.9 tok/s
Time per forward pass32.4 ms
Accepted tokens per pass2.79
5 varied prompts × 286.2 tok/s
4 users, --max-num-seqs 4 (before #1166/#1165)143-156 tok/s aggregate
#1166 alone (1,432 passes, identical text)32.465 vs 32.566 ms/pass = 0.10 ms

measured on: 4× V100-SXM2-32GB (NVLink), TP4, MTP k=3, card build from a clean clone

MTP vs DFlash2

Greedy, 4 chat prompts × 700 tokens, same build.

DrafterDraft tokenstok/sAccepted / stepMax context
MTP374.82.77131,072
MTP470.42.95131,072
MTP565.93.08131,072
DFlash2369.42.8898,304
DFlash2469.13.1098,304
DFlash2769.73.6598,304

measured on: 4× V100-SXM2-32GB (NVLink), TP4, FP8 KV; 1Cat main + EXL3 + MTP + glm-attn-v100 branch build

Prefill

Time to first token, unique prompts, no prefix-cache reuse.

PromptWith #1125Without
~1.1K tokens~0.8 s (1,250-1,475 tok/s)~1.8 s (~645 tok/s)
~29K tokens~21 s (~1,390 tok/s)~94 s (~314 tok/s)
70.8K-token needle55 s, pass266 s, pass

measured on: 4× V100-SXM2-32GB (NVLink), TP4, MTP k=3, FP8 KV

Eight GPUs, 1M context
MetricValue
Max context1,048,576
KV pool2,944,982 tokens (2.81× of 1M)
Decode, 1 stream39.5 tok/s
Decode, 4 streams100.8 tok/s aggregate (25.2 each)
Decode, 8 streams175.1 tok/s aggregate (21.9 each)
Needle, 210,565-token promptpass, 94.6 s

measured on: 8× V100-SXM2-32GB, TP4 × PP2 (one stage per NVLink quad), NCCL_P2P_LEVEL=NVL, no MTP, card build

Fidelity to FP8

KL and top-1 agreement of next-token predictions vs the unquantized FP8 model, three held-out texts of 1,536 tokens.

BuildKL READMEKL design docKL PythonMean KLMean top-1
First build (all K2)0.0880.1860.2200.16585.5%
Build 20.0960.2000.2310.17685.4%
Build 30.0860.1810.1870.15186.4%
Release (build 4)0.0700.1390.1880.13287.8%

measured on: Layer-streamed reference eval on V100, exllamav3

Task battery

Same harness as my Qwen3.8-27B quants (that row was measured on 2× V100). gsm8k strict match; factual and fabrication sets are LLM-judged. Greedy.

ModelFactual (75)Fabrications accepted (of 75)gsm8k-200
GLM-5.3-Flash EXL3 2.1 bpw1.00020.985
Qwen3.8-27B NVFP4, best of mine0.973270.965

measured on: 4× V100-SXM2-32GB (NVLink), TP4, MTP k=3

07

Findings

  • The 2-bit quant breaks down in long agent sessions. In multi-turn agentic coding it started looping past ~85K context and corrupted paths and JSON by ~120K. Needle retrieval at 210K still passed and serving checked out clean, which points at the weights.
  • An earlier symptom of the same thing: at ~56K context in a live session the run-3 build could no longer copy an address exactly. Cloud GLM got the replayed prompt right 3/3; my builds 0-4 of 6. Calibrating on real sessions helped (0/6 to 3-4/6) but did not fix it.
  • On three real bug-fix tasks it passed two, the same as the cloud GLM-5.3-Flash API, at a lower judge score (17 vs 20 of 30). One run per task.
  • TP8 is impossible with this EXL3 path: the per-rank MoE intermediate must be a multiple of 512, and TP8 gives 256. Expert parallelism is refused outright. MTP has no pipeline-parallel support. So eight GPUs means TP4 × PP2 without MTP.
  • The model card's build recipe was broken as published. Merging the PR branches onto the pinned base pulled in a newer main that fails at MTP draft-model init, so the pin did nothing. I rebuilt from the card verbatim, found it, switched the card to cherry-picks and re-verified from a clean clone.
  • Under MTP, tok/s on any single prompt depends on its text through draft acceptance. A kernel that changes numerics slightly changes the greedy text and therefore the tok/s. Milliseconds per pass is the fair comparison.
  • My #1166 gain was first posted as 0.16 ms/pass. It was measured on a stack with an extra unrelated kernel enabled; the clean A/B on the exact PR stack gave 0.10 ms (0.3%), and I corrected the PR.
  • Cross-socket PCIe peer-to-peer copies on my box silently returned corrupt data, with no error. Every non-NVLinked pair was affected; every NVLinked pair was clean. It first showed up as non-finite calibration rows.
  • MTP vs no-speculation greedy output is not token-identical (0/8). A teacher-forced check showed MTP's picks were near-ties at the same rate as plain decode (1.25% vs 0.89% of tokens, median margin 0.14 nats).