DeepSeek-V4.1-Flash-1017 EXL3

1. 2026-10-17

   Start

   First time running DeepSeek locally. 1Cat 1.5.1 supports V4-Flash, not V4.1; V4.1 lives in open PR #1031, which is 1.5.1 plus 5 commits. Pulled the 510 GB official checkpoint and built the SM70 tree against CUDA 12.8.

2. 2026-10-18

   First tokens: 2.82 tok/s

   Served the official MXFP4 weights with experts spilled to host RAM, 32K context, one sequence. Chat and a tool call were correct; decode was 2.82 tok/s. Spilling the least-routed experts first took it to 4.07.

3. 2026-10-18

   Graphs, staging, 1M

   Found CUDA graphs had been silently disabled; PIECEWISE graphs doubled decode to 8.53 tok/s. Staging spilled experts into HBM gave 10.20. Moving to DeepSeek's 1M context cost speed (9.02); a bit-exact fused Sinkhorn kernel got it to 11.85.

4. 2026-10-19

   EXL3 experts in VRAM

   Wrote the mul1 codebook, K5 and per-expert-K paths for the SM70 EXL3 MoE kernel and loaded 0xSero's 2.7 bpw pack expert-parallel. No more spill: 16.54 tok/s, TTFT about 0.2 s instead of 2.4 s, at 256K context.

5. 2026-10-19

   Packed FP4 KV, native 1M

   Stored the compressed KV packed instead of as fp16. Greedy output stayed byte-identical and the KV pool reached 1,427,403 tokens, so 1M fit. A 533,739-token needle passed.

6. 2026-10-19

   30.1 tok/s, exact

   Fused hyper-connection kernels, split-K decode attention, per-layer graphs with device-side builders, async scheduling, deferred zero-copy Engram and a 1530 MHz clock lock. 30.1 tok/s single stream at KL 0.0000 vs the exact kernels, 74.0 aggregate at four streams.

7. 2026-10-20

   CPU oracle and the franken pack

   Built an fp32 CPU oracle from DeepSeek's reference code and measured packs against it: 0xSero 0.0663, HeNryous 0.0421, franken 0.0364. Same decode speed for all three. Fixed the DSpark PP bookkeeping and drafter graph bugs, and trimmed the stage-2 budget to 22.0 GiB so the drafter and 1M KV fit together.

8. 2026-10-21

   Published

   Batch-invariant GEMM buckets, and DSpark at two sequences once the drafter's all-reduces moved to NCCL. Uploaded franken-v100 to Hugging Face and opened PR #1243 against 1Cat-vLLM.

9. 2026-10-23

   Memory that stays put

   Every OOM so far was memory allocated after vLLM's profiler ran. Added demand-mapped KV through CUDA VMM, one shared pool for the decode graphs and fail-fast workers. A 30-minute agent-style soak passed with 70+ completions and no errors.

10. 2026-10-24

    Prefill set

    Exact decoder replay (layers 21-39 only on each request's last 3,072 rows), fused WMMA prefill attention at 1.45 ms per layer instead of 9.46, a tiled EXL3 prefill MoE kernel and Engram lookahead. Solo prefill went from 527 / 626 / 617 to 989 / 1,636 / 2,274 tok/s at 8K / 32K / 128K, KL 0.0358.

11. 2026-10-24

    Engram in RAM

    Moved the Engram tables from a ZFS dataset into hugepage RAM, split across both NUMA nodes. Solo prefill reached 2,186 / 2,841 / 2,376 tok/s.

12. 2026-10-24

    Layer-sliced prefill

    A long prefill now runs in slices of a few layers between decode steps. A stream decoding during a 32K prefill went from 6.0 to 52.6 tok/s, with sliced and unsliced prefill bit-identical.

13. 2026-10-24

    Fused indexer kernels

    Fused kernels for indexer candidate scoring, scan-and-mask and top-512. Solo prefill 2,985 / 3,231 / 3,194 tok/s, 56.9 tok/s for a stream decoding through a 32K prefill. Oracle KL unchanged at 0.0358.

14. 2026-10-25

    DSpark fix and fused decode, current build

    Scaled the DSpark taps before the fp16 cast so attention-sink tokens no longer overflow, and made the decode indexer a fused, graph-capturable kernel. Single-stream code decode went from 43.8 to 64.9 tok/s, and decode at 128K context from 28 to 53. KL unchanged at 0.0358.

15. 2026-10-25

    Image input

    Turned on the vision tower and aligner, which the port had skipped, with the weights in host RAM so the 1M context stays. Image tokens route through the MoE gate's vision bias and skip Engram, as in DeepSeek's reference. Also fixed a crash when a prefix-cache hit landed on a staged prefill. KL unchanged at 0.0358; 6 / 6 image tests answered correctly.

1. 1

   Check the hardware

   You need 8× V100-SXM2-32GB on NVLink, two quads of four. Stage 1 takes 27.77 GiB of weights per GPU and stage 2 25.29 GiB plus the drafter, so 16 GB cards are out. Plan for host RAM above the Engram tables plus 32 GiB, and disk for 218.6 GB of pack plus 203 GB of Engram shards.

   shellcopy

   nvidia-smi topo -m && free -g

2. 2

   Download the pack and the Engram shards

   The pack repo holds the EXL3 experts and every official non-expert tensor. The Engram tables are unmodified, so they come from DeepSeek's repo: download shards 47 and 48 into the same directory. Then run verify_engram.sh from the engine repo: it checks both shards against the Hub's sha256 and writes the .sha256-ok sidecars the launcher's preflight wants.

   shellcopy

   hf download philbert440/DeepSeek-V4.1-Flash-1017-EXL3-franken-v100 --revision 99d85a463b446e4ff07e173be6a8faf9b920b97d --local-dir ds41-franken-v100
   hf download deepseek-ai/DeepSeek-V4.1-Flash-1017 model-00047-of-00048.safetensors model-00048-of-00048.safetensors --revision 1bae0e1088993b9b82de08f59de5be5a34e42170 --local-dir ds41-franken-v100
   DS41_MODEL=$PWD/ds41-franken-v100 deepseek-v4.1-v100/v100/verify_engram.sh

   Model cardBase model

3. 3

   Install the engine

   Install the prebuilt SM70 wheel from the deepseek-v4.1-v100 release: the engine as served on 2026-10-25, image input included, with the three prebuilt DS41 extensions bundled. You still need the CUDA 12.8 toolkit, a C++ compiler and ninja at runtime, because the Engram reader and the TileLang kernels compile on first start. To build from source instead, clone the tag and follow the repo README. PR #1243 is the engine as of 2026-10-21, without the later prefill, decode and DSpark fixes.

   shellcopy

   git clone https://github.com/philbert440/deepseek-v4.1-v100
   # wheel + SHA256SUMS from the ds41-vision-20261025b release
   sha256sum -c SHA256SUMS
   uv venv --python 3.12 .venv
   uv pip install --python .venv/bin/python ./1cat_vllm-1.5.2.dev1149+ds41vision1025b-cp312-cp312-linux_x86_64.whl ninja --torch-backend=cu128

   Release ds41-vision-20261025b (wheel)deepseek-v4.1-v100: build from sourcePR #1243 (2026-10-21 engine)

4. 4

   Launch

   v100/serve_ds41.sh is the live configuration: PP2×TP4 cut at layer 20, --max-model-len 1048576, --max-num-batched-tokens 1024, --gpu-memory-utilization 0.943, DSpark with 5 draft tokens, --max-num-seqs 2, and every VLLM_DS41_* knob the published numbers were measured with. Copy the env example and set your paths and GPU quads, then dry-run: preflight checks that the GPUs are free, each quad is all-NVLink, and the shards, port and memory are in place, without launching. DS41_DSPARK=0 switches to plain decode. With passwordless sudo the launcher sets the CPU governor and locks SM clocks at 1530 MHz. With Engram loaded into RAM, a boot takes about 12 minutes.

   shellcopy

   cd deepseek-v4.1-v100
   cp v100/ds41.env.example ds41.env   # DS41_MODEL, DS41_PYTHON, DS41_GPUS_A / DS41_GPUS_C
   source ds41.env
   v100/serve_ds41.sh --dry-run --strict
   v100/serve_ds41.sh   # detached tmux session; --foreground to stay attached

   serve_ds41.shds41.env.exampleHF recipe (2026-10-21 launcher)

5. 5

   Smoke test

   Served model name is deepseek-v4.1-flash, on DS41_PORT from ds41.env. Healthy looks like /health 200, 391 for the prompt below, and a get_weather call with {"city": "Toronto"} and finish_reason tool_calls from v100/smoke.py, which also reports streamed decode tok/s. Thinking is off by default; pass chat_template_kwargs: {"thinking": true} and it arrives in the reasoning field. Images go in as OpenAI image_url content parts (data URLs work), also inside tool results, up to 64 per prompt; each image becomes up to 1,024 prompt tokens.

   shellcopy

   curl -s http://127.0.0.1:$DS41_PORT/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"deepseek-v4.1-flash","messages":[{"role":"user","content":"What is 17 * 23? Answer with just the number."}],"max_tokens":400,"temperature":0}'

   v100/smoke.py

PR #1031 (PLI Labs) is the V4.1 bring-up on 1Cat 1.5.1. PR #1243 stacks on it and carries the engine as of 2026-10-21: EXL3 MoE kernels, graph-captured decode attention, fused hyper-connection kernels, the Engram host service, DSpark, batch-invariant GEMMs.

The current build adds decoder replay, fused prefill attention, a tiled EXL3 prefill MoE kernel, RAM-resident Engram, layer-sliced prefill, fused indexer kernels, a graph-capturable fused decode indexer, the DSpark fp16 tap fix, a sliced-prefill fix for prefix-cache hits and image input. None of that is in PR #1243; deepseek-v4.1-v100 carries it.

deepseek-v4.1-v100 is the engine as served on 2026-10-25: #1031's tree imported unmodified, then the engine squashed on top. Its tree equals engine commit 714597f124 except for publication edits (host names in comments, extension and bench-script default paths, the Engram reader also finding its sources inside the package, internal gate drivers removed). No kernel or model logic differs. The two engine commits after it, the sliced-prefill fix and image input (engine commit 71c3e6e14f), are replayed as they are. The release ds41-vision-20261025b carries the current prebuilt SM70 wheel (up to 64 images per prompt); ds41-final-20261025 is the text-only one.

Every addition sits behind a VLLM_DS41_* knob. Each one was gated against the fp32 oracle before it went live.

CUDA 13 dropped Volta, so the stack stays on CUDA 12.8.

• CUDA graphs were silently off at the start. The engine asked for FULL_AND_PIECEWISE, the attention backend reported NEVER, and vLLM fell back to no graphs. Forcing PIECEWISE took decode from 4.04 to 8.53 tok/s with byte-identical output.
• The EXL3 trellis decode is ALU-bound on V100, not bandwidth-bound. MoE cost scales with unique experts touched per step. Split-K bought -18% at one route per rank and nothing at two or more; a persistent stream-K decode kernel came out at 0.55-0.85x of the grouped kernel.
• DSpark with the eager proposer was net-negative: 18.7 vs 30.5 tok/s plain, with normal acceptance. The drafter graphs gave 11% acceptance until I found that each drafter layer has its own KV group and block table. The B>1 crash turned out to be vLLM's custom all-reduce inside the drafter graph; running only the drafter's all-reduces on NCCL fixed it.
• DSpark drafts were all zeros for the first ~100 steps of every chat request. The drafter's taps were captured in fp16, and the attention-sink tokens at the start of a chat (BOS, the user tag, the first word) overflow 65,504. The resulting NaNs sat in the drafter's 128-token window until those positions scrolled out. Completion-style prompts have no BOS, so benchmarks never showed it. Scaling the taps down before the fp16 cast fixed it: steps with NaN drafts went to zero, first-draft acceptance on code went from 0.49 to 0.90, and single-stream code decode from 43.8 to 64.9 tok/s.
• The same bug explains an earlier ceiling. First-draft acceptance had topped out near 0.49 even on easy content and came in streaks, 0.20 after a step that accepted nothing and 0.67-0.91 after one that accepted something. A trace had ruled out an off-by-one in the PP bookkeeping. It was the NaN window: while it was in place, nothing the drafter proposed could be accepted.
• Batched decode flipped tokens that single-request decode picked by a 0.19-nat margin, by 0.2-0.7 nats per copy. Cause: cuBLAS picks its kernel by row count, amplified through 40 MoE layers at 2.8 bpw. Padding GEMM rows to fixed buckets (≤16 → 16, else multiples of 64) took mixed-step KL from 0.12 to 0.013 at no measured speed cost.
• The port's hidden-state drift vs the reference was the 2.7 bpw experts, not missing activation quantization. Per-layer cosine fell smoothly from 0.999 at layer 0 to 0.87 by layer 30. An oracle without activation QAT matched the port worse (0.875 vs 0.898).
• Capping prefill tokens per step was a poor fix for head-of-line blocking. On a cold-Engram bench, a 128-token cap lifted the co-running decoder from 0.8 to 3.20 tok/s and cut prefill from 480 to 307 tok/s. Chunk cost is mostly fixed: a 1024-token chunk already touches 86 of 94 local experts. Layer-sliced prefill is what fixed it.
• A decode-priority lane on its own was rejected: no decoder gain, prefill down 25-30%. Prefill chunks of 2048 did not fit 1M; the window and state groups cap context at about 791K there.
• Engram on a 16K-record ZFS dataset capped cold reads at about 2.0 s per chunk's 49K records, regardless of thread count. Holding the tables in RAM cut a 1024-token chunk's gather to 0.9 ms from about 400 ms of cold pread per rank.
• Binding workers to their NUMA node changed nothing measurable (4.04 vs 4.03 tok/s). Locking SM clocks at 1530 MHz did: 26.90 to 27.68 tok/s median, minimum 23.05 to 26.92, because auto-boost dipped between latency-bound decode bursts.
• The checkpoint always had a vision tower; the V100 port just skipped it. Running it without costing context meant keeping its 926 MiB of weights in pinned host RAM and streaming them through two small GPU buffers per image, because the 1M KV pool leaves about 8 MB free per GPU. One surprise: PyTorch attention takes the math path on Volta and builds the full score matrix, 4.2 GiB for a large image. Chunking the queries keeps it near 128 MB with bit-identical output.
• prompt_logprobs on long prompts runs out of memory in the 1M DSpark configuration. There is no slack for full-vocabulary chunk logits.