Qwen3.6-40B Deckard Uncensored

1. 2026-05-21

   First W4A16 quant published

   Quantized DavidAU's 40B Deckard checkpoint to W4A16 AWQ, asym g128 with an MSE observer, MTP head kept high-precision, and published it. First uncensored model I shipped, and the start of the V100 track.

2. 2026-05-24

   Recipe sweep

   Worked through the 4-bit recipe space: group 32 vs 128, GPTQ vs AWQ, SpinQuant, mixed precision, and several calibration corpora. MSE-observer AWQ at group 128 won on served behavior and became the production recipe.

3. 2026-05-26

   Quant-degradation check

   Ran a weight-space diff against BF16. It flagged q/k/v at about 5.5 dB SNR with o_proj pristine, but gsm8k 0.99 and factual 1.0 confirmed behavior intact. Logged the lesson: raw-weight SNR does not predict behavioral harm under AWQ.

4. 2026-05-27

   Hermes-calibrated build

   Re-calibrated on a Hermes function-calling set to sharpen tool-use and published it as the agentic build. This is the one I serve.

5. 2026-05-28

   gsm8k budget trap

   gsm8k scored 0.40 until I realized the default generation budget was truncating the model's chain of thought. Raised max_gen_toks to 3072 and it jumped to 0.99. The model was never wrong; the harness was cutting it off.

6. 2026-05-29

   Agentic harness fixed and locked

   Standardized the τ-bench user-sim on grok-4.3, then chased two bugs: a 120 s timeout turning long thinking into false failures, and the user-sim stopping instead of confirming writes. With a 1200 s timeout and a patched user-sim, locked the harness and re-ran the full 115-task retail set.

7. 2026-05-29

   Agentic final and MoE comparison

   Deckard W4A16 landed at τ-bench 0.783 (90/115), zero infra errors. A 4-bit MoE comparison hallucinated less and ran ~2× faster but lost on reasoning and agentic, and was a different lineage, so it is a serving choice rather than a clean quant comparison.

8. 2026-05-30

   FP8 explored, 4-bit kept

   An FP8+MTP build of the lineage served on V100 and beat 4-bit throughput at concurrency 16, at equal quality, but its larger footprint cannot hold the KV and GatedDeltaNet state at production concurrency and gives up context. Confirmed 4-bit plus MTP as the right fit for two cards.

1. 1

   Pick a build

   Use the Hermes-calibrated repo for tool-use and agentic work; it is the one I serve and the one the agentic numbers were measured on. The base ultrachat-calibrated repo is the plain quant. Both are the same 4-bit recipe and the same size.

2. 2

   Download the weights

   About 38 GB. Point the server at the local directory.

   shellcopy

   hf download philbert440/Qwen3.6-40B-DeckardUncensored-OpusDistilled-HermesCalibrated-W4A16-AWQ --local-dir ./deckard-40b-hermes

   Hermes-calibrated build (Hugging Face)Base quant

3. 3

   Install the engine

   Install the current upstream 1Cat-vLLM release (1.5.1). No fork is needed; the SM70 FLASH_ATTN_V100 attention and the TurboMind W4A16 path are in the stock wheel. On Ampere or newer it also runs on stock vLLM.

   1Cat-vLLM releases

4. 4

   Serve on 2× V100

   Tensor-parallel 2 with FLASH_ATTN_V100, TurboMind W4A16, fp8_e5m2 KV, and MTP speculative decoding. One thing to know for benchmarks: this is a thinking model, so give it room. gsm8k needs max_gen_toks of at least 3072; truncating the chain of thought at 2048 drops the score to 0.40 as an artifact, not a real regression.

   shellcopy

   export VLLM_SM70_FLASH_ATTN_V100=1 VLLM_SM70_QUANT_BACKEND=turbomind
   python -m vllm.entrypoints.openai.api_server \
     --model philbert440/Qwen3.6-40B-DeckardUncensored-OpusDistilled-HermesCalibrated-W4A16-AWQ --trust-remote-code --dtype half \
     --attention-backend FLASH_ATTN_V100 --tensor-parallel-size 2 \
     --kv-cache-dtype fp8_e5m2 --max-num-seqs 6 \
     --speculative-config '{"method":"mtp","num_speculative_tokens":4,"attention_backend":"FLASH_ATTN_V100","draft_sample_method":"greedy"}' \
     --compilation-config '{"cudagraph_mode":"full_and_piecewise","cudagraph_capture_sizes":[1,2,4,8]}'

   Model card: serve

Runs on upstream 1Cat-vLLM, no fork. Serve with the SM70 FLASH_ATTN_V100 backend and the TurboMind W4A16 path.

The base checkpoint's GatedDeltaNet layers are kept as-is: there is no better SM70 kernel for them, and swapping them out would be a different model and lose the long-context behavior.

Benchmarks were run under 1Cat-vLLM on 2× V100; the current upstream release is 1.5.1.

• The W4A16 quant is not noticeably degraded against BF16. gsm8k 0.99 and factual 1.0 settle it; the weight-space SNR scare on q/k/v did not translate into behavioral harm because AWQ smoothing games exactly that metric.
• The gsm8k 0.40 I first saw was a harness artifact, not the model. This is a thinking model, and the default generation budget truncated its chain of thought. Raising max_gen_toks to 3072 gave 0.99. The same class of trap hit later FP8 runs where a short generation budget or an early stop token read as a wrong answer.
• The agentic harness needed two fixes before the number meant anything. A 120 s request timeout was turning the model's long multi-turn thinking into false reward-zero failures, and the grok-4.3 user-sim was emitting a stop token when the model asked for the confirmation that retail policy requires, instead of answering yes. With a 1200 s timeout and a patched user-sim, the clean run came in at 0.783.
• Where it actually fails on τ-bench: compound and conflicting multi-operation tasks, and tasks that require computing and reporting an exact refund total. It nails authentication, lookups, single writes, and confirmation; it drops precision and completeness on the hard compound cases.
• An FP8 build of the same lineage with MTP turned out to serve on V100 after the fork added a dense SM70 FP8 path, and at concurrency 16 it beat the 4-bit prod throughput (about 112 against 80 tok/s) at equal quality. But FP8's ~43 GB footprint leaves too little room for KV and the GatedDeltaNet state at production concurrency, and it cannot reach the context the 4-bit build serves. 4-bit plus MTP stays the right fit for two cards.
• A MoE alternative at the same 4-bit (a community Qwen3.6-35B-A3B) hallucinated less (confab 0.067) and ran roughly twice as fast, but scored lower on reasoning and agentic (gsm8k 0.95, τ-bench 0.687). It is a different checkpoint and lineage, so it answers which to serve, not dense-versus-MoE in isolation.
• Dense FP8 without MTP, 8-bit on V100 generally, and a per-channel compressed-tensors FP8 build were all dead ends on this hardware: no SM70 kernel, or the quant scheme is gated above compute capability 70. That is the wall that makes 4-bit the move.
• The quant inherits one deterministic quirk from the base: a reproducible typo on a narrow internal-vocabulary identifier that survives every calibration and stack change, so it is intrinsic to the model, not the quant. A LoRA fix was scoped but not shipped, so it is not in these numbers.