Local LLMs

Cloud models are great for Rivet's daily work, but I wanted serious local capacity for embeddings, batch evals, and experiments without a running API bill. pve3 (GERTY) hosts two Tesla V100 32GB cards — enough VRAM for a 40B-class model with the right quantization story.

Inference runs through vLLM on the 1Cat-vLLM fork, with work on KV-cache configuration, compressed-tensors quirks on Volta, and selective FP8/MTP experiments (including community Deckard-40B checkpoints). We learned the hard way that the on-disk BF16 base and the quantized artifact are different beasts — and that full FP8 on V100 is usually the wrong tradeoff versus W4A16 plus speculative decoding.

The bigger picture is agentic use: tool-calling harnesses generate training and eval data, so fixing the scaffold before SFT matters. Rivet Local (CT114) runs tuning sessions on local models and smaller Qwen graders; results land under /rivet-shared/deckard-eval/ for comparisons over time.

Tighter integration with RivetOS memory (local embedder offload), MTP speculative paths on 4-bit weights, and more disciplined eval loops before any fine-tune. Peak quality on existing hardware beats chasing the next download.