Tuning Local AI for Hermes on DGX Spark

On August 16 I published a benchmark of local open-weight models on a single DGX Spark-class machine and ended the tuning section with a number I was pretty happy about. Out of the box, Qwen3.8-27B (NVFP4) on vLLM gave me 11.3 tok/s. After pointing an agent at a community tuning repo, it reached roughly 20 tok/s — nearly double, from settings alone.

That post's real conclusion was not a model recommendation. It was this: every number in this article moved, sometimes by 2×, based on settings, not hardware and not model choice.

This is the follow-up. Same box. Different model. 75 to 79 tok/s single-stream.

Roughly 4× the tuned Qwen result, and the hardware never changed. Two things got me there: a better-suited model, and a serving configuration I had to learn the hard way. This post is mostly the second part, because that is the part nobody publishes.

What is Ornith?

Ornith comes from the DeepReinforce team. Ornith-1.0 was not trained from scratch — it was post-trained on top of Qwen 3.5 and Gemma 4 with additional continued pretraining, mid-training and post-training, and released under the MIT licence with no regional limitations. Ornith-1.5 extends that into what they call an end-to-end self-improvement loop: the model proposes its own training tasks, generates the scaffold to solve them, produces solution rollouts, and optimizes all three stages with reinforcement learning (ornith.ai).

The variant that matters for a 128 GB workstation is Ornith-1.5-35B-A3B — a 35B mixture-of-experts model that activates only about 3B parameters per token. That architecture is the whole reason it works here: you get the judgment of a 35B model at roughly the decode cost of a 3B one.

The vendor-published agentic coding numbers for that size class:

Source: ornith.ai/ornith_1_5.html and the Ornith-1 repo. These are the publisher's own figures, averaged over five runs — treat them the way you should treat any vendor benchmark.

The reason I care about the coding and agentic columns rather than the essay-writing ones is that this box is not running a chatbot. It is running Hermes, an open-source agent harness that talks to any OpenAI-compatible endpoint. Ornith exposes exactly that, with tool calling and a separate reasoning channel, so the harness works against it unmodified — you point OPENAI_BASE_URL at your own server and the agent never leaves your network.

The one constraint that governs everything else

The DGX Spark (GB10, sm_121a) has 119.7 GB of usable unified memory. CPU and GPU share one LPDDR5X pool. Two consequences surprise everyone arriving from discrete-GPU deployments:

• --cpu-offload-gb is a no-op. There is no separate host RAM to offload into.
• The model competes with the desktop, the browser and every agent process for the same pool. Size it as if it owns the box and you will put the machine into swap.

119.7 GB is a hard ceiling, and three candidates failed it outright. Ornith-1.5-397B-NVFP4 needs 221.7 GiB. Qwen3.8-Flash-Next needs 335 GiB at full precision and still 126 GiB as a 4-bit NVFP4 quant. The practical ceiling on this hardware is roughly a 35B–70B class model at 4 bits.

Ornith-1.5-35B-A3B-NVFP4 loads its weights in 21.97 GiB. That is what makes the rest of this possible.

The serving command

vLLM 0.27.1 (vllm/vllm-openai:latest), in Docker:

vllm serve ornith-ai/Ornith-1.5-35B-A3B-NVFP4 \
  --revision 94e431d9cc47fa1986a7a1a4e9a80f7f118b03aa \
  --served-model-name ornith-1.5-35b-a3b-nvfp4 \
  --max-model-len 262144 \
  --kv-cache-memory 34359738368 \
  --gpu-memory-utilization 0.80 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_xml \
  --speculative-config '{"method":"qwen3_5_mtp","model":"ornith-ai/Ornith-1.5-35B-A3B-NVFP4","num_speculative_tokens":3}'

Plus two environment variables that are not optional:

Why each parameter is set the way it is

--kv-cache-memory 34359738368 — the most important setting

This is an absolute byte count (32 GiB), and it should be preferred over tuning --gpu-memory-utilization alone. Utilization is a fraction of whatever happens to be free at startup, so it silently re-inflates whenever the rest of the box is idle when vLLM launches.

I learned this the expensive way. At 0.90, vLLM reserved 82.55 GiB of KV cache. Peak actual usage across the entire life of the container was 5.3%. The box went into swap, about 10.7 GB of the inference server itself was paged to disk, and agents began failing. Dropping to 0.80 was not enough — it still reserved 70 GiB and swapped again two days later.

Why 32 GiB is provably safe, not a guess. vLLM reports 6,183,859 tokens for 70.05 GiB, i.e. ~88,278 tokens per GiB. The scheduler can never demand more KV than every slot full at maximum context:

max_num_seqs 8 × max_model_len 262,144 = 2,097,152 tokens = 23.8 GiB

32 GiB is 35% above a ceiling the engine is structurally incapable of exceeding. That is arithmetic, not an estimate.

Recompute if you change either input. The formula is seqs × model_len / 88,278 = GiB floor. Raising max_num_seqs to 16 moves the floor to 47.5 GiB, and 32 GiB would begin preempting.

--max-model-len 262144

Serve the model's full window. On this architecture only a few layers are full-attention — the rest are GatedDeltaNet with constant-size state — carrying 2 KV heads at head_dim 256, so long contexts are cheap here in a way they are not on a conventional transformer. I verified judgment quality at a 219,998-token prompt: 7 of 7 planted anchors recalled correctly, no degradation. The window ran out before the model did.

The real limit is arithmetic: prompt + max_tokens must fit under 262,144. With a 32,768 output cap, usable prompt is 229,376 tokens.

--max-num-seqs 8

This is not a memory lever — vLLM allocates KV on demand and does not reserve per slot. It is a stability ceiling. 8 is the highest value I measured clean; a concurrency-16 sweep killed the engine with a CUDA illegal memory access.

Do not set it to 1. One agent is not one request: background review, title generation and subagent delegation each issue their own completions, and a cap of 1 head-of-line blocks the live turn behind a background task. Across thousands of samples I never saw more than 5 running or 1 waiting.

--max-num-batched-tokens 8192

vLLM auto-clamps the prefill chunk to 2048 whenever speculative decoding is enabled, and warns about it on every boot. That is a 4× cut from its own default, and it lands squarely on the slow path that matters: the ~10% of agent turns that miss the prefix cache and must prefill a fresh ~77k-token transcript. Setting the flag explicitly overrides the clamp.

--enable-prefix-caching

Must be requested explicitly — vLLM leaves it off by default for this hybrid GatedDeltaNet + full-attention architecture. It is the single biggest win for an agent loop, which re-sends a growing transcript every single turn. Measured on a 15.6k-token prompt, time-to-first-token dropped from 3.65 s to 0.90 s. Steady-state hit rate runs 86–90%.

Note the interaction with the section above: the prefix cache lives in the same KV pool. Cached blocks are evictable and do not appear in the usage figure, so shrinking KV too aggressively trades swap pressure for cache misses. Watch the hit rate after any change; below ~85%, raise --kv-cache-memory.

--speculative-config ... num_speculative_tokens 3

MTP (multi-token prediction) speculative decoding, output-preserving. Draft depth must be measured, not assumed — and it changed when upstream re-uploaded the MTP head mid-week:

On the older weights, k=3 was the worst option and k=1 the best. On the new weights that inverts. If you are running different weights than the pinned revision above, re-measure before trusting k=3.

And k=3 is provisional even here. It leads k=1 by only 2.1% on a single run, which is inside noise — k=2 landing below both proves the sample is noisy. k=1 has about 30% better median inter-token latency (22.65 ms vs 31.69 ms) because k=3 emits in bursts. Choose k=3 for agent throughput, k=1 for anything a human watches stream. If it looks like a coin flip, prefer k=1.

The parsers

--reasoning-parser qwen3 and --tool-call-parser qwen3_xml with --enable-auto-tool-choice. Without these, reasoning traces leak into message content and tool calls do not parse. The Qwen lineage shows up here too — Ornith inherits Qwen's chat template and tool-call format, which is why the Qwen parsers are the correct choice.

Client-side settings (Hermes)

The server is only half of it. Three agent-side settings mattered:

Where it landed

For contrast, the 8/16 post's tuned figure on this same box was roughly 20 tok/s, and the untuned figure was 11.3.

Read this before you go live

Three of these cost me real hours.

1. The engine deadlocks, and /health lies about it. vLLM's V1 engine core can deadlock while the API server stays alive. /health returns 200 throughout, /v1/models answers normally, and zero tokens are generated. I lost three hours to this before detecting it, then watched it recur the next day. This is a known open upstream defect (vllm-project/vllm#37729, related: #41530, #40926); the reported trigger is FP8 KV cache + prefix caching + a Qwen3.5-family model under concurrency, and this configuration has all four. The only documented workaround is --enforce-eager, at roughly 8× throughput cost, which is not viable.

Mitigation: monitor with a synthetic completion, not a health check. Send a real generation every five minutes and restart on failure. Two details matter. Use a generous max_tokens (512+) and require finish_reason: "stop" — Ornith spends its opening tokens on reasoning, so at max_tokens: 16 it returns content: null with finish: "length", which looks exactly like a failure and is not. That false signature produced three bogus results in an earlier benchmark and one spurious restart. And never restart on a single miss: probe, wait ~20 s, probe again, and add a cooldown so a second failure escalates to a human instead of looping restarts.

2. Swap is the silent killer. When the model is oversized the box does not OOM — it swaps, and the inference server itself gets paged to disk. The symptoms look like a network fault: the server reports healthy and logs no errors while clients time out. Monitor vLLM's own VmSwap in /proc/<pid>/status, not total system swap; a couple of GB of inert desktop-process residue sits in swap permanently and means nothing.

3. Never let a restart resolve main. Pin --revision. I did not, and an automated crash-recovery restart silently pulled a newly published upstream revision mid-incident — the model changed underneath me during an outage. It happened to be an improvement. It might not have been. Keep the prior revision in the Hugging Face cache so rollback is one flag and not a 22 GiB download.

One smaller one: vLLM logs in UTC while the host runs local time. A five-hour offset makes simultaneous events look unrelated when you are correlating an agent error against a server log. Use docker logs --timestamps.

Ten-minute validation checklist

1. docker logs vllm-server | grep "GPU KV cache size" — expect ~2.7M tokens. Much larger means the KV flag was not applied.
2. Confirm the boot warning max_num_scheduled_tokens is set to 2048 is absent. Present means --max-num-batched-tokens did not take.
3. Send a real completion with max_tokens: 512; expect finish_reason: "stop" and non-empty content.
4. free -g — swap in use should be near zero.
5. After an hour of traffic, confirm prefix cache hit rate is above 85%.
6. Confirm VLLM_MARLIN_USE_ATOMIC_ADD=1 is in the container environment. Its absence degrades quality silently.

Things I tried that did not work

What this does not change

The 8/16 post's capacity warning still stands, and tuning does not repeal it. One Spark comfortably serves one user, and at a stretch two. Aggregate throughput climbs with concurrency, but per-user speed collapses, and I capped max_num_seqs at 8 for stability reasons, not because 8 people can share this box happily.

The other thing that has not changed is the conclusion. The effort it took to tune a local AI LLM to go from 11.3 to 20 to 76 tok/s took zero additional dollars of hardware but a meaningful number of engineering hours. If your plan is to trade a token bill for a $4,000 box, budget for the engineer who tunes and maintains it. That salary is part of the total cost of ownership, and it is the line item that decides whether local AI is actually cheaper for you.

Every figure above was measured on a single DGX Spark between 19 and 27 August 2026, at vLLM bench serve 1024 input / 1024 output, concurrency 1, unless noted. Where a number is provisional I have labelled it as such.

Sources and references:

• Ornith-1.5 model release and benchmarks: https://ornith.ai/ornith_1_5.html
• Ornith-1 repository, benchmarks and serving recipes (MIT licence): https://github.com/ornith-ai/Ornith-1#benchmarks
• Prior Patriot benchmark and tuning result on the same hardware: https://www.patriotconsulting.com/blogs/2026/08/16/your-right-to-a-local-llm-for-self-defense
• vLLM V1 engine-core deadlock (open upstream defect): https://github.com/vllm-project/vllm/issues/37729
• NVIDIA DGX Spark: https://www.nvidia.com/en-us/products/workstations/dgx-spark/
• Measurements taken on a single DGX Spark (GB10, sm_121a, 119.7 GB usable unified memory), 19–27 August 2026.

About the author. Joe Stocker is the Founder and Chief Technical Officer of Patriot Consulting, a former Microsoft Security MVP (2020-2026), and author of the book "Securing Microsoft 365." In his spare time Joe volunteers with several Microsoft programs including Microsoft's Defending Democracy (AccountGuard) program, Microsoft Tech for Social Impact, and the Microsoft Software and Systems Academy (MSSA), which serves military service members desiring to transition into the civilian workspace.