vLLM served the same tokens on 4× less energy than Ollama

Everything you see on this blog is cooked in-house. The image above was generated on one of my own RTX 3090s, offline, in about eighty seconds — no stock photography, no cloud image API. Full recipe below.

Feature image made in my lab — image model, prompt and settings

Image model

Qwen-Image 2512 — fp8 (e4m3fn) variant. The bf16 pair does not fit a 24 GB card; fp8 weights are stored compressed and cast at compute time

Text encoder

Qwen2.5-VL 7B — a vision-language model doing the prompt understanding, which is most of why the composition follows a long prompt

Generated on

1× NVIDIA RTX 3090 (24 GB) in my own lab, through a self-hosted ComfyUI — my hardware, my electricity, nothing leaving the building

Settings

1664×928 · 20 steps · cfg 2.5 · euler / simple · shift 3.1 · seed 1794249188 · ~81 s

Prompt

a dense bundle of network cables sweeping in from every direction and converging into one small matte-black hub device sitting alone on a dark workbench, the hub lit from within by a single warm amber glow, racks of server hardware far out of focus behind it, moody editorial hardware photography, cold blue and cyan light with a single warm amber accent, shallow depth of field, 50mm, dark background, fine detail, cinematic, high dynamic range, no text

Negative prompt

text, words, letters, watermark, logo, signature, caption, ui, people, faces, hands, cartoon, illustration, cgi render, oversaturated, blurry, low quality, jpeg artifacts

Licence

Apache-2.0 — both the image model and the text encoder. Commercial use permitted with no use-based restrictions, which is exactly why this stack and not a prettier one

---

I cannot keep up with open-source AI. I say that as someone who does it for a living. A new model family every week, a new serving engine every month, a new quantisation format just as the last one settled. Following all of it is not a full-time job. It is several, and I have one.

For a while I assumed that was the price of building on open source. I no longer think so.

Open-source AI is consolidating onto hubs

Something quieter is happening underneath the release noise. The ecosystem is settling onto a small number of load-bearing projects that many people maintain, many companies depend on, and nobody can afford to let rot. Call them hubs. The hundred projects that do the same thing are still there; the weight has moved. When a new model lands, the question is no longer “which of twelve runtimes will support it” but “when does the hub merge it”.

That is what turns a stream of repositories into a foundation. You cannot build a company on a firehose. You can build one on a handful of hubs with stable interfaces and a community that outlives any single contributor. I run a company on exactly that, and this post is about the three hubs I lean on most and one measurement that shows why it works.

LiteLLM is the gateway hub

I have been at a lot of events lately, and one name keeps coming up when the conversation reaches “so how do you actually connect all of this”: LiteLLM. It is what I run, so I can say why.

Every client in my lab talks to one gateway. My coding CLI, the operator daemon, Open WebUI, the portal — none of them knows which engine is answering. The gateway speaks OpenAI /v1 and Anthropic /v1/messages, routes by capability tier (general, vision, code, reason, embed, image, video), fails over, logs everything, and enforces a budget per key.

The value is not any one of those features. It is that swapping what sits underneath — Ollama for vLLM, one model for another — is invisible to everything above. One seam, and everything talks to it. That is the whole point of a hub, and it is why the benchmark below exists: it only mattered because the seam above it made the swap cheap.

Under the gateway: two engines, both excellent

Beneath the gateway sit the serving engines. I run two. Ollama and vLLM are both hubs in the sense above: widely used, heavily maintained, quick to adopt new models. I want to say plainly that both are excellent, because what follows will look like a verdict and it is not one.

Ollama is built so that one machine serves one person beautifully. Pull a model, run it, done; that is how a great many people ran their first local model. vLLM is built so that one machine serves many people at once. Those are different design centres, and each engine is very good at its own.

I run Ollama in production for embeddings, on the same machine as the vLLM I am about to compare it with. Nothing below changed that.

Where they differ, and by how much

Same Qwen3 14B-class model, same 400-token prompt, temperature 0, each engine on its own RTX 3090 in the same machine, so the per-GPU energy counters read directly. Two loads: one request at a time, then four at once.

Engine	Concurrency	Throughput	J/token
vLLM	1	83.8 tok/s	4.13
Ollama	1	73.5 tok/s	4.45
vLLM	4	324.6 tok/s	1.07
Ollama	4	78.6 tok/s	4.36

Start with the serial rows, because they are the honest ones. One request at a time, the two engines land within about 7 % of each other on energy per token. That is close enough to make “both are excellent” a measurement rather than diplomacy. Part of that gap may not even be the engine: the quantisations were not identical (GGUF Q4_K_M under Ollama, AWQ under vLLM), and each configuration got a single trial.

Then four requests arrive together, and the engines stop looking alike. vLLM scaled to 3.9× the throughput while each request took about as long as before: roughly 4.8 s became 4.9 s. Ollama’s total throughput stayed flat and each request took three times longer, 5.4 s to 16.6 s. It served them one after another. vLLM batches continuously; Ollama queues. That difference is architectural, not a quantisation artefact, and it is why the energy column ends 4.1× apart — the number in the title.

There is an obvious objection. vLLM draws more power while it works; I saw it peak around 344 W. But it holds that draw for a quarter as long. Energy is power multiplied by time, and energy is what the meter counts. Instantaneous watts are not a cost. Joules are. A card that works hard and finishes early is the cheap one.

So the 4× is real, and it is not “better software”. It is a design centre meeting the workload it was designed for. Put one user in front of each engine and you would struggle to tell them apart.

The 250 W free lunch

With the energy counters live, I ran one more experiment. These are stock 350 W RTX 3090s. Capping them at 250 W cost 7 % of throughput and saved 23 % of the energy per token. That cap is now applied permanently.

It is not a slider you can keep pulling, though. There is a knee, and I only know it exists because I swept for it. Below the knee the card can no longer hold its clocks, the work stretches out, and joules per token climb back up. Cap too hard and you lose performance and efficiency together. 250 W is on the right side of it.

Why I can run a company on this

None of this was a project. Swapping the engine behind the gateway did not touch a single client, and both engines simply worked with the same model on the same hardware. That is consolidation paying off: a stable seam on top, two well-maintained engines underneath, and a lab where a 4× energy saving is a configuration change rather than a migration.

I still cannot follow everything. Nobody can. But I no longer have to, because the things that matter are converging on the places I already watch. That is what open source looks like when it wins — not a thousand options, but a few that many hands keep standing.