Superfluid Fleet Turns Mixed Local Machines Into One Inference Queue
Superfluid Fleet routes complete LLM sessions across heterogeneous local machines behind one endpoint. The design favors aggregate concurrency and fault tolerance, but every node must still fit the full model.

Fleet replication can improve local multi-user inference, but it scales concurrent sessions rather than the maximum model size.
Local AI often begins with a single workstation and an uncomplicated question: can this model run here? Once several people or automated tasks share it, the harder question becomes: where should the next conversation run? A studio, lab, or small office may already have multiple capable machines, but different accelerators and software stacks make them awkward to operate as one service.
Superfluid Fleet approaches that problem as request scheduling rather than model sharding. One head exposes a shared API while participating nodes each host a complete copy of the same model. A conversation is assigned as a whole to a node, and later turns can return to the machine holding the most useful cached prefix. This makes the fleet look like one endpoint to clients without requiring every device to cooperate on every generated token.
Replication changes the scaling target
There are two very different meanings of distributed inference. Tensor or pipeline parallelism divides one model across devices, which can make a model runnable when it does not fit on one accelerator. Fleet replication does not solve that problem. Instead, it increases the number of independent sessions that a collection of machines can handle.
That distinction determines the best workload. A single long generation cannot automatically consume the combined memory and compute of the fleet. Many simultaneous chats, coding agents, document jobs, or internal assistants are a more natural match because complete sessions can be spread across nodes. The useful unit of capacity is therefore concurrent work, not the sum of all device memory.
The strict boundary is model fit: every participating node needs enough memory for the full model copy and its runtime overhead. A small machine may still contribute, but only if it can load the chosen quantization and retain adequate cache space. Teams selecting a model should start with the least capable node they intend to include, not the aggregate specifications printed across the fleet.
Heterogeneity becomes a scheduling problem
Mixed hardware rarely delivers equal performance. Treating all nodes as interchangeable would give a fast workstation and a modest laptop the same arrival rate, creating queues on the slow device while capacity remains elsewhere. Superfluid reports that its head considers measured prefill and decode speed, queue depth, and cache occupancy when estimating where a new session should finish earliest.
This is important because inference has at least two performance phases. Prefill processes the prompt, while decode produces output tokens. A node that is attractive for a short prompt may not be the best destination for a long context or lengthy answer. Queue-aware placement can make uneven machines useful without pretending they are equal. Prefix affinity adds another tradeoff: reusing a conversation's cache may save work, yet an overloaded cache-holding node may still be a worse choice than recomputing elsewhere.
The announced implementation also keeps weights, activations, and KV caches on their serving nodes; token identifiers move across the network. Fleet traffic is described as encrypted and authenticated using a token-keyed Noise handshake. Those properties reduce data movement, but they do not remove operational duties. The head remains an important coordination point, fleet tokens need careful handling, and administrators must still decide which local clients may reach the API.
Failure recovery has a semantic cost to test
If a node disappears during generation, Fleet can place already delivered tokens into the prompt and continue the session on another node. This is a practical way to avoid emitting the same token twice, and the authors report completing requests during a node-loss experiment. It should not be interpreted as bit-for-bit continuation. Sampling state, runtime differences, and regenerated cache state can affect what comes next. For interactive use, a coherent continuation may be enough; reproducible batch pipelines should explicitly test interruption behavior.
The same caution applies to benchmark figures. The project reports higher aggregate output throughput and lower median time to first token as machines were added in its six-device, 64-user test. It also reports a twelve-machine Apple silicon run. These are useful demonstrations of the intended concurrency pattern, not universal capacity promises. Prompt lengths, output lengths, quantization, cache pressure, network quality, and arrival bursts can all change the result.
A practical evaluation plan
A team can evaluate this architecture without starting from peak throughput. First, confirm that the exact model build fits every candidate node with room for realistic context caches. Next, replay a representative mixture of short chats, long prompts, and multi-turn sessions. Measure per-user time to first token, output rate, queue time, and tail latency rather than only fleet-wide tokens per second.
Then remove the fastest node and the head in separate tests. Check whether streams remain usable, whether clients retry safely, and how quickly capacity stabilizes. Compare warm follow-up turns with cold placements to expose the value of prefix affinity. Finally, add nodes one at a time and record marginal improvement; a device that adds little throughput or worsens tail latency may be better left outside the serving pool.
Superfluid Fleet is best understood as a local inference load balancer with cache-aware session placement. It can turn scattered, individually useful computers into a shared concurrency pool while allowing each machine to use an appropriate runtime. Its value will be clearest where privacy, offline operation, and existing hardware matter more than running a model larger than any single node can hold.
Source: Distributed Inference for Local AI via Superfluid ↗. How we write


