A Three-Layer Pattern for Scaling CI Beyond Hosted Runner Limits

Llama.cpp's CI redesign shows how selective testing, shared compiler caches, and ephemeral external runners can relieve pressure without replacing GitHub Actions as the control plane.

Source artwork for How we optimized our workflows and overcame GitHub Actions limitations
Source artwork · Hugging Face / credited contributors ↗
THE SHORT VERSION

Scale CI by separating test selection, reusable build state, and execution capacity, then measure each layer independently while preserving a correctness-first fallback.

Large native-code projects often encounter several continuous-integration bottlenecks at once. More pull requests increase queue depth, broad hardware support expands the test matrix, and expensive compilation makes every cache miss visible. Treating all of that as a single "CI is slow" problem usually produces local fixes rather than a durable system.

The llama.cpp team has described a more useful decomposition. Its full CI spans more than 100 jobs, while GitHub-hosted concurrency and cache capacity are finite. The team responded with three coordinated changes: select pull-request jobs according to the diff, retain the complete suite for the master branch, move non-release compiler-cache snapshots to Hugging Face Buckets, and dispatch many Linux x64 workloads to ephemeral Hugging Face Jobs. GitHub Actions remains the workflow interface and source of status information.

That design is worth examining as an architectural pattern, not merely as a recipe for one repository. It separates decisions about what must run, what reusable state should survive, and where the work executes.

Start by reducing unnecessary work

Adding runners can shorten a queue, but it does not answer whether every change needs every job. A documentation edit, a platform-specific backend change, and a modification to shared tensor code have different risk profiles. A diff-aware workflow can map each class of change to the smallest credible test set, while a protected branch still receives comprehensive coverage.

This split creates two feedback loops. Pull requests get a focused loop optimized for developer response time. The integration branch gets a broad loop optimized for confidence and artifact production. The important constraint is that selection rules become part of the safety model: shared build files, dependency manifests, workflow definitions, and common headers should generally fan out to wider testing than narrowly scoped changes.

Teams adopting this pattern should maintain an explicit change-to-job map and test the map itself. A useful review question is not simply "did CI pass?" but "did this diff activate every relevant risk domain?" Periodic full runs can reveal gaps where the selector has become stale as the repository evolves.

Make cache ownership explicit

Compiler caches are valuable because native builds repeatedly produce identical object files under identical inputs. Yet a cache is not just a speed feature; it is shared state with keys, writers, readers, retention rules, and failure modes. When a hosted CI cache is too small, frequent eviction turns nominal cache use into unpredictable cold builds.

The llama.cpp arrangement establishes a useful ownership boundary. Eligible master builds publish cache snapshots, while pull-request jobs consume that shared baseline. A PR can still compile missing objects locally, but it does not become an authoritative writer. This reduces cache churn and limits the risk that arbitrary branch state displaces or contaminates the baseline used by other builds. Recent archives are retained per build key and older ones are cleaned up.

A robust key should encode every input that can change object compatibility: compiler family and version, target architecture, backend, important build flags, and relevant dependency revisions. Keys that are too broad risk incorrect reuse; keys that are too narrow destroy hit rates. Cache restore must also remain an optimization. If storage is unavailable or a key misses, the build should fall back to compilation rather than lose correctness.

External object storage adds operational questions. Teams should define retention, access permissions, integrity checks, and a budget for both stored bytes and transfers. They should measure restored bytes and time saved, rather than assuming a large archive is automatically beneficial.

Decouple orchestration from execution

Moving compute does not require moving the developer experience. In the reported design, GitHub Actions still queues jobs and displays logs and results. A dispatcher reacts to specially labeled jobs, validates the webhook and repository permissions, obtains a short-lived runner registration token, and starts an appropriately sized Hugging Face Job. The ephemeral runner registers, claims the queued work, reports through GitHub, and exits.

This is a clean control-plane/data-plane split. GitHub owns workflow definitions, checks, and pull-request visibility. The external service supplies execution capacity. Labels form the contract between them, so a CPU job and a GPU job can request different profiles without embedding provisioning logic throughout every build step.

Ephemeral runners also reduce dependence on long-lived machines, but they shift responsibility to bootstrap reliability. Image startup time, toolchain pinning, network access, cleanup, and dispatcher availability all affect the critical path. External capacity may remove a hosted-runner concurrency ceiling while introducing service quotas or per-minute cost. Queue time, startup time, execution time, and failure rate should therefore be measured separately.

Evaluate the layers independently

A staged rollout makes this pattern easier to reason about. First, establish a baseline for queue delay, job duration, cache hit rate, and failed reruns. Next, introduce selective PR coverage and audit missed-job risk. Then move cache snapshots while keeping execution unchanged. Only after cache restoration is dependable should stateless jobs migrate to ephemeral runners.

Each stage has a distinct success criterion. Selection should reduce scheduled work without weakening required coverage. Shared caches should reduce compilation time without making builds depend on cache availability. External runners should reduce queue pressure while preserving logs, cancellation, and status reporting in the existing interface.

The broader lesson is that CI scale is best handled as three related resource problems. Scheduling policy controls demand, shared cache design controls repeated work, and elastic runners control supply. Keeping those concerns separate lets a project tune or replace one layer without redesigning the other two—and keeps correctness as the invariant when any optimization is unavailable.

Source: How we optimized our workflows and overcame GitHub Actions limitations ↗. How we write

← Back to all articles