Storage Buckets gave working AI artifacts a different home

The Storage Buckets launch introduced a non-versioned storage option on the Hub, accessible through the browser, Python, and the hf CLI. Built on Xet, it was designed for frequently changing artifacts such as training checkpoints, processed data, logs, and agent traces.

The important distinction is between a final artifact worth publishing as a versioned repository and working files that need frequent updates. Treating them identically can add unnecessary friction. A separate storage primitive makes that choice explicit without abandoning the Hub’s permissions model.

Map your workflow into published outputs and temporary or evolving artifacts before choosing storage. Follow the source to current documentation for permissions and operations. Non-versioned storage requires its own retention and recovery plan when previous states matter.

Working artifacts are not all repositories

An AI project produces more than final models and datasets. It also creates intermediate files, evaluation outputs, logs and checkpoints that may be temporary or frequently replaced. Treating every working artifact like a finished release can make storage organization cumbersome; treating everything as an unstructured file dump makes it hard to recover the experiment later.

A useful storage design distinguishes what is being worked on from what is being published. The same file may move through both stages, but the expectations for retention, naming and access can differ.

Organize around an experiment identity

Give each run a stable identifier and keep its related artifacts under that identity. A checkpoint, configuration and evaluation report should be discoverable together. Names such as “final-new-2” are convenient in the moment and expensive when several people need to reconstruct what happened.

Keep a compact manifest describing the inputs and outputs. It can record model revisions, dataset references and the purpose of important files without duplicating the full artifacts. This makes a storage collection navigable even when its contents are large.

Separate permission from convenience

A shared storage location should not automatically make every artifact public or writable by every collaborator. Decide who needs to read, upload and delete. Temporary credentials for a job should have the smallest useful scope and should not be embedded in filenames, scripts or logs.

When artifacts contain user data or derived sensitive information, the fact that they are “intermediate” does not make them harmless. Apply the same data-handling expectations that would apply to the original input.

Plan for partial work

A job can stop during an upload or finish computation without successfully publishing its outputs. The downstream process should know whether an artifact set is complete. One practical pattern is to publish a completion manifest only after all required files are available and verified.

Avoid making consumers guess from the presence of a single file. A checkpoint without its matching configuration can be worse than an obviously failed run because it looks ready to use.

Retention is part of optimization

Not every intermediate needs to live forever. Define which artifacts are essential for reproducibility and which can be regenerated cheaply. Keep successful release artifacts and selected diagnostic failures; remove routine temporary outputs according to an explicit policy rather than an emergency disk cleanup.

The benefit of a working-artifact store is flexibility. The benefit of a well-managed one is that flexibility does not become confusion. Clear identities, permissions, completion signals and retention rules make stored computation useful long after the process that produced it has exited.