Search can keep more than one vector per document
Late-interaction models preserve finer matching information at the cost of larger indexes.
Finer-grained retrieval is worth evaluating when relevance gains justify the bigger index.
Sentence Transformers added a MultiVectorEncoder interface for late-interaction retrieval. Instead of reducing a text to one embedding, these models retain token-level representations and compare query and document features through a dedicated scoring operation.
That can preserve details a single vector smooths away. The trade-off is storage and retrieval complexity: more representations need to be encoded, indexed, and scored. The source also covers visual document retrieval, where page images can be matched directly to text queries.
Compare a dense baseline with a multi-vector model on a small relevance set first. Measure index size alongside quality and query latency. The guide covers supported checkpoint families, scoring, and pooling options, which help turn an interesting model into a manageable search system.
Compression creates a retrieval trade-off
A single-vector representation compresses a document into one point in a learned space. That can be efficient, but a long or varied document contains several ideas that may compete for representation. A query about one small detail can be difficult to match if that detail disappears inside an overall average meaning.
Keeping multiple representations gives the scoring process more local information. The trade-off is straightforward: richer matching can require more storage and more work at query time. Whether the improvement is worth it depends on the documents and the queries, not just on a model’s headline evaluation.
Think about the unit you are searching
For a collection of short product descriptions, one vector per item may already work well. For long technical pages containing tables, specifications and exceptions, preserving local detail may matter much more. Before changing models, ask whether the current failure comes from representation, poor chunking or missing source material.
A multi-vector model cannot recover a specification that was removed during document ingestion. Similarly, a strong scoring method will not fix an index whose pages are attached to the wrong document identifiers. Data preparation remains part of retrieval quality.
A practical comparison
Take a set of real queries with known relevant documents. Run the current retrieval approach and the proposed one against the same collection. Include queries that require matching a small detail, not only broad subject similarity. Compare the first few results as well as the total number of relevant items retrieved.
Record index size, build time and query latency alongside relevance. If a larger index improves only a small subset of queries, it may be sensible to use the richer method as a later stage rather than applying it to everything. That decision should be driven by measured failures and the available infrastructure budget.
Visual documents need their own tests
When the searchable material includes page images, preserve the association between a match and the actual page. A reader needs to see the table, chart or passage that made the result relevant. Returning an entire long report without locating the useful page can erase the benefit of a better retrieval method.
Test rotated pages, small text and visually similar documents. Also include queries for which no suitable document exists. A confident nearest neighbour is not automatically a correct answer.
The practical attraction of late interaction is selective detail: it can retain evidence that a single summary vector loses. Its practical challenge is paying for that detail only where it makes the search experience meaningfully better.
Source: Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers ↗ · tomaarsen, NohTow, raphaelsty. How we write
