Sentence Transformers extended retrieval beyond text

The Sentence Transformers v5.4 article introduced multimodal embedding and reranking workflows. Embedding models place different kinds of inputs into a shared representation space, while rerankers score mixed-modality pairs for relevance. The guide connected these capabilities to search and retrieval-augmented applications.

This opens a useful design space: a query need not have the same modality as the material being searched. But different models support different input combinations. A shared library interface is not a promise that one checkpoint can handle every possible pairing.

Choose a retrieval task first, then select a model whose supported modalities match it. Build a small evaluation set with positive and confusing negative examples. Use the source’s embedding and reranking sections to understand where each component belongs.

Matching across media changes the retrieval problem

A text-only search system assumes that both the question and the searchable material can be represented as text. That assumption becomes limiting when the useful evidence is a photograph, a diagram or another non-text input. Multimodal representations offer a way to compare different kinds of material within a retrieval workflow.

The important question is what the match is supposed to mean. A picture can be relevant because it contains an object, communicates a scene or includes readable text. Those are different signals. A system that is good at recognizing a broad scene may still struggle with a small label inside the image.

Define the collection before the model

For an image library, decide whether users search whole images or regions within them. For a document collection, decide whether the unit is a page, a figure or an entire file. Store enough metadata to take the user back to the relevant location.

Clean identifiers are essential. An excellent similarity score is not useful if the result points to a thumbnail that cannot be traced to the original asset. Keep the relation between the representation, the source item and its access permissions intact during indexing.

Evaluate different query types

Build examples that test the intended use: objects, spatial relationships, visual styles, text within images or combinations of these. Include negative examples that are visually similar but semantically wrong. A red object is not necessarily the particular red product the user requested.

Compare the proposed approach with simple baselines such as captions or existing metadata search. Multimodal retrieval should solve a real gap rather than replace a simpler method that already answers the task reliably.

Consider preprocessing as part of the model

Image resizing, cropping and normalization affect what information reaches the encoder. A diagram with tiny labels can lose its most important details during an aggressive resize. Test the complete preprocessing path, not only the model on a carefully prepared example.

When mixing text and image queries, check that scores are meaningful for each mode. Avoid assuming that a threshold chosen for one input type transfers unchanged to another. Calibrate decisions using the same kinds of queries the application will receive.

Present evidence in the medium that matters

Show the matched image or page alongside enough context to judge it. If a generated explanation accompanies the result, it should not invent details that were never established by the retrieval process.

Multimodal search is most useful when it makes previously hard-to-find evidence accessible. The engineering work is to preserve that evidence through ingestion, representation, ranking and presentation so the user can verify why a result belongs in the answer.