Multimodal embeddings and CLIP: how vectors connect text and image

By Ivo Donker — compiled with AI assistance (Claude & Gemini)

Traditional search engines and language models process text and image through strictly separated pipelines. Where a language model converts sentences into dense numerical vectors, images were analyzed for decades using convolutional networks trained only on predefined categories. Multimodal embeddings fundamentally break down this separation. By projecting both textual descriptions and visual information into one shared geometric vector space, it becomes possible to compare a photo directly with a written search query.

The introduction of CLIP (Contrastive Language-Image Pre-training) by OpenAI in 2021 marked the turning point for this field. Instead of classifying millions of manually labeled images, a multimodal embedding model learns to draw connections between arbitrary images and their accompanying descriptions from the public web. Anyone who wants to understand how modern visual search systems, zero-shot classification, and multimodal RAG architectures work must first look at the mathematical and architectural basis of these shared vector spaces. For a broader perspective on generative pipelines and generative capabilities across modalities, the overview of multimodal AI models offers a systematic comparison.

What is a shared vector space for text and image?

In a unimodal system, a text encoder generates vectors that reflect only the semantic relationships between words or paragraphs. Two synonymous sentences lie close together, but an image of the same concept has no equivalent position in that space. Consult the article in which embedding models compared are discussed to see how purely textual embedding models abstract semantics.

A multimodal vector space removes this barrier by training two different encoders — one for visual input and one for text — on one shared objective function. The result is a $d$-dimensional space (often 512, 768, or 1024 dimensions) in which both a photo of a red sports car and the text string "een snelle rode Italiaanse auto op het circuit" are converted into a normalized vector. The degree of substantive similarity between the two modalities is then expressed directly via cosine similarity:

similarity = cos(theta) = (v_tekst . v_beeld) / (||v_tekst|| * ||v_beeld||)

Because both vectors are normalized to unit length (length 1.0) during the transformation, the calculation reduces to a simple dot product. A high cosine value (close to 1.0) means that the textual description and the visual elements of the image share the same semantic meaning, regardless of the original form of the data.

The architecture of CLIP: dual encoders and contrastive learning

The power of CLIP lies in the simplicity of its architectural setup: the dual encoder framework. The model consists of two parallel components that operate independently of each other during inference:

• The Vision Encoder: Usually implemented as a Vision Transformer (ViT), such as ViT-B/32 or ViT-L/14, or as a modified ResNet network (such as ResNet-50). The Vision Transformer divides an image into fixed patches (for example 14x14 or 32x32 pixels), linearly projects these into patch embeddings, and processes the spatial relationships via self-attention layers.
• The Text Encoder: A standard Transformer decoder or encoder that processes a tokenized text sequence and summarizes it via a pooling layer into one representative vector of identical dimensionality to the visual vector.

During the training phase, batches of $N$ pairs (image, corresponding text) are loaded simultaneously. For a batch size of $N$, the model generates $N$ image vectors and $N$ text vectors. This creates an $N \times N$ matrix of cosine similarities. The contrastive learning objective (the InfoNCE loss) forces the model to simultaneously optimize two specific goals:

1. Maximize the cosine similarity on the main diagonal of the matrix ($N$ correct image-text pairs).
2. Minimize the cosine similarity for all $N^2 - N$ incorrect combinations within the same batch (negative examples).

# Vereenvoudigde representatie van de CLIP contrastieve matrix
import torch
import torch.nn.functional as F

# I_e: beeldvectoren (N x D), T_e: tekstvectoren (N x D)
I_e = F.normalize(image_encoder(images), dim=-1)
T_e = F.normalize(text_encoder(texts), dim=-1)

# Bereken logits met temperatuurschaal tau
logits = torch.matmul(I_e, T_e.T) * torch.exp(temperature)
labels = torch.arange(N)

# Symmetrische cross-entropy loss over rijen en kolommen
loss_i = F.cross_entropy(logits, labels)
loss_t = F.cross_entropy(logits.T, labels)
loss = (loss_i + loss_t) / 2.0

Because the model learns exclusively by contrasting pairs at enormous scale (originally 400 million image-text pairs), it builds a robust understanding of objects, styles, context, and actions without ever using specific category labels.

Zero-shot classification: recognition without labeled training data

Classic computer vision requires a labeled dataset for every new task: anyone who wants to recognize dog breeds trains a linear layer on thousands of photos of labradors, poodles, and terriers. CLIP makes this process unnecessary via zero-shot classification.

In zero-shot classification, the desired class names are embedded in a prompt template, such as "Een foto van een {klasse}" or "Een close-up van een {klasse}". The text encoder precomputes the embeddings for all possible classes. When a new image comes in, the vision encoder generates the corresponding image vector. The cosine similarity is then calculated between the image vector and all precomputed text vectors. The class with the highest score is selected as the prediction via a softmax function:

In practice, prompt ensembling — averaging embeddings across multiple prompt variants such as "een foto van een kleine [X]", "een korrelige foto van de [X]" and "een gecentreerde afbeelding van een [X]" — often improves classification accuracy by several percentage points without additional inference costs on the image side.

Cross-modal retrieval and visual search in practice

The primary industrial application of multimodal embeddings is cross-modal retrieval: searching large image collections with natural spoken language (text-to-image), or finding relevant documents and products based on a photo (image-to-image or image-to-text).

In a production environment, all images are passed through the vision encoder once. The resulting vectors are stored in a specialized vector database (such as Qdrant, Milvus, or pgvector with HNSW indexing). When a user types a search query, the text encoder converts this search term into a vector, after which the database performs an Approximate Nearest Neighbor (ANN) search. To technically set up such a search system, the guide on building semantic search via an API explains step by step how vectors are compared efficiently.

Because vectors in production environments change continuously due to additions, mutations, and recalculations, solid index management is crucial. Read how to maintain a vector index to prevent performance loss and memory fragmentation with millions of vectors.

Evolution of CLIP: OpenCLIP, SigLIP, and ColPali

Since the publication of the original CLIP model, substantial methodological improvements have been made that have significantly increased accuracy and computational efficiency:

• OpenCLIP: An open-source reproduction and extension trained on public datasets such as LAION-5B. OpenCLIP produced larger model variants (such as ViT-H/14 and ViT-bigG/14) that perform significantly better on fine-grained visual recognition tasks.
• SigLIP (Sigmoid Loss for Language Image Pre-training): Developed by Google Research. SigLIP replaces the classic softmax normalization over the entire batch with a simple pairwise sigmoid loss. This allows each image-text pair to be treated as an independent binary classification problem. This enables more efficient training at very large batch sizes and consistently yields higher zero-shot scores.
• ColPali and Late Interaction: Where standard CLIP compresses a complete image into one global vector, ColPali (based on PaliGemma) uses a multi-vector representation. Each document or visual page retains its individual patch tokens. Via ColBERT-like late interaction, complex document layouts, tables, and diagrams can be searched accurately without traditional OCR.

Anyone who wants to dive deeper into how modern models interpret visual content outside of embedding spaces will find practical guidance in the article about vision models that understand images. For applications focused on document processing and text recognition, the comparison between vision models and traditional OCR highlights when dense visual embeddings make traditional text extraction unnecessary.

Audio, video, and broader modalities: beyond still images

CLIP's contrastive learning model is not mathematically limited to text and static images. The same dual-encoder principles are now widely applied to other sensory modalities:

• CLAP (Contrastive Language-Audio Pre-training): Connects audio signals with textual descriptions. Audio signals are converted into log-mel spectrograms and processed by a 2D convolutional network or audio transformer. This makes it possible to search for concepts such as "blaffende hond in de verte met regen" directly in unannotated audio files. See the overview of AI for music and audio to see how specialized audio models relate to broad multimodal systems.
• Video-CLIP: Extends the vision encoder with a temporal dimension (3D convolutions or spatio-temporal attention) to link actions, camera movements, and dynamic sequences directly to text prompts.
• ImageBind: A model developed by Meta that combines six different modalities (image/video, text, audio, depth maps, thermal data, and motion sensors/IMU) into one single vector space by linking all modalities to a central visual anchor space.

To understand how raw signals from various sensory sources are broken down into manageable inputs for transformer networks, the guide on multimodal tokenization of images and audio offers a thorough mathematical foundation.

Fundamental limitations and blind spots of CLIP

Although CLIP excels at general semantic categorization, the architecture has clear shortcomings that engineers must take into account:

1. The "bag-of-words" phenomenon and lack of compositional understanding
Because CLIP is trained on global image-text similarities, the model regularly fails at tasks that require a strict grammatical relationship or spatial order. The model has great difficulty recognizing the difference between "een rode kubus op een blauwe cirkel" and "een blauwe kubus op een rode cirkel". Both sentences generate nearly identical cosine scores for both images.

2. Exact counting and numerical insight
CLIP can distinguish concepts such as "few," "many," or "a group of people," but reliably fails when asked to identify exactly seven apples on a table versus eight apples.

3. Resolution loss and small visual details
Because images are scaled to fixed resolutions during preprocessing (such as 224x224 or 336x336 pixels) and then divided into patches (such as 14x14 pixels), fine textures, small text elements, and distant objects disappear completely from the final embedding.

For those who want to validate model performance on these specific blind spots, the benchmark dossier on evaluating multimodal AI offers standardized methodologies to quantify reliability and hallucinations in image processing.

Privacy, GDPR, and compliance in multimodal data processing

Processing image and audio material with multimodal embedding models brings specific legal obligations under the General Data Protection Regulation (GDPR). Images of people contain personal data by definition, and in specific contexts, embeddings of faces can be classified as biometric data for the purpose of unique identification (special category of personal data under Article 9 GDPR).

When embeddings are generated based on camera footage, medical scans, or employee photos, organizations must apply a valid legal basis for processing and apply data minimization. For a detailed legal review, the dossier on AI models and privacy regarding GDPR compliance covers how data processing agreements, retention periods, and local hosting strategies should be arranged.

When building a robust API architecture for multimodal inference, integrating gateways and model routers can help route requests dynamically and absorb outages. Read the explanation of API aggregators and model gateways to see how proxy architectures manage rate limits and provider redundancy.

Comparative matrix of leading embedding models

The table below compares the most important multimodal embedding models currently deployed in production environments based on architecture, resolution, and application area:

Conclusion: the role of multimodal vectors in modern AI architectures

Multimodal embeddings form the connective tissue between unstructured visual and auditory media and structured computational systems. Where generative vision-language models (such as GPT-4o, Claude 3.5 Sonnet, or Gemini Pro) excel at in-depth interactive analysis and reasoning about individual images, dual-encoder models such as CLIP and SigLIP are indispensable for large-scale indexing, clustering, and real-time retrieval across millions of files.

The choice of a specific multimodal embedding architecture depends on the trade-off between latency, memory usage, and level of detail. For simple e-commerce search functions, a compact SigLIP or OpenCLIP model with one fixed vector per image suffices. For demanding document processing and technical manuals, multi-vector late-interaction approaches such as ColPali offer the highest accuracy by keeping visual page elements intact in the search index.