Skip to content

Representation Space Alignment

What Is Representation Space Alignment

The core idea of representation space alignment is to map data from different modalities (e.g., images, text, audio) into a shared vector space through their respective encoders, such that semantically similar samples from different modalities are placed close to each other in that space.

For example, an image of a cat and the sentence "a photo of a cat" should be mapped to very similar locations in the vector space after being encoded by their respective encoders.

This kind of alignment is the cornerstone of multimodal AI systems. Without alignment, there is no way to perform meaningful comparisons or interactions across modalities.

Why Alignment Is Needed

  • Foundation for multimodal learning: A wide range of downstream tasks (image-text generation, visual question answering, multimodal dialogue) all depend on a unified semantic space. The quality of alignment directly determines the upper bound of downstream task performance.
  • Cross-modal retrieval: Once aligned, text can be used to retrieve images, or images to retrieve text (Text-to-Image / Image-to-Text Retrieval). This has broad applications in search engines and e-commerce recommendation systems.
  • Zero-shot capability: An aligned model can perform classification, detection, and other tasks through natural language descriptions without any task-specific fine-tuning, dramatically reducing data annotation costs.
  • Value for data engineering: Alignment models can be used for data quality assessment, automatic annotation, deduplication, and cleaning — making them a key component of the data flywheel.

CLIP (Contrastive Language-Image Pre-training)

CLIP is a landmark work proposed by OpenAI in 2021 that established the paradigm for image-text alignment.

Architecture: Dual-Encoder Structure

CLIP adopts the classic dual-encoder architecture (Dual Encoder):

  • Image Encoder: Can be a ResNet or Vision Transformer (ViT). It encodes an image into a fixed-dimensional vector \(\mathbf{v}_i\).
  • Text Encoder: Based on the Transformer architecture. It encodes text into a fixed-dimensional vector \(\mathbf{t}_i\).
  • Projection Head: The outputs of both encoders are passed through linear projections to map them into a shared space of the same dimensionality, followed by L2 normalization.

The two encoders are completely independent, each encoding its own modality. They interact only at the loss function level through contrastive learning. This design allows features to be extracted independently at inference time, making it highly suitable for large-scale retrieval scenarios.

Contrastive Learning Loss Function

CLIP uses the InfoNCE Loss (also known as NT-Xent Loss) for training. Given \(N\) image-text pairs \(\{(\mathbf{v}_i, \mathbf{t}_i)\}_{i=1}^{N}\) in a batch, the loss function is defined as follows:

Image-to-Text direction:

\[ \mathcal{L}_{i \to t} = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(\text{sim}(\mathbf{v}_i, \mathbf{t}_i) / \tau)}{\sum_{j=1}^{N} \exp(\text{sim}(\mathbf{v}_i, \mathbf{t}_j) / \tau)} \]

Text-to-Image direction:

\[ \mathcal{L}_{t \to i} = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(\text{sim}(\mathbf{t}_i, \mathbf{v}_i) / \tau)}{\sum_{j=1}^{N} \exp(\text{sim}(\mathbf{t}_i, \mathbf{v}_j) / \tau)} \]

Total loss:

\[ \mathcal{L} = \frac{1}{2} (\mathcal{L}_{i \to t} + \mathcal{L}_{t \to i}) \]

Here, \(\text{sim}(\cdot, \cdot)\) denotes cosine similarity, and \(\tau\) is a learnable temperature parameter. The temperature controls the sharpness of the distribution: a smaller \(\tau\) concentrates the distribution on the most similar samples.

In essence, this loss function treats the \(N\) correct pairings in the batch as positive samples and the remaining \(N^2 - N\) incorrect pairings as negative samples, forming an \(N \times N\) similarity matrix where the objective is to maximize the diagonal entries. Therefore, the larger the batch size, the more negative samples, and the more effective the contrastive learning. CLIP was trained with a batch size of 32,768.

Training Data

CLIP was trained on the WIT (WebImageText) dataset, which contains approximately 400 million image-text pairs crawled from the internet. These data were not manually annotated — they were all sourced from images and their corresponding alt-text or captions found on web pages.

This large-scale weakly-supervised data strategy is one of the key factors behind CLIP's success: it avoids expensive manual annotation while leveraging the scale of the internet to achieve sufficiently rich semantic coverage.

How Zero-Shot Classification Works

CLIP's zero-shot classification process is remarkably intuitive:

  1. Construct text prompts: Fill each class name into a template such as "a photo of a {class_name}" to produce \(K\) text descriptions.
  2. Encode: Use the Text Encoder to encode these \(K\) texts and the Image Encoder to encode the image to be classified.
  3. Compare: Compute the cosine similarity between the image vector and each of the \(K\) text vectors.
  4. Predict: The class corresponding to the text with the highest similarity is the prediction.

This approach requires no training data — only the class names. On ImageNet, CLIP achieves a zero-shot accuracy of 76.2%, on par with a supervised ResNet-50.

CLIP Variants and Follow-Up Work

OpenCLIP

An open-source reproduction of CLIP developed by the LAION community. Trained on larger open-source datasets (such as LAION-5B with 5 billion image-text pairs), some configurations surpass the original CLIP in performance. OpenCLIP is currently the most widely used alignment model in the community.

SigLIP (Sigmoid Loss for Language-Image Pre-training)

Proposed by Google in 2023. It replaces the Softmax in InfoNCE with a pairwise Sigmoid loss, eliminating the need for global normalization. Each image-text pair is evaluated independently without depending on other samples in the batch, enabling the use of even larger batch sizes and improving training efficiency. SigLIP generally outperforms CLIP under the same compute budget.

ALIGN (Google)

Google's ALIGN model was trained on over 1 billion noisy image-text pairs using EfficientNet as the Image Encoder. ALIGN demonstrated an important insight: data scale can compensate for data noise. Even without careful data cleaning, a sufficiently large dataset allows the model to learn high-quality alignments.

EVA-CLIP

A work from the Beijing Academy of Artificial Intelligence (BAAI). Through improved training strategies (such as initialization via Masked Image Modeling pre-training) and larger-scale ViT architectures, EVA-CLIP achieved state-of-the-art results on multiple benchmarks at the time. It highlighted the importance of Image Encoder pre-training quality for alignment effectiveness.

Applications of Alignment in Data Engineering

Data Quality Filtering

The CLIP Score is a widely used metric for measuring image-text matching quality, defined as the cosine similarity between the image and text vectors:

\[ \text{CLIP Score} = \cos(\mathbf{v}, \mathbf{t}) = \frac{\mathbf{v} \cdot \mathbf{t}}{||\mathbf{v}|| \cdot ||\mathbf{t}||} \]

When constructing training datasets, the CLIP Score can be used to filter out low-quality image-text pairs. For example, the LAION-5B dataset was curated from Common Crawl by applying a CLIP Score threshold (typically \(\geq 0.28\)). This filtering effectively removes noisy data such as mismatched image-text pairs, meaningless text, and corrupted images.

Annotation Assistance

Leveraging CLIP's zero-shot classification capability, large-scale unlabeled data can be given weak labels (Weak Labeling):

  • Perform coarse-grained classification of images (e.g., scene classification, content categorization).
  • Generate candidate labels for images, which are then reviewed by human annotators, significantly reducing annotation costs.
  • Detect labeling errors in datasets: if CLIP's prediction disagrees with the original label, the sample is worth a manual review.

Cross-Modal Data Retrieval

Alignment models make cross-modal retrieval possible:

  • Text-to-Image: Input a text description and retrieve the most semantically relevant images. Useful for finding specific types of images when building datasets.
  • Image-to-Text: Input an image and retrieve the best matching text descriptions. Useful for verifying the matching quality of image-text pairs.
  • Deduplication: Clustering or nearest-neighbor search in the embedding space can efficiently identify semantically duplicate data.

Other Alignment Methods

ImageBind (Meta)

ImageBind, proposed by Meta in 2023, extends alignment to 6 modalities: images, text, audio, depth maps, thermal maps, and IMU data.

Its key insight is that paired data between all modalities is not required. ImageBind uses images as the "anchor" modality and learns pairwise alignments — image-text, image-audio, image-depth, etc. Thanks to transitivity, all modalities are naturally pulled into the same space. This design dramatically reduces the data requirements for multimodal alignment.

Alignment via Knowledge Distillation

When large-scale paired data is unavailable for a target modality, knowledge distillation can transfer the alignment capability of an existing model to a new modality:

  • Teacher: An already-aligned model (e.g., CLIP's Image Encoder).
  • Student: An encoder for the new modality (e.g., a point cloud encoder or a medical imaging encoder).
  • Objective: Make the Student's output vectors as close as possible to the Teacher's output vectors, thereby inheriting the Teacher's alignment properties.

This approach is widely applied in specialized domains such as medical imaging, remote sensing, and 3D point clouds.

Evaluation Metrics for Alignment Quality

Recall@K

Recall@K is the most commonly used metric in cross-modal retrieval tasks. It is defined as the proportion of queries for which the correct match appears among the top \(K\) most similar candidates.

  • Recall@1: The strictest — requires the top-ranked result to be the correct match.
  • Recall@5 / Recall@10: More lenient — measures whether the correct result appears among the top 5 or top 10 candidates.

Evaluations are typically conducted on the COCO and Flickr30k datasets, reporting Recall in both the Image-to-Text and Text-to-Image directions.

CIDEr

CIDEr (Consensus-based Image Description Evaluation) is primarily used to evaluate the quality of image captioning, measuring the similarity between generated text and reference text. It is based on TF-IDF weighted n-gram matching, assigning higher weights to more informative words. Although it does not directly assess alignment quality, it is frequently used in alignment evaluations that involve generation tasks.

Other Common Metrics

  • Zero-shot Classification Accuracy: Zero-shot accuracy on classification datasets such as ImageNet, directly reflecting the alignment model's semantic understanding capability.
  • Linear Probe Accuracy: Freeze the encoder and train only a linear classification head to measure the quality of the learned representations.
  • CLIP Score Distribution: Analyzing the distribution of CLIP Scores across a dataset provides an overall assessment of image-text matching quality.

Summary

Representation space alignment serves as the bridge connecting different modalities. CLIP established the classic paradigm of contrastive learning combined with a dual-encoder architecture, and subsequent works such as SigLIP, ALIGN, and EVA-CLIP have continued to improve upon it through advances in loss functions, data scale, and training strategies. In the field of data engineering, alignment models have become standard tools for data filtering, weak labeling, and cross-modal retrieval. As works like ImageBind extend alignment to more modalities, a unified representation space is becoming foundational infrastructure for general-purpose AI systems.

评论 #