"Multimodal" is one of the most frequently appearing terms in model releases: capable of viewing images, hearing sounds, reading charts, and answering questions about a picture. Originally, this term described only one thing—namely, which input channels are used. Yet in public discourse, it is often misread as another kind of promise: that once a model receives images and sounds, its perception becomes as reliable as human perception. A batch of studies released between 2022 and 2024 examined this promise from different angles—and the results were not optimistic: state-of-the-art models commit widespread errors on visual tasks where humans expect perfect performance; they achieve high scores on mainstream benchmarks even without looking at images; and when describing pictures, they generate content not present in the images. This article examines six abstract-level materials and unfolds its analysis across four layers: the representation layer, the perception layer, the evaluation layer, plus an additional layer for audio—and asks one question: Are models truly *seeing* the image, or are they merely *guessing* what should be there based on linguistic habits?
## A Measurable Gap Between Image and Text
A paper posted in March 2022 (as listed on its abstract page: published at NeurIPS 2022) provides a precise answer at the representation layer: images and text do not truly merge inside the model. The authors call this the "modality gap" (modality gap), a geometric phenomenon that can be measured: within the shared representation space of multimodal models like CLIP, representations of the two modalities reside in separate locations, separated by a measurable distance—the authors’ phrase is "at arm's length" (at arm's length, quoted verbatim).
This gap is not a training accident but arises jointly from two stages. First is initialization: representations in deep neural networks are confined to a "narrow cone" (narrow cone, quoted verbatim), and two encoders initialized separately begin in distinct locations from the outset. Second is the contrastive learning optimization process: the loss function maintains a distance between modalities, with the magnitude of that distance governed by the temperature parameter. The authors also find that altering the width of this gap significantly affects downstream zero-shot classification performance and fairness.
The significance of this paper for this article lies in the following: image-text fusion is an engineering goal—not a default, completed state. Before discussing perceptual quality, "multimodality" must first cross this geometric gap at the representation layer.
## Reproducible Blind Spots
A paper posted in January 2024 traces the problem directly to the vision component itself. Its starting point: recent multimodal large models’ visual capabilities typically rely solely on CLIP—a component pretrained via instance-level contrastive learning. The authors construct a set of "CLIP blind pairs": image pairs that CLIP deems similar yet are visually clearly distinct. Based on these, they build the MMVP benchmark, covering nine fundamental visual patterns. State-of-the-art systems—including GPT-4V—commit widespread errors on direct questions targeting these patterns, and further provide "hallucinated explanations" (quoted verbatim) for their incorrect answers. More troublingly, the visual patterns that stump CLIP strongly correlate with those that stump multimodal large models: errors in the component propagate to the entire system.
The authors also propose a preliminary fix: integrating features from visual self-supervised learning (the MoF method) significantly enhances visual grounding (visual grounding) capability. They explicitly label this work a "preliminary effort" (quoted verbatim). Their conclusion: visual representation learning remains an open challenge, and accurate visual grounding is critical for future multimodal systems.
## The Simplest Tasks, the Largest Gaps
A paper posted in July 2024 shifts the testing direction: instead of probing complex understanding, it tests the most basic observation. The authors design BlindTest, comprising seven extremely simple tasks: whether two circles overlap, how many times two lines intersect, which letter is circled in a word, and how many circles appear in an Olympic-style emblem. Human expectation for accuracy on these tasks is 100%. Four state-of-the-art vision-language models average only 58.07%; the best-performing model, Claude 3.5 Sonnet, achieves only 77.84%.
The failure pattern is highly regular: varying image resolution and line width—or even switching to slower-thinking models—does not resolve persistent errors in scenarios where geometric primitives are close or overlapping; yet increasing the spacing between shapes and letters lifts accuracy nearly to 100%. In other words, the issue is not "not seeing," but rather failing to process spatial relationships when elements are proximate.
The most important finding of this paper appears in its subtitle (the current version’s title includes the subtitle, translated roughly as: "Failure in Translating Detailed Visual Features into Language"): linear probing (linear probing) experiments show that the visual encoder already contains sufficient information to solve these tasks—but the language model fails to decode that information into correct answers. The information exists; the breakdown occurs precisely at the stage of translating it into language.
## Generating Things Not Present in the Image
A paper posted in May 2023 (as listed on its abstract page: accepted at EMNLP 2023) investigates another type of evidence: object hallucination—i.e., when models generate objects inconsistent with the target image while describing it. The authors claim this is the first systematic study (explicitly stated in the original text) of object hallucination in large vision-language models, finding severe object hallucination across most representative models.
The key finding is that hallucinations are not random: objects that appear frequently in visual instructions—or that commonly co-occur with objects in the image—are markedly more likely to be hallucinated. In other words, models do not merely describe what they see; they also fill in content based on linguistic habits—i.e., "what usually appears in such scenes." This is precisely one manifestation of "guessing what should be there": what gets filled in follows data frequency.
The authors also find that existing evaluation methods are sensitive to instruction phrasing and generation style, so they propose POPE—a polling-style questioning evaluation method—to make object hallucination assessment more stable and flexible.
## Scoring Without Looking at Images
A paper posted in March 2024 targets the evaluation methodology itself. The authors conduct an in-depth investigation of current vision-language model evaluations and identify two problems. First, many samples fundamentally do not require visual content: answers can be derived directly from the question and options, or drawn from world knowledge embedded within the language model—and this phenomenon is pervasive across current benchmarks. Empirical test: GeminiPro achieves 42.9% on the MMMU benchmark without any visual input whatsoever, and averages over 24% higher than random-choice baselines across six benchmarks. Second, unintentional data leakage exists during training: Sphinx-X-MoE—which does not access images—achieves 43.6% on MMMU, outperforming its own language-model backbone by 17.9%, indicating that some questions supposedly requiring image inspection were already memorized by the model from large-scale training data.
Both problems point to the same consequence: the true multimodal gain is misjudged, potentially misleading subsequent research. The authors’ response is MMStar: 1,500 human-curated samples covering six core capabilities and 18 fine-grained dimensions; each question is first coarsely selected via an automated pipeline, then manually verified to ensure visual dependency, minimal data leakage, and requirement of advanced multimodal capabilities; additionally, two metrics are introduced—one measuring data leakage, the other measuring genuine multimodal training gains. The authors use MMStar to evaluate 16 leading models and re-examine leakage and genuine gains across seven benchmarks.
## The Audio Side
A paper posted in June 2024 extends the inquiry to audition. AudioBench covers eight tasks and 26 datasets (seven of which are newly proposed), evaluating three aspects: speech understanding, audio scene understanding, and voice understanding (paralinguistic, paralinguistic). The authors point out a gap: large audio models lack a comprehensive benchmark for instruction-following capability under audio conditions. After evaluating five popular models, their conclusion—quoted verbatim—is: "No single model consistently leads across all tasks."
This article cites only one benchmark abstract for the audio side, lacking the kind of systematic blind-spot research available for vision. As regards this single source, the conclusion is structurally identical to that for vision: simply adding an input channel does not automatically confer reliable performance on that channel; moreover, the audio evaluation landscape is even more fragmented, making claims of "universality" even harder to justify.
## Three Judgments
First, failures are systemic—not random noise (this article’s synthesis). Four lines of evidence, employing distinct methodologies yet converging in direction: the gap between image and text is geometrically measurable in representation space (2022); blind spots can be systematically constructed as pairs across nine fundamental visual patterns (early 2024); failures concentrate in scenarios where geometric primitives are proximate or overlapping, and accuracy recovers immediately when spacing increases (mid-2024); hallucination frequency correlates with occurrence frequency in visual instructions and co-occurrence relationships (2023). Random errors would not exhibit such structure. Thus, "multimodal" describes input channels only—it carries no implicit promise of perceptual quality; the two must not be conflated.
Second, high scores do not imply genuine visual usage (this article’s judgment). The figures 42.9% and 43.6% are empirically measured scores achieved under zero visual input on mainstream benchmarks; linguistic priors and training memory alone can inflate scores. Before interpreting a multimodal benchmark score as evidence of multimodal capability, one must first verify whether each question genuinely requires visual input; otherwise, the "progress" reflected in scores may merely reflect advances in language modeling and memory—not multimodal perception.
Third, reliability must be verified separately per modality and per task; models’ self-reports cannot serve as verification (this article’s judgment). Linear probing shows visual information resides in the encoder, with the bottleneck lying precisely in translation into language—so the same model, when applied to different task formats, exhibits shifting error distributions; audio-side evaluation is even more fragmented, with no model dominating across all tasks. Moreover, hallucination research shows models’ self-reports inherently contain elements absent from the image—thus, "the model says it saw it" cannot serve as evidence of having seen it.
## What Different Readers Can Take Away
For product teams deploying multimodal models: treat "support for image and sound input" as a description of available channels—not a guarantee of quality; before launch, test perception using real samples from your own business domain—especially for spatial relationships, fine details, and counting—rather than inferring reliability in your specific context from public benchmark scores. For evaluation researchers: whether a sample truly requires visual input is empirically testable—simply remove the visual input and re-run the test; the numbers 42.9% and 43.6% demonstrate that this test merits becoming a routine step in multimodal benchmarking. For agent developers: perceptual errors have fixed high-frequency zones—errors increase sharply as geometric primitives approach or overlap; implement explicit verification steps for critical actions, anchoring verification criteria externally—not relying on the model to self-attest to having seen anything. For general readers: next time you see a model confidently expounding on an image, remember fluency has two sources—one is actual seeing, the other is linguistic habit—"what usually appears in such images"; with the same set of models, widening the spacing between graphical elements lifts accuracy from widespread failure to near-perfect—what you perceive as "seeing" may not be seeing at all.
Scope and Boundaries: All six sources cited herein are arXiv abstract-level materials; full texts were not obtained, and content absent from the abstracts is excluded. Dates refer to the initial posting dates listed on each abstract page; one paper from July 2024 underwent multiple revisions, and its current official title includes the subtitle—this article refers to it by that current title, and reports experimental figures as stated in the current abstract version. Conference status is recorded verbatim only as indicated in the abstract-page metadata: one 2022 paper is listed as published at NeurIPS 2022, one 2023 paper as accepted at EMNLP 2023; the remaining four papers lack conference metadata, and this article makes no assertions regarding their publication status nor infers status from project pages or code repositories. Model and benchmark names appearing in the text (GPT-4V, Claude 3.5 Sonnet, GeminiPro, Sphinx-X-MoE, and CLIP, MMVP, BlindTest, POPE, MMMU, MMStar, AudioBench, MoF) are proper nouns as they appear in the original abstracts, cited strictly as tested systems and evaluation methods—no product evaluation is implied. Author names are omitted from the main text; papers are referenced by year of posting and conference metadata. Conflict of Interest: None of the six abstract pages include disclosure sections; this article makes no assertion regarding presence or absence of conflicts. Repair proposals are mentioned only within the scope reported in the abstracts (e.g., MoF), without elaboration on training or engineering details; discussion of the audio side is confined to the scope of a single abstract and does not represent the field’s full landscape. Translation conventions: "modality gap" rendered as "modality gap", "at arm's length" as "at arm's length", "narrow cone" as "narrow cone", "linear probing" as "linear probing", "visual grounding" as "visual grounding", "paralinguistic" as "paralinguistic"—all first occurrences accompanied by original English terms.
Sources: Six arXiv abstract pages: 2203.02053 "Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning", March 3, 2022, published at NeurIPS 2022; 2305.10355 "Evaluating Object Hallucination in Large Vision-Language Models", May 17, 2023, accepted at EMNLP 2023; 2401.06209 "Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs", January 11, 2024; 2403.20330 "Are We on the Right Way for Evaluating Large Vision-Language Models?", March 29, 2024 (https://arxiv.org/abs/2403.20330); 2406.16020 "AudioBench: A Universal Benchmark for Audio Large Language Models", June 23, 2024; 2407.06581 "Vision language models are blind: Failing to translate detailed visual features into words", July 9, 2024