Modalities Explained: From Text-Only to Omni Models

In November 2023, a production content-moderation pipeline built for text started routing image attachments through it after GPT-4V launched. Nobody ran a token-cost estimate first. Within a week, the daily API bill had tripled — not because vision was particularly expensive per token, but because every image was adding 1,500 tokens to a prompt that had previously been 200 tokens. Cost-per-request went up 8×. The pipeline worked fine technically. The budget didn't survive first contact with images.

That's the unglamorous side of multimodal AI that the model launch announcements don't mention. Every modality beyond text has hidden costs — in tokens, in latency, in hallucination rate, in API availability — and understanding those costs is what separates a working multimodal system from a demo that surprises you in production.

What "modality" actually means

A modality is a type of input or output signal a model can process. Text is one modality. Images are another. Audio and video are two more. Structured data (tables, JSON) is sometimes treated as a fifth, though it usually enters as formatted text.

The word gets used loosely enough to create confusion. "Multimodal" sometimes means "takes images" and sometimes means "takes images, audio, and video and outputs audio and images." These are very different capabilities. When you see a model described as multimodal, the useful follow-up questions are: which input modalities, which output modalities, and which of those are available in the production API today versus planned for later?

The mental model that makes this concrete: text is the base modality because transformers were designed for token sequences. Every other modality is a conversion problem — how do you turn an image, or an audio clip, or a video, into something that looks like a token sequence to the attention layers?

The shared embedding space

The answer to that conversion problem is a shared embedding space. This is the architectural idea that makes cross-modal reasoning possible, and it's worth understanding precisely because it explains both the capabilities and the limits of current models.

A transformer's attention mechanism operates on a sequence of vectors, each of fixed dimensionality. For text, a tokenizer converts the input to subword tokens and a learned embedding matrix maps each token to a vector (typically 4,096 or 8,192 dimensions for a frontier model). The attention layers then process this sequence — tokens attending to other tokens, each position building up a representation informed by its context.

For images: an image is divided into a grid of fixed-size patches — 16×16 pixels is common — and a Vision Transformer (ViT) encoder projects each patch into a dense embedding vector of the same dimensionality. For a 512×512 image at 16×16 patches, that's 1,024 patch embeddings. Those embeddings are prepended to (or interleaved with) the text token embeddings, and the attention layers see a single flat sequence of vectors, indifferent to whether each came from a patch or a word.

For audio: the raw waveform is converted to a mel spectrogram — a 2D time-frequency representation — and then sliced into short frames, each projected by an audio encoder into the shared dimensionality. Whisper-style audio encoders have been the standard approach since OpenAI's whisper model, and GPT-4o extended this to live, low-latency audio.

sequenceDiagram
    participant USER as User input
    participant ENC as Modality encoder
    participant EMB as Shared embedding space
    participant ATT as Transformer attention
    participant DEC as Output decoder

    USER->>ENC: "image: 512×512 photo"
    ENC->>EMB: 1,024 patch vectors (dim=8,192)
    USER->>ENC: "text: 'What is in this image?'"
    ENC->>EMB: 9 token vectors (dim=8,192)
    EMB->>ATT: flat sequence of 1,033 vectors
    ATT->>ATT: self-attention across all positions
    ATT->>DEC: contextualized representations
    DEC->>USER: text output tokens

The key consequence: the transformer itself does not know or care about modalities. It runs attention over the flat vector sequence identically regardless of input type. Whether a particular position in the sequence is "image token 47" or "text token 3" is implicit in how the encoder placed it there, not in how attention handles it. This is also why you can have one model that handles both — there's no architectural seam between the modalities once encoding is done.

Native multimodal vs. bolt-on

Not all multimodal models are created equal. There's a meaningful architectural difference between models that are natively multimodal (jointly pretrained on all modalities from scratch) and models with bolt-on encoders (a pretrained language model with a separately pretrained vision or audio encoder whose output is projected into the LM's embedding space).

The bolt-on approach is faster to ship. Take a strong language model, freeze its weights, train a relatively small projection layer that maps vision encoder outputs into the LM's embedding space, then fine-tune the whole system. GPT-4V and the earliest Claude vision support worked roughly like this. The upside is that you keep the LM's language capabilities while adding vision. The downside is that the two components have never learned to reason jointly at the pretraining level — the LM learned about the world in text, and the vision encoder learned about images, and the projection layer is doing its best to bridge the gap.

Natively multimodal models — Gemini 1.0 Pro was an early example, then Gemini 1.5, and Meta Llama 4 which was jointly pretrained on text, images, and video across 200+ languages — develop cross-modal representations from the beginning. The attention layers learn what "image showing a red traffic light" means in a world where text, images, and audio have been seen together since step 1 of pretraining.

In practice, natively multimodal models handle tasks requiring tight cross-modal reasoning more reliably: captioning an unusual image with precise spatial relationships, responding to an audio clip where tonal cues change the interpretation, or reading a document with mixed text and figures. The gap between native and bolt-on is narrowing as projection layer training gets more sophisticated, but for precise vision tasks like reading charts, identifying fine-grained differences between similar images, or reasoning about spatial relationships, native tends to win.

The vision modality in practice

Vision has been the most widely deployed non-text modality. Nearly all major frontier APIs as of mid-2026 accept images — DeepSeek's V3 line is the text-only holdout. The practical questions are resolution, cost, and task-specific failure modes.

Resolution modes. Most providers offer a low-detail mode (one fixed-size representation of the entire image, cheap) and a high-detail mode (the image is tiled at higher resolution, each tile encoded separately, expensive). GPT-4o's high-detail mode first scales the image to fit within 2048×2048, then rescales it so the shortest side is 768 px, and only then counts 512×512 tiles — each tile costing approximately 170 tokens, plus an 85-token base. A 1024×1024 image rescales to 768×768 and produces four tiles: 4 × 170 + 85 = 765 tokens. A 2048×1024 image rescales to 1536×768 and produces six tiles: 6 × 170 + 85 = 1,105 tokens. Low-detail mode is always 85 tokens regardless of input resolution.

The choice is not always "use high-detail for accuracy." For tasks like identifying that an image contains a cat, or classifying a product image into one of ten categories, low-detail is sufficient. For reading text in an image, interpreting a chart, or identifying small objects, you need high-detail. Choosing the wrong mode in the wrong direction costs you either accuracy or money.

{ "type": "token-cost", "title": "What image tokens do to the monthly bill" }

The widget has no resolution-mode toggle, so simulate it yourself: drag the input-tokens slider between ~85 tokens/request (low-detail) and ~1,500 (high-detail) and watch what the mode choice does to the monthly bill.

What vision models get wrong. Image hallucination — confidently describing image content that isn't there — is the primary failure mode. Spatial relationships are particularly unreliable: "what is to the left of the red circle?" trips up all current models with some frequency. Text in images is better than it was in 2023 but still problematic for handwriting, unusual fonts, and low contrast. Models will sometimes confabulate text that is partially obscured or blurry.

import anthropic
import base64
from pathlib import Path

client = anthropic.Anthropic()

def encode_image(path: str) -> tuple[str, str]:
    """Return (base64_data, media_type) for a local image."""
    data = Path(path).read_bytes()
    media_type = "image/jpeg" if path.endswith((".jpg", ".jpeg")) else "image/png"
    return base64.standard_b64encode(data).decode("utf-8"), media_type

def analyze_image(image_path: str, question: str) -> str:
    b64, media_type = encode_image(image_path)
    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=1024,
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "image",
                        "source": {
                            "type": "base64",
                            "media_type": media_type,
                            "data": b64,
                        },
                    },
                    {
                        "type": "text",
                        "text": question,
                    },
                ],
            }
        ],
    )
    return response.content[0].text

# Example: chart reading
result = analyze_image(
    "quarterly_revenue.png",
    "What was Q3 revenue and what trend do you observe from Q1 to Q4?"
)

For OpenAI's API, the structure differs slightly but the concept is identical — content is a list containing image and text items, the model receives them as a flat token sequence.

The audio modality

Audio is where the biggest divergence in API support exists today. GPT-4o and the GPT-5 family accept and produce audio natively, enabling real-time voice conversations with sub-300ms latency in optimized setups. Gemini 2.x/3.x has similar native audio support and adds audio generation. Claude 4.x Sonnet and Opus as of mid-2026 have no native audio input — voice applications on Claude go through a cascaded pipeline: Whisper or Deepgram for speech-to-text, Claude for text reasoning, then a TTS model for output.

The architectural difference: native audio means the audio tokens participate in the same attention computation as text, so the model can respond to tone of voice, pacing, and non-verbal cues (a laugh in a customer service call, hesitation before answering). Cascaded pipelines lose all of that — by the time the audio hits the LLM it's been transcribed to flat text.

For voice applications where the conversation is primarily informational — "what's my account balance," "set a reminder for 3 PM" — cascaded pipelines work fine and give you more control over each stage. For applications where the emotional register of the audio matters — coaching, therapy support, entertainment — native audio matters and the choice of model becomes more constrained. The voice pipeline deep-dive covers the end-to-end engineering.

flowchart TD
    A["Audio input"] --> NATIVE_PATH["Native path"]
    A --> CASCADE_PATH["Cascaded path"]

    NATIVE_PATH --> NATIVE_ENC["Audio encoder\n(mel spectrogram → embeddings)"]
    NATIVE_ENC --> SHARED["Shared attention\n(text + audio tokens together)"]
    SHARED --> NATIVE_OUT["Audio + text output"]

    CASCADE_PATH --> STT["STT model\n(Whisper / Deepgram)"]
    STT --> TRANSCRIPT["Text transcript"]
    TRANSCRIPT --> LLM["LLM\n(text reasoning)"]
    LLM --> TTS["TTS model\n(ElevenLabs / gTTS)"]
    TTS --> AUDIO_OUT["Audio output"]

    style NATIVE_PATH fill:#15803d,color:#fff
    style CASCADE_PATH fill:#00e5ff,color:#0a0a0f
    style SHARED fill:#a855f7,color:#fff

The video modality

Video is the most expensive and least mature modality in current APIs. The core challenge is density: a 30-second clip at 30 fps is 900 frames, each potentially thousands of tokens if encoded at full resolution. No API processes video at frame-by-frame resolution — instead, models sample frames at lower rates (1 fps or less is common), encode those, and combine them with any audio stream.

Gemini 1.5 Pro and later Gemini 2.x/3.x accept video input directly, leaning on Google's 1M-token context window to hold many frame embeddings at once. Meta Llama 4 Scout was trained jointly on video as part of its pretraining data. OpenAI's GPT-5 family added limited video support in 2025.

The practical constraint: video input is not suitable for real-time applications in any current API. It's appropriate for batch analysis tasks — "describe what happens in this product demo video," "extract the key moments from this 5-minute recording," "identify safety violations in this manufacturing footage." Even at low frame-sampling rates, a 5-minute video at 1 fps with 200 tokens per frame is 60,000 tokens — roughly $0.30 at mid-2026 pricing before the text prompt and output.

What breaks: failure modes by modality

Vision. Cross-modal hallucination is the headline failure: the model describes image content that isn't there, and it gets worse exactly where the stakes are highest — small text, small objects, spatial relationships. Low-contrast inputs make it worse still: text on busy backgrounds, diagrams with overlapping elements, and dense screenshots all score well below what clean product photos suggest. Resolution mode compounds the problem from the other direction — pick low-detail for a task that requires reading fine print and the image looks fine to a human while the tile embeddings simply don't contain the answer, so the model fills the gap with a plausible guess. And the image itself strains a budget you may not have re-checked: adding one to a prompt that also carries a long system prompt and tool definitions can push a request over the context limit that its text-only version cleared comfortably.

The failure mode none of that covers is adversarial: multimodal prompt injection. Text rendered inside an image — a sign in a photo, a line in a screenshot, near-invisible text in an uploaded document — is read by the model like typed input, and it sails straight past every text-side input filter you've built. Treat anything extracted from an image as untrusted input; the indirect prompt injection article covers the attack class and which defenses actually hold up.

Audio failure modes:

• Cascaded error accumulation: in a text-to-speech-to-text pipeline, transcription errors compound. "Revenue fell 15%" transcribed as "revenue fell 50%" produces a qualitatively wrong analysis.
• Latency spike at scale: each stage in a cascaded pipeline adds latency and a new point of failure. The voice AI latency article has the numbers, but in brief: STT adds 200-800ms, LLM adds 300-1500ms, TTS adds 200-500ms. That's 700ms-2800ms before the user hears anything.
• Native audio hallucination: models that process audio natively can hallucinate spoken content, attributes of the voice, or emotional states that weren't present.

Video failure modes:

• Temporal reasoning failures: current models struggle to answer "at exactly what point in the video does X happen?" with accuracy.
• Frame sampling misses key moments: if the sampling rate is 1 fps and the key event happens in a 0.5-second window, it's missed entirely.
• Cost at scale: 100 video analysis tasks per day at $0.30 each is $900/month before any other costs. This modality essentially requires batching and careful task selection.

The connecting thread across all three: always validate outputs before presenting them to users. The capability limits article covers the general failure taxonomy; the modality-specific failures above are additive.

Picking the right model for your modality mix

The model landscape article covers the full comparison, but the modality-specific decision is different enough to spell out here.

flowchart TD
    Q1{"What input modalities\ndoes your app need?"}
    Q1 -->|"text only"| T1["Any frontier model\nPick by cost/quality"]
    Q1 -->|"text + images"| Q2{"Volume of images\nper day?"}
    Q1 -->|"audio input required"| Q3{"Real-time voice?"}
    Q1 -->|"video analysis"| V1["Gemini 2.x/3.x or\nLlama 4 Scout via API\n(batch tasks only)"]

    Q2 -->|"< 1k/day"| IM1["Any frontier model\nHigh-detail OK"]
    Q2 -->|"> 10k/day"| IM2["Model choice + resolution\nmode matters for cost\nSee token budget math"]

    Q3 -->|"yes — real-time"| AUDIO1["GPT-4o / GPT-5\nor Gemini 2.x/3.x\n(native audio)"]
    Q3 -->|"no — batch/async"| AUDIO2["Cascaded pipeline\nSTT + any LLM + TTS\nMore control, higher latency"]

    style Q1 fill:#00e5ff,color:#0a0a0f
    style AUDIO1 fill:#15803d,color:#fff
    style V1 fill:#0e7490,color:#fff

Task	Modality needed	Best fit (mid-2026)	Gotcha
Document Q&A with PDFs	text + images	Claude Sonnet 4.x, GPT-4o	Image token cost at volume
Real-time voice assistant	audio in + out	GPT-4o, Gemini 2.0 Flash	Latency sensitive; see voice pipeline
Product image classification	images	Any vision-capable model	Low-detail mode often sufficient
Meeting transcription + summary	audio in	GPT-4o or cascaded Whisper+Claude	Cascaded gives more control over STT errors
Video content moderation	video + text	Gemini 3.1 Pro	Batch only; expensive at volume
Chart interpretation	images (high-detail)	GPT-4o, Gemini 3.1 Pro	Spatial reasoning still unreliable; validate output
OCR on scanned documents	images	Specialized: see Document AI article	VLMs often lose to classic OCR on structured forms

The token cost reality

The worked numbers above gave a top-line monthly cost. Here's the finer breakdown that changes decisions.

Back-of-envelope: 50,000 image API calls per day

High-detail mode (avg 1,500 tokens/image):
  Image tokens: 50,000 × 1,500 = 75M input tokens/day
  At $5/M (illustrative mid-tier): $375/day = $11,250/month

Low-detail mode (85 tokens/image):
  Image tokens: 50,000 × 85 = 4.25M input tokens/day
  At $5/M: $21.25/day = $637.50/month

Savings from resolution mode selection: ~$10,600/month
Accuracy loss: minimal for classification, significant for chart reading

Mixed strategy: 80% low-detail (classification) / 20% high-detail (reading):
  Image tokens: 50,000 × (0.8 × 85 + 0.2 × 1,500) = 50,000 × 368 = 18.4M/day
  Monthly: $2,760/month — 75% cheaper than all high-detail

→ Route requests by task type, not by default to the most accurate setting

Adding the detail parameter is one line of code. Doing the cost math before shipping is the actual work.

For Claude, the equivalent is image resizing: Claude's image tokens scale with input resolution. Sending a 2000×2000 photo and a 512×512 version of the same photo for a classification task is a 15× token difference. The How Vision-Language Models Work article covers the architectural reasons why resolution scaling works the way it does.

What the "omni model" label actually means

Marketing teams reached for "omni model" in 2024-2025 to describe models that handle arbitrary modality combinations. GPT-4o was the prototype: it accepts text, images, and audio natively and produces text and audio, and in 2025 it gained native image generation — exposed in the API as a separate model id (gpt-image-1) rather than through the chat endpoint. That's the pattern worth noticing: the capability exists in the model, but the API surface splits it out. Gemini follows the same shape — native image output shipped in the 2.x Flash line, but as its own image-generation surface, not a response type of the main chat models.

A genuinely any-to-any model — arbitrary input and output modalities behind a single API surface — does not exist in production as of mid-2026. What exists is multimodal input with text (and in some cases audio) output, plus image and video generation exposed through separate endpoints or model ids, even when the generation capability comes from the same underlying multimodal model. When vendor documentation says "omni," verify the specific supported input/output combinations in the API reference.

The trajectory is clearly toward more modalities per model, and the native-vs-bolt-on distinction is likely to matter less as jointly pretrained models become the default. But engineering for today means knowing which capabilities are live in the API you're calling, not which capabilities the press release described.

The model landscape overview has the full comparison across closed frontier and open-weight options, and the open vs. closed decision framework covers when it makes economic sense to run a multimodal open-weight model yourself versus paying API rates.

The judgment call: when to go multimodal

Adding a modality to a system is not free in any dimension — cost, latency, complexity, hallucination risk. The default should be: stay text-only until the task genuinely requires another modality.

Signs that a task genuinely requires vision: the information is only in the image (not extractable as text), the accuracy requirement is low enough to tolerate hallucination risk, or the throughput is low enough that image token cost is not a budget concern. Signs that you might be reaching for vision when you don't need to: you're OCR'ing a structured form (use a classic OCR engine — the Document AI article makes the case), you're sending a screenshot of a table (parse the HTML instead), or you're passing an image because the user uploaded it but your actual task only needs the text they typed alongside it.

For audio: the question is whether you're building a real-time voice application (the user is speaking and expects to hear a response) or a batch transcription and analysis task. Real-time requires native audio and the constraints that come with it. Batch works fine with cascaded STT + text LLM and gives you more control over each stage.

The practical default for 2026: start with text plus images (universally supported), add native audio only when the use case is genuinely conversational and real-time, and treat video as a batch-analysis modality until the cost and API maturity improves. When you do add a modality, budget the token cost before you ship, add output validation for the hallucination risk specific to that modality, and check the live API docs — not the model announcement — for what's actually available.