Multimodal models: vision, audio and documents
A multimodal model can take more than one kind of input, usually text plus images, and often audio, video and PDFs, and sometimes produce more than text. Under the hood, each kind of input is converted into tokens or vectors that the language model can process alongside text. They are very good at describing images, answering questions about them, reading documents and screenshots, and transcribing speech, but they still miss small details, miscount objects and can describe things that aren't there, and for fast, high-volume or safety-critical vision tasks, a specialised model like YOLO is often the better choice.
What multimodal means
Text, images, audio and video are all unstructured data: there are no rows and columns, and every example is different in size and content. For years, each type had its own models. Convolutional neural networks handled images, recurrent networks handled text and speech, and to caption an image, you combined them, a CNN to find the objects and an LSTM to write the sentence.
Multimodal models bring these together in one system. You can pass an image with a question and get a text answer, pass text and get an image back, or speak and get speech back. Vision language models (VLMs), models that take images and text, are the most common kind.
How images become something an LLM can read
A language model works on sequences of vectors, so an image has to be turned into vectors first.
The common approach uses a vision transformer (ViT). The image is cut into small square patches, for example 16 by 16 pixels each, every patch is turned into a vector, and the vision transformer processes them the same way a text transformer processes tokens. The resulting vectors are projected into the same space as the language model's token embeddings, so the model sees a sequence that contains both image tokens and text tokens, and attention connects them.
Two consequences matter for you as a builder. Images cost tokens, often hundreds or more per image, depending on the resolution and the provider, so they add to your bill and your context window. And images are often resized before processing, so very small text or fine details can be lost.
Audio and video
Audio is usually converted into a spectrogram, a picture of which frequencies are present over time, and then encoded in a similar way. Some systems transcribe speech to text first with a speech recognition model like Whisper and pass the text to an LLM. Newer native audio models process speech directly, which keeps tone and timing, and can respond with speech, which is what real-time voice assistants use.
Video is a sequence of frames, often 24 to 30 per second. Models usually sample some of those frames, treat them as images, and add the audio track, so very fast events between sampled frames can be missed.
Documents and PDFs
Many models now accept PDFs directly. Depending on the provider, the PDF is converted into page images, extracted text, or both, and the model reads the pages much like a person looking at them, including tables, charts and layout.
That is powerful for documents that text extraction handles badly: scanned pages, forms, tables, slides and diagrams. It is also more expensive and slower than extracting the text once, and for large document collections in a RAG system, you usually still parse documents up front, as document parsing explains.
What multimodal models are good at
Image captioning
Captioning turns an image into a text description, and once something is text, you can search it, filter it, store it and check it against rules. This is why captioning is one of the most important uses of vision models. Search systems caption images so you can find them with words, and content moderation systems detect what is in an image, so they can check it against a policy, for example detecting harmful content, or checking whether an ad image has too much text on it.
Visual question answering
You ask a question about an image and get an answer: "Is there a red SUV in this frame?", "What is the total on this receipt?", "Which step of the form is the user stuck on?". With video, the same idea lets you search footage with a description instead of watching hours of it.
Reading screenshots, forms and charts
Models can read text in images (OCR), understand the layout of forms and receipts, and describe charts and diagrams. Combined with structured outputs, that turns photos of invoices, forms or whiteboards into data.
Speech
Transcription, translation, summarising meetings and voice interfaces. See voice agents.
Generating images, audio and video
Some models generate media from text, like image generators and video models such as Sora. These are separate capabilities with their own pricing, rules and risks, especially around real people and copyrighted characters.
Where they fail
Multimodal models make some characteristic mistakes:
- Counting: "how many people are in this photo" is often wrong beyond a handful.
- Small details: small text, distant objects and fine print are often misread, especially after resizing.
- Spatial reasoning: left and right, exact positions and measurements are unreliable.
- Hallucinated details: describing objects or text that aren't in the image, especially when the question suggests they should be.
- Charts: reading exact values off a chart is error-prone.
Test them on your real images, including bad lighting, blur, rotated photos and handwriting, not on clean examples.
When a specialised model is better
Not every vision problem needs an LLM.
For real-time detection, like spotting a person, a vehicle or a fall in a camera feed, object detection models like YOLO (You Only Look Once) are much faster and cheaper, run on small devices, and give you exact bounding boxes. They can process every frame of a live video, which a large multimodal model can't do at a reasonable cost.
A common pattern combines both: a fast specialised model watches everything and detects events, and a multimodal LLM is called only on the frames that matter, to describe what happened or answer a question about it.
The same applies elsewhere. A dedicated OCR engine can be cheaper and more predictable for clean, high-volume scans. A dedicated speech recognition model can be cheaper for plain transcription.
A multimodal call
Most APIs accept images as files, URLs or base64 data inside the message.
import os
from pathlib import Path
from google import genai
from google.genai import types
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
image = Path("receipt.jpg").read_bytes()
response = client.models.generate_content(
model=os.environ.get("MODEL_NAME", "gemini-2.5-flash-lite"),
contents=[
types.Part.from_bytes(data=image, mime_type="image/jpeg"),
"List each item on this receipt with its price, then the total.",
],
)
print(response.text)
For anything you will store or act on, ask for structured output and validate it, and check totals with code rather than trusting the model's arithmetic.
Running multimodal models locally
Several open-weight vision language models run on a laptop through Ollama or Hugging Face, for example versions of Gemma, Qwen-VL and LLaVA. They are useful for private images that shouldn't leave your machine, and for learning, but they are usually less accurate than the best hosted models, so test them on your task.
Privacy
Images and audio often contain more personal data than the text around them: faces, licence plates, addresses on envelopes, ID cards, voices. Before you send them to an API, check your data rules, remove or blur what isn't needed, and be careful about storing them.
From the floor
In our session on multimodal systems, the example I used for vision in the real world was fall detection on a large construction site. Nobody can watch every worker on a big site, and CCTV records a fall without alerting anyone, so a detection model like YOLO v8 watches the frames, detects a fall, and raises an alert with the location, and an LLM can be added on top to describe what happened. The point I wanted builders to take from it: vision is not always about LLMs, and deep learning models are still the right tool for many of these jobs.
The task I gave builders after that session was to build an image captioning system, first with an older deep learning approach and then with a vision model from Hugging Face like BLIP, and then a small multimodal app that takes an image or text through a web interface, so they could see the difference between the two generations of models themselves.
In another workshop, the agent we built took a photo of a homework question, used OCR to extract the text, and passed it to an LLM to solve, a simple example of combining a vision step with a language model step.