Research Papers
EN ZH

Multimodal Models · Language Models

LLaVA Explained: Visual Instruction Tuning for a Vision-Language Chat Model

LLaVA bolts a CLIP vision encoder onto a Vicuna LLM with one linear projection, then trains on GPT-4-generated image instructions — hitting 85.1% of GPT-4's score and 92.53% on ScienceQA.

Microsoft Research · Paper published 2023-04-17 · 5 min read

LLaVA Explained: Visual Instruction Tuning for a Vision-Language Chat Model

Quick answer

LLaVA shows you can build a capable visual chat assistant cheaply by connecting a frozen CLIP vision encoder to a Vicuna language model through a single trainable linear projection, then fine-tuning on instruction data that a text-only GPT-4 wrote for it. The released model scores a 85.1% relative score against GPT-4 on the paper’s own multimodal instruction benchmark, and after fine-tuning on ScienceQA the LLaVA + GPT-4 combination hits 92.53% accuracy — a new state of the art at the time. The data, model, and code were all released openly.

Generating instruction data with GPT-4

The central trick is that the data generator never sees an image. Instead, the authors feed text-only GPT-4 the symbolic representations an image already carries in datasets like COCO — its captions and its object bounding boxes — and ask GPT-4 to produce instruction-following conversations as if it could see the picture. From those text descriptions GPT-4 writes three kinds of data: multi-turn conversations about the image’s contents, detailed descriptions, and complex reasoning questions that require inference rather than literal lookup.

This is the cleverest and the most fragile part of the paper at once. It sidesteps the absence of large human-annotated multimodal instruction corpora, which barely existed in early 2023. But it also means the training signal inherits GPT-4’s blind spots: GPT-4 is reasoning over captions and boxes, not pixels, so anything the caption omitted is invisible to the teacher and therefore never taught to the student. The resulting dataset is 158K samples — small by pretraining standards, which is precisely the point.

Connecting a vision encoder to an LLM

LLaVA’s architecture is deliberately minimal. A frozen CLIP ViT-L/14 encodes the image into visual features; a single linear layer projects those features into the LLM’s word-embedding space so they look like ordinary input tokens; and a Vicuna LLM (a LLaMA fine-tune) consumes the projected visual tokens alongside the text prompt. That is the whole connector — no cross-attention stack, no Q-Former, just one matrix.

Training runs in two stages. Stage one freezes both the vision encoder and the LLM and trains only the projection matrix on image-caption pairs, aligning visual features to the language space. Stage two then fine-tunes the projection and the LLM together on the GPT-4-generated instruction data while keeping the vision encoder frozen. The honest takeaway is that the projection-as-connector design is what made multimodal LLMs reproducible on an academic budget — it works well enough that the field largely copied it before later moving to richer connectors.

Key results

  • 85.1% relative score vs GPT-4 on LLaVA-Bench, the synthetic multimodal instruction-following benchmark the authors built to score open-ended visual chat against text-GPT-4 judgments.
  • 92.53% on ScienceQA when LLaVA is fine-tuned on it and combined with GPT-4, setting a new state of the art on that benchmark at publication.
  • 158K instruction samples generated entirely by text-only GPT-4 — the model never trained on a single human-written visual instruction.
  • One linear projection layer is the only new visual-to-language component, on top of an off-the-shelf CLIP ViT-L/14 and Vicuna.

Why LLaVA mattered

LLaVA arrived right after GPT-4’s multimodal demos but before any open model could chat about images, and it handed the community a full, cheap, reproducible recipe: use a strong text LLM to bootstrap multimodal instruction data, glue a vision encoder on with a trivial projection, and fine-tune in two stages. The “visual instruction tuning” framing became the default pattern for open multimodal models — LLaVA-1.5, LLaVA-NeXT, and a wave of imitators all descend from this paper. Its real contribution is less the specific architecture than the demonstration that GPT-4 could be used as a data factory to teach a smaller model a capability GPT-4 itself had.

Limits and open questions

The biggest weakness is baked into the data pipeline: because GPT-4 generated instructions from captions and bounding boxes rather than pixels, LLaVA can confidently hallucinate details that were never in the image, and it inherits whatever the source annotations missed. The paper’s own benchmark is GPT-4-judged, so the headline 85.1% partly measures agreement with a model in the same lineage rather than ground truth. The single linear projection, elegant as it is, is a thin bottleneck — later work replaced it with an MLP and higher-resolution inputs for real gains, which tells you how much was left on the table. And the authors themselves frame these as early experiments on a small dataset, so the chat quality, while striking in demos, was uneven on fine-grained perception.

FAQ

What is LLaVA and what does Visual Instruction Tuning do?

LLaVA (Large Language and Vision Assistant) is an open multimodal chat model that connects a CLIP vision encoder to a Vicuna LLM. Visual instruction tuning is the method of fine-tuning that combined model on instruction-following conversations about images so it can answer open-ended questions about a picture.

How does LLaVA generate its training data with GPT-4?

LLaVA feeds text-only GPT-4 the captions and object bounding boxes of an image — not the image itself — and asks it to write conversations, detailed descriptions, and reasoning questions about that image, producing 158K instruction samples without any human visual annotation.

How does LLaVA connect the vision encoder to the language model?

A single trainable linear projection maps the frozen CLIP ViT-L/14 visual features into Vicuna’s word-embedding space, so the image becomes a sequence of tokens the LLM reads alongside the text prompt. No cross-attention or Q-Former is used.

How good is LLaVA compared to GPT-4?

On the paper’s LLaVA-Bench it reaches a 85.1% relative score versus GPT-4, and the LLaVA + GPT-4 combination scores 92.53% on ScienceQA. These are early-2023 results on a small dataset, and the benchmark is itself GPT-4-judged.

One line: let a text-only GPT-4 write the visual lessons, glue a vision encoder onto an LLM with one matrix, and you get an open chat model that sees. Read the original paper on arXiv.