Most teams building with Vision Language Models are walking into a production trap. They treat the VLM as a magical all-in-one black box: upload an image, ask a question, and pray the answer is right. It looks fun in a demo, but it burns through capital the moment it hits the real world. Here is the architectural flaw that is likely costing you millions.
THE SETUP
You’re the Lead AI Enigneer at a major auto insurer. Policyholders submit high-resolution (4K) photos of vehicle damage via a mobile app. Your company processes over 2,000 claims per day, and the current system sends raw crash photos directly to a VLM (like (like Gemma 3, Gemini 3.1 Pro, Sonnet 4.6, Qwen3-VL or GPT-4o) and asks for a repair estimate.
It’s failing in three ways, costing the company millions in claims leakage:
It misses visible damage. Cracked tail light housings, scraped bumper edges, dented trim panels, damage that is perfectly clear in the original 4K photo doesn’t appear in the VLM’s estimate. A human adjuster catches these in seconds.
It mistakes the car’s design for damage. A Lamborghini’s aggressive air intakes get flagged as “puncture damage.” A sport sedan’s sculpted body crease is called a “dent.” Anything that deviates from a “normal sedan” shape gets interpreted as a deformation.
It pays for damage that isn’t from this accident. Old scratches, prior dents, rusted trim, the VLM has no way to distinguish what was caused by this collision from what was already there. Here’s the audit snapshot:
The task: Design a multi-stage pipeline that fixes all three failure modes.
⏳⏳ How would you solve this? 💭
🙋 The Instinct Everyone Has FIRST — Not Wrong, Not Enough
Most engineers immediately reach for better prompting. Write a detailed system prompt, list damage types, require structured JSON, instruct the model to be thorough and careful. And honestly? That doesn’t get you far here!
In production, you’re treating structural problems with a behavioral fix. Here’s why each failure mode is immune to prompting:
The missed damage is a resolution problem. VLMs don’t see your 4K photo at 4K. They downsample every input into a fixed grid of low-resolution “patches”, typically 512×512 or 1024×1024. A hairline crack spanning 200 pixels in the original might occupy 3 pixels after downsampling. The VLM isn’t ignoring the damage, it physically cannot see it. No prompt fixes missing pixels.
The design-as-damage confusion is a ground truth problem. VLMs are trained on billions of general internet images. They’ve learned what a “normal” car looks like, roughly a mid-range sedan. Anything that deviates gets flagged as abnormal. The model has no concept of what this specific vehicle is supposed to look like. No prompt gives it one.
The pre-existing damage is a context problem. The model evaluates each crash photo in complete isolation. It sees a scratch and includes it in the estimate. It has no mechanism to ask: “Was this scratch here before the accident?” No prompt creates a temporal baseline that was never provided. And before you think:
“These sound like small-model problems, just use the biggest/latest one.”
They’re not. These are architectural limitations that apply to every current VLM, including frontier models. Downsampling is baked into the vision encoder. Lack of vehicle-specific ground truth is a training data gap. Single-image context blindness is a pipeline design choice. Frontier models are better at reasoning about what they see, but they still can’t see pixels that were thrown away, and they still can’t reference a baseline they were never given. The degree varies; the fundamental limitations don’t.
The fix isn’t a better prompt. It’s a better pipeline.
⚡ THE PRO SOLUTION: THREE-STAGE VISUAL CLAIMS ENGINE
The core principle: stop treating the VLM as an all-in-one perception-and-reasoning engine. Specialized tools handle perception, the input is engineered to bypass VLM limitations, and the VLM is invoked only at the end — as a reasoning engine comparing processed evidence against a known baseline.
STAGE 1 — HYBRID DETECTION: Offload Perception to a Specialist
Fixes: Missed visible damage
Don’t ask the VLM to find things. VLMs are powerful reasoners but unreliable perceivers, they struggle with precise localization, exact counting, and detecting small objects in large scenes. Traditional computer vision models are fast, deterministic, and surgically precise at exactly these tasks.
Run every uploaded photo through a lightweight detection or segmentation model (e.g., YOLO, fine-tuned on vehicle parts and common damage types). It doesn’t understand what the damage means. It just finds it:
[
{"part": "rear_bumper", "box": [120, 340, 580, 610], "anomaly": true, "confidence": 0.94},
{"part": "tail_light_left", "box": [590, 280, 720, 430], "anomaly": true, "confidence": 0.87},
{"part": "trunk_lid", "box": [200, 100, 560, 340], "anomaly": false, "confidence": 0.91}
]The VLM never searches the image. It receives a curated manifest of regions flagged by a model purpose-built for spatial precision.
STAGE 2 — CROP-AND-ANNOTATE: Control What the VLM Sees
Fixes: Missed visible damage (continued) + guides VLM attention
Now you have bounding boxes. Two operations:
Crop at native resolution. For each flagged region, extract a crop from the original 4K image — not the downsampled version. That hairline crack that disappeared at 512×512 is now the centerpiece of a tight, high-fidelity crop. You’ve bypassed the VLM’s biggest limitation without touching the model.
Annotate before sending. Draw colored bounding boxes, arrows, and text labels directly onto the crop pixels. If the detector found a dent and a paint scrape on the rear bumper, draw a red box around each and label them “Area A” and “Area B.” This is visual prompting — physically anchoring the VLM’s attention rather than hoping its attention mechanism finds the right spot.
The VLM now receives annotated, high-resolution crops with a text prompt referencing the annotations: “Examine Area A and Area B. For each, describe the damage type, severity, and whether repair or replacement is needed.” You’ve converted an open-ended visual search into a focused, guided reasoning task, which is exactly what VLMs excel at.
STAGE 3 — CONTRASTIVE REASONING: Give the VLM a Baseline
Fixes: Design-as-damage + Pre-existing damage
This is the stage that separates a demo from a production system.
The second and third failure modes share the same root cause: the VLM evaluates crash photos in isolation. It doesn’t know what the vehicle looked like before the accident or what it’s supposed to look like by design. So every visual anomaly, a Lamborghini intake, a five-year-old scratch, or fresh collision damage, gets treated identically.
The fix: shift from absolute evaluation to relative comparison. Don’t ask “What damage do you see?” Ask “What is different between these two images?”
When a claim is filed, your pipeline queries a vehicle reference database using the policyholder’s VIN, returning pristine stock photos of that exact make, model, and trim. Crop the same region from both the reference and the claim image, and present them together:
You are an auto insurance damage assessor. You will receive two images
of the same vehicle region:
- IMAGE 1 ("Reference"): The vehicle in factory-original condition.
- IMAGE 2 ("Claim"): The vehicle as submitted in the current claim.
Identify ONLY the structural differences between Reference and Claim.
Any feature present in both images is factory-original and must NOT be
listed as damage.
For each difference:
1. Damage type (dent, crack, scratch, deformation)
2. Severity (cosmetic / moderate / structural)
3. Recommended action (no action / repair / replace)
4. Confidence (high / medium / low)
If no differences are detected: "No new damage detected in this region."The Lamborghini air intake appears in both images — not damage. The five-year-old scratch, if you have onboarding photos — appears in both — not damage either. Only the delta gets reported. The VLM is no longer guessing what’s “normal.” It’s comparing what is against what was.
🧠 WHAT A SENIOR ENGINEER WOULD ACTUALLY SAY
“The VLM is the last thing that touches the data, not the first.”
This pipeline has a natural confidence-tiering structure. Claims where the detector finds clear damage, the crops are high quality, and the contrastive analysis shows obvious deltas, those get auto-estimated. Claims where the detector is uncertain or the VLM flags low confidence, those get routed to a human adjuster with the full annotated evidence package already prepared. The adjuster isn’t starting from scratch; they’re reviewing a pre-built case.
And the three-stage design has a side benefit most teams miss: auditability. When a policyholder disputes an estimate, you can show exactly what happened at each stage, what the detector found, what was cropped and annotated, the reference comparison, the VLM’s reasoning. That’s a defensible audit trail. A single-prompt VLM call is a black box that says “trust me.”
The broader lesson goes beyond insurance: in production, VLMs fail when you treat them as all-in-one systems. The pattern is always the same, offload perception to specialists, engineer the visual input, give the model a reference to reason against. The VLM isn’t the pipeline. It’s the reasoning engine at the end of one.
📖 This scenario is drawn from The Agentic AI Book — a production-first guide to building AI systems that actually work.
Grab early access: book.ryanrad.org
Until next dose — Dr. Ryan Rad