Rejected97/100
- Shape match
- 0.75 lowest view 0.66
- Genuine re-photos
- 0.83–0.95 same pair, new photos
- Colour shift
- 2.6 ΔE looks alike
Beta Multi-agent return verification for marketplaces
Catch return fraud at the doorstep, not the warehouse.
The delivery agent photographs the product at handover. When it comes back, ReturnGuard checks it is the same unit: a free visual screen first, Gemini agents only when they're needed, and an explainable verdict before the refund goes out.
Day 0 · Delivery
Return fraud detectedRefund held · evidence report sent to the seller
5 days later…
—
F0.5 score
—
Precision, stress-tested
—
Recall (fraud)
—
Decided with no LLM call
—
LLM calls per return
Loading evaluation results…
The problem
Return fraud is a $100-billion problem hiding in the refund queue.
Most platforms only inspect returns at the warehouse, days after pickup and often after the refund has gone out.
$103B
lost by US retailers to fraudulent returns and claims in 2024
Appriss Retail & Deloitte, 2024
15%
of all returns to US retailers in 2024 were judged fraudulent
Appriss Retail & Deloitte, 2024
₹5.5 cr
taken from Meesho by a ring that returned different items than were sold; caught by an audit of sold vs returned items
Bengaluru police case, Dec 2024
Sources: Appriss Retail press release · Deccan Herald
Real-world pattern · old-for-new swap
New headphones delivered. The old pair sent back.
A customer orders new headphones, keeps them, and returns the pair they already owned in the new box. Same brand, same shape at a glance, so a quick look at the warehouse can pass it. The seller refunds a new product and gets a used one back.
Fingerprint at the door
The shape and colour of the exact unit handed over are stored at delivery, so the return is compared with that unit, not with a catalogue photo.
Shape, not just colour
Dark navy and black are close for a colour check (2.6 ΔE, inside the normal range). Here the object-only DINOv2 shape match is what flagged the swap.
Label read blind
If the unit has a printed serial or model label, Gemini is told only where it was and must read it back. The match is checked in code.
Real photos and scores from ReturnGuard's test set (pair 49, python scripts/evaluate.py). Headphones are a hard case: the reverse swap (black delivered, navy returned) scored 0.82, close to genuine re-photos, so it went to a human reviewer instead.
How it works
Cheap evidence first. Expensive agents only when needed.
01
Fingerprint at delivery
3 photos at handover. Background removed, object-only DINOv2 embeddings and colour stored once. The identity agent reads labels, serials, marks and tags.
1 Gemini call per delivery
02
Free visual screen
Return photos are compared locally on shape and colour. Clearly different products are rejected and clear matches accepted with no LLM.
0 LLM calls · runs in parallel with policy + history
03
Agents when it's ambiguous
If a label exists, it's read blind and compared in code. If evidence still conflicts, a visual inspector compares all six photos.
0–2 Gemini calls per return
04
Explainable decision
Evidence becomes a 0–100 risk score with every contribution shown. Accept, human review or reject, plus a seller-ready evidence report.
Full audit trail in SQLite
Coverage
What it catches, and which agent catches it.
Product substitution
A different or cheaper item comes back in the box.
Visual screen → fast reject
Same-model swap
An old or broken unit of the same model replaces the new one.
Identity agent (blind label read) + colour
Damaged after delivery
Handed over intact, returned scratched or broken.
Visual inspector + review rule
False damage claim
"Arrived damaged", but the handover photos show it intact.
Inspector vs delivery photos
Policy abuse
Returns outside the category's return window.
Policy agent (no LLM)
Repeat offenders
Customers with earlier flagged returns get stricter cut-offs.
Customer history agent
See a verdict in under a minute.
Register the blue speaker as a delivery, then return the black one. Sample photos are one click away.
Step 1 · at handover
Register a delivery
Photograph the product as it is handed to the customer. This creates the order and its identity fingerprint.
What gets recorded
- Background-removed cut-out of each view, checked for blur and framing.
- Object-only DINOv2 embedding and median colour per view, computed once and stored.
- Identity fingerprint from Gemini: printed identifiers (serial, batch, model), distinctive marks, accessories and tags.
- An order ID assigned by the database at this moment.
Step 2 · at pickup
Process a return
Photograph the item the customer hands back. The pipeline runs in the background and you can watch each agent work.
What happens next
- Intake: background removal, capture-quality gate, features.
- In parallel: policy agent (return window), customer-history agent, visual screen.
- Triage router: fast reject / fast accept with no LLM, or call an agent.
- Identity agent: blind read of the label recorded at delivery.
- Visual inspector: all six photos, only if evidence still conflicts.
- Decision agent: evidence-weighted risk score and verdict.
Return
Verifying return…
What we checked, in plain words
Waiting for the agents…
Gemini's verdict
Waiting for the agents…
Pipeline · live
Technical details: evidence weights, raw label readout, inspector output, LLM calls
Evidence · how each finding moved the risk score
Your activity
My dashboard
Only your own deliveries and returns, linked to your customer ID. This is the same history the customer-history agent reads.
What the customer-history agent sees for your ID
My returns
My deliveries
Measured, not claimed
Evaluation
Every number on this page is produced by a script and read from its results file: python scripts/evaluate.py for the labelled pair dataset and python scripts/stress_test_precision.py for the robustness test.
Loading…
System design
Architecture
Two entry points: delivery registration builds the fingerprint once; return verification is a LangGraph state machine where every LLM agent sits behind a cheaper check.
Design decisions
Why it's built this way
Object-only pooling, not plain patch-mean
Cut-outs are padded to a square. For tall items 65–77% of patches were padding, which made a shaker and a bottle score 0.77 "similar". Pooling only object patches drops that to 0.38.
A colour signal next to DINOv2
DINOv2 is trained to ignore colour. Median object chroma (ΔE in a*b*) separates the blue and black speakers that DINOv2 calls near-identical.
Blind label reading
The return-time identity agent is told where the label was, never what it said, and the comparison happens in code with fuzzy matching. It can't just agree with the expected answer.
Independent evidence
The visual inspector never sees the similarity scores or the label result, so its verdict is a second opinion, not an echo. Its prompt doesn't claim the items are the same product.
Structured outputs, transparent score
Every agent returns schema-validated JSON. The decision is an evidence-weighted log-odds score with each contribution shown, not a number the LLM made up.
Customer accounts feed the history agent
Signing in with email and phone creates a customer ID. Every delivery and return is tied to it, so the history agent can count earlier flagged returns and tighten the auto-accept cut-off for repeat offenders.
Every number explained in plain words
Reviewers see "colour difference 2.1, normal is under 2.8" rather than a bare score, plus which way each finding pushed the decision. Gemini's verdict, or the reason it wasn't needed, is always shown, and a second opinion can be requested on demand.
Fail safe, never stuck
Timeouts, retries with backoff and rate-limit handling. If an agent still fails, the return goes to a human instead of being auto-accepted, and nothing stays "processing" forever.
Stack
OrchestrationLangGraph state graph: parallel fan-out, conditional routing, streamed progress
VisionDINOv2-base (object-only patch pooling) + CIE Lab colour
LLM agentsGemini 3.8 Flash via LangChain, schema-validated structured output
Background removalrembg (U²-Net), cached by image hash
BackendFlask REST API, background worker, live progress polling
StorageSQLite: orders, returns, per-stage audit log
Welcome back
Sign in with the email and mobile number you registered with.
or
Photos you upload are processed only to verify returns. This is a demo, so please don't upload personal photos.
About
About ReturnGuard
The project
ReturnGuard verifies that a returned product is the unit that was delivered, using photos taken at the doorstep. It is decision support: clear cases are automated, ambiguous ones reach a person with the evidence already laid out.
Current evaluation data covers 6 physical items. Larger real-world datasets, a mobile capture app and integration with marketplace return flows are on the roadmap.
RG
Rajveer Gupta
IIIT Nagpur · Amazon ML Summer School