MC MenuConflicts-RAG
methods / v0.1
METHODS / SYSTEM DESCRIPTION

Dataset construction and inference pipeline

This page records how the menu conflict corpus was assembled, how controlled price edits were produced, and how the explorer retrieves and answers from image evidence.

Evaluation unit One untouched menu photograph paired with one counterfactual version containing a single intended price replacement.
01

End-to-end overview

Corpus preparation and online inference are separate phases.

01Yelp menu photosPublic source corpus
→
02Source selection10 original menus
→
03Price intervention50 counterfactuals
→
04Structural + visual QALabels retained
→
05SigLIP2 retrievalRestaurant-scoped top K
→
06Vision answerCited image evidence
02

Conflict dataset construction

The image edit is localized; the rest of the photograph is copied from the source.

  1. 2.1

    Prepare the menu-photo corpus

    Menu-labelled photographs are extracted from the Yelp Open Dataset photo archive. The prepared local corpus contains 1,678 menu images associated with 1,338 businesses. Business and photo identifiers are preserved as metadata.

    data/yelp-menu/images · data/yelp-menu/metadata.json
  2. 2.2

    Select source photographs

    Ten real, user-contributed menu photographs are selected as the working source set. Each source contributes five independent conflict cases. A case is a pair—not a six-image temporal sequence.

    data/conflict-sources.json · data/menu-conflicts-50/originals
  3. 2.3

    Locate an item-price target

    A vision model identifies a readable menu item, its displayed price, and a normalized price bounding box. The detector output is cached so reruns reuse the same localization.

    Detector gpt-5.6-terra Cached output target_cache.json
  4. 2.4

    Construct deterministic edit inputs

    The normalized box is converted to pixel coordinates. A small context crop is calculated around the target, then an edit mask is restricted to the price region. Crop, mask, and paste coordinates are stored in the manifest.

    input cropalpha maskprice boxedit boxcrop box
  5. 2.5

    Generate and composite the replacement

    A deterministic seed selects a different target price. The crop and mask are sent to the image editor, which renders the new price in local photographic context. Only the permitted edit region is pasted back into the full-resolution source.

    Editor gpt-image-2 Seed 20260729 Variants 5 per source
  6. 2.6

    Assign experimental metadata

    Each sample records source and synthetic paths, item name, intended prices, edit coordinates, model provenance, QA state, and a synthetic observation date. Synthetic dates support conflict experiments; they are not claims about when a real menu was photographed.

    data/menu-conflicts-50/manifest.json · samples.jsonl
03

Worked image example

One source image and one localized counterfactual from the released corpus.

Original source_01 restaurant menu photograph
Original evidence original_source_01
localized price edit →
Counterfactual source_01 menu with an edited Galbitang price
Counterfactual evidence conflict_f9f4d0487b4b
Context crop

Inspect the localized edit

The full-menu pair above establishes image-level context. These stored generation artifacts expose the small region sent to the editor before compositing.

Original context crop around the Galbitang price
Original context crop
Generated context crop containing the changed Galbitang price
Generated context crop
Target item
Galbitang
Manifest annotation
14.99 → 16.99
Source
Yelp menu photograph
Intervention
Price-region inpainting
Case semantics
Independent original/counterfactual pair
Inclusion
Retained in the 50-case corpus

The full image is retained at its source dimensions. Generation operates on a context crop, and compositing copies only the permitted edit region back into the original photograph.

04

Verification and QA policy

Structural validity is enforced. Visual QA is reported but does not remove samples.

Structural verification

Checks that all files exist, dimensions match their sources, each source has five cases, and pixels outside the allowed region remain unchanged.

Missing files
0
Size mismatches
0
Outside-region violations
0
Result
valid

Batch visual audit

A separate vision pass assesses whether the intended price is readable and whether the edit appears visually plausible.

Exact-price passes
34 / 50
Realism passes
49 / 50
Accepted
33
Needs review
17
05

Explorer RAG pipeline

The explorer replays the frozen restaurant-scoped retrieval output used by the current E001 rerun.

Input

Benchmark question + top K

The user selects one of the 50 item-price questions and a top K from one to six. The known restaurant determines which images are eligible.

↓
Candidate construction

Images from the same restaurant

The full corpus contains 10 original menu photos and 50 edited photos. Each query is restricted to the 6 or 12 candidates sharing its Yelp business ID.

↓
Retrieval

SigLIP2 text-to-image similarity

The question is embedded as text and each eligible candidate is embedded from pixels with google/siglip2-base-patch16-224. Cosine similarity determines the natural top six within that restaurant.

↓
Leakage boundary

No expected-image injection

Only the Yelp business ID is used to restrict the candidate pool. Expected image IDs, item labels, and prices are not used for similarity ranking, and neither expected image is added by hand after retrieval.

↓
Answering

Multimodal evidence package

The vision model receives the question, each retrieved image, its image ID, evidence type, source ID, and available observation metadata. When “Report conflicts” is enabled, the prompt also requests explicit comparison and disagreement reporting; when disabled, that instruction is omitted.

↓
Output

Direct answer with citations

Claims cite image identifiers in the form [image:IMAGE_ID]. When images disagree, the answer is instructed to report both observations and cite each side.

06

Models and retrieval components

Model assignments for corpus construction, the current explorer, and the general image index.

Frozen retrieval artifact

The explorer uses the exact SigLIP2 rankings saved for experiment E001. Replaying the artifact keeps the interactive site aligned with the reported results without loading the embedding model on the web server.

Phase Stage Model or method Role
Construction Target localization gpt-5.6-terra Reads a source menu and returns item name, original price, and normalized price box.
Construction Price mutation Deterministic Python Samples a changed numeric price from the fixed experiment seed.
Construction Crop and mask Pillow geometry Converts boxes to pixels and creates the localized crop, mask, and paste region.
Construction Generative editing gpt-image-2 Inpaints the replacement price within the cropped menu context.
QA Generation validation gpt-5.6-luna Checks expected-price visibility and association with the intended item.
QA Batch visual audit gpt-5.6-terra Audits exact rendered price and visible editing artifacts after generation.
Explorer RAG Image retrieval google/siglip2-base-patch16-224 Filters candidates by Yelp business ID, then ranks the eligible images by question-text to image-pixel cosine similarity. The explorer replays the frozen top-six ranking.
Explorer RAG Answer generation GPT-5.6 → GPT-4o Selected per query from vision-capable OpenAI models: GPT-5.6 Sol/Terra/Luna, GPT-5.5, GPT-5.4, GPT-5, GPT-4.1, or GPT-4o. Terra is the configured default.
Benchmark Answer-model comparison 8 OpenAI + 5 Fireworks Runs the same 50 questions, retrieved images, automatic scoring, and two prompt conditions across all 13 models shown on the Results and Predictions pages.
General RAG Image/text embedding google/siglip2-base-patch16-224 Places queries, images, and retrieval text in a shared normalized embedding space.
General RAG Caption generation gpt-5.6-luna Produces retrieval transcriptions containing readable menu items, prices, and sections.
General RAG Rank fusion RRF + diversity penalty Combines image similarity, text similarity, and lexical rankings, then discourages redundant images.
07

Interpretation and limitations

Important constraints for reading baseline results.

Counterfactual, not longitudinal

The conflicting price is generated. It does not establish that the restaurant charged that price at the synthetic date.

Independent pair semantics

Five edits from one source are five independent evaluation cases. They should not be interpreted as five historical versions of one menu.

Restaurant identity is known

The restaurant's Yelp business ID is supplied by the page context and used only to restrict candidates. The expected item, price, and image identities remain evaluation labels and do not rank the images.

Vision extraction can fail

Small, blurred, angled, or stylized prices may be misread. QA labels and retrieved pixels should be inspected alongside aggregate accuracy.

Small working corpus

Ten sources and fifty cases support rapid baseline experiments, not claims about general restaurant-menu performance.

Metadata is incomplete

Original capture times are generally unavailable. Missing metadata is retained as missing rather than inferred.

08

Implementation artifacts

Primary modules and outputs used by the current prototype.

Conflict generationsrc/place_rag/conflicts.py
Explorer retrievalsrc/place_rag/explorer.py
Vision answeringsrc/place_rag/vision.py
HTTP interfacesrc/place_rag/api.py
Dataset manifestdata/menu-conflicts-50/manifest.json
Verification reportdata/menu-conflicts-50/verification.json
Visual auditdata/menu-conflicts-50/audit.json