MC MenuConflicts-RAG
methods / v0.1
METHODS / SYSTEM DESCRIPTION

Dataset construction and inference pipeline

This page records how the menu conflict corpus was assembled, how controlled price edits were produced, and how the explorer retrieves and answers from image evidence.

Evaluation unit One untouched menu photograph paired with one counterfactual version containing a single intended price replacement.
01

End-to-end overview

Corpus preparation and online inference are separate phases.

01Yelp menu photosPublic source corpus
→
02Source selection10 original menus
→
03Price intervention50 counterfactuals
→
04Structural + visual QALabels retained
→
05Scoped retrievalOriginal / conflict / both
→
06Vision answerCited image evidence
02

Conflict dataset construction

The image edit is localized; the rest of the photograph is copied from the source.

  1. 2.1

    Prepare the menu-photo corpus

    Menu-labelled photographs are extracted from the Yelp Open Dataset photo archive. The prepared local corpus contains 1,678 menu images associated with 1,338 businesses. Business and photo identifiers are preserved as metadata.

    data/yelp-menu/images · data/yelp-menu/metadata.json
  2. 2.2

    Select source photographs

    Ten real, user-contributed menu photographs are selected as the working source set. Each source contributes five independent conflict cases. A case is a pair—not a six-image temporal sequence.

    data/conflict-sources.json · data/menu-conflicts-50/originals
  3. 2.3

    Locate an item-price target

    A vision model identifies a readable menu item, its displayed price, and a normalized price bounding box. The detector output is cached so reruns reuse the same localization.

    Detector gpt-5.6-terra Cached output target_cache.json
  4. 2.4

    Construct deterministic edit inputs

    The normalized box is converted to pixel coordinates. A small context crop is calculated around the target, then an edit mask is restricted to the price region. Crop, mask, and paste coordinates are stored in the manifest.

    input cropalpha maskprice boxedit boxcrop box
  5. 2.5

    Generate and composite the replacement

    A deterministic seed selects a different target price. The crop and mask are sent to the image editor, which renders the new price in local photographic context. Only the permitted edit region is pasted back into the full-resolution source.

    Editor gpt-image-2 Seed 20260729 Variants 5 per source
  6. 2.6

    Assign experimental metadata

    Each sample records source and synthetic paths, item name, intended prices, edit coordinates, model provenance, QA state, and a synthetic observation date. Synthetic dates support conflict experiments; they are not claims about when a real menu was photographed.

    data/menu-conflicts-50/manifest.json · samples.jsonl
03

Worked image example

One source image and one localized counterfactual from the released corpus.

Original source_01 restaurant menu photograph
Original evidence original_source_01
localized price edit →
Counterfactual source_01 menu with an edited Galbitang price
Counterfactual evidence conflict_f9f4d0487b4b
Context crop

Inspect the localized edit

The full-menu pair above establishes image-level context. These stored generation artifacts expose the small region sent to the editor before compositing.

Original context crop around the Galbitang price
Original context crop
Generated context crop containing the changed Galbitang price
Generated context crop
Target item
Galbitang
Manifest annotation
14.99 → 16.99
Source
Yelp menu photograph
Intervention
Price-region inpainting
Case semantics
Independent original/counterfactual pair
Inclusion
Retained in the 50-case corpus

The full image is retained at its source dimensions. Generation operates on a context crop, and compositing copies only the permitted edit region back into the original photograph.

04

Verification and QA policy

Structural validity is enforced. Visual QA is reported but does not remove samples.

Structural verification

Checks that all files exist, dimensions match their sources, each source has five cases, and pixels outside the allowed region remain unchanged.

Missing files
0
Size mismatches
0
Outside-region violations
0
Result
valid

Batch visual audit

A separate vision pass assesses whether the intended price is readable and whether the edit appears visually plausible.

Exact-price passes
34 / 50
Realism passes
49 / 50
Accepted
33
Needs review
17
05

Explorer RAG pipeline

The current baseline favors a transparent retrieval path over a large precomputed index.

Input

Question + evidence scope

The user supplies an item-price question, top K, one of three scopes—originals, conflicts, or both—and whether conflict reporting should be requested.

↓
Candidate construction

Manifest-backed image records

Original records aggregate the item labels associated with their source. Each conflict record carries only its target item label and non-price provenance.

↓
Retrieval

Lexical + fuzzy item matching

Question tokens are matched against item names with overlap, substring, and sequence-similarity signals. The result is a small ranked image set.

↓
Leakage boundary

Expected prices are removed

Manifest prices are available for dataset inspection but are not placed in retrieval captions or answer metadata. The answer model must read prices from retrieved pixels.

↓
Answering

Multimodal evidence package

The vision model receives the question, each retrieved image, its image ID, evidence type, source ID, and available observation metadata. When “Report conflicts” is enabled, the prompt also requests explicit comparison and disagreement reporting; when disabled, that instruction is omitted.

↓
Output

Direct answer with citations

Claims cite image identifiers in the form [image:IMAGE_ID]. When images disagree, the answer is instructed to report both observations and cite each side.

06

Models and retrieval components

Model assignments for corpus construction, the current explorer, and the general image index.

Current explorer baseline

The 50-case conflict explorer does not use embedding retrieval. It uses supervised item labels from the manifest, then sends the retrieved pixels to the answer model. The SigLIP2 embedding path belongs to the broader place-photo RAG service and is included below for completeness.

Phase Stage Model or method Role
Construction Target localization gpt-5.6-terra Reads a source menu and returns item name, original price, and normalized price box.
Construction Price mutation Deterministic Python Samples a changed numeric price from the fixed experiment seed.
Construction Crop and mask Pillow geometry Converts boxes to pixels and creates the localized crop, mask, and paste region.
Construction Generative editing gpt-image-2 Inpaints the replacement price within the cropped menu context.
QA Generation validation gpt-5.6-luna Checks expected-price visibility and association with the intended item.
QA Batch visual audit gpt-5.6-terra Audits exact rendered price and visible editing artifacts after generation.
Explorer RAG Conflict retrieval Lexical + fuzzy scoring Ranks manifest item labels using token overlap, substring match, and sequence similarity. No embedding model is called.
Explorer RAG Answer generation GPT-5.6 → GPT-4o Selected per query from vision-capable OpenAI models: GPT-5.6 Sol/Terra/Luna, GPT-5.5, GPT-5.4, GPT-5, GPT-4.1, or GPT-4o. Terra is the configured default.
General RAG Image/text embedding google/siglip2-base-patch16-224 Places queries, images, and retrieval text in a shared normalized embedding space.
General RAG Caption generation gpt-5.6-luna Produces retrieval transcriptions containing readable menu items, prices, and sections.
General RAG Rank fusion RRF + diversity penalty Combines image similarity, text similarity, and lexical rankings, then discourages redundant images.
07

Interpretation and limitations

Important constraints for reading baseline results.

Counterfactual, not longitudinal

The conflicting price is generated. It does not establish that the restaurant charged that price at the synthetic date.

Independent pair semantics

Five edits from one source are five independent evaluation cases. They should not be interpreted as five historical versions of one menu.

Retrieval labels are supervised

Item names come from target detection and are used for retrieval. This baseline evaluates conflict-aware visual answering more directly than open-world OCR retrieval.

Vision extraction can fail

Small, blurred, angled, or stylized prices may be misread. QA labels and retrieved pixels should be inspected alongside aggregate accuracy.

Small working corpus

Ten sources and fifty cases support rapid baseline experiments, not claims about general restaurant-menu performance.

Metadata is incomplete

Original capture times are generally unavailable. Missing metadata is retained as missing rather than inferred.

08

Implementation artifacts

Primary modules and outputs used by the current prototype.

Conflict generationsrc/place_rag/conflicts.py
Explorer retrievalsrc/place_rag/explorer.py
Vision answeringsrc/place_rag/vision.py
HTTP interfacesrc/place_rag/api.py
Dataset manifestdata/menu-conflicts-50/manifest.json
Verification reportdata/menu-conflicts-50/verification.json
Visual auditdata/menu-conflicts-50/audit.json