Back to projects

Evaluation prototype

MealScout Agent

An AI agent prototype for evaluating meal recommendations against budget, distance, dietary needs and pickup or delivery preferences.

The problem

A meal can fit someone’s taste and still fail their actual requirements. Menu price alone omits tax and fees, an open restaurant may lack the requested delivery service, and a menu description may leave dietary questions unresolved.

Who it serves

People choosing meals around a firm budget, dietary needs, and practical pickup or delivery constraints.

Contribution
Built the scenario-based prototype, designed the benchmark, and reviewed and compared model replies.
Timeline
October–November 2026
Technology
  • React
  • TypeScript
  • Vite
  • Python
  • FastAPI
  • Ollama

Product decision

The same scenarios for every model

Fictional menus kept prices, availability and dietary information consistent across models. That made it possible to compare how each model handled the same constraints. The prototype evaluates recommendations using fictional data. Live restaurant search is outside its current scope.

The solution

In the React chat, reviewers can choose a model and scenario, change constraints in follow-up questions, and compare replies with reference answers.

How the scenario chat works
  1. React sends the question, history and scenario ID
  2. FastAPI adds the scenario facts
  3. The selected model returns a reply

Technical implementation

The Vite client calls FastAPI. Scenario chat and batch evaluation share the prompt and provider adapter. General chat uses a separate local Ollama path. Python functions for cost, distance, hours, coverage, dietary checks and ranking run in a separate test harness.

Model evaluation

I compared three models on the same 25 fictional scenarios. An answer passed only if it met every criterion for its scenario. I reviewed the replies and used AI assistance to check the grading.

Model comparison · one run per model on November 5, 2026
  • OpenAI GPT-5.4 Mini

    60% passed (15/25)

    Critical failures
    7
    Request errors
    0
    Mean response
    2.06 s
    Est. API fee / 1,000
    $2.20
  • Ollama Qwen 2.5 3B

    24% passed (6/25)

    Critical failures
    15
    Request errors
    0
    Mean response
    49.26 s
    Est. API fee / 1,000
    $0.00
  • Ollama Llama 3.2 3B

    16% passed (4/25)

    Critical failures
    15
    Request errors
    0
    Mean response
    47.82 s
    Est. API fee / 1,000
    $0.00

Timing includes network time and model loading. Costs apply to successful requests at the rates recorded for this run. The local models had zero API fees, excluding hardware and power.

Failure cases

These paraphrased examples use fictional restaurant data. A critical failure breaks a firm constraint or makes an unsupported claim someone could act on.

  • A $12 budget including a $2 tip

    Expected
    The bowl exceeds the budget at $12.80, including $0.80 tax and a $2 tip.
    Observed
    GPT-5.4 Mini opened with yes, while its own table showed the meal was over budget.
  • A severe peanut allergy

    Expected
    Cross-contact safety could not be verified from the available information.
    Observed
    Qwen 2.5 3B called the bowl safe while acknowledging that cross-contact could not be verified.
  • A priced meal with unknown pickup availability

    Expected
    The $10.80 price was known. Pickup availability was unverified.
    Observed
    GPT-5.4 Mini presented a verified pickup match before disclosing that availability was unknown.

What the results changed

GPT-5.4 Mini scored highest and responded fastest in this run, but seven answers still had critical failures. That result informed a design decision to check eligibility in Python and use the model to explain the resulting shortlist. The constraint checks currently run separately from the chat.