SamaChama
A portal for a Kenyan welfare and investment group to manage contributions, loans and shared financial records.
Evaluation prototype
An AI agent prototype for evaluating meal recommendations against budget, distance, dietary needs and pickup or delivery preferences.
A meal can fit someone’s taste and still fail their actual requirements. Menu price alone omits tax and fees, an open restaurant may lack the requested delivery service, and a menu description may leave dietary questions unresolved.
People choosing meals around a firm budget, dietary needs, and practical pickup or delivery constraints.
Product decision
Fictional menus kept prices, availability and dietary information consistent across models. That made it possible to compare how each model handled the same constraints. The prototype evaluates recommendations using fictional data. Live restaurant search is outside its current scope.
In the React chat, reviewers can choose a model and scenario, change constraints in follow-up questions, and compare replies with reference answers.
The Vite client calls FastAPI. Scenario chat and batch evaluation share the prompt and provider adapter. General chat uses a separate local Ollama path. Python functions for cost, distance, hours, coverage, dietary checks and ranking run in a separate test harness.
I compared three models on the same 25 fictional scenarios. An answer passed only if it met every criterion for its scenario. I reviewed the replies and used AI assistance to check the grading.
60% passed (15/25)
24% passed (6/25)
16% passed (4/25)
Timing includes network time and model loading. Costs apply to successful requests at the rates recorded for this run. The local models had zero API fees, excluding hardware and power.
These paraphrased examples use fictional restaurant data. A critical failure breaks a firm constraint or makes an unsupported claim someone could act on.
GPT-5.4 Mini scored highest and responded fastest in this run, but seven answers still had critical failures. That result informed a design decision to check eligibility in Python and use the model to explain the resulting shortlist. The constraint checks currently run separately from the chat.