Food detection

The capstone brief: 16,256 photographs across 17 dishes, class labels only, no bounding boxes, and a fictional consumer client, NutriScan, that wants food recognition from a phone photo. The open question is what is most useful to a business given exactly that data. Recognition alone is a commodity; what a business can deploy is a system that knows what it is confident about, quantifies that confidence, and declines what it does not know instead of guessing. This demo is that system, live: photograph a plate, get per-item dish identification with calibrated uncertainty, a kilocalorie estimate per item and for the plate, and an explicit decline for anything outside the 17-dish menu. On the held-out test set it answers 94.9% of items, the correct dish appears in 97.6% of those answers, and the remainder is declined, not guessed.

How it is achieved: the dataset ships without bounding boxes, so detection is zero-shot Grounding DINO with SAM 2.1 segmentation; classification is a SigLIP2-B backbone fine-tuned on duplicate-aware splits with an audited label set and seed-averaged weights, reaching 0.9706 macro-F1 on held-out test. The served model is deliberately not the benchmark winner: a variant retrained on detection crops scores 0.9421 on the same test, yet on real plates it recovers more true dishes and rejects the condiment bowls the benchmark model labels confidently, so it serves; the trade-off is measured, committed, and reversible in one line. Uncertainty is split-conformal calibration wired as a two-tier gate in the serving API. A zero-shot vision-language model was evaluated as the buy option: 0.856 dish accuracy against the classifier's 0.982 on the same images, with kilocalorie error too large to use, settling make-versus-buy with numbers. Every figure here traces to a committed file.

Benchmark, held-out test

0.9706

macro-F1 (top-1 0.9740), the seed-averaged SigLIP2-B soup

Served model

0.9421

retrained on detection crops; lower on test, but recovers more true dishes on real plates, so it serves

Answered on test

94.9%

of items; the correct dish is in 97.6% of those answers, and 5.1% are declined, not guessed

Make versus buy

0.982 vs 0.856

supervised classifier against a zero-shot vision-language model; the VLM's calorie error (SMAPE 115.7%) is too large to use

Built on the Food-101 dataset (non-commercial licence). A portfolio and research demonstration only, not a commercial product. NutriScan is a fictional client. Not medical or dietary advice.

Sources, all committed in the capstone repository: benchmark and served macro-F1 from reports/M2_RESULTS.md; answered, correct and declined fractions from reports/appendices/serving_operating_point.json; the classifier-versus-VLM comparison and calorie error from reports/VLM_BENCH.md; the dataset counts from reports/dataset_profile.json.