The problem
Nobody can read 660,000 reviews. A model can label them, but its answer is only useful if it is affordable, checked, and every number traces back to a real review.
What I did
- Accounted for all 660,622 reviews in code before any model saw them, matching the course's own checksum.
- Labeled a declared 100,000-review sample with a small, cheap model, 50 reviews per request, with a cache so each distinct text is labeled once.
- Had a different model check a blind 1% sample, and tested that checker with 20 planted wrong labels and 5 prompt-injection reviews.
- Hand-labeled 50 reviews myself and compared them to the model, field by field.
- Grouped complaints into issues, ranked four business areas by severity, and wrote a memo where every number cites a claim and a review.
- Shipped a public dashboard backed by a database and an API, so anyone can follow the ranking down to a single review.
Results
$4.36for 100,000 reviews, under a $10 cap
20 / 20planted wrong labels caught
5 / 5prompt injections resisted
35.5%of complaint severity is usability
The recommendation: put next quarter's effort into usability, ahead of billing and support at 19.1%.
What I'd do differently
Get more human labels in early. Only 50 labels are mine, and the model matched me on topic 42% of the time, so the answer leans on the model and its checker. Next: more human labels, and the full run over every review, estimated at $25 to $48.
Artifacts
Open the dashboard ↗ · Source and evidence on GitHub ↗
