Olu Akele / Portfolio 2026Now building: ShelfReadyPrefer the technical details? Flip this.

Case study · Build 01In progressUpdated

ShelfReady

Cleans up an online store's product details so AI shopping assistants can find, compare and buy its products.Makes a Shopify catalog clean and structured enough for AI shopping agents to find, compare and buy from.

In plain English

Problem
More people now ask an AI assistant to shop for them. When a product's details are vague or missing, the assistant skips it, and the store never knows why.
Bet
Check every product the way an AI shopper would, fix what's missing, and have a person approve each change. Then test real AI shopping before and after.
Result
Almost every product is now fully readable by an AI shopper: 99.3%, up from 78%. In 360 test shopping trips, the AI never bought the wrong item.

Customer & problem

A mid-size outdoor-gear brand hears that shoppers are starting to buy through AI assistants. Its catalog has vague titles, missing sizes and materials, and inconsistent variants, so agents skip its products. The brand doesn't know how big the gap is, or which fixes matter.

ShelfReady asks one question of every product: could an AI shopping agent, given only this product's data, confidently match it to a shopper's request and answer their follow-up questions? Then it closes the gap and measures whether agents actually shop better.

Strategy & scope

The MVP is four pieces: a readiness audit that scores each product 0–100, an AI fixer with a human review queue, an MCP server so agents can shop the store, and an eval that runs the same shopping tasks on the catalog before and after fixing.

What I chose not to build, and why:

  • No payments in the agent path. The agent's last step returns a Shopify checkout link, and the shopper pays on Shopify. That keeps payment and personal data away from the agent entirely.
  • No auto-applied fixes. The fixer proposes and a person approves. Wrong product data causes returns and brand damage, and every approval becomes labeled data for improving the fixer.
  • One store, Shopify only, no app-install flow. Multiple stores, other platforms and OAuth install are out of scope for v1. They add surface area without changing whether the core idea works.
  • $0 a month until it proves its value. That meant free tiers everywhere and Gemini as the default model, with a model comparison planned once there's an eval to compare on.

The demo store is seeded rather than scraped. A deterministic generator built 150 products and injected 8 kinds of defects into 50 of them, so every audit and eval is scored against an exact answer key.

What I built

Fix the catalog

  1. Shopify store

    150 products, 50 seeded with defects

  2. Sync

    Mirror into Postgres with a content hash

  3. Audit

    7 rubric checks + one grounded model call per product

  4. Fixer

    Rule fixes + model fixes that must quote the product's data

  5. Human review

    Approve, edit or reject each fix

  6. Write-backlive Shopify

    Approved fixes to Shopify, then re-sync and re-audit

Let agents shop

  1. AI agent

    e.g. Claude, via a custom connector

  2. MCP server

    4 tools on Vercel, rate-limited, every call logged

  3. Search + details

    Structured filters on the Postgres copy

  4. Stock + cartlive Shopify

    Live from Shopify's Storefront API

  5. Checkout link

    The shopper pays on Shopify; the agent never sees payment

ShelfReady through P5. Search reads the synced copy because it has the structured attributes; stock and carts always go to Shopify live.
  • P0 Setup: Shopify auth with a cached, early-refreshed token; Postgres on Neon; a live /status page that checks every connection.
  • P1 Catalog: the generator plus a ground-truth file, an idempotent and resumable seed, and a sync that mirrors the store into Postgres.
  • P2 Audit: a rubric of seven checks, category detection from the product's own data (150/150 correct), one model call per product that must quote its evidence, and an /audit page.
  • P3 Fixer: rule-based and grounded model fixes, a /review queue (anyone can view, only an admin can change), write-back to Shopify, and a before/after view.
  • P4 MCP: search_products, get_product, check_availability and create_cart on a public, rate-limited endpoint, tested end to end with Claude.
  • P5 Eval + demo: 60 shopping tasks with verified answers, run 3 times per catalog by an agent using the same tools as the live server, with an /eval page showing every transcript. Plus a public /playground where anyone can chat with the shopping agent: rate-limited, capped at 200 messages a day, and falling back to recorded replays of real sessions instead of an error.

Key decisions & trade-offs

  • Ground truth before measurement. A deterministic generator rather than model-written products. Trade-off: the copy reads more templated.
  • Weak checks cap the band. The first audit ranked products correctly but called 89% of the store agent-ready, because strong checks hid fatal gaps. Any check below 0.5 now caps a product at Partial, and two or more make it Not ready. Trade-off: bands no longer follow from the score alone, so the UI shows which check capped each product. Raising the cutoffs gave a similar split, but with thresholds picked after seeing the data.
  • Grounded fixes or no fixes. Model fixes need a verbatim quote from the product's own data. Titles may only use words already there, and descriptions only numbers already there. Anything else goes to the merchant. Trade-off: some reasonable rewrites get dropped.
  • Duplicate variants go to the merchant. Deleting a variant moves inventory and can break orders, which isn't a call the fixer should make.
  • Search on the synced copy, stock and carts live. Shopify's storefront search can't filter on the structured attributes the fixer improved. Trade-off: search is only as fresh as the last sync.
  • A public MCP endpoint, rate-limited, with no login. Anyone can connect an agent to the demo. The tools are read-only except cart creation, which takes no payment or customer data.
  • Fix the agent's tools, not just the data. The first fair eval run failed mostly on tool behavior, on both catalogs: search dropped numbers, so "6 person tent" returned the cheapest tents, and invented attribute names silently matched nothing. I fixed the tools mid-eval. Trade-off: every earlier run was discarded, so before and after always use the same tool version.
  • Repeat every eval 3 times. At 40 tasks, one task is 2.5 points, and the same model varies run to run. Trade-off: 3× the cost, still only about $1 for a clean run.
  • A live demo with a budget. The playground spends real money, so it has a hard daily ceiling (about $0.50 at worst) and falls back to recorded real sessions, so visitors never see an error.

Results

The headline: an AI agent bought 0 wrong products in 360 shopping sessions, while the catalog's readiness score went from 94.1 to 98.3. The eval ran 60 shopping tasks, 3 times each, on the original messy catalog and on the fixed one, using the same model (Gemini 3.5 Flash-Lite) and the same tools.

Store readiness score
from 94.1to 98.3
Products agent-ready
from 78%to 99.3%
Products not ready
from 11to 0
Agent shopping success
from 98.9%to 99.4%
Wrong product bought
from 0%to 0%
Tokens per shopping task
from 9,773to 9,370
Audit: same rubric and model before and after the review queue (105 fixes applied, 2 rejected, 15 left for the merchant). Eval: 60 tasks × 3 runs per catalog. Sources: Episodes 04–06.

The finding: agent-readiness is data and tools. Before I fixed the agent-facing search tool, the first fair run scored 95% on the messy catalog and 92.5% on the fixed one. Fixing it moved both catalogs to 97.5–100%. With good tools, a capable agent absorbed most of the catalog mess. The few remaining failures all trace to missing or damaged facts. One of them is a product whose weight is still a merchant gap, and the agent declined rather than guess.

What this doesn't show yet. The success rate barely moved (98.9% → 99.4%) because many tasks have 2–4 correct products, so an agent can route around one damaged listing. Single-answer tasks would separate the two catalogs more sharply. The clean products also score exactly 100 on the audit because the generator makes them perfect, which a real catalog won't.

In a live test through the MCP connector, Claude went from "waterproof hiking boots, men's 10, under $150" to a working checkout link in 7 tool calls. There was no boot under budget, so it offered the closest match: a waterproof hiking shoe at $115.95.

What broke

  • A laptop freeze mid-seed. 54 of 150 products had landed. Because resume state lives on each Shopify product, the re-run skipped those and created the other 96 with 0 failures.
  • A score that hid the defects. The first audit said 89% agent-ready and 0% not ready. It was fixed with the band gate described above.
  • Free-tier quotas. An unpaced audit hit the per-minute limit after about 100 calls and saved a partial run. Calls are now spaced out, and partial runs are never saved.
  • An "all-or-nothing" apply that wasn't. The first test landed 6 of 8 changes and marked all 8 failed. Shopify has no transaction across mutations, so each change now records its own result.
  • A fixer running on a stale audit. It drafted 11 rewrites of descriptions that were already fixed. They were deleted before review, and the fixer now refuses stale products.
  • An invisible store. Products were Active but not published to a sales channel, so the Storefront API saw nothing. Active isn't the same as published.
  • "Lynx 30f." The fixer had treated the product's URL handle as evidence, so a bad spec looked grounded. Specs must now match the product's own text in number and unit. A re-check of all 50 live fixes found 3 wrong, and all 3 were corrected.
  • An eval harness bug. Stand-in product ids for the messy catalog were longer than the tools allow, so on long-named products the agent found the item but couldn't add it to the cart. That unfairly lowered "before". I found it from a crash, fixed the ids, and discarded the flawed run.
  • A model outage. Gemini's free tier returned "high demand" errors and 115-second calls for hours. Moving to the paid tier cost about $1.54 for the whole eval, including discarded runs.

Roadmap

  • P6 Ship: record a Loom walkthrough and write the launch post.
  • Second model: Groq's gpt-oss-120b is running the same eval on its free tier, and its results go on /eval when it finishes.
  • Sharper tasks: add single-answer tasks so the eval can separate a messy catalog from a fixed one more clearly.
  • Search sort option: questions like "warmest" or "lightest" need sorting, not just relevance. One recorded demo answer got this wrong and was replaced before it shipped.
  • Known rough edges: drafted descriptions can only answer what the data covers, and some attribute gaps still need the merchant.

Build log

  1. Episode 00
    Sep 24, 2026
    Before any code: six accounts, a $0 stack, and three gotchas
  2. Episode 01
    Sep 25, 2026
    A green status page before a single feature
  3. Episode 02
    Sep 27, 2026
    Seeding 150 products, a third of them broken on purpose
  4. Episode 03
    Sep 27, 2026
    My audit said 89% of the store was ready. That was the bug.
  5. Episode 04
    Sep 28, 2026
    101 fixes, one human reviewer, and a store score from 94.1 to 98.3
  6. Episode 05
    Sep 28, 2026
    Claude shopped my store and came back with a checkout link
  7. Episode 06
    Sep 29, 2026
    360 agent shopping sessions, 0 wrong products, and a finding I didn't expect
  8. Episode 07
    Sep 30, 2026
    A public AI shopping demo that can't run up a bill or show an error