Agentic
Ask AutoOracle is a focused, tool-calling chat interface for the NZ used car market: buyer advice, known issues, live prices, fleet counts and trends, all grounded in AutoOracle's market data, NZTA data and the M15/M16 enrichment content, and nothing outside it. The service backbone is built, the tool contract is frozen, and the public UI and rollout controls are the part still being finished.
Scope
Architecture
The design is a hybrid: a base set of tools the model calls when it knows the entity it needs (family enrichment, make enrichment, market stats, a live sample), plus one additional tool backed by a vector search over the enrichment text for queries where the user doesn't name a make or family up front. The vectors live inside the existing production database as a Postgres extension, not a separate service, and are generated automatically after each M15/M16 enrichment write. The Ask service itself is already built as its own backend, with the contract frozen and the rollout gates now becoming the remaining product work.
A discovery query such as "find me a reliable 7-seater grey import under $15k" walks through three steps:
Vector search
The query is embedded and matched against enrichment text by cosine similarity, returning a ranked shortlist of candidate families rather than a single guess.
Structured lookup
For the top candidates, a second tool call fetches real listing counts, median price and fleet size, so the answer reflects the current market, not just the narrative text.
Synthesis and citation
The model combines both results into an answer with a source cited against each figure, and says so plainly if a claim can't be grounded in a tool result.
Six tools in total, already frozen so the build isn't re-guessing the contract mid-way:
get_family_enrichment
Buyer signals, known issues, NZ context and era notes for a named make and family.
get_make_enrichment
Brand history, NZ presence and buyer notes for a named make.
get_family_stats
Listing counts, median price and fleet size for a named family.
get_price_trend
Monthly median price by model-year cohort, so the assistant describes a trend rather than quoting one pre-baked number.
get_listings_sample
A small page of live listings, filterable by price, year, mileage and region.
search_enrichments
The vector search tool, used when the user hasn't named a make or family.
Orchestration is left to the model, not a fixed script. Given a question, the LLM decides which of the six tools to call and in what order, chains a second call when the first result isn't enough on its own (a discovery query calls search_enrichments, then get_family_stats on the shortlist it returns), and stops once it has enough to answer. A session context tracks which make and family are currently in focus and any constraints the user has stated, so a follow-up question doesn't repeat a tool call it already has the answer to. Phase 1 already carries a hard cap on how many tool calls a single query can chain, so a query can't spiral into an open-ended loop.
1.000
Retrieval precision@5, labelled benchmark
$0.006
To embed the full 1,198-chunk corpus
Investigation
Every finding below came from the real production database, the real enrichment corpus and the real Bedrock model AutoOracle would run in production, not a synthetic prototype. That mattered: several of the more useful results were things a stub-tool test would have missed entirely. Those findings are now reflected in the service contract and the build plan, not just in a research note.
Safety held under red-teaming
52 of 52 test cases passed across in-scope, out-of-scope and adversarial suites, including four attacks planted inside tool results rather than the user's own message, such as a seller description instructing the assistant to disable an immobiliser, and an enrichment field asking it to leak the system prompt. All were resisted.
Citations had to be machine-checked, not eyeballed
Free-text citations passed a casual read while quietly failing a machine check. Rebuilding the schema around atomic figures got quantitative citations verbatim-grounded in tool output, with the "I don't have that" path never emitting a fabricated source.
The eval harness itself had a bug worth finding
A naive grounding check over-reported fabrication by roughly six times (17.8% down to 2.7%) until a three-tier rebuild caught a text-embedded-number blind spot the original 4-query check had missed.
Real cost came in lower than the original estimate
The original cost model used retired pricing by mistake. A real 52-query run against real DB-backed tools measured cost about 19% below the estimate, and a head-to-head test against a larger model found no quality gap worth its 3-7× higher cost.
104
Prompts in the golden eval baseline set
94%
Usefulness on the baseline run
2.7%
Residual range-synthesis rate, post-fix
What's next
The architecture review confirmed the hybrid design as a single build rather than a phased tool-calling-then-vector-search migration, since the vector search step turned out to be nearly free and demonstrably works. The Ask service backbone is already built; what remains is the public UI, trust surfaces, and rollout controls. Phase 1 covers:
Rollout is staged in three steps, each gated on measured thresholds, admin-alpha, then paid-beta, then broad-premium, rather than opened to everyone at once. It's a premium feature from day one: each query carries a real LLM cost, so daily limits are enforced before any model call is made.
| Tier | Daily limit |
|---|---|
| Trial | No access |
| User | 40 queries |
| Reporting | 80 queries |
| Admin | Unlimited |
Want to see what's already live?
Open AutoOracle