ML & AI

Machine learning, LLM enrichment, and human review. Each doing its job.

Price prediction from a nine-model competition. LLM condition scoring with a formal multi-model evaluation. Family profiles generated, then reviewed before anything reaches the database. And a development process that used AI as a tool throughout.

Price prediction

Nine models entered. XGBoost won.

Price prediction starts with a model race, not a model assumption. Nine algorithms are trained and evaluated on every run: linear regression, Lasso, Ridge, support vector regression, K-nearest neighbours, decision tree, bagging, random forest, and XGBoost. Each goes through two tuning stages: a randomised search across a broad parameter space, then a grid search focused on the best region. The model with the lowest RMSE on the held-out test set is selected.

AutoOracle trains two separate models: one for each major NZ car-market source. The platforms differ structurally. One captures seller type (dealer or private) and region from the listing tile; the other does not. Keeping the models separate means each one uses the signals actually available for that source, and a data quality problem on one platform does not affect the other.

Training uses a time-based split: the oldest 80% of listings by first-seen date form the training set, and the most recent 20% form the test set. A random split would allow the model to see future prices during training, since a listing from April 2025 in the training set can be compared to an otherwise identical one from October 2024 in the test set. A time-based split is the only honest simulation of a task that is always about predicting future listings from past patterns. At Iteration 2, XGBoost won both sources.

R²=0.893 Market source model RMSE $5,296 – Iter 2.1
R²=0.844 Market source model RMSE $5,821 – Iter 2.1
183,023 Market source training rows Time-based 80/20 split
294,190 Market source training rows Time-based 80/20 split

Features

What the model actually looks at.

Every record is represented by the same structured feature set: canonical taxonomy identifiers (make, family, and model), vehicle age, odometer reading, engine size, fuel type, gearbox type, and the calendar month the record was first published. On one market source, seller type and region are additional signals that the other tiles do not carry.

The taxonomy identifiers are resolved to integer foreign keys during ingestion and let the model learn that a 2019 Toyota RAV4 and a 2019 Mazda CX-5 have completely different price distributions, even when their age, mileage, and engine size are identical. The exploratory analysis confirmed vehicle age has the strongest single correlation with price, followed by odometer reading and engine displacement. But within any age/mileage/size band, taxonomy identity is what explains most of the remaining variance.

-0.651

Vehicle age vs price (Pearson r)

-0.483

Odometer vs price (Pearson r)

+0.277

Engine size vs price (Pearson r)

Value scoring

A number that tells you if it is a deal.

The price prediction on its own is not enough. A car listed below the estimate is only a genuine deal if the gap is meaningful, not statistical noise. The value score makes that judgment explicit on a 1-to-10 scale.

So how does the model decide what a car should cost? Not with a simple average. A median lumps a tired older car in with a nearly new one. Instead, a machine learning model trained on around 600,000 New Zealand listings predicts the going rate for that exact car, weighing its year, mileage, make, model, engine, fuel, transmission and region together. That predicted figure is the benchmark every asking price is measured against.

Take a 2019 Toyota Corolla, 60,000 km, automatic petrol, in Auckland. The model predicts a going rate of $21,000. Three sellers, three very different scores.

Asking price Versus estimate Score Verdict
$18,000 14% below 9.0 Good value
$21,000 At estimate 5.5 Fairly priced
$25,000 19% above 1.6 Overpriced

Under the hood, the gap is turned into the score with a sigmoid function. The listed price is compared to the estimate as a percentage deviation; that percentage passes through a logistic curve calibrated with steepness constant k=7. A 0% deviation (listed at exactly the estimate) maps to 5.5, the neutral midpoint. Underpriced listings curve toward 10; overpriced ones toward 1. The sigmoid handles extreme outliers naturally: a car listed at 50% below the estimate approaches 10 asymptotically rather than being clamped.

Deviation Score Band
+20% underpriced 9.5 Green
+10% underpriced 8.3 Green
+5% underpriced 7.0 Green
At market 5.5 Amber
-5% overpriced 4.0 Amber
-10% overpriced 2.7 Red
-20% overpriced 1.5 Red

After training, k=7 was validated against 596,517 active records. The two market-source medians came out at 5.90 and 5.80. Green, amber, and red bands came out roughly balanced at one-third each. The slight upward skew is consistent with a competitive market where records tend to price fractionally below model estimates. No adjustment was required.

5.90

Market source median value score

5.80

Market source median value score

596k+

Active listings validated at k=7

LLM classification

Condition scoring: the evaluation is still running.

Each listing description is passed to a language model to extract four structured signals: condition, damage, service history, and modification. These supplement the price model. A car with full service history or described as exceptional condition commands a premium that age and mileage alone cannot explain.

The reason for using an LLM rather than a traditional classifier is straightforward: there was no labelled training data. AutoOracle had hundreds of thousands of listing descriptions but no ground truth for what each one actually meant in terms of condition or history. A supervised model requires labelled examples to learn from. An LLM can apply the task from a prompt, which made it the only practical starting point. The question was never whether to use one, but which one to trust.

Four models have been put through the evaluation so far. Amazon Nova Lite via Bedrock, gpt-5.4-nano, gpt-5-nano, and now Mistral Ministral 8B, which is currently running. Nova Lite had a calibration problem: 43.7% of its condition classifications came back as Excellent, and 50% of its service history classifications as Full, even for listings with thin or ambiguous descriptions. A confidence threshold cannot fix a calibration bias. It would still leave high-confidence incorrect classifications in the output.

Two OpenAI models followed. gpt-5.4-nano produced more distributed outputs but had a service history skew in the opposite direction: 35% of rows fell below 0.50 confidence, meaning the model was frequently unsure. gpt-5-nano is currently the best-performing model from the completed runs. Mistral Ministral 8B is the fourth candidate, running now on Bedrock. The goal is to find a model with well-calibrated distributions across all four signals before committing to a full production run.

What a model evaluation looks like

A fixed sample set, output distribution analysis per signal, and a check on how many classifications cluster at the extremes. A model that returns Excellent for 44% of listings has failed the evaluation regardless of its published benchmarks. The test is the distribution, not the label.

Why the process matters

The classifier directly affects the condition adjustment factors applied to Oracle price estimates. A systematic calibration bias propagates into every condition-adjusted price shown to users. Getting the calibration right is worth the iteration.

What the evaluation framework allows

Add a new model, run it against the same fixed sample, compare the distribution tables. The structure is reusable. When Mistral's results are in, they will be compared directly against the gpt-5-nano baseline before anything changes in production.

Family enrichment

Combining data and AI to build insight profiles.

Each vehicle family in AutoOracle has a rich intelligence profile built from two sources: internal data gathered from 18 months of listings and the NZTA motor vehicle register, and LLM-generated content reviewed section by section before it is written to the database.

The pipeline for each family runs in a fixed sequence: resolve all taxonomy aliases so that data from MINI, BMW Mini, and Mini Countryman all aggregate together, gather internal stats (listing counts, median price, price trend across three time windows, MVR fleet size, regional distribution, top variants by volume), run a pre-enrichment variant quality audit, build a structured prompt from all of that data, call the language model, then step through the six output sections one at a time: accept as generated, edit in the terminal, or skip. Nothing is written to the database until the review is complete.

The edit log is part of the data model. Every field records whether the generated content was accepted, edited, or skipped. That log is a training signal for future automation of the review process and a running record of prompt quality over time.

The same pipeline then iterates variants. Each active variant for the family gets its own listing statistics, is compared against the family context already written, and receives LLM-generated notes reviewed before writing. At the time each family was first enriched, that meant 18 active variants for the Mini Countryman, 38 for the Delica, and 51 for the Cayman. The taxonomy keeps evolving, so those counts shift as variants are added and consolidated; the families enriched earliest stay the most deeply documented in the catalogue, and each enrichment pass deepens the record further.

01

Alias resolution

All taxonomy aliases collapsed to a single canonical family. Listing and fleet data from every source name variant counts together.

02

Data gather

Internal stats assembled: listing counts, median price, three-window price trend, MVR fleet size, regional distribution, and top variants by volume.

03

Variant quality audit

Variant linkage checked before enrichment begins. Taxonomy issues are fixed first so generated content is grounded in clean data.

04

Prompt and generate

Structured prompt built from all gathered data. LLM called to produce six content sections: history, NZ context, era notes, buyer signals, seller signals, and known issues.

05

Section-by-section review

Each output section reviewed in the terminal: accept, edit, or skip. Nothing writes to the database until the full review pass is complete. Every decision is logged.

06

Write and iterate variants

Approved content written to the database. Pipeline then repeats for each active variant, generating and reviewing notes specific to that variant's listing data.

How it was built

AI from first commit to deployment.

The ML pipeline uses AI. The enrichment pipeline runs through an LLM. And the code, documentation, planning, and deployment work that produced AutoOracle was done with AI assistance throughout. Different models, different tasks.

Planning and documentation

Module specs, decision logs, architecture notes, and this website were planned and written with AI assistance. Maintaining a living technical spec across 18 modules and hundreds of decisions at a one-person pace required it.

Code and infrastructure

Backend endpoints, frontend components, ETL scripts, Alembic migrations, CDK stacks, and GitHub Actions pipelines were built with AI pair programming. Not generated wholesale: reasoned through, reviewed, tested, and iterated.

Data analysis and enrichment

Taxonomy quality audits, deduplication runs, NZTA matching passes, and family enrichment sessions ran as interactive AI sessions. Complex multi-step operations (relinking 700 Delica records, resolving MINI alias chains, auditing Cayman variant linkage across 157 variants) were planned and executed in dialogue.

Model evaluation and iteration

LLM model comparisons, ML training run reviews, value score calibration checks, and confidence distribution analysis were conducted as structured AI-assisted analysis sessions rather than solo investigation.

Want to see the results?

Open AutoOracle