Architecture
Eighteen months of data collection, taxonomy work, and iterative model development. The story of making three incompatible data sources agree on what a car is.
The starting point
The NZ used car market is documented by three primary sources: two major NZ car-market sites and the NZTA Motor Vehicle Register. Each captures the same vehicles from a different angle, and none of them use the same language to describe them.
One market site enters make and model as free text. The other has a structured taxonomy but applies it inconsistently across listing generations. The NZTA register uses type approval codes and submodel fields derived from compliance documentation. A 2019 BMW 320i, a "BMW 3 Series", and a vehicle registered under a type approval code may all refer to the same car. Without a layer that resolves all three to a common identifier, there is no way to compare prices across sources, train a model on combined data, or know how many of a given vehicle are actually registered in New Zealand.
That resolution layer is the AutoOracle taxonomy.
3
Primary data sources
19,116
Canonical vehicle families
85,198
Variants catalogued
The foundation
The taxonomy is a three-level hierarchy: makes, families, and variants. Every record from every source is matched against it using a year-aware lookup: exact year first, then a two-year tolerance window to account for NZ compliance year vs manufacture year, then a family-level fallback.
Resolving the NZTA data was the hardest part. The register holds 3.86 million vehicle records with type approval codes and submodel strings that have no direct equivalent in listing data. Matching them required a parallel resolution path using NZTA-specific normalisation rules alongside the standard make and family lookup.
The result is a single database where a market record from either NZ car-market site and an NZTA registration record for the same vehicle family all resolve to the same canonical identifiers. That shared foundation is what makes cross-source price comparison, model training on combined data, and fleet-aware market analysis possible.
87.5%
Records resolved to a canonical variant
94.9%
NZTA records resolved to a family
3.86M
Fleet records linked
Data collection
AutoOracle selectively collects public market data from the two major NZ car-market sites on a rolling basis. Each collection run captures tile-level data first (price, make, model), then queues detail pages for vehicles that are new, updated, or have gaps in condition and mileage data. Images are stored separately for LLM-based condition analysis.
Each record passes through a normalisation and deduplication layer before entering the database. Relisted vehicles (the same car appearing under a new listing ID after a price drop) are tracked via registration plate matching and fuzzy deduplication. Price changes are logged, and vehicles that leave the market are marked rather than deleted. The database is designed as a longitudinal record, not a snapshot.
Collect
Tile and detail ingestion from the two major NZ car-market sites on a rolling cadence. Images captured and stored to S3 for downstream LLM processing.
Normalise
String cleaning, source-specific field mapping, mileage banding, and structured field extraction from free-text description fields.
Deduplicate
Plate-based and fuzzy matching to collapse relists and cross-source duplicates. Price history tracked across market sessions.
Taxonomy resolve
Year-aware lookup against canonical families and variants. NZTA records resolved via a parallel normalisation path.
ML pricing
XGBoost models per source, trained on resolved records with complete taxonomy resolution, mileage, and variant identification.
AI enrichment
LLM condition scoring per record. AI-generated family profiles reviewed and published per vehicle family.
Pricing intelligence
With a resolved, cleaned dataset, training a price model becomes tractable. AutoOracle uses XGBoost regression, one model per source, trained on records with complete taxonomy resolution, mileage data, and variant identification. Models are evaluated against a held-out test set and iterated as data quality improves.
Above the price model sits an LLM classification layer. Each listing's description and condition signals are passed to a language model to extract structured condition scores and service history indicators. This is not a single model decision: GPT-5-nano, Amazon Nova Lite via Bedrock, and Mistral Ministral 8B each went through a formal evaluation cycle with quality gates before a production configuration was selected.
A third AI layer generates human-readable vehicle family profiles: history, NZ import context, buyer and seller signals, known issues, and era-by-era variant notes. Each output is reviewed before being published to the Insight screen. The enrichment pipeline is designed to be re-run per section as new data arrives, rather than regenerating everything from scratch.
Delivery
All application code lives in a single GitHub repository. GitHub Actions handles testing, Docker image builds, and deployment to AWS. Pushes to main trigger the pipeline; environment targeting is controlled by branch and tag rules. All infrastructure is defined as code using AWS CDK in TypeScript. No resources are provisioned manually.
Every service runs as a Docker container. ECS Fargate handles orchestration: no server management, and ETL workloads scale to zero between runs. The API, ETL jobs, and ML training environment each have their own container definition and task configuration. Credentials are managed via Secrets Manager and injected at runtime; nothing is baked into images.
Automation gates
ETL jobs run on schedule via ECS scheduled tasks, but data-altering operations (taxonomy fixes, bulk LLM enrichment, NZTA reloads) require a manual trigger. Automation is trusted for reads and aggregations; writes that affect canonical data have a human decision point before execution.
Cost management
Fargate Spot is used where workloads are interruptible. The ML training environment spins up on demand and terminates on completion. RDS runs on a right-sized single-AZ instance with storage autoscaling rather than a multi-AZ configuration at this stage. LLM API usage is batched and provider-switched based on cost-per-token vs quality tradeoffs.
Role-based access control
Four access tiers (Trial, Reporting, Dealer Guide, and Admin) are enforced at the API level on every endpoint. The frontend reflects the same structure: routes, sidebar entries, and feature components are conditionally rendered by role, not hidden after load. Trial accounts get full Oracle, Insight, and Explore access. Dealer Guide adds DGest valuation and export. Admin adds ETL monitoring and data quality tooling.
The stack
Python collectors with two-stage ingestion (tile then detail), image capture to S3, incremental update detection, and rolling deduplication across sources.
PostgreSQL 16 on AWS RDS. Alembic migrations. Records, taxonomy, NZTA fleet, price history, ML estimates, and AI enrichment all in one schema.
XGBoost regression with separate market-source models. Scikit-learn pipeline with k=7 value score calibration. Iterative evaluation against a held-out test set.
Multi-provider LLM layer via OpenAI-compatible API. Condition scoring and family enrichment. Production model selected after formal evaluation across GPT, Amazon Bedrock, and Mistral.
FastAPI, Python, JWT authentication, role-based access control enforced per endpoint, Pydantic response schemas. Deployed on AWS ECS Fargate.
React, TypeScript, Vite, Tailwind CSS. GitHub Actions CI/CD pipeline with Docker image builds. AWS Amplify with CloudFront distribution for global delivery.
All infrastructure as code via AWS CDK (TypeScript). ECS Fargate for containers, RDS PostgreSQL, S3 and CloudFront for static assets and the promo site, Secrets Manager for credential management, and Amplify for frontend hosting.
Want to see the output?
Open AutoOracle