prtotype.com / builds / token-spend-toolkit tech review · 2026-08-26
Build showcase · Token-Spend Toolkit

Every token,
accounted for.

A full pipeline for seeing, forecasting and acting on LLM token spend across Claude and ChatGPT. Seven tools over one normalised data layer, built in Python on DuckDB and Parquet, proven by a 204-test suite and signed off under a governed delivery plan.

204tests passing
7tools, one pipeline
2providers normalised
$0.000001max cost drift
100%deterministic output
01

Why it matters

Every team running workloads on Anthropic or OpenAI models shares the same blind spot: the bill arrives after the fact, aggregated into a console nobody watches. The toolkit exists to close that gap before invoice time.

There is no built-in way to answer the questions that actually matter about LLM spend. Usage data sits in two provider consoles in two shapes, with no unified view, no forecast, no alerting and no link from a number to an action. Finance finds out at month end; engineering finds out never.

  • What did each client cost this month, and where is it heading?
  • Which model family is eating the budget?
  • Did spend spike yesterday, or creep for three weeks?
  • What should we do, and what is each move worth in dollars?

The purpose is not another dashboard. It is financial control of LLM usage: catch an over-pace client mid-month, catch a runaway loop in hours, and hand a decision-maker a briefing that already contains the numbers and the recommended move.

02

At a glance

FactValue
Tools delivered7 core tools plus supporting infrastructure (milestones M0-M7, remediation backlog R1-R8)
Providers normalisedAnthropic (Claude) and OpenAI (ChatGPT) into one canonical schema
Ingest sources6: two deterministic mocks, two key-free file parsers, two live API adapters
Test suite204 passing; 2 live-API tests skip cleanly without admin keys
Reconciliation accuracyDerived vs provider-reported cost agrees within $0.000001
DeterminismSame store always renders byte-identical reports and dashboards
GovernanceEvery milestone independently board-signed; v1 signed off 2026-06-14
ScheduleWeekly report automated, Mondays 08:03
03

Architecture

One data layer, many thin readers. Tools never talk to each other; every tool reads the canonical store through a hardened SQL interface.

Sourceswhere usage comes from
deterministic mocks (seeded) console-exported JSON files live admin APIs
Ingest enginesrc/ingest.py
provider adapters Pydantic validation collision guard + reject ledger latest-wins merge
Canonical storethe single interface
Parquet + DuckDB (embedded) rate_card.json · effective-dated prices
Toolspure functions of the store
forecast + budgets spend explorer anomaly detector back-test replay
Outputswhat people receive
Markdown report HTML Command Centre ranked actions SMTP delivery

Money flows one way: raw usage in at the top, ranked dollar-costed actions out at the bottom. Every figure between carries its period, its unit and its provenance.

04

The tools, reviewed

01

Ingest engine & adapters

Get real usage data from anywhere into one clean shape without losing or duplicating a row.

Six interchangeable sources cover every operating mode: seeded mocks for reproducible demos, key-free parsers for console-exported JSON so operators without admin credentials can feed it real spend today, and paginated live API adapters for hands-free pulls.

Why it matters
  • Every row validated against a 17-column canonical schema before touching the store
  • Bad rows land in a reject ledger with reasons; strict mode raises one diagnostic listing all of them
  • Re-ingesting restates rather than duplicates, so scheduled re-pulls can never double-count
  • Duplicate IDs with differing token counts fail immediately instead of vanishing silently
02

Canonical schema & rate card

One definition of a usage record and one source of prices, shared by everything.

The schema keeps billed and computed cost side by side: what the provider charged, and what the toolkit computed from published prices. The rate card holds effective-dated pricing for both providers plus the USD-to-GBP reference, so no currency rate is ever hardcoded.

Why it matters
  • Two independent cost fields turn every ingest into an ongoing audit
  • A price change overnight, or a parsing bug, exposes itself as drift between the fields
  • Unknown models raise instead of costing $0, killing the classic silent budget leak
  • Cross-field validators enforce timestamp ordering and flag source-versus-derived mismatch over $1
03

Forecast & budget tracker

Know mid-month whether each client will finish over budget.

Per client: month-to-date spend, projected month-end, budget consumed, projected percentage and an over-pace flag. Money renders as $1,202.50 (£949.98), the GBP reference sourced from the rate card, never a literal.

Why it matters
  • Projects over active days, not calendar days: a client starting on the 20th is judged fairly
  • The projected overage figure is carried verbatim into the actions layer, one chain of custody
  • No downstream recomputation that could drift from the source number
04

Back-test replay

Prove the forecast is actually any good.

Replays the projection at day 7, 14, 21 and 28 using only data available up to each checkpoint, then scores every projection against the realised month-end actual.

Why it matters
  • Anyone can ship a straight-line forecast; this measures how its error shrinks through the month
  • Zero data leakage by construction, so the score is honest
  • Part of a wider habit: claims are measured, not asserted
05

Spend explorer & Command Centre

One self-contained HTML page answering "where is the money going?" at a glance.

KPIs, daily trend, breakdowns by client, model and key, period deltas, efficiency views and a budget panel render into a single file. No server, no build step: data embedded as JSON, charts from a pinned CDN, opens in any browser.

Why it matters
  • Chart.js pinned with a Subresource Integrity hash, so a compromised CDN cannot inject scripts
  • Secrets reported as presence-only booleans; values structurally cannot reach the page
  • Colour-blind-safe palette with labels too, so meaning never rides on colour alone
  • Every figure carries its period and unit; indicative prices are stated plainly
06

Anomaly detector

Surface genuine spend spikes without crying wolf.

Scans daily cost series per scope: total, project, model family or API key. An alert fires only when a day clears both gates: at least 3 standard deviations above baseline AND at least $50 above it.

Why it matters
  • Leave-one-out baseline: the day under test is excluded from its own statistics, so a spike cannot mask itself
  • Dual conditions kill the false-positive machine where tiny-variance series flag $3 Tuesdays
  • Survivors rank by magnitude with baseline, deviation and z-score attached
07

Automated insight report

Turn the whole data layer into a briefing read in ninety seconds.

A TL;DR-first Markdown report weaving provenance, headline KPIs, drivers, anomalies, forecast versus budget and costed recommendations from a pluggable advisor registry. It classifies its own sources in plain English so demo data can never masquerade as real spend.

Why it matters
  • Pure and deterministic: same store renders byte-identical output, no wall clock anywhere
  • That property lets 204 tests pin exact figures and catch regressions to the cent
  • It is what makes the Monday 08:03 automated delivery trustworthy
08

Actions layer

Close the loop from signal to recommended action.

A stateless deriver converts report signals into ranked actions, each carrying estimated dollar impact, confidence, caveats and a declarative done-check describing how a human would verify completion. When nothing clears the impact floor it says so explicitly.

Why it matters
  • Most analytics stop at a chart; this stops at what to do, what it is worth, how to check it worked
  • Impact figures copied verbatim from their source signal, never recomputed
  • Deliberately advisory: execution needs a state store, deferred honestly rather than faked
05

Engineering principles worth stealing

These rules are enforced in code and covered by tests, not aspirational wall art.

PrincipleHow it shows up
Evidence or it didn't happenEvery milestone claim backed by executed tests; status tables cite evidence
Independent reconciliationMock data emits a provider cost computed independently of the derived path, so cost tests cannot pass circularly
Fail loudUnknown model, price-period gap, ID collision, drifted costs, corrupt file: all raise with diagnostics instead of degrading quietly
DeterminismNo wall clock in analysis; identical inputs give byte-identical outputs
Hardened SQLAllow-listed scopes and escaped identifiers; no interpolated user input near the query engine
Log hygieneExternally derived text sanitised before logging (CR/LF stripped, length capped), defeating log forging
Single source of truthOne schema module owns columns, records and cost math; one formatter owns money strings

An independent audit surfaced eight defects: circular reconciliation, silent record-ID collisions, stale rows surviving re-ingest, unknown models costing $0, crashes on empty windows, shallow validation, unformatted money and interpolated SQL. All eight were fixed, re-gated and covered by new tests before sign-off. Shipping is not the story here. Shipping after adversarial review, then fixing everything it found, is.

06

Proof in numbers

The canonical fixture pins the whole suite to exact figures. New tests must pin to these, so a regression anywhere breaks loudly.

Claude mock · seed 42 · May 2026
$1,202.50
186 rows
190,387,278 input tokens
47,875,719 output tokens
(£949.98 reference)
Model split
$665.84
Opus lead
Sonnet $400.47
Haiku $136.19
OpenAI mock · seed 99
$768.58
second provider
same store shape
reconciles clean
Mixed store
$1,971.08
Claude + OpenAI
one canonical schema
drift ≤ $0.000001
07

Honest limits

A tool that hides its limits is a liability. This one publishes them:

  1. Live API adapters are implemented and wired but unverified against real endpoints while admin keys are pending; the key-free file path covers real data today.
  2. The OpenAI parse path returns no billed amount, so its reconciliation runs derived-only and says so.
  3. Deeper advisors such as right-sizing, caching gains and unit economics need per-request prompt logs, and are deferred rather than promised.
  4. Live fetches lack timeout and retry/backoff, planned with a written design.

Knowing what a system does not do is part of knowing what it does.

08

Verdict

"Full-stack data engineering judgement: a normalised multi-provider data layer, seven tools that each do one job well, security treated as a requirement, and every number traceable to evidence."

The Token-Spend Toolkit solves a real, expensive problem every team buying LLM capacity shares, and does it with rigour usually reserved for regulated financial software. The build is complete, tested, board-signed and running on a schedule. That is the work this page showcases.