ARIA: Case Study

Where It Started

Why A Chatbot Was The Risk

I had built the dashboards. People still asked me the questions.

2Apps that already held every number a stakeholder could want

1Fabricated figure in an executive deck is enough to discredit every number after it, including the correct ones

3Independent guards it took to make an invented number a path the system does not have

Finding a number meant knowing which of the two apps to open, which tab to click and how to filter it. So the questions kept landing in my inbox: what is a business unit tracking at this fiscal year, which subscriptions are over budget, how much are we spending on AI.

A chatbot was the obvious interface and the obvious risk. A model that confidently produces a plausible dollar figure is worse than no tool at all. The engineering question was never whether it can answer. It was whether it can be built so that making one up is not an available move.

How It Works

One Question, Followed

A representative question, “which subscriptions are over budget this fiscal year?”, from keystroke to audit row.

The Question Arrives

The request passes an input check first: messages are capped at 2,000 characters and each person is limited to 20 questions per 60 seconds, keyed to their SSO identity rather than an IP address. The question needs no date, because the agent is fiscal-year aware by default (July through June).
The Model Picks Its Tools

Claude Sonnet, called through Vertex AI, reads a roughly 280-line system prompt that is cached between requests. Extended thinking (a 10,000 token budget) makes it reason about which tool fits before it requests one. For this question, that is the over-budget tool.
The Tools Run Together

Every tool the model requests in a round executes concurrently, which is safe because each one is read-only and independent. Business unit names are resolved to IDs through a cached lookup, and pagination runs up to 20 pages of 500 records without the model seeing it. The UI shows a live pill such as “Fetching: get_budget_variance,” so the wait looks like work. If one lookup is needed to ask the next, the loop runs again, up to 10 rounds.
It Reads Totals, Not Rows

Each financial tool returns a summary object computed on the server: actual total, budget total, forecast, delta, and the COGS split. The prompt tells the model to read summary.actual_total and never to add up rows itself.
The Answer Gets Checked

Before anything reaches the user, the agent scans the response for currency patterns. If it contains dollar figures and every tool failed (or none ran), the response is discarded and replaced with the actual tool errors, not a vague apology.
Depth Follows The Words

A single-metric question gets one sentence and one drill-down link into the source app at the exact filtered view. Three follow-up suggestions appear, each one level deeper. A comparison would get a table instead, and only the words “monthly” or “breakdown” would unlock the full report.
The Turn Is Logged

The audit log records the user identity, question, response, tools invoked, token counts (including cache reads) and response time. Nothing about the exchange depends on anyone remembering it.

Anti-Fabrication

The Three-Layer Guard

A prompt instruction is a request, not a control. If the FinOps API is down and the model still produces a plausible figure, no “please don’t hallucinate” saves you.

Layer	Where It Lives	What It Does	What It Cannot Do Alone
Prompt	System prompt	Eight data-integrity rules that override every other instruction: every number traces to a tool result, empty results are stated as empty, uncertainty is disclosed	Enforce anything; the model can still ignore it
Code	Agent loop, after the final response	Scans for dollar figures and discards the answer if every tool errored or none ran	Judge a figure when a tool did succeed
Tool	Every failure payload	Returns an `instruction` field: “STOP. Tell the user this data source is unavailable. Do NOT fabricate numbers.” The model reads the guardrail inside its own context	Act when no tool fails

Arithmetic closes the gap between the layers. Early versions returned raw records and the model summed them. It was right most of the time, which is the worst failure mode, because “mostly right” looks identical to “right” until someone checks. Aggregation now happens in Python, where it is deterministic and testable.
Grading is code too. Scorecard weights, budget-adherence bands and coefficient-of-variation stability scores are calculated server-side, never reasoned about in prose.
The detector exempts greetings and off-topic refusals, so a canned reply that mentions a dollar amount does not trip it. The accepted trade: a legitimate answer could in principle be blocked. A false “I can’t reach the data” costs a retry; a false number costs the project.

The Tool Layer

Thirty Tools, All Read-Only

Every tool is a GET. ARIA cannot change a budget, a resource or a record.

Group	Tools	Covers	Reads From
Financial	12	Spend, budget variance, scorecards, vendors, COGS, over-budget, AI and DR spend	FinOps Reporting App
Infrastructure	18	GKE workloads and clusters, GCE instances and recommendations, GCS buckets, AI model spend	GCP Spotlight

Tool schemas are a UX surface. Vague parameter descriptions produce confidently wrong tool calls, so the descriptions got the same care as the prompt.
Two tools are legitimately named “AI spend”: one infrastructure-level, one attributed to a business unit. They needed explicit disambiguation rules to stop the agent picking the wrong one.
Every failure is structured, and carries an explicit instruction not to guess, which is the tool layer of the guard.
Concurrency has one precondition. Running four tools at once is safe only because none of them writes; there is no ordering constraint to violate.

Response Depth

Tiered Answers

The first version answered everything with a wall of analysis. Executives stopped reading at line two.

Tier	Triggered By	Response Shape
Tier 1 (default)	A single-metric lookup	One sentence and one drill-down link
Tier 2	“compare,” “which,” or three or more items	Summary sentence and a table (5 columns, 10 rows at most)
Tier 3	“monthly,” “trend,” “breakdown”	Full breakdown with analyst commentary

The tier is set by the user’s words, not by how much data came back.
A tool that returns 40 rows for a one-entity question means the agent aggregates to a single total and stays at Tier 1. The volume of available data is not a reason to spend the reader’s attention.

Cost And Latency

Spending Where It Pays

Two model features that push the bill in opposite directions, paired on purpose.

10,000Token budget for extended thinking, spent on the part that varies

~280Lines of system prompt, cached between requests because it never changes

10Tool rounds at most: a confused agent burns budget, not the afternoon

Extended thinking cut a class of error. It measurably reduced the “grabbed the wrong field from the JSON” mistake. It also forces temperature to 1 and needs an output ceiling that covers both thinking and the answer; miss that and responses truncate silently. The budget is configurable, so thinking can be turned off without a code change.
Caching makes the savings measurable. Cache reads are logged per request next to token counts, so the effect is visible in the audit data rather than assumed.
Streaming fixes perceived latency for free. Tool calls reach the UI as they happen, so a 20-second answer shows its work instead of a spinner.
A 25-second heartbeat exists because production taught it. The service mesh proxy closes idle connections, so a background task emits a keepalive during long multi-round answers. Testing never showed this.
Upstream calls use a pooled async client with two retries on timeout or connection error and none on HTTP errors, because a 404 will still be a 404.

Trust And Identity

Zero Stored Keys

ARIA calls Claude through Vertex AI instead of the direct API. That one choice removed a whole category of operational risk.

Control	Setting
Model Access	Workload Identity: the pod authenticates as a GCP service account, so no key exists to rotate, leak or commit
Permissions	A custom IAM role granting only `aiplatform.endpoints.predict`; the agent can call the model and nothing else in the project
Data Residency	Inference runs inside the organization’s own GCP project and region, which made it approvable for corporate financial data
Local Development	The same code path works with application default credentials, so dev and prod share one identity model
Data Access	Read-only by construction; upstream calls use cluster-internal addresses and public access sits behind SSO on the corporate network
Attribution	Every question is recorded against the SSO identity that asked it, with the tools used to answer

The trade accepted: Vertex model identifiers lag the direct API slightly, and newer features arrive later. For financial data, removing credentials was worth more than being first to a model release.

Architecture

A Thin Agent Over Two Platforms

ARIA owns no cost data. It owns the reasoning, the guardrails and the audit trail.

Interfaces

React chat UI (full page and embedded widget in both source apps) and a Python CLI with interactive and one-shot modes.

FastAPI Backend

Server-sent events for streaming, a 25-second heartbeat, and routes for chat, audit, feedback, suggestions and export. SQLite in WAL mode.

Agent Loop

Claude Sonnet on Vertex AI with extended thinking, a cached system prompt and up to 10 tool rounds.

Two Platforms

FinOps Reporting App (12 tools) and GCP Spotlight (18 tools), over the Ternary API, GCP APIs and BigQuery billing export.

ARIA can never contradict the dashboards it links to.
The upstream apps already handle collection, alias mapping, COGS classification and cost calculation. Querying their databases directly would have duplicated that logic in a third place and guaranteed the copies would eventually disagree, so ARIA goes through their APIs.

SQLite over PostgreSQL. WAL mode gives concurrent reads during writes, and a 30-second busy timeout absorbs contention. The honest cost is one replica; if the app ever needs to scale out, the database is the migration.
No login of its own. The platform already enforces Okta SSO at the edge. Conversation IDs live in browser local storage so each person sees their own history, and identity from the proxy headers feeds the audit log only.
Both choices are reversible and documented as such. Choosing the smaller thing is defensible only when you know exactly what would force you to change it.

Python 3.11 FastAPI httpx React 18 TypeScript Vite Tailwind CSS Click Rich SQLite Vertex AI Docker Kubernetes ArgoCD

Governance

After The Answer

Everything asked is logged, exportable and fed back into what the tool suggests next.

The audit log is complete and exportable. User, question, response, tools, tokens and latency, with CSV and JSON export for compliance review and a usage dashboard of queries per day, unique users and top users.
Feedback has a queue. Thumbs up or down roll into a satisfaction rate, and recent negatives land in a review list.
Questions become templates. Normalization turns raw questions into reusable patterns with dropdowns for the variable parts, and the popular ones return to the sidebar as one-click suggestions.
Readiness is verified, not assumed. A deep health check confirms both upstream apps and model access before the service reports ready.
Refusals are designed. Off-topic questions get a rotating reply from a pool of eleven (movie quotes, Victorian demurrals), cycled so nobody sees a repeat until they have seen them all. Adjacent questions get a redirect that says what ARIA can answer. A blunt refusal reads as broken; a wink reads as designed, and it made people comfortable testing the edges during rollout.
Conversations expire. History can be deleted by the user, with an automatic 180-day cleanup; the audit log is retained separately.

Retrospective

What Each Choice Cost

A decision log: what I chose, and the price I accepted for it.

Decision	Chosen	Price Accepted
Guard failure mode	Fail closed on any dollar figure with no successful tool behind it	A legitimate answer can be blocked; the user retries
Model hosting	Vertex AI with Workload Identity	Model identifiers lag the direct API
Database	SQLite in WAL mode	One replica; the database changes first at scale
Login	None; SSO at the platform edge	Identity comes from proxy headers and feeds the audit log only
Scope	Decline anything off-topic	Too strict refuses real FinOps questions; too loose lets scope creep back
Concurrency	All tools in a round run at once	Valid only while every tool stays read-only

What I Would Change

Build the evaluation suite on day one. Prompt engineering has no compiler: a rule added to fix one behavior can quietly change three others, and only usage reveals it. Question and expected-answer pairs would make prompt changes regression-testable instead of eyeballed.
Version the system prompt alongside the audit log, so a past answer can be explained by the rules in force when it was given.
Capture full tool inputs and outputs per message from the start. The schema has the columns, but early answers cannot be fully replayed.
Link negative feedback straight to its audit entry, turning a thumbs-down into a one-click investigation.
Plan for multiple replicas earlier, since SQLite is the one thing that would change first.

With financial data, an assistant’s credibility is set by its worst answer, not its average one.
Most of the engineering went into the paths where something has gone wrong: a source is down, a tool returns nothing, a question falls outside scope. Getting those right is what let people rely on the answers when everything worked. The interesting work was never the conversation. It was the constraints around it.

ARIA is the conversational layer over two platforms I built; both are case studies of their own:

FinOps Reporting Platform GCP Spotlight

Back to Projects

ARIA

Why A Chatbot Was The Risk

One Question, Followed

The Question Arrives

The Model Picks Its Tools

The Tools Run Together

It Reads Totals, Not Rows

The Answer Gets Checked

Depth Follows The Words

The Turn Is Logged