GCP Spotlight: Case Study

Where It Started

Waste With Four Owners

The data existed in GCP’s APIs. Nobody had connected it.

~600Projects, each a place for over-provisioned pods, idle VMs and unread buckets to hide

4Service types, each with its own console, its own mental model and its own responsible team

0Places to answer where the waste is across the organization, and what to fix first

Foundation

The Cost Model Comes First

Every recommendation is a dollar claim, so every module reads from the same three-tier pricing engine.

1. Billing Export

Real invoiced cost from the BigQuery billing export, including Committed Use and Sustained Use credits. Cached for 6 hours. Labeled “Billing.”

2. Derived Rates

Where billing data exists, per-core and per-GB rates are actual cost divided by capacity, reflecting real machine types, regions and discounts. Labeled “Estimated.”

3. On-Demand Fallback

A machine-type catalog of 9 families (e2 through t2d) with regional multipliers, and generic on-demand pricing when the type is unknown. Labeled “Estimated.”

Every figure is labeled “Billing” or “Estimated.” FinOps decisions cannot be made against numbers nobody believes, so the UI never lets an estimate pass as an invoice.
The fallback chain is the cost of partial data. Billing export availability varies by project, and handling that without silent failures was the largest source of complexity in the cost engine.

What I Built

One Module At A Time

Each resource type has its own signals and its own rules, so each gets its own module.

GKE: Kubernetes

59 Clusters, every pod measured

Stability Class	Variability (CV)	Headroom Applied
Stable	Under 25%	25% above P95
Variable	25 to 50%	35% above P95
Spiky	Over 50%	45% above P95

A flat “P95 plus 20%” fails when some workloads spike 5× and others are perfectly flat, so each workload is classified by its coefficient of variation.
A 70% waste recovery rate recommends capturing most of the savings, not all of it, to leave a practical safety margin.
Recommendations end in a patch. Bulk YAML generation with multi-select and preview bridges “identify waste” and “fix it,” and a fleet efficiency score carries a 7-day trend forecast.
Label compliance is tracked because untagged workloads cannot be allocated to anyone.

GCE: Compute Engine

2,222 Virtual machines

Signal	Rule	Recommendation
Idle	Below 5% CPU utilization	Flag the instance
Oversized	Usage fits a smaller machine type	Downsize
Interruptible	Spot candidate score	Convert to Spot
Business hours only	Usage confined to working hours	Auto-schedule
Steady load	CUD coverage gap	1-year and 3-year savings projections

GCS: Cloud Storage

2,989 Buckets

Signal	Rule	Output
Abandoned	No reads in 90 or more days	Candidate for cleanup
Wrong class	Access pattern does not match storage class	Downloadable lifecycle policy JSON
Exposure	Public access enabled	Risk flag
Versioning	Old versions accumulating	Waste tracked
Egress	Outbound transfer cost	Trend monitoring

AI: Vertex AI

2× The rolling average that flags an anomalous day

Signal	Rule	Output
Model spend	BigQuery billing export, by model	Spend by model
Cheaper option	Cost per token across models	Alternative recommendation
Anomaly	A day above 2× the rolling average	Flagged day
Attribution	Spend grouped by project	Chargeback view

Putting It Together

Hub, Project Pages And ARIA

Four modules answer four questions. Three layers on top answer the one that matters.

The hub dashboard is a four-quadrant executive summary of cost, waste and savings across all four services.
Cross-service project pages combine every metric for one GCP project, which answers “what are we spending on this project, and where is the waste?” in seconds.
A conversational layer sits on top. ARIA, the FinOps AI agent I built over this platform and the FinOps Reporting App, reads from Spotlight with 18 of its 30 tools. The app hosts a full-page assistant and a navbar chat widget, backed by a reverse-proxy route that streams responses from the agent service. Users who do not want to learn the filters can ask.
Everything is inspectable. CSV export from every table, methodology tooltips on every calculated metric, global filter bars with cascading dropdowns, and a collection history per service.

Architecture

Collect In Parallel, Store Apart

Three collectors run side by side into three separate databases.

GCP APIs

Cloud Monitoring, Container, Compute Engine, Cloud Storage and the BigQuery billing export.

Three Collectors

GKE, GCE and GCS run twice daily on a configurable schedule, in parallel with asyncio.gather, behind a concurrency lock.

Three SQLite Files

One database per service in WAL mode, with 1-year retention that prunes data older than 365 days.

API And UI

FastAPI with 123 endpoints feeding a React app of 13 pages and 61 components.

Isolation removes four problems at once: parallel runs do not contend for writes, schema migrations touch one service, a restore does not need downtime for the others, and a corrupt database stays inside its own failure domain.
PostgreSQL was the wrong tool. The workload is single-writer, read-heavy and has no cross-service joins, so SQLite with WAL avoids running a database server for its own sake.
The collection audit trail earns its keep. When a stakeholder asks why a cluster has only 30 days of data, the answer is in the collection history, not a debugging session. Every run logs its duration, resource counts and errors.
Python and FastAPI were chosen for the problem. Google’s Python client libraries are first-class, NumPy handles percentiles, regression and CV across thousands of workloads, and SQLAlchemy with Alembic auto-initializes the schema. The price: no shared backend code with the Node.js apps elsewhere in the suite.

Hardening

Correctness Under Failure

A dedicated pass after daily use found bugs that only appear under conditions you do not hit while developing.

Condition	What Went Wrong	What I Changed
A pod dies mid-run	The single-flight guard was a plain thread lock that wedges forever if its holder dies; every later collection returned 409	A flag with a timestamp that auto-reclaims after two hours, longer than any plausible run
A restart interrupts a run	Threads killed on shutdown left run records stuck “in progress,” so status endpoints reported a collection that would never finish	On startup, any run still in progress is marked failed with an “interrupted by restart” reason
A resource is decommissioned	Latest-snapshot views kept showing deleted pods and VMs as though they still existed	A 48-hour staleness cutoff on every “current” view
Same names in two places	Grouping by resource name alone collided same-named resources across projects and zones	Full natural keys: project, zone and instance, or project, cluster, namespace and pod
Two requests race	A billing cache refresh cleared and refilled a dictionary, so a concurrent reader saw half a cache	Build the full result first, then swap the reference under a lock
Load arrives	Synchronous database work inside async handlers blocked the event loop	Read routes are plain synchronous functions run in the threadpool; only truly awaiting routes stayed async

None of this is visible in a demo.
All of it is the difference between a tool people use once and a tool people rely on.

Trust And Access

Read-Only Across 600 Projects

Terraform applies identical IAM bindings to every monitored project in a single plan and apply.

Control	Setting
Permissions	Viewer roles only (container, monitoring, compute) plus a custom storage role; the app cannot modify any cloud resource
Storage Role	Custom, limited to `buckets.list/get/getIamPolicy` and `objects.list`, replacing `storage.objectViewer`
Credentials	Workload Identity maps the Kubernetes service account to a GCP IAM principal; no keys are stored or written to disk
Runtime	Non-root container (UID 1001) on a minimal base image
Network	No public internet exposure; VPN-only on the internal corporate network
Traceability	Structured JSON logs with a correlation ID on every request
Auditability	Every permission grant is version-controlled in the Terraform module and reviewed in pull requests
Onboarding	A new GCP project is a one-line Terraform variable addition

The stock storage role was too generous. storage.objectViewer also grants storage.objects.get, which is permission to read object contents in every monitored project. A FinOps tool needs only bucket metadata and IAM policy, so a custom role removed a large exfiltration surface that nothing in the app used.

Retrospective

Harder Than The Code

The hardest problems were data problems, not technical ones.

Quotas shaped the collector. Cloud Monitoring API limits meant batching queries carefully across 59 clusters to avoid rate limiting.
Billing and metrics never line up. Reconciling the billing export with API-reported metrics meant living with lag between a resource changing and monitoring data catching up.
One cost model, many discount shapes. It had to handle CUD credits, sustained use discounts and regional pricing across 9 machine families.
Aggressive versus safe. The recommendation engine had to work across wildly different workload types without recommending something that breaks production.

What I Would Change

Detect Autopilot clusters from day one to skip right-sizing where GKE already optimizes automatically.
Build namespace exclusion filters first (kube-system, monitoring); retrofitting them was tedious.
Track whether a recommendation was applied, and measure the real savings that followed.
Revisit PostgreSQL early if multi-user writes or cross-service queries become requirements.

React 18 TypeScript Vite TanStack Query Recharts Tailwind CSS Python 3.11 FastAPI SQLAlchemy Alembic SQLite BigQuery Terraform Docker Kubernetes ArgoCD

The most valuable FinOps tool is the one that makes the invisible visible.
Most of the waste this platform identifies was technically knowable before. It just was not surfaced in a way that made action easy.

Back to Projects