GCP Spotlight

Getting accurate cloud cost data is surprisingly hard. I built a three-tier pricing engine that uses the best data available:

• Actual billing data (preferred) — Queries the GCP BigQuery billing export for real invoiced costs, including Committed Use Discount and Sustained Use Discount credits. Cached with a 6-hour TTL.
• Derived per-unit rates — When billing data exists, per-core and per-GB rates are derived from actual cost divided by capacity, reflecting real machine types, regions, and discount programs.
• On-demand fallback — Machine type catalog covering 9 families (e2 through t2d) with regional multipliers, falling back to generic on-demand pricing when machine type is unknown.

The UI transparently labels every cost figure as “Billing” or “Estimated” so users know exactly what they’re looking at. This was critical for trust — FinOps decisions can’t be made against numbers nobody believes.

A flat “use P95 + 20% buffer” doesn’t work when some workloads spike 5× and others are perfectly flat. I classify every workload by its coefficient of variation and apply variable headroom:

This prevents aggressive recommendations on unpredictable workloads while still capturing savings on stable ones. A 70% waste recovery rate further guards against over-optimization — the system recommends capturing most of the savings, not all of it, to leave a practical safety margin.

Each service module (GKE, GCE, GCS) gets its own SQLite database. This was a deliberate architectural choice:

• Eliminates cross-service write contention — parallel collection runs don’t block each other
• Independent schema evolution — Alembic migrations for GKE schema changes don’t touch GCE or GCS
• Simplified backup and restore — a GCS database restore doesn’t require downtime for GKE data
• Isolated failure domains — a corrupt GCE database doesn’t affect GKE metrics

Why not PostgreSQL? The workload is single-writer, read-heavy, and doesn’t require complex cross-service joins. SQLite with WAL mode handles it without the operational overhead of a separate database server.

• Twice-daily schedule — runs automatically on a configurable cron, no manual trigger required
• Parallel collection — all three services collect simultaneously using asyncio.gather
• Concurrency lock — prevents overlapping runs if a collection takes longer than expected
• 1-year retention policy — data older than 365 days is automatically pruned to control storage growth
• Full audit trail — every collection run is logged with duration, resource counts, and error details

The audit trail was essential for trust: when a stakeholder asks “why does this cluster only have 30 days of data?” the answer is in the collection history, not a debugging session.

A dedicated correctness pass after the platform was in daily use. Every item below is a bug that only appears under conditions you don’t hit while developing — a pod dying mid-run, two requests racing, a resource being decommissioned:

• Self-healing collection lock — the single-flight guard was a plain thread lock, which wedges permanently if the holder dies before releasing. Every subsequent collection would return 409 forever. It’s now a flag with a timestamp that auto-reclaims after two hours, longer than any plausible run.
• Orphaned-run reconciliation — collection threads are killed abruptly on shutdown, leaving run records stuck “in progress” and status endpoints reporting a collection that would never finish. On startup, any run still marked in-progress necessarily predates this process, so it’s marked failed with an explicit “interrupted by restart” reason.
• Staleness cutoff on “current” views — latest-snapshot queries now ignore metrics older than 48 hours, so decommissioned VMs and deleted pods stop appearing in the fleet as though they still exist.
• Full natural keys in snapshot joins — grouping by resource name alone collides same-named resources across projects and zones. Queries now key on project + zone + instance, or project + cluster + namespace + pod.
• Atomic cache rebind — the billing caches were refreshed by clearing and repopulating a dictionary, leaving a window where a concurrent reader saw half a cache. Refresh now builds the full result first and swaps the reference under a lock.
• Read endpoints off the event loop — synchronous database work in async handlers was blocking the loop under load. Read routes are now plain synchronous functions that the framework runs in its threadpool; only genuinely awaiting routes stayed async.

None of this is visible in a demo. All of it is the difference between a tool people use once and a tool people rely on.

This was a deliberate departure from the Node.js backend pattern used in other apps in the suite — driven by the problem domain:

• GCP SDK maturity — Google’s Python client libraries (monitoring, container, bigquery, resource-manager) are first-class with rich typing and well-documented APIs
• NumPy for statistical calculations — percentiles, linear regression, and coefficient of variation computed efficiently across thousands of workloads
• FastAPI provides automatic OpenAPI docs, Pydantic validation, and async support out of the box
• SQLAlchemy + Alembic — mature ORM with migration tooling, auto-initializing the schema on first startup

Trade-off accepted: Two languages across the broader suite (Python + Node.js) means no shared backend code — but each app uses the right tool for its specific problem.

• No service account keys — Workload Identity maps the Kubernetes service account directly to a GCP IAM principal, eliminating key rotation and secret storage entirely
• ~600 projects at scale — Terraform applies identical IAM bindings across all monitored projects in a single plan/apply cycle
• Principle of least privilege — read-only roles only (container.viewer, monitoring.viewer, compute.viewer) plus a purpose-built custom role for storage
• Custom role instead of storage.objectViewer — the stock role also grants storage.objects.get, meaning permission to read object contents across every monitored project. A FinOps tool only needs bucket metadata and IAM policy, so I replaced it with a custom role limited to buckets.list/get/getIamPolicy and objects.list — removing a large data-exfiltration surface that nothing in the app was using
• Auditability — all permission grants are version-controlled in the Terraform module, reviewable in PRs
• Reproducible — onboarding a new GCP project is a one-line Terraform variable addition