Observability

We measure GrinGo through Google SRE's four golden signals — Latency, Traffic, Errors, Saturation — applied to our specific boundaries. This doc captures the plan: what we measure, what the targets are, what alerts fire and when, and what runbooks respond. It's forward-looking: SLIs and SLOs described here are targets to instrument, not values we've already got dashboards for.

Tooling choice is deferred until we're ready to instrument (see Tooling). The framework — signals, SLIs, SLOs, alert policy — is vendor-agnostic and stands regardless of what we pick.

Why o11y matters specifically for us

Under Model A (custody-model) the entire product proposition rests on end-to-end PIX latency being fast at PoS. Without measurement we can neither defend that promise nor detect when it regresses. Everything else (deposit visibility, withdraw settlement, auth first-launch) is a supporting SLO in service of the same principle: the app must feel like a fintech, not a crypto payment.

Measurement boundaries

Five hops, one load-bearing business journey (PIX). Every signal below is anchored to at least one of these.

Golden signals applied

Signal	GrinGo API (edge)	Infinia calls (upstream)	Neon (data)	Business journeys (end-to-end)
Latency	Request duration per route, p50 / p95 / p99	Per-endpoint call duration	Query duration by kind	PIX quote→settled, deposit visibility, withdraw settlement, auth exchange
Traffic	Requests / sec per route	Calls / sec per Infinia endpoint	Queries / sec per table	PIX payments / hour, deposits / day, active users / day, webhook events / hour
Errors	5xx rate; 4xx as a separate stream (client noise vs real issues)	Non-2xx rate, timeouts, transport errors	Connection failures, query errors	Payout failure rate, webhook signature failures, KYC decline rate
Saturation	Cloudflare CPU-time budget per request, isolate cold-start rate	Timeout rate as a proxy (Infinia's own saturation isn't visible)	Hyperdrive pool utilisation, connection wait time	Webhook processing backlog, retention cron duration, reconciliation drift count

Service-Level Indicators (SLIs)

Concrete metrics we emit, with labels. Vendor-agnostic — every viable backend (Cloudflare Analytics Engine, Axiom, Honeycomb, Grafana, Datadog) supports labelled time-series.

Edge (GrinGo API)

SLI	Metric	Labels
API request duration	api.request.duration_ms	route, method, status_class (2xx / 3xx / 4xx / 5xx)
API request rate	api.request.count	same
API error rate	derived from api.request.count{status_class="5xx"} / total	—
Cloudflare CPU-time used	api.request.cpu_time_ms	route
Isolate cold start	api.request.cold_start (boolean → count)	route

Upstream (Infinia + JWKS)

SLI	Metric	Labels
Infinia call duration	infinia.call.duration_ms	endpoint, outcome (success / client_error / server_error / timeout / transport_error)
Infinia call rate	infinia.call.count	same
JWKS fetch duration	oauth.jwks.duration_ms	provider (apple / google)
JWKS cache hit rate	oauth.jwks.cache_hit	same

Data (Neon via Hyperdrive)

SLI	Metric	Labels
Query duration	db.query.duration_ms	query_kind (bucket queries by shape — e.g. select_user, insert_transaction, update_pix_payment_status), outcome
Query rate	db.query.count	same
Hyperdrive pool wait	db.pool.wait_ms	—

Business journeys (end-to-end)

The load-bearing SLIs. Measured from timestamps in our own ledger, not from external instrumentation.

SLI	How computed	Labels
PIX end-to-end duration	transactions.completed_at − transactions.created_at, for type = 'payment'	outcome (completed / failed), fallback_reversal_triggered
PIX quote latency	api.request.duration_ms{route="POST /pix/quote"}	—
Deposit visibility	For each type = 'deposit' row: movement.created webhook received_at → webhook_events.processed_at	currency
Withdraw settlement	transactions.completed_at − transactions.created_at, for type = 'withdraw'	currency, outcome
Auth session exchange	api.request.duration_ms{route="POST /auth/session"}	provider
KYC completion	kyc_verifications.completed_at − kyc_verifications.created_at	outcome, kyc_mode
Payout success rate	transactions.status = 'completed' / (completed + failed) for type in ('payment', 'withdraw') over a rolling window	type, currency
Reconciliation drift count	Nightly cron output — number of Infinia accounts whose balance disagreed with our computed ledger	—
Webhook processing backlog	Count of webhook_events where received_at more than N seconds ago and processed_at is null	provider

Business / product metrics (adjacent to SRE, still worth capturing)

Not for alerting — for decisions:

Metric	Why
Realised GrinGo spread per PIX	(rate_at_execute - rate_at_quote) - gringoSpreadBps-net revenue
Average deposit size, deposit-to-first-PIX time	Onboarding funnel health
BRL residual accumulated (per Infinia account)	Sweep cron sizing; drift signal
Withdraw rate as % of deposits	Refund / dissatisfaction signal
KYC decline rate	Product quality (bad prompts) vs fraud (bad actors)

Service-Level Objectives (SLOs)

Opening bids. Real SLOs get set with a month of real data. Every SLO has an error budget = 1 − SLO per window; alerts trigger when budget burn rate is projected to exhaust it too quickly.

User-facing (highest priority)

Journey	SLO	Window	Error budget
PIX end-to-end duration	p95 ≤ 45s, p50 ≤ 25s	28-day rolling	5% of PIX payments may exceed p95 target
PIX quote latency	p95 ≤ 500ms	28-day rolling	5% of quote calls
Auth session exchange	p95 ≤ 800ms	28-day rolling	5% of exchanges
Deposit visibility	p95 ≤ 60s from webhook receipt	28-day rolling	5%
Withdraw settlement (FPS)	p95 ≤ 30s	28-day rolling	5%
Withdraw settlement (SEPA)	Same-day	28-day rolling	Business-day dependency; measured against next-business-day

System-level

SLI	SLO	Window
API availability (non-webhook 5xx rate)	≥ 99.9%	28-day rolling → 43 min budget / month
Webhook processing success	≥ 99.5% within 5s of receipt	28-day rolling
Payout success rate (payment + withdraw, excluding user-caused invalid destinations)	≥ 99%	28-day rolling
Reconciliation drift	Zero on any single nightly run	Per-run

Note: PIX end-to-end includes Infinia's payout latency, which is outside our control. The SLO holds us accountable for what we can influence (queueing, our own delays), and holds Infinia accountable via the SLA in our vendor contract. If Infinia consistently misses the composite SLO due to their leg alone, that's a vendor conversation, not an internal fix.

Alerts

Two tiers — page (wake somebody up) and ticket (fix during business hours). Every alert links to a runbook in ops/runbooks.

Page (SEV-1)

Alert	Condition	Runbook
API 5xx spike	5xx rate > 5% over 5-min window	api-5xx-spike
Infinia unreachable	Non-transient error rate > 50% for > 5 min on any Infinia endpoint	infinia-outage
Webhook backlog	> 100 unprocessed webhook_events older than 60s	webhook-backlog
Payout failure spike	Payout failure rate > 5% in a 10-min window	payout-failure-spike
Neon unavailable	Connection failures > 30s continuous	neon-outage
Auth JWKS failure	JWKS fetch failure rate > 20% for > 5 min	jwks-outage

Ticket (SEV-2 / SEV-3)

Alert	Condition	Runbook
SLO fast-burn	Burning error budget at 14× rate over any 1h window (would exhaust in 2 days)	slo-burn-fast
SLO slow-burn	Burning error budget at 6× rate over any 6h window	slo-burn-slow
Reconciliation drift	Nightly cron reports any drift	recon-drift
KYC decline spike	Decline rate ≥ 2× rolling 7-day average	kyc-decline-spike
Retention cron failed	Nightly retention cron reported error	retention-cron-fail
BRL residual above threshold	Any user's BRL account > R$100 outstanding for > 24h	brl-residual-high

Multi-window multi-burn-rate is the SRE-recommended pattern for SLO alerts — it avoids the false positives of simple threshold alerts. See Google SRE workbook, "The Art of SLOs" chapter. Fast-burn and slow-burn together give sensitivity to both quick outages and slow degradation without alert fatigue.

Runbook pattern

Every pageable alert has a one-page runbook. The template (ops/runbooks):

• What fired — the alert condition
• Severity — page or ticket
• First checks — dashboards to open, logs to grep
• Common causes + fixes — enumerated
• Escalation — who to page next; when to open a postmortem
• Postmortem trigger — thresholds that make this incident postmortem-worthy

Runbooks are code — they live in apps/docs/docs/ops/runbooks/ (or the single index page at ops/runbooks.md until we outgrow it), reviewed on PRs, kept current.

Health checks & synthetic monitoring

• Liveness: GET /health returns 200 { status: "ok" } unauthenticated. External monitor (Cloudflare Analytics or third-party pingdom-equivalent) checks it every 60s from multiple regions.
• Readiness: implicit — every real request exercises DB and Infinia. If those are down we surface it as an API error, not a distinct readiness signal.
• Synthetic PIX (Phase 1.5): a scheduled Worker that runs a full onboarding + deposit + PIX + withdraw cycle in Infinia sandbox on a canary user, once an hour. If any step fails, page.

Tracing (deferred to Phase 1.5)

OpenTelemetry-instrumented distributed traces across mobile → API → Infinia → webhook → ledger. Enables answering "why was this specific PIX slow?" instead of "why is p95 slow?"

Backend candidates: Honeycomb, Grafana Tempo, Axiom (all OTel-compatible). Choice made when instrumentation lands.

Tooling (deferred decision)

The o11y framework above is vendor-agnostic. Tooling choice is deferred to the point where we start instrumenting, so we're picking with concrete requirements in hand. The options landscape at time of writing:

Stack	Fit for CF Workers	Cost at MVP	Notes
Cloudflare Analytics Engine + Sentry	Native	Free tier + ~$26/mo Sentry	Analytics Engine handles labelled numerical metrics; Sentry handles errors + stack traces. One new vendor.
Axiom	Purpose-built	Free tier; ~$25/mo per user thereafter	Logs + metrics in one tool. Popular with Workers apps.
Honeycomb	Excellent for debugging	Free tier; expensive at scale	Event-driven, best-in-class high-cardinality queries. OpenTelemetry-first — pairs with a future tracing decision.
Grafana Cloud	Good, more setup	Free tier generous	OSS-friendly, OpenTelemetry-native, we own the dashboards.
Datadog / New Relic	Best-in-class	££££	Enterprise. Overkill for MVP.

Decision criteria (to apply when picking):

• Cost curve — must stay affordable to ~1k active users
• Cloudflare Workers ergonomics — SDK size, cold-start cost, edge-friendly ingestion
• Cardinality — every SLI has at least 2–3 dimensions; some (per-user metrics for debugging) go higher
• Alert routing — must integrate with PagerDuty or equivalent (or offer built-in on-call)
• Migration cost — vendor lock-in should be at most 1–2 weeks to swap

Where this shows up in the roadmap

Phased delivery (see product/roadmap):

• Phase 1 (MVP): Instrument the edge SLIs (API request duration, error rate, traffic). Emit business SLIs (PIX end-to-end, deposit visibility, withdraw settlement) computed on read from the ledger. Wire the six page-worthy alerts + runbooks. Pick tooling.
• Phase 1.5 (operational hardening): SLO burn-rate alerts (multi-window multi-burn-rate). Ticket-tier alerts. Synthetic PIX cron. Reconciliation drift alerting.
• Phase 2: OpenTelemetry-based distributed tracing. Business metrics dashboards (unit economics live). Product-funnel metrics.

Cross-references

• privacy → PII in logs — the PII-scrubbing helper is the same helper that structured metrics/logs go through. No o11y backend receives raw PII.
• ops/runbooks — runbook index and template.
• ops/deployment § pre-production checklist — o11y instrumentation is on the pre-production list.
• payment-providers § circuit breaker — should we add a router with breakers back in the future, its state changes become an o11y source.