Observability for Serverless Without the Bill Shock

How to actually see what your serverless app is doing without CloudWatch quietly becoming your biggest line item.

Hey, it’s Lefteris 👋 I’m the voice behind the weekly newsletter “The Cloud Engineers.”

Here’s a story every serverless team eventually lives through. You ship a Lambda-based system, it works beautifully, and you feel great. Then the monthly bill arrives and there’s a line item that makes you squint: CloudWatch. Sometimes it’s bigger than the Lambda compute it’s supposed to be observing.

Observability is non-negotiable. You cannot run a distributed serverless system blind. But in serverless, the very thing that gives you visibility is metered, and the defaults are quietly expensive. The goal isn’t to log less and fly blind. It’s to be deliberate: capture what you need, drop what you don’t, and design your observability the same way you design everything else, with cost as a first-class constraint.

Let’s break down the three pillars — logs, metrics, and traces — where each one silently runs up the bill, and how to get production-grade visibility without the shock.

Why Serverless Observability Is Different

In a traditional server, you SSH in, tail a log, and watch. In serverless, there’s nothing to SSH into. Your system is dozens of ephemeral functions, queues, and managed services, each emitting its own telemetry. Visibility has to be designed in, not bolted on.

And every signal has a price:

• Logs are billed for ingestion and storage and Lambda makes it trivially easy to emit enormous volumes.

• Metrics are cheap by default but get expensive the moment you reach for high-cardinality custom metrics.

• Traces are sampled for a reason trace everything at scale and the cost climbs fast.

The three pillars are your toolkit. Knowing when to reach for each, and when not to, is the whole game.

Pillar 1: Logs — Where the Bill Usually Hides

Logs are the number one source of surprise CloudWatch bills. The culprit is almost always volume: verbose logging left on in production, giant payloads dumped into logs, and infinite retention.

Three habits fix most of it:

1. Log structured JSON, not strings. Structured logs are queryable, filterable, and far more useful per byte. One rich JSON line beats ten print statements.

2. Log at the right level. DEBUG is for development. In production, log INFO for the meaningful events and ERROR for failures and make the level configurable per environment so you’re not redeploying to change it.

3. Set a retention policy. Always. By default, CloudWatch Logs keep your data forever and you pay for that storage forever. Almost no one needs 90-day-old function logs. Set retention to something sane (7, 14, 30 days) on every log group.

A structured log line looks like this — one line, fully queryable in CloudWatch Logs Insights:

{
  "level": "INFO",
  "event": "cv_enhanced",
  "requestId": "abc-123",
  "durationMs": 842,
  "inputTokens": 310,
  "outputTokens": 190
}

The single highest-leverage change most teams can make: turn off debug logging in production and set log retention. That one afternoon of work often cuts the CloudWatch bill more than any other single action.

Pillar 2: Metrics — Cheap by Default, Expensive When Careless

Metrics answer “how is the system behaving overall?” such as invocation counts, error rates, duration, throttles. The good news: Lambda emits the essential ones for free, and they’re your first line of defense.

Where cost sneaks in is custom metrics, and specifically cardinality. Every unique combination of dimensions on a custom metric is billed separately. Add a customerId dimension to a metric and you don’t have one metric but one per customer. That’s how a “small” custom metric becomes a four-figure line item.

Keep metrics affordable:

1. Lean on the free built-ins first. Errors, duration, throttles, and concurrency are already there. Build your core dashboards and alarms on these before adding anything custom.

2. Guard your cardinality. Never put unbounded values (user IDs, request IDs, email addresses) into metric dimensions. Those belong in logs, not metrics.

3. Emit metrics from logs where you can. Deriving metrics from structured log fields avoids a separate metric-publishing cost and keeps your telemetry in one place.

Metrics tell you that something is wrong. For why, you need the third pillar.

Pillar 3: Traces — Powerful, and Worth Sampling

A trace follows a single request across every hop: API Gateway → Lambda → Bedrock → DynamoDB and shows you exactly where the time (or the failure) went. In an async, event-driven system, this is the difference between “it’s slow somewhere” and “it’s this DynamoDB call, right here.”

Tracing tools (like AWS X-Ray) are indispensable for debugging distributed flows. But tracing every single request at high volume gets expensive and rarely adds insight.

Use tracing wisely:

1. Sample, don’t capture everything. A small percentage of requests is usually enough to understand behavior and catch anomalies. Reserve full capture for when you’re actively hunting a problem.

2. Propagate a correlation ID. Pass a single request ID through every hop, including across queues and events, and log it everywhere. This is the cheapest, highest-value tracing habit, and it works even where full tracing doesn’t reach.

3. Trace the boundaries that matter. Focus tracing on the hops most likely to hurt: external API calls, model invocations, and database access. Those are where latency and failures actually live.

A Practical, Cost-Aware Setup

If you want a default posture that gives strong visibility on a small budget, start here:

1. Structured JSON logging at INFO in production, DEBUG only in dev.

2. Log retention set explicitly on every log group (7–30 days).

3. Dashboards and alarms built on the free Lambda metrics — errors, duration (p95, not just average), and throttles.

4. Custom metrics only for a handful of true business KPIs, with strict low cardinality.

5. Tracing enabled with sampling, plus a correlation ID threaded through every function and event.

6. A billing alarm on CloudWatch spend itself so your observability can never surprise you again.

That setup catches real problems fast and keeps the observability bill proportional to the value it delivers.

Conclusion

Observability isn’t optional in serverless. You’re operating a distributed system, and you need to see it. But “see everything, forever, at full fidelity” is a choice that quietly wrecks your bill without making you meaningfully wiser. The fundamentals hold here just like everywhere else: capture what matters, drop what doesn’t, and treat cost as a design constraint from the start.