Two kinds of clients call us about observability. The first has nothing: kubectl logs and prayer, and incidents that start with twenty minutes of grepping. The second has a Datadog bill that has quietly crept toward the cost of an engineer, and a CFO asking questions. Both end up in the same conversation, and OpenTelemetry is the center of it.

This is the primer I wish I could send in advance of that call.

Three signals, three jobs

The "three pillars" framing is overused, but the division of labor is real:

  • Metrics are cheap aggregates over time. They answer "is something wrong?": error rate up, latency P99 climbing, queue depth growing. They're what you alert on, because they're cheap enough to keep at high resolution for months.
  • Logs are discrete events with context. They answer "what exactly happened?": the stack trace, the rejected payload, the specific tenant ID.
  • Traces follow one request across services. They answer "where did it happen?": which of the six hops in checkout added the 3 seconds, whether the timeout was the database or the retry storm in front of it.

Small teams over-invest in logs (easy, familiar, expensive at volume) and under-invest in traces, which is backwards once you run more than three services: the questions that actually burn incident hours are "where" questions, and grep can't answer them across service boundaries. The trace ID is the thread that ties all three together: a metric alert fires, you pivot to example traces (exemplars), and from a trace span you jump to exactly the logs for that request. That pivot is the workflow; everything else is plumbing.

Why OpenTelemetry specifically

Before OTel, instrumentation meant vendor SDKs: sprinkle Datadog's libraries through your code, and your telemetry (and your codebase) belongs to Datadog. Migrating means re-instrumenting everything, which is why nobody migrates, which is why the pricing is what it is.

OTel breaks that lock:

  • One SDK per language, vendor-neutral. You instrument once against a CNCF standard. Auto-instrumentation covers the common frameworks (HTTP servers/clients, gRPC, database drivers) with near-zero code for Java, Python, Node; Go asks a bit more of you.
  • One wire protocol (OTLP) that effectively every backend now speaks: Grafana, Datadog, Honeycomb, New Relic, AWS, all of them.
  • The Collector in the middle, so the backend decision becomes configuration. Switching vendors is editing an exporter block, not a re-instrumentation project.

For a small company, that last point is the budget-relevant one: OTel keeps the exit door open, and the existence of the exit door is negotiating leverage even if you never use it.

The Collector: receivers, processors, exporters

The Collector is a pipeline daemon that sits between your apps and your backends. Apps send OTLP to the Collector; the Collector shapes the data and fans it out. Three concepts:

  • Receivers ingest: OTLP from your SDKs, but also Prometheus scrapes, host metrics, filelogs.
  • Processors transform: batch, add metadata, drop noise, sample.
  • Exporters ship to backends.

A realistic starter config:

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  memory_limiter:
    limit_percentage: 80
    check_interval: 1s
  k8sattributes: {}        # enrich with pod/namespace/deployment
  batch:
    timeout: 5s
  filter/drop-health:
    traces:
      span:
        - 'attributes["http.route"] == "/healthz"'

exporters:
  prometheusremotewrite:
    endpoint: http://mimir:9009/api/v1/push
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [loki]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, filter/drop-health, k8sattributes, batch]
      exporters: [otlp/tempo]

Deployment-wise on Kubernetes: an agent DaemonSet close to the pods, optionally forwarding to a small gateway Deployment where the sampling and vendor credentials live. Start with just the gateway if you like; add the DaemonSet when you want host metrics and log tailing.

The strategic value hiding in this YAML: cost controls live here, in config you own (dropping health-check spans, filtering debug logs before they're billed, sampling), not in a vendor's pricing tier.

Sampling: you don't need every trace

Tracing everything at production volume is how trace storage becomes the new Datadog bill. Options, in ascending order of sophistication:

  1. Head sampling: the SDK keeps a fixed fraction (say 10%) decided at request start. Trivial, predictable cost, but it's blind: it discards 90% of your errors too.
  2. Tail sampling: the Collector holds spans briefly and decides once the trace completes: keep all errors, keep everything slower than 2s, keep 5% of the boring rest. This is the right answer for most teams. Cost: the gateway needs memory to buffer, and all spans of a trace must route to the same collector instance (load-balancing exporter handles this).
  3. Proportional/adaptive schemes: worry about these at a scale you probably aren't at.

My default prescription: head-sample at 100% in staging, tail-sample in production with "all errors + all slow + small percentage of normal." A team of five will never look at the normal traces anyway.

A pragmatic stack for a small team

The self-hosted path that works: Grafana + Prometheus (or Mimir) + Loki + Tempo, with the OTel Collector feeding all three. It's coherent (trace-to-logs and exemplar links work out of the box) and it runs comfortably on a couple of small nodes for most sub-50-engineer companies. Budget real engineering time for care and feeding: Loki's storage config and Tempo's compaction are not fire-and-forget.

Honest alternative: Grafana Cloud's free/low tiers (or Honeycomb's) with the same OTel instrumentation. For a team with no one to own the observability stack, paying a modest managed bill beats running four stateful systems badly. The instrumentation is identical either way. That's the entire point of OTel. I've moved a client from self-hosted to Grafana Cloud and another in the opposite direction; both migrations were collector-config changes, done in a day.

What I steer small teams away from: starting with the big-ticket vendors on default settings. Not because the products are bad (they're excellent) but because uncapped per-host, per-GB pricing plus default-everything instrumentation is how you get the CFO phone call from paragraph one.

What to instrument first

Not everything. In order:

  1. The edge. Auto-instrument your ingress/API service. RED metrics (rate, errors, duration) per route from day one.
  2. The request path. Auto-instrument every service in the money path (checkout, signup, whatever pays the bills) so traces connect end to end. Verify context propagation across queues; that's where traces silently break.
  3. The database calls. Query spans expose the N+1s and missing indexes that cause most "why is it slow" tickets.
  4. A handful of business metrics. Orders placed, jobs processed. One counter each. Alerts on business metrics catch what infrastructure metrics miss. I've seen "orders per minute dropped to zero" fire twenty minutes before any CPU graph moved.

Structured logging with the trace ID injected comes along for the ride with most auto-instrumentation. Turn it on, then stop logging request bodies; that habit alone routinely halves log volume.

Observability on a budget isn't about tolerating a worse setup. It's auto-instrumentation, one collector, tail sampling, a boring Grafana stack, and owning your telemetry so that whatever backend you use next year is a config change, not a rewrite.