Observability: when it is worth investing seriously

Observability is not “having logs”. It is being able to answer new questions about production without releasing new code.
Start early with the basics: structured logs, critical business metrics, centralised errors and correlation identifiers.
Invest seriously when you already have real users, external integrations, payments, internal SLAs or repeated incidents with no clear cause.
OpenTelemetry is now the sensible choice for instrumentation, because it reduces vendor lock-in and supports metrics, traces and logs.
The biggest mistake is collecting everything without judgement. Observability without a budget for cardinality, retention and alerts becomes expensive noise.

Observability is not monitoring by another name

Monitoring answers questions you already know: “is the API up?”, “has CPU gone over 80%?”, “are there 500 errors?”. It is useful, but limited. Observability answers questions you did not yet know you would need: “why do customers on a specific plan have a slow checkout?”, “which external integration increased p99?”, “which queue is accumulating jobs after the deploy?”.

The practical difference is in the type of data and how you connect it.

The four signals I usually consider mandatory are:

Metrics: aggregated time series. Example: p95 latency, p99 latency, error rate, requests per second, jobs processed per minute.
Structured logs: discrete events with searchable fields, not loose text.
Distributed traces: the path of a request through APIs, queues, databases and external services.
Errors and exceptions: grouped stack traces, with application version, affected user where permitted and operational context.

CPU and memory profiles also come into the conversation, but they are not usually the first investment. They are valuable when the application already has enough load to justify continuous performance analysis.

The definition I use in small teams is simple: if a customer reports “it is slow”, you should be able to find out in under 15 minutes whether the problem is in the browser, API, database, queue, external service or that customer’s specific data. If you cannot, you do not have enough observability.

The minimum viable setup to start well

I do not recommend starting with an expensive platform, pretty dashboards and 40 alerts. That gives a false sense of control. The minimum viable setup should be small, consistent and applied from the start.

For a typical B2B SaaS application in Node.js 22 LTS, PostgreSQL 16 and an HTTP API, I would start with this:

Logs in JSON, with request_id, user_id where permitted, tenant_id if it is multi-tenant, route, status_code, duration_ms and service_version.
RED metrics for APIs: rate, errors, duration. This comes from Google SRE practice and remains an excellent foundation.
USE metrics for infrastructure: utilisation, saturation, errors. CPU, memory, disk, database connections and queues.
Centralised exception capture. Sentry is often enough at the start.
Real health checks. It is not enough to return 200 OK; check the database, Redis, queues and critical dependencies.
A correlation identifier that follows the request across frontend, backend, jobs and webhooks.

A short example in Express shows the idea. It is not the tool that matters, it is the discipline of context:

app.use((req, res, next) => {
  const startedAt = performance.now();
  req.requestId = req.header("x-request-id") ?? crypto.randomUUID();

  res.setHeader("x-request-id", req.requestId);

  res.on("finish", () => {
    logger.info({
      event: "http_request",
      request_id: req.requestId,
      method: req.method,
      route: req.route?.path ?? req.path,
      status_code: res.statusCode,
      duration_ms: Math.round(performance.now(), startedAt)
    });
  });

  next();
});

This seems basic, but this is where many teams fail. If logs do not have consistent fields, you will end up doing text search during an incident. Text search in production, at 02:00, is an expensive way to discover that technical discipline was missing.

In very early stage projects, such as clients without billing defined yet or in product validation, it does not make sense to sell a heavy observability stack. It makes sense to leave the application prepared: structured logs, centralised errors and essential metrics. The marginal cost is low and avoids a rewrite when the application starts to have real traffic.

When to start investing more seriously

The short answer: before the first expensive incident. The honest answer: almost nobody does that. So I use concrete signals.

Start investing seriously when at least two of these points are true:

You have paying customers and a production failure already has commercial impact.
You have critical external integrations: payments, ERP, CRM, carriers, banks, healthcare services, external authentication.
You have asynchronous jobs, webhooks or synchronisations where the error does not appear immediately to the user.
You have more than one application or service in production.
The p95 or p99 of critical endpoints already matters. For example, checkout above 2 seconds or search above 500 ms.
You have incidents where the team takes more than 30 minutes to understand the cause.
You have frequent deploys and need to know quickly whether a version worsened latency, errors or resource consumption.
You have contractual obligations, even if informal. For example, responding to problems for enterprise customers during business hours.

For an internal application with 20 users, good logs and Sentry may be enough for a long time. For a SaaS with multi-tenancy, billing, permissions and integrations, traces start paying for themselves quickly. For eCommerce, where a checkout problem has immediate impact, I would add funnel metrics and error alerts very early.

The metric that changes the conversation is MTTR, mean time to recovery. If every incident takes 2 hours to diagnose, and you have 2 incidents per month, you are losing technical time that could be spent on product. Observability does not eliminate incidents. It reduces the time you spend working blind.

Tools: buy, host or mix

There is no universal answer. There are trade-offs.

Sentry is a very good choice for application errors. It groups exceptions, shows releases, browser, stack trace and impact by user. For small teams, it often brings value in hours. The limitation is that it does not replace infrastructure metrics or complete traces in more distributed systems.

Grafana, Prometheus and Loki are attractive when you want control and predictable cost. Prometheus 2.x remains excellent for metrics. Grafana 11 is a mature interface for dashboards and alerts. Loki is interesting for logs when you accept the labels model carefully. The hidden cost is in operations: retention, backups, updates, storage, tuning and alerts. A VM at 20 to 80 euros per month may look cheap, but if someone spends half a day per month maintaining the platform, that cost has already changed.

Datadog, New Relic, Honeycomb and Grafana Cloud reduce operations and speed up adoption. They are strong in traces, APM, dashboards and correlation. The problem is variable cost. Logs with 10 GB per day, high cardinality in metrics or traces without sampling can make the bill unpleasant. Always check the current pricing page and test with real traffic for one or two weeks.

OpenTelemetry, documented at opentelemetry.io, is now the layer I would try to put between the application and the vendor. The W3C Trace Context specification is also important, because it defines how to propagate context between systems through headers such as traceparent.

Practical comparison:

Sentry plus Prometheus: good for starting out. Low effort, good coverage of errors and metrics. Less strong in complex distributed traces.
Self-hosted Grafana stack: good control and predictable cost. Requires operational knowledge and retention discipline.
Datadog or New Relic: excellent time to value. Can become expensive if you send logs and traces without a policy.
OpenTelemetry with a managed vendor: good portability and modern instrumentation. Requires more careful initial design.

If you are in small B2B SaaS, I would start with Sentry, structured logs and essential metrics. When integrations and jobs grow, I would add OpenTelemetry and traces. Only then would I decide whether a complete platform is worth it.

The technical design that avoids regrets

The worst observability is observability that grows by accident. Each library sends logs in a different format, each service invents metric names, each alert shouts for different reasons.

Define conventions early:

Service name: billing-api, customer-portal, worker-sync.
Environment: production, staging, preview.
Version: commit SHA or semantic version.
Tenant or organisation: only if safe and necessary.
Normalised routes: /customers/:id, not /customers/123.
Explicit units: duration_ms, payload_bytes, queue_lag_seconds.

Route normalisation is more important than it seems. If you create a time series for every URL with an ID, you destroy cardinality. High cardinality means too many combinations of labels. In Prometheus, this increases memory usage and can degrade queries. In managed vendors, it increases cost.

A minimal OpenTelemetry configuration in Node.js shows the main point: service, version, exporter and automatic instrumentation. There is no need to fill the codebase with manual spans on day one.

import { NodeSDK } from "@opentelemetry/sdk-node";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { resourceFromAttributes } from "@opentelemetry/resources";

const sdk = new NodeSDK({
  resource: resourceFromAttributes({
    "service.name": "customer-api",
    "service.version": process.env.GIT_SHA ?? "unknown",
    "deployment.environment": process.env.NODE_ENV ?? "development"
  }),
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
  }),
  instrumentations: [getNodeAutoInstrumentations()]
});

sdk.start();

Then you add manual spans only at the points that matter: calls to ERP, payments, PDF generation, stock synchronisation, import jobs, critical queries. Manually instrumenting everything is wasteful and increases noise.

You should also separate technical metrics from product metrics. http_requests_total does not replace checkout_completed_total. If a retail platform has a healthy API but checkouts are dropping, the infrastructure is “green” and the business is on fire.

Alerts that are useful and alerts that only wake people up

A good alert forces action. A bad alert informs you that the world is imperfect.

Do not create alerts for CPU above 80% for 1 minute unless you know that this causes real impact. High CPU may be normal in batch jobs. Also do not alert for every isolated 500 error. An error can be irrelevant; a 5% error rate for 10 minutes on a critical endpoint is a different conversation.

Useful alerts:

5xx error rate above 2% for 5 minutes on critical endpoints.
p95 above 1 second for 10 minutes on the main API.
p99 above 3 seconds on checkout or payment.
Queue with lag above 5 minutes.
Consecutive failed webhooks for the same vendor.
Database with connections above 80% of the limit for 10 minutes.
TLS certificate expiring in less than 14 days.
Disk space above 85% with a growth trend.

I also like alerts based on symptoms, not only causes. “Customers cannot pay” is better than “high CPU on worker 3”. The cause matters for solving the issue; the symptom matters for prioritising.

The Prometheus documentation on alerting rules and Alertmanager is a good foundation. But the hard part is not the syntax. It is deciding who receives it, when they receive it and what they do. Every alert should have an owner, severity and short runbook. If nobody knows what to do, the alert is not ready yet.

Common production traps

The most common trap is confusing volume with visibility. Teams send all logs to a platform, with no structure, no sampling and no thought-out retention. During the incident, they have millions of lines and little information.

Another trap: putting user_id, email, complete payloads or personal data in logs. In Portugal and the European Union, this touches on GDPR. Logs should follow data minimisation. Store internal identifiers when necessary, avoid sensitive data, apply short retention and control access. In healthtech or fintech, this stops being a technical detail and becomes a serious risk.

Third: dashboards to impress, not to operate. A dashboard with 30 charts is nice in a meeting and useless in an incident. I prefer three simple screens:

Product health: logins, checkouts, payments, synchronisations, errors by customer.
API health: traffic, p50 latency, p95 latency, p99 latency, errors, saturation.
Worker and integration health: queues, retries, webhooks, times by vendor.

Fourth: not linking deploys to metrics. If you cannot see that deploy abc123 increased p95 from 180 ms to 650 ms, you will waste time debating opinions. Link releases, commits and feature flags to observability.

Fifth: absence of sampling in traces. Tracing 100% of requests may be acceptable with 1000 requests per day. With 300 requests per second, it can become expensive and unnecessary. A practical approach is to keep 100% of traces with errors, a sample of normal requests and higher sampling on critical endpoints.

How much to invest at each stage

Think in stages, not abstract maturity.

Stage 0, prototype or validation: local structured logs, Sentry or equivalent, simple health check. Low cost. Do not overcomplicate.

Stage 1, first real customers: centralised logs, RED metrics, API dashboard, basic alerts, releases associated with errors. Objective: diagnose simple incidents in under 30 minutes.

Stage 2, product with integrations and jobs: traces with OpenTelemetry, metrics for queues, webhooks, external vendors, business dashboards, runbooks. Objective: understand asynchronous failures and partial degradations.

Stage 3, scale and larger team: SLOs, error budgets, formal on-call, retention by data type, sampling, monthly review of alerts, load tests. Here it already makes sense to discuss p99 below 100 ms on specific internal endpoints, or throughput of 300 RPS with a known margin.

Observability is a technical investment with an operational return. It should not come from fear, nor from fashion. It should come from concrete questions that the team needs to answer quickly.

Conclusion

Good observability starts small, but it has to start with discipline. Structured logs, the right metrics, centralised errors and correlation context solve more problems than an expensive, poorly configured platform.

Invest more when the impact of not knowing is greater than the cost of instrumentation. If you are facing a similar problem, book a call at https://impact-origin.com/agendamento.