TL;DR
- Observability is different from monitoring: you need structured logs, metrics and distributed traces, not just CPU alerts. Monitoring says "what failed", observability says "why it failed".
- Start investing when you have: 2+ services communicating, error rate > 0.5%, or degraded P99 latency in production. Before that, good structured logging with Postgres + visualisation is enough.
- The real cost is high: DataDog, New Relic and similar cost €200-500/month for small startups (without scale). Open alternatives (Grafana + Prometheus + Loki + Jaeger) require ops but only cost infrastructure.
- Common trap: implementing full observability before you have a production problem. Result: 20GB/day of logs nobody reads, unnecessary monthly cost, and debugging still slow because traces are disorganised.
- Start simple: structured JSON logs, 3-4 critical metrics (latency, errors, RPS, resource saturation), and distributed trace only between services that call each other. Expand as you grow.
What Observability Is (And Isn't)
Observability is the ability to understand the internal state of a system through observation of its external behaviour. This is different from monitoring.
Monitoring is binary: the server is up or down, CPU above 80%, response under 200ms. Good for alerting. But when your API takes 5 seconds in production and CPU is normal, traditional monitoring tells you nothing.
Observability has three pillars.
First: structured logs. Not "error processing payment". But {"timestamp": "2025-01-15T14:32:10Z", "level": "error", "service": "payments", "user_id": "1234", "method": "POST /charges", "error_code": "insufficient_funds", "duration_ms": 145}. This allows you to filter, aggregate, correlate.
Second: metrics. Not CPU snapshots. But time series: latency P50, P95, P99 per endpoint; errors by type; request throughput. So you see patterns over time.
Third: distributed traces. A user makes a request. It goes through load balancer, API gateway, authentication service, billing service, database. A distributed trace shows the complete path and time at each step. Without this, you knew it was slow, but not where.
The practical difference: monitoring says "P99 went up to 2 seconds". Observability says "P99 went up because the billing service is making 10 queries per request instead of 1, because a data migration introduced an unindexed query".
When to Start Investing
There's a grey area. Month-old startups with a single server don't need sophisticated observability. Companies with 50+ services in Kubernetes do. Where's the turning point?
Phase 1: Small monolith (up to ~10M requests/month)
A single Node.js or Python server. One database. One Redis cache.
Here, simple logging to file or stdout, parse from terminal with grep and jq, is enough. You can do post-mortems on bugs in 10 minutes because it's your code, the history is linear. Cost: zero in specialised tools.
Exception: if you're already facing difficult debugging (concurrency, race conditions, intermittent timeouts), structured JSON logging in Postgres (a logs table with JSONB) takes 2 hours to set up and will save you 40 hours/month in frustrated debugging.
Phase 2: Emerging microservices (10M to 100M requests/month, 2-5 services)
This is where the first problems appear: a call from one service to another takes longer than expected. Error rate on checkout went up. But which of the 3 services failed?
This is the moment to start serious observability. It's not optional. An error that takes 2 days to trace in production costs more than 6 months of tools.
Concrete criteria:
- Error rate > 0.5% in production (unexpected).
- Latency P99 degraded > 2x in 2 weeks.
- You have 2+ services in production communicating with each other.
- You have at least one "incident" per week that took > 30 minutes to resolve.
Phase 3: High scale (> 100M requests/month, 10+ services)
Observability is critical infrastructure, like databases. It's not optional. Costs of specialised tools (DataDog, New Relic) become justifiable because you gain in incident resolution speed.
The Real Cost of Observability
This is where many get it wrong.
A SaaS solution like DataDog realistically costs €200-500/month for a startup with 20M requests/month. It includes 5GB/day log ingestion, 15 days retention, some traces. Scale? At €0.50 per GB of logs, at 100M requests/month you hit €1500+/month quickly. It's not cheap.
Open alternatives (Grafana Cloud + Prometheus + Loki + Jaeger):
- Prometheus/Grafana Cloud: €50-150/month for metrics and dashboards (pro plan).
- Loki for logs: €30-100/month or self-hosted (costs infrastructure: 1 small server, €20/month).
- Jaeger for traces: €50-200/month managed or self-hosted (another server).
- Total: €150-450/month for infrastructure, but requires internal DevOps.
The truth: if you don't have dedicated DevOps, the time to set up and maintain an open stack can be > SaaS cost. If you do, the ROI of open-source is better.
For a Phase 2 startup, I recommend: start with simple structured logging (JSON to stdout, parsed by a Python script to Postgres) + 3 metrics in Prometheus (you install the Prometheus node exporter agent on one server, takes 5 minutes). Cost: zero. Then, as it grows, move to a more robust stack.
Practical Setup Starting from Zero
I'll assume you have a Node.js/Express API in production. Here's the bare minimum without spending money.
Step 1: Structured JSON logs
Replace console.log("error") with:
const logger = require('pino');
const l = logger({ level: process.env.LOG_LEVEL || 'info' });
app.post('/charges', (req, res) => {
const start = Date.now();
try {
// ... processing
l.info({
method: 'POST',
path: '/charges',
user_id: req.user.id,
amount: req.body.amount,
duration_ms: Date.now() - start,
status: 200
});
} catch (err) {
l.error({
method: 'POST',
path: '/charges',
user_id: req.user.id,
error: err.message,
error_code: err.code,
duration_ms: Date.now() - start,
status: 500
});
}
});
This writes structured JSON to stdout. Redirect to file or aggregator.
Step 2: 3 metrics in Prometheus
const promClient = require('prom-client');
const httpDuration = new promClient.Histogram({
name: 'http_request_duration_ms',
help: 'Duration of HTTP requests in ms',
labelNames: ['method', 'route', 'status_code'],
buckets: [50, 100, 200, 500, 1000, 2000, 5000]
});
const httpErrors = new promClient.Counter({
name: 'http_errors_total',
help: 'Total HTTP errors',
labelNames: ['method', 'route', 'error_code']
});
app.use((req, res, next) => {
const start = Date.now();
res.on('finish', () => {
const duration = Date.now() - start;
httpDuration.labels(req.method, req.route?.path || 'unknown', res.statusCode).observe(duration);
if (res.statusCode >= 400) httpErrors.labels(req.method, req.route?.path || 'unknown', res.statusCode).inc();
});
next();
});
app.get('/metrics', (req, res) => {
res.set('Content-Type', promClient.register.contentType);
res.end(promClient.register.metrics());
});
Install Prometheus, point it to http://localhost:3000/metrics, set up a simple graph. Takes 2 hours, costs zero euros.
Step 3: Traces between services (if you have 2+)
Use OpenTelemetry with exporter to local Jaeger:
const opentelemetry = require('@opentelemetry/api');
const { NodeTracerProvider } = require('@opentelemetry/node');
const { JaegerExporter } = require('@opentelemetry/exporter-jaeger-http');
const jaegerExporter = new JaegerExporter({ endpoint: 'http://localhost:14268/api/traces' });
const tracerProvider = new NodeTracerProvider({ exporter: jaegerExporter });
opentelemetry.trace.setGlobalTracerProvider(tracerProvider);
const tracer = opentelemetry.trace.getTracer('api-service');
app.post('/charges', (req, res) => {
const span = tracer.startSpan('POST /charges');
span.setAttributes({ user_id: req.user.id, amount: req.body.amount });
// call to another service propagates trace automatically
const childSpan = tracer.startSpan('call-billing-service', { parent: span });
// ... code
childSpan.end();
span.end();
});
Local Jaeger runs in a Docker container. You see the complete trace of a request across services.
Common Pitfalls
1. Implementing full observability too early.
I've seen startups spend €300/month on DataDog with 5M requests/month, collecting logs nobody reads. Result: 2 years later, €7200 accumulated bill, and debugging still as slow as ever.
Start simple. Expand as the problem appears.
2. Logs without context.
error: timeout is useless. error: timeout, user_id: 123, endpoint: POST /api/transfer, backend_service: payments-v2, duration_ms: 30000 is traceable.
Always structure logs. Costs 5% more in performance (JSON compression), saves 95% of debugging time.
3. Too many metrics.
I've seen dashboards with 50 graphs. Nobody looks at 50 graphs. Keep 5-7 critical metrics. Everything else is logs.
4. Indefinite data retention.
Storing 2 years of structured logs in Elasticsearch costs. Typical retention should be: error logs, 30 days; normal logs, 7 days; traces, 24 hours; metrics, 1 year.
When Not to Invest in Advanced Observability
Be honest: there are cases where it's not worth it yet.
You have a SaaS dashboard with 50K active users, 5M requests/month, all in a Ruby on Rails monolith. Rails already comes with decent logging. Error rate is < 0.1%. P99 is 150ms. Nobody complains.
Here, investing in DataDog is waste. Simple structured logging in Rails (use Lograge with JSON output) and a Prometheus scraper pulling metrics from Puma is more than enough.
The cost of sophisticated observability is only justified when the cost of downtime or slow debugging is > tool cost.
Practical Roadmap: Months 1-12
Months 1-2: Structured JSON logging (Pino or similar). Exporter to Postgres or S3 for analysis. No external tools.
Months 3-4: 4 critical metrics in Prometheus. A Grafana dashboard (can be local). Team can run queries like SELECT COUNT(*) FROM logs WHERE error_code IS NOT NULL AND created_at > NOW() - INTERVAL '1 hour'.
Months 5-6: If you already have 2+ services, start distributed traces with local Jaeger. DevOps installs on a single machine. Cost: ~€20/month cloud instance.
Months 7-12: Growth, scale increases. You evaluate: cost of open stack on DevOps vs SaaS cost. If you have 50M+ requests/month, DataDog might be more efficient than maintaining infrastructure.
Conclusion
Observability is not a luxury for large companies. But it's also not necessary on day 1. The real metric: when an incident takes > 30 minutes to debug, it's time to invest. Start with structured logging (takes 2 hours, will save you weeks). Expand as the system grows. Avoid the graveyard of expensive tools nobody uses.
If you're facing a similar problem, book a conversation at https://impact-origin.com/agendamento.
