Background Jobs in Node.js: BullMQ vs pgmq vs Building Your Own

TL;DR

• BullMQ is the safest choice for startups: it offers automatic retry, DLQ (dead letter queue), and observability at no operational cost (managed Redis is around £15/month).
• pgmq eliminates an external dependency (Redis) if you already use PostgreSQL, but adds polling CPU overhead and is around 30% slower in high concurrency scenarios (>500 jobs/s).
• Building your own job queue makes sense in very specific cases: fintech with extreme audit requirements, critical processing with SLA <100ms p99, or when you have senior eng available with dedicated bandwidth for continuous maintenance.
• The most common gotcha: assuming that "it's in cache, so it's fast" without actually measuring p99 latency, many custom job systems suffer from stalls when backup or lock contention happens.
• For 90% of B2B SaaS, BullMQ + managed Redis is the sweet spot: cost/complexity minimised, large community, and lets you focus on the business.

Why This Problem Matters in 2024

If you're building any Node.js application in production, at some point you'll need to process work asynchronously: send emails, generate PDFs, integrate with external APIs, process webhooks, aggregate data. Putting that in a queue is critical to keep HTTP latency low and offer a responsive experience to the user.

The choice of background jobs tool is deceptively important. A bad decision here multiplies quickly: it can prevent scalability, create single points of failure, increase operational complexity by 10x, or leave you locked into technology that doesn't make sense for what you actually need.

In Portugal, we see many SaaS build unnecessary proprietary solutions (tables in PostgreSQL with polling, in-memory queues that lose work on restart) or, at the other extreme, use Celery + RabbitMQ in a Node.js project (waste of complexity).

This article focuses on practical reality: what are the real numbers, where does each tool fail in production, and when is it really worth building your own.

BullMQ: The Pragmatic Choice for 95% of Cases

BullMQ is the most mature job queue library in the modern Node.js ecosystem. Built around Redis, it offers solid semantics: automatic retry, exponential backoff, dead letter queues, scheduling, rate limiting, and integrated observability.

Real numbers:

• Throughput: around 1000-3000 jobs/s per worker (typically 1-4 workers per server), depending on job complexity.
• Latency (p50): 5-20ms between enqueue and pickup.
• Latency (p99): 50-150ms under normal load (Redis cluster with persistence).
• Overhead per job: around 500 bytes in Redis (metadata + retry info).
• Monthly cost: £15-35 for managed Redis (Upstash, Redis Cloud) in typical SaaS with 10M jobs/month.

Concrete advantages:

1. Automatic retry with exponential backoff: If a job fails, BullMQ retries at 2s, then 4s, then 8s, up to 60s. This resolves 70% of transient issues (API timeout, rate limiting, network failures).

2. Observability without manual instrumentation: BullMQ exposes events (processing, completed, failed, delayed). You can hook this into any observability tool (DataDog, Sentry, Grafana) in 30 minutes.

3. Durability: Redis, especially with persistence enabled, guarantees that jobs aren't lost between restarts, unlike in-memory queues.

4. Scheduled jobs: Scheduling tasks for specific times is trivial (queue.add(data, { delay: 3600000 }) for 1 hour later).

5. Rate limiting per second: You can limit jobs to, say, 100/s to avoid overwhelming an external API.

Production pitfalls:

The biggest failure I see: assuming Redis is "always fast". It's not. If your Redis cluster is overloaded (e.g., concurrency in the backend is filling Redis whilst workers are reading), p99 latency can jump to 500ms-1s. This creates invisible backlog.

Mitigation: monitor connected_clients, used_memory, and Redis ping latency. If Redis is rejecting connections, you've already lost.

Second gotcha: forgetting that await queue.process() blocks forever. If you have multiple workers but don't scale the number of instances as volume grows, there will be a point where workers crush the server (CPU at 100%, memory growing).

When to use BullMQ:

• Any B2B SaaS with volume <10M jobs/month.
• When you need automatic retry + observability without effort.
• If managed Redis isn't a categorical "no" by your company policy.

pgmq: Removing Redis as a Dependency

pgmq is the growing alternative. It's a Postgres extension that implements a job queue directly in the database, without Redis. The premise is: if you already use PostgreSQL (and almost everyone does), why add another infrastructure tool?

Real numbers:

• Throughput: around 300-800 jobs/s per worker (5-10x slower than BullMQ in push throughput).
• Latency (p50): 20-50ms (polling overhead).
• Latency (p99): 200-500ms in polling (depends on poll_interval).
• Overhead per job: around 2KB per row in PostgreSQL.
• Monthly cost: Zero operational extra (you already pay for Postgres). But Postgres CPU +15-20% if jobs are high volume.

Concrete advantages:

1. One fewer tool: You don't need to manage Redis as separate infrastructure. Simplifies deploy, backups, monitoring.

2. Full ACID transactions: You can enqueue a job and update business logic in the same transaction. Impossible with Redis.

BEGIN;
  UPDATE users SET balance = balance + 100 WHERE id = $1;
  INSERT INTO pgmq.queue ('payment_jobs') VALUES (jsonb_build_object('user_id', $1, 'amount', 100));
COMMIT;

If the UPDATE fails, the job never enters the queue. This is powerful for fintech or domains where audit is critical.

3. No extra operational burden: Postgres backups already cover everything, including unprocessed jobs.

Production pitfalls:

The biggest: pgmq uses polling by default. This means a worker periodically (normally every 1-5 seconds) queries SELECT * FROM pgmq.queue WHERE consumed=false LIMIT 1. This is inefficient and creates high variable latency.

If you have 100 workers polling every 1s, that's 100 unnecessary queries per second on your database.

Second gotcha: contention in PostgreSQL. If multiple workers do UPDATE at the same time on the same row (to mark job as processing), PostgreSQL needs to serialise this. With >10 workers processing concurrently, you start seeing lock timeouts.

Third: pgmq was created recently (2023-2024). Community is small. If you hit edge case, there aren't many people with experience.

When to use pgmq:

• Job volume is low (<1M/month), and p99 latency of 500ms is acceptable.
• You need ACID transactions between job enqueueing and business logic (fintech, regulated domains).
• Your company has explicit policy: "no Redis in production".
• You're comfortable being early adopter.

Building Your Own: When It Really Makes Sense

This is the most dangerous scenario. Building a proprietary queue in Node.js is possible (table in PostgreSQL + polling, or in-memory with file persistence), but should only be done in very specific circumstances.

Real costs of building your own:

• Initial effort: 40-80 hours of senior eng (or 200+ hours of junior eng).
• Maintenance: 10-20 hours/month investigating edge case failures (stalls under load, state corruption if crash at wrong time).
• Testing: complex. You need chaos testing (failure injection, network partition simulation) to have confidence.
• Observability: you have to instrument everything manually.
• Scale: most proprietary solutions I've seen start failing at >100 jobs/s under sustained load.

Cases where building your own is defensible:

1. Extreme audit requirements in fintech: If you need immutable log of every job attempt, with cryptographic signature chain, or precise timing for regulatory compliance, and no off-the-shelf tool offers it.

2. Latency <100ms p99 is hard requirement: For example, real-time fraud detection system that needs <50ms feedback. BullMQ and pgmq may not guarantee this under load.

3. SLA of 99.99%+ uptime with multi-region: You want active-active replication of jobs across regions with automatic failover. Standard job queues don't offer this.

4. Deep integration with domain-specific requirements: For example, in online games, queue of player actions needs guaranteed ordering per session, with state rollback. This is custom enough to justify proprietary solution.

5. You have senior eng (5+ years) with protected bandwidth: Not "when we have a dev free". It's when you have a dedicated architect for this.

What we got right in a proprietary project that saw production:

Three or four years ago we worked on a file processing platform for a retail client. Volume was 50K jobs/day, with 30-minute SLA. We built queue in PostgreSQL with a specific occupancy model:

• Jobs were immutable after enqueue (append-only log).
• Worker process did exclusive lease for 5 minutes (with heartbeat).
• If worker died, job returned to queue automatically.
• Retry logic was explicit: job could have up to 3 attempts, then went to DLQ for manual review.

This worked well because it was specific to the problem (large files, long-running processing, integration with S3 storage), and the cost of getting it wrong was high (customer data at risk).

But the effort was 60 hours initial + 5 hours/month maintenance. If we'd used BullMQ, it would be 5 hours initial + 0 hours/month.

Direct Technical Comparison

Pragmatic Recommendation by Scenario

Typical B2B SaaS (volume <5M jobs/month, p99 latency <500ms acceptable):

Use BullMQ. Don't overthink it. The operational cost is trivial, community is massive, and it lets you focus on functionality.

Fintech, compliance-heavy, or critical audit:

pgmq if volume is low and ACID guarantee is crucial. If volume is high, consider BullMQ + separate audit log (table in PostgreSQL where you record each job enqueued + outcome).

Real-time processing (recommendation, fraud, dynamic pricing):

BullMQ with distributed workers and monitored SLA. If you need <50ms p99 guaranteed, it may be that no standard solution works and you need custom solution. But this is rare.

Legacy codebase already using Celery, RabbitMQ, or SQS:

Migrate to BullMQ when you have a refactor cycle. The payoff in reducing operational complexity is huge.

What Really Matters

The real decision isn't which tool is "better" technically. It's:

1. What's your burn rate on senior eng? If each hour costs £150, and building your own costs 100 hours, that's £15K. BullMQ costs £200/year. Math is clear.

2. What's the cost of job queue downtime? If jobs are lost and nobody complains (e.g., email notifications users don't see), maybe lower reliability is acceptable. If it's payment processing, absolutely not.

3. How much will you spend troubleshooting when something fails in production? If you choose obscure tool and it fails, you could spend 20 hours investigating. Popular tool? There are 100 people with the same problem on GitHub.

Conclusion

For most B2B SaaS in Portugal, the answer is BullMQ. Minimal operational cost, large community, proven reliability, and lets you focus on what matters: your product.

pgmq is a valid alternative if you want to eliminate Redis as a dependency and volume is low. Building your own is a seductive trap that in retrospect costs a lot of engineering time.

If you're facing a similar problem (scaling background jobs, or questioning your tech stack for asynchronous processing), book a conversation at https://impact-origin.com/agendamento.