56% of Kafka clusters process under 1 MB/s. You might be paying to host an architecture diagram.
The Architecture Resume Problem
There’s a pattern I keep seeing across job descriptions, Notion docs and pull request descriptions: Kafka appears in the tech stack before anyone has defined what problem it solves.
Junior leads propose it because it sounds serious. Senior engineers validate it with phrases like “future-proof” and “event-driven architecture.” Then the team spends the next three months managing consumer groups, retention policies and partition rebalancing instead of shipping features.
According to fleet data from Confluent and Aiven, around 56% of Kafka clusters operate at or below 1 MB/s throughput. That’s not my hot take. The vendor is telling you, in survey form, that the majority of their paying customers don’t have a Kafka-scale problem. They have a job queue problem.
Those are different problems. They need different tools.
What Kafka Actually Solves
Kafka is genuinely excellent at a narrow thing: high-throughput, fault-tolerant, distributed event streaming where multiple completely independent consumers need to read the same events at their own offsets, with durable replay and guaranteed ordering per partition.
That’s the use case. Not “I need to notify my email service when a user signs up.”
Kafka was built at LinkedIn to handle hundreds of billions of events per day across dozens of internal systems. Its architecture reflects those requirements: distributed brokers, ZooKeeper for coordination (finally removed in Kafka 4.0, which dropped ZooKeeper entirely after being deprecated since version 3.5), partition rebalancing, consumer group coordination and ISR management to guarantee replication durability.
Running Kafka for a SaaS product with 500 daily active users is operationally equivalent to deploying a cargo ship to deliver lunch.
What Most Applications Actually Need
Strip away the architecture diagrams and look at what most small-to-mid backend systems actually require from a queue or event system. The list is short:
- Run a task asynchronously (send email, process an image, trigger a webhook)
- Retry failed tasks with exponential backoff
- Fan out a single event to two or three consumers
- Observe job throughput and failure rates
That’s it. Nothing in that list requires Kafka. In fact, you already have the tools.
The PostgreSQL Option Most Teams Ignore
PostgreSQL has had LISTEN/NOTIFY since the early 2000s. It's a native pub/sub mechanism built directly into the database engine. A producer calls NOTIFY channel_name, 'payload' and every connected client listening to that channel receives the message immediately over their existing connection.
-- Producer
NOTIFY user_signups, '{"user_id": 42, "email": "user@example.com"}';
-- Consumer (any connected PostgreSQL client)
LISTEN user_signups;The payload is limited to 8,000 bytes, per the official PostgreSQL documentation. If you need to pass larger data, send a record ID and fetch the full row. That’s the right pattern regardless of which notification system you’re using.
Notifications are transactional. A NOTIFY inside a transaction only fires on commit. If the transaction rolls back, the notification never fires. This eliminates the entire class of "we sent the event but the data wasn't ready yet" bugs that plague systems using external message brokers alongside relational databases.
No ZooKeeper. No consumer groups. No partition math. No separate process to maintain. If PostgreSQL is already your application database, LISTEN/NOTIFY costs you exactly nothing to run.
For workloads under roughly 10,000 notifications per second on a single instance, this handles the job with sub-50ms delivery latency and near-zero operational overhead. The only hard limitation is that LISTEN/NOTIFY does not replicate across PostgreSQL instances. If you're running a sharded cross-region setup, this does not fit. For a team running one to three application servers against a single PostgreSQL primary, the limitation is irrelevant.
Laravel Queues: The Three-Tier System You Already Have
If your stack is Laravel-based, you have a queue system built into the framework with three tiers that match different scale points. Most teams skip directly to tier two or three without thinking about whether tier one already covers their actual load.
Tier 1: Database Driver
Laravel’s database queue driver ships out of the box. Your queue backend is the same PostgreSQL or MySQL instance you’re already running. No new infrastructure, no new failure modes to learn.
// config/queue.php
'default' => env('QUEUE_CONNECTION', 'database'),
// Dispatching a job from anywhere in your application
SendWelcomeEmail::dispatch($user)->delay(now()->addMinutes(1));
// Worker process
php artisan queue:work --tries=3 --backoff=60The database driver polls the jobs table at a configurable interval. Failed jobs land in failed_jobs with their exception message and stack trace, readable with plain SQL. Retries are a single Artisan command: php artisan queue:retry all.
This is appropriate for applications where jobs are dispatched at rates well under a hundred per second and sub-second processing isn’t a product requirement. Internal admin notifications, weekly digest emails, PDF generation, webhook delivery to external services: the database driver handles all of it without complaint.
Tier 2: Redis and Laravel Horizon
When processing speed matters, Redis replaces the database as the queue backend. Jobs live in memory and workers pull from Redis lists with far lower latency than polling a SQL table.
composer require laravel/horizon
php artisan horizon:install
php artisan horizon
// config/horizon.php
'environments' => [
'production' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['high', 'default', 'low'],
'balance' => 'auto',
'maxProcesses' => 10,
'minProcesses' => 2,
],
],
],Horizon gives you a real-time dashboard for throughput, wait time, job failures and worker saturation. Queue priorities work by naming your queues and listing them in order. Jobs dispatched to the high queue get processed before anything in default.
Redis Streams (introduced in Redis 5.0, released October 2018) adds consumer groups and at-least-once delivery semantics on top of Redis. If you need message persistence or a simple fan-out with acknowledgment, Redis Streams closes most of the gap between Redis Pub/Sub and a full message broker, without the operational surface area.
For a team of one to five engineers, Horizon’s dashboard does more useful work than a Kafka control plane will, because you’ll actually open it. A Kafka control panel at low throughput is a dashboard that tells you a lot about a system doing nothing interesting.
Tier 3: External Broker (When the Data Justifies It)
If your Redis queue is processing tens of thousands of jobs per hour, multiple fully independent services need to consume the same event stream at different offsets and you need durable event replay for auditing or system recovery, an external broker earns its keep.
The threshold is concrete throughput and concrete fan-out requirements, not an estimate of what you might need in two years.
The Real Cost of Early Kafka Adoption
Most teams underestimate what it costs to run Kafka before they’re already in it.
Self-hosted Kafka means managing broker configuration, replication factors, partition counts, topic retention policies (Kafka is an append-only log store and disk fills up), consumer lag monitoring and JVM garbage collection tuning. Kafka 4.0 removed ZooKeeper, which simplifies the setup considerably, but the baseline operational complexity of running a distributed log system is still significant. The documentation is long. The failure modes are not obvious until you hit them.
Managed Kafka reduces the ops burden and introduces cost variability that surprises most teams on their first invoice.
Amazon MSK Serverless charges $0.75 per cluster-hour plus $0.0015 per partition-hour plus data ingress and egress fees. The cluster-hour charge alone is $540 per month at 24/7 operation before a single message is processed. A Redis instance on a $6/month Hetzner VPS handles most small queue workloads without a per-cluster floor charge.
The math is not competitive at low throughput. You’re paying for operational simplicity and the Kafka protocol, not for capacity you’re actually using.
When Kafka Actually Makes Sense
There are real use cases where Kafka is the right answer. The problem is that the teams encountering those use cases usually know it before they start the architecture conversation.
Kafka makes sense when:
- Multiple completely independent services must consume the same event stream at different offsets and different speeds
- You need to replay past events for audit, debugging or feeding a new downstream service months after the original event occurred
- Sustained write throughput is above 10 MB/s and you need guaranteed ordering per partition key
- You have dedicated infrastructure engineers who can own the deployment, not developers who also write product code
If you answer yes to all four, you probably already work somewhere that has Kafka running. The teams asking “should we add Kafka?” typically answer no to at least three of these.
The Recommended Path for Small Teams
Start with the database queue driver in Laravel, or a simple job table for any framework. If your database is already the bottleneck, you have a different problem to solve first and Kafka will not fix it.
Move to Redis when your job volume grows to the point where polling latency matters or you need the Horizon visibility layer. This step covers the vast majority of production workloads a small team will ever run.
If you reach sustained high throughput with genuine multi-consumer fan-out requirements and event replay needs, consider Kafka, Redpanda (API-compatible with Kafka, lighter to operate) or a managed service sized to your actual traffic. That decision should be driven by metrics, not projections.
Most teams take years to reach that tier. Many never do.
The Real Pattern Worth Watching
The deeper issue here isn’t Kafka specifically. It’s the habit of pre-solving scale problems with tools that impose real operational cost today.
Adding Kafka to a system with 500 users doesn’t prepare you for 500,000 users. It adds the overhead of scale without the traffic that would justify it. When you finally reach real traffic, the Kafka deployment you built for a small system will need a full redesign anyway, because the topic structure, partition counts and consumer group topology that made sense at low scale are wrong at high scale.
The best architecture for a small team is the simplest one that solves the problem you actually have today. Complexity is easy to add. Removing it from a production system is a project.
Start with what you have. Upgrade when the data tells you to. Keep Kafka on the shelf until you have a throughput problem that genuinely requires it.
If your queue is processing under 10,000 jobs per hour and your biggest pain point is debugging a failed job, open the failed_jobs table. The answer is probably there.
