Can Small Services Survive? Limits of Polling and Criteria for Introducing Queues

Database polling processes tasks, but at some point you hit a situation where CPU has headroom while response times keep getting slower. Reduce polling interval from 1 second to 100ms and the database connection count grows exponentially until the connection pool breaks. This article analyzes the limits that polling-based processing must endure and presents concrete criteria for when to introduce a message queue system like Redis.

The textbook answer for when polling works

In a small service, polling can be the answer. Periodically fetching and processing tasks with WHERE status = 'pending' LIMIT 100 queries works without any complex infrastructure. You don't need to provision a separate system like Redis or RabbitMQ, and deployment is just a few lines of code added to the existing application.

The problem is how far this approach can go. With a 1-second polling interval, maximum latency is 1 second. Cutting the interval to 100ms brings average latency down to 50ms, but the database now receives 10 queries per second. If 10 workers poll simultaneously, that's 100 additional polling queries per second hitting the database.

At this level, even a small service can sustain the database. PostgreSQL's connection limit is typically 100~200, but with proper connection pool tuning it can handle up to 500. The trouble starts when you shorten the polling interval further. What about 10ms polling? The database receives 100 queries per second. With 10 workers, that's 1,000. You've already hit the connection limit ceiling.

Signals that the system is entering saturation

The symptoms are simple. Response times keep getting slower while CPU utilization remains low. Memory is still sufficient. The database has reached its connection limit, yet the database's own load metrics don't look particularly heavy. A common mistake in this situation is to shorten the polling interval even more.

Shortening the polling interval causes counterproductive effects. Database lock contention worsens and connection pool exhaustion begins. If polling queries start failing, workers have to wait until the next polling cycle, creating a vicious cycle that further increases latency.

The most reliable way to detect actual saturation is to measure connection pool wait time. If requesting a database connection takes 10ms, that's normal. But if it takes 100ms or 500ms, that's the saturation signal of the polling system.

The list of responsibilities that queue adoption brings

Introducing a message queue like Redis brings new operational burdens. What was once a single polling query now has additional concerns. The number of variables to consider grows rapidly: queue clustering, failover, memory management, message expiration policies, and more.

Redis operations are trickier than they seem. With a single instance there's little room for problems, but if it fails the entire system goes down. Even with replicas, replication lag occurs and consistency issues arise when slaves separate from the master. Even Redis's AOF (Append Only File) setting alone can cause a 2~3x performance difference.

However, the biggest advantage of a queue system is the fundamental resolution of polling latency. Workers don't need to poll; they can start processing immediately as soon as a new task arrives. Average processing time drops to 12ms and system throughput improves by 510x.

Infrastructure determines cost

Polling system costs are simple: just the application server and database server costs. But introducing a queue means additional costs for Redis instances, monitoring agents, backup systems, and more.

Looking at AWS prices: if the database can be handled on a t3.micro ($0.023/hour), Redis can be handled on cache.t3.micro ($0.024/hour). But in real operations, you need at least two Redis nodes for HA, plus additional Lambda functions and CloudWatch costs for backup and monitoring.

A bigger issue is operational cost. In a polling system, managing the database was enough. Introducing Redis requires a Redis specialist and knowledge of backup strategies, failure recovery procedures, and performance tuning. In a small team, this cost becomes particularly heavy.

Both late adoption and premature adoption can fail

Common mistakes come in two forms. One is introducing a queue when the polling system can still handle it, and the other is rushing to introduce a queue once the system has already saturated.

Introducing a queue too early increases operational burden and wastes budget. Development becomes more complex and deployment pipelines need to be refined. On the other hand, rushing adoption leads to a chaotic state where existing polling code and the queue system coexist. During migration, you bear the burden of running both systems simultaneously.

Criteria for adoption timing

The moment to consider introducing a queue in a polling system is clear.

First criterion: connection pool wait time exceeds 50ms. This means polling queries are overloading the database.

Second criterion: polling query ratio exceeds 30% of all database queries. This indicates database resources are being excessively consumed by polling.

Third criterion: average latency exceeds 100ms and concurrent throughput is at saturation. This means you've reached the lower bound of polling latency.

If any one of these three criteria is met, you should consider queue adoption. Conversely, if none are satisfied, maintaining the polling system is the wiser choice.

Migration strategy

Once you've decided to adopt a queue, you don't need to migrate everything at once. A gradual approach is safer.

Start by moving low-load tasks to the queue first. For example, moving auxiliary tasks like notification delivery lets you gain operational experience with the queue system. Gradually expand to major business logic over time.

During migration, you must have a fallback mechanism. If the queue system has issues, you need the ability to roll back to the polling system immediately. This can be implemented with feature flags or routing logic.

Operational checkpoints are straightforward: monitor Redis memory usage, queued message count, worker processing speed, and failure rate. Pay special attention to messages that wait too long in the queue, as this indicates the worker's processing limit has been reached and requires immediate response.

In the end, it comes down to trade-offs

Polling and queues each have their own limitations. Polling is simple and intuitive, but has clear scaling limits. Queues offer high throughput but increase operational complexity.

For developers of small services, the most important thing is judgment at the right time. If polling can still handle it, it's better not to add unnecessary complexity. But once you recognize that system metrics are hitting their limits, don't hesitate and move to a queue.

The key is securing evidence that polling hasn't yet gone 'sufficiently wrong'. By comprehensively analyzing multiple metrics—database connection count, polling query ratio, average latency, worker CPU usage—you can get closer to the right answer.

Comments

Loading comments.