Squandering Money: Fear-Driven Decisions in Streaming

Most messaging and streaming technologies have the ability to persist data to replay later or allow for stateless application design (persistence) and to guarantee a quality of service - ensuring message delivery (durability). In most messaging technologies this involves storing data on the file system.

Some systems, like Kafka, require at least some level of persistent storage and can be tuned via data retention configuration and replication factor. In other systems, like MQTT, a per message QoS dictates durability from none at all to exactly once delivery where replication options vary by vendor. NATS.io enables data durability and persistence when the JetStream subsystem is used, allowing for both persistence/durability and at-most-once delivery at the same time.

Regardless of your messaging system choice, configuration around data storage and replication has implications. Many organizations struggle with finding the best choices for their business and make the mistake of defaulting to what is perceived as the safe bet - a data persistence strategy where data is replicated far beyond what is required. This comes with a cost that should not be ignored.

Should organizations default to maximizing data persistence and minimizing risk?

No! I’ve seen teams default to producer and consumer acknowledgements and choose to replicate every message, all the time, in five places. When asked why, the answer is simply “we cannot miss a message,” without knowing business SLAs (Service Level Agreements) or the consequences of a SLA violation.

This stance of replication-to-the-max is certainly understandable, especially in large enterprises where there’s a disconnect between business requirements and technical requirements. When there’s a missing or delayed transaction, who takes the blame? What is the cost of temporarily lost data or a lost transaction? Is the transaction truly lost or can it be retried? Fear and uncertainty around these questions result in overprovisioned systems that are expensive and negatively impact performance and scalability. While you can reduce the probability of downtime (or even data loss) to something tenable, you’ll never get to zero. Ever.

Back to the question, I challenge you to never default to persistence. In fact, go the other way, and work up from the minimum to what you need and will let you sleep at night. Default to doing what is the best fit for your business in terms of cost and meeting SLAs through balancing throughput (units of work per second), latency (how fast a unit of work can be completed), and availability (the probability a unit of work will complete). Consider the notion that persistence is not even necessary in some data flows and retries/errors can be ignored or handled at the application layer.

Think of the many resilient applications built atop HTTP while still adhering to stringent business SLAs. It is of course cumbersome and pushes a lot of logic to the application layer, but the point is you don’t always need to use persistence provided by your distributed communication technology. Durability and persistence is a tremendous convenience which comes at the cost of some increased operational complexity, additional hardware/network resources, or additional SaaS cost (if you’re using a service).



Determining Persistence and Durability Needs

There are a number of factors to consider around data persistence and durability. Note that these come from business requirements - which should drive technical decisions.

Some questions to think about include:

• What is the business SLA for a transaction or unit of work?
• What is your risk tolerance? Businesses should look at what the consequences are for not meeting SLAs in both the tangible and intangible. Is a lost/delayed transaction or some downtime a rare and minor inconvenience? How much potential revenue will you lose per hour of downtime? Will you incur hefty financial penalties? Or worse, will you lose a customer?
• How important is total cost of ownership? Is spending significantly more money for a few hundredths of a percentage of uptime actually worth it? In some sectors, like Fintech, this is a justified resounding “yes!”, but for other lines of business it may be difficult to rationalize the additional expenditure.
• Is the transaction retryable outside the messaging system?
• Which applications require what quality of service? Most distributed messaging systems have flexibility in selecting durability options so you can mix and match based on the business needs behind groups of applications. Optimization here will reduce complexity and improve performance while meeting your SLAs.

Statistics over Intuition

It’s easy and safe just to decide you want to cover your bases and go all out, then request the additional budget for more machines, clusters, storage, etc. However, let’s run through a quick scenario.

Assuming you require a 99.9% uptime guarantee for your business and are deciding between a server cluster of three nodes versus five nodes with quorum based replication. We’ll start with a cloud provider instance level guarantee of 99.5% uptime per instance (messaging server). For the sake of argument, let’s pessimistically subtract 0.5 % for software/network errors to have a 99.0% uptime for software and hardware resources. This results in a downtime of several hours per month by instance… but remember, we’re clustered.

With three nodes in a quorum based system and a replication factor of three (meaning at least two nodes must be available for business continuity), using binary probability calculations with each node having a 99.0% percent uptime, you’ll have a cumulative probability of P(X >= 2) of 99.97% uptime.

With five nodes, you’ll need at least three nodes available to continue business, resulting in a cumulative probability of P(X >= 3) of 99.999% uptime.

In this example, either choice meets the business SLA requirements. If you choose the three node deployment and you’ve saved yourself provisioning two extra nodes and potentially have increased performance with the lower replication factor - lean and mean. This could reduce TCO in terms of resources by 40% at the expense of potentially losing 0.029% uptime. In large multi-clustered deployments, this can represent hundreds of thousands of dollars per year.

If you measure your uptime and continue to meet the business SLAs, then you’ve justified the cost savings and become a hero to those that budget. Bonuses, anyone?

Buffers, Not Databases

Wrapping it Up