Partial Failure

You send a request to charge a customer. Two seconds pass, and the request times out.

Was the customer charged?

The timeout cannot tell you. The request might never have arrived. The payment might still be processing. The charge might have succeeded, with only the response getting lost.

Your code must decide what to do next, even though each possibility calls for a different decision.

This is what makes partial failure difficult: one part of a system can fail while others keep running, and the surviving parts may not know what happened.

When a function call crosses the network

Consider a function that calculates an order’s total entirely within your process:

const total = calculateTotal(order.items);

The caller and the function execute in the same process. If that process dies, both stop. There is no independently running service that might still be calculating the total after the caller disappears.

Local execution is not perfectly predictable. A function can loop forever, modify state before throwing, or encounter a process crash. But this call does not need to send a message somewhere else and wait for another message to learn the result.

A network call does.

Consider an order system that asks a payments service to charge a customer:

async function placeOrder(order: Order) {
  const response = await fetch(CHARGES_URL, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(order),
    signal: AbortSignal.timeout(2_000)
  });

  if (!response.ok) {
    throw new Error(`Charge request returned HTTP ${response.status}`);
  }

  const payment = await response.json();

  return save({ ...order, paymentId: payment.id });
}

The caller and the payments service now run independently. Either can stop while the other continues. The network can also interrupt communication while both remain healthy.

one process
two machines

If the request times out, your process is still alive and has to decide what to do. But it lacks the information that decision depends on.

Moving a call across a network separates doing the work from learning that the work was done.

The same symptom can mean different things

When the charge request times out, several things may have happened:

  • The request never reached the server.
  • The server received it but failed before doing the work.
  • The server is still doing the work, just slowly.
  • The server completed the work but failed before replying.
  • The response was sent but never reached the client.

Which of these possibilities can the client rule out from the timeout alone?

None of them. Without additional evidence, all remain possible.

Underneath the same symptom are three different situations: the work did not happen, the work is still happening, or the work already happened.

The difference matters:

  • If the work never started, another attempt may be necessary.
  • If the work already happened, another attempt may charge the customer twice.
  • If the work is still happening, another attempt may race the original, and both may succeed.

A missing response is not the same as a failed operation. Unknown is an outcome the client must be prepared to represent.

Why another acknowledgement is not enough

An acknowledgement helps when it arrives. If the payments service confirms that a charge succeeded, the client has learned something useful.

But what if that acknowledgement is lost?

You could acknowledge the acknowledgement. That message could be lost too. Every additional confirmation creates another message whose delivery is uncertain.

The Two Generals Problem captures this limit.

Two armies are camped on opposite sides of a valley, with an enemy between them. They must coordinate an attack, but their only means of communication is a messenger who may be captured.

The first general sends “attack at dawn.” The second receives it and sends a confirmation. If that confirmation arrives, the first general knows the plan was received. But the second still does not know whether the confirmation arrived.

Another message can move that uncertainty one step further. It cannot eliminate it.

When any message can be lost, no finite exchange guarantees common knowledge: both sides knowing the plan, both knowing that the other knows, and so on.

Acknowledgements remain useful. Practical systems rely on them. But a protocol cannot guarantee that every exchange ends with all uncertainty resolved.

Your system still needs a way to behave when a response never arrives.

Timeouts limit waiting

If the other side might never answer, the caller needs a limit on how long it will wait.

A timeout provides that limit. It does not establish what happened remotely.

The server may have crashed. It may be overloaded. The network may be delayed. Without additional evidence or timing guarantees, a crashed server and a slow server can look identical to the caller: both are silent.

The operation might even complete just after the timeout expires.

too short
too long

A timeout that is too short turns ordinary latency into apparent failure and may trigger unnecessary retries. A timeout that is too long holds resources and delays recovery when a dependency really is unavailable.

Start with the dependency’s latency distribution and the amount of time the caller can afford to spend. Tail percentiles such as p99 or p99.9 help reveal how often a chosen timeout would interrupt otherwise successful requests.

The whole operation needs a budget too. Three attempts with a two-second timeout, plus waits between attempts, can take much longer than two seconds.

What cancellation can stop

Stopping the wait and stopping the work are separate actions. A system can connect them.

Suppose a server starts several tasks to handle a request. If those tasks belong to the request’s lifetime, the server can cancel them when the request is abandoned and run cleanup to release their resources.

This is part of structured concurrency: tasks have an owner, and their lifetime is managed with that owner.

An HTTP server can connect a detected client disconnect to this cancellation mechanism. Once it observes that the client has gone away, it can interrupt the request handler and the tasks owned by that handler.

That is useful. An abandoned request should not keep doing unnecessary work indefinitely.

But there are limits.

First, the server has to observe the disconnect. A network interruption or an intermediary can delay that observation. The client’s timeout is not an instant notification to every service involved.

Second, cancellation needs support from the work being cancelled. A task may defer interruption while finishing a protected operation, and an external library may keep working unless its cancellation mechanism is connected.

Third, stopping local execution does not undo work already performed elsewhere.

Suppose the handler has already sent a charge request to a payment provider. Cancelling the handler can stop it from waiting for the response. Aborting its outgoing connection may let the provider detect the disconnect and cancel its own work.

But the provider might have committed the charge already.

client
request
payments
abort

Even if the handler confirms that it has stopped and finished cleanup, that tells you about the handler’s lifetime. It does not prove that the customer was never charged.

Cancellation helps control resource use. Recovering a payment’s outcome still requires evidence from the system that performed it.

Partial failure spreads

So far, this has been a story about one client and one server. Real systems contain chains of dependencies.

A slow dependency causes its callers to wait. Waiting consumes resources: memory, connections, queue capacity, and sometimes workers or threads.

As more requests accumulate, those resources run out. The caller becomes slow too, and now its own callers start waiting.

Notice that nothing had to crash in the upstream services. They failed by being patient.

Cancelling abandoned work helps release capacity, but requests can arrive faster than the system can finish or cancel them. Waiting still needs limits.

Retries can amplify the load

A retry may recover from a temporary failure. But it also sends more work to a dependency that may already be struggling.

If many clients time out together and retry immediately, the service receives a burst of additional traffic at the moment it has the least capacity to spare.

Retries also multiply across layers.

Suppose three layers each make up to three attempts for every incoming call: one initial attempt and two retries. A single request at the edge can produce up to 27 attempts at the bottom.

1
3
9
27

Each layer sees a small retry allowance. The dependency at the bottom sees their product.

Some of the original attempts may still be running when the retries arrive. A timeout does not establish that the previous attempt has stopped consuming capacity.

Recovery behavior has become another source of load.

Design for uncertainty

There are two problems to solve:

  • How should the system proceed when an operation’s outcome is unknown?
  • How can it prevent a struggling dependency from taking everything else down?

They require different mechanisms.

Make repeating an operation safe

For an operation that may be retried, idempotency means repeating it does not produce an additional effect.

A common approach is an idempotency key. The client assigns an identifier to a logical operation and reuses that identifier for every attempt. The server uses it to recognize attempts that belong to the same operation.

Here is the charge request with a key:

async function requestCharge(order: Order) {
  const response = await fetch(CHARGES_URL, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      // Stable across retries of this logical charge.
      'Idempotency-Key': `charge:${order.id}`
    },
    body: JSON.stringify(order),
    signal: AbortSignal.timeout(2_000)
  });

  if (!response.ok) {
    throw new Error(`Charge request returned HTTP ${response.status}`);
  }

  return response.json();
}

This assumes one logical charge per order and a payments service that implements the idempotency guarantee. Adding the header alone does not provide it.

The request can still time out. Within the service’s idempotency contract, retrying the same operation with the same key can recover the result without creating another charge.

Once the operation has completed and its result has been recorded, subsequent attempts can return that result.

If the original is still running, the server must prevent another attempt from independently performing the same work. It must also handle crashes between starting the operation and recording its outcome. We will examine those details in the next lesson.

This protects the charge, not the entire order workflow. Saving the order must also tolerate retries, and the system still needs to recover if charging succeeds but saving fails.

Give unknown outcomes a path to resolution

Safe retries reduce the risk of acting twice. The system still needs to learn what happened.

An operation ID can support:

  • A status endpoint that reports whether the operation is pending, succeeded, or failed.
  • A lookup that finds the resulting payment or order.
  • A reconciliation process that checks unresolved operations against durable records.

The goal is to make an unknown outcome a state the system can track and investigate.

Until it has enough evidence, it should preserve that uncertainty. Marking a timed-out payment as definitively failed can lead a user or another service to initiate a second payment.

Contain the spread

A dependency’s problems should consume a bounded amount of the caller’s resources.

  • Use timeouts and overall deadlines to bound waiting.
  • Propagate cancellation to request-owned work and release resources when that work is no longer needed.
  • Limit concurrency and queue size so pending work cannot grow indefinitely.
  • Use backoff and jitter to spread retries over time.
  • Set retry limits, and choose where retries belong so they do not multiply across layers.
  • Fail fast, for example with a circuit breaker, when recent failures suggest that more calls are unlikely to help.

Work that must survive a disconnect needs a different owner. Once a system accepts a durable job, finishing that job should not depend on the client keeping an HTTP connection open. Its lifetime belongs to the job system.

Backoff and jitter address another part of the problem: when to make another attempt.

Backoff increases the delay between attempts. But if every client follows the same schedule after failing together, they may return together too.

Jitter adds randomness to those delays.

backoff
+ jitter

Spreading retries reduces synchronized bursts. It does not create capacity. A service that remains overloaded also needs callers to limit or stop additional attempts.

Retries should be selective: another attempt may help with a temporary connection failure, but it will not fix an invalid request.

Separate what happened from what you know

An operation has an actual outcome. The caller has evidence about that outcome. Across a network, the two can diverge.

A charge can succeed while the client sees a timeout. A handler can be cancelled after a downstream service has committed its work. A retry can repeat an operation that already completed.

Designing for partial failure means keeping those possibilities explicit.

Represent unknown outcomes. Provide ways to recover the result. Bound how much time and capacity uncertainty can consume, and cancel work whose result is no longer needed.

Then, when another attempt is necessary, make sure repeating the request does not repeat the effect.

That is the subject of the next lesson: idempotency.