Don't Trust Webhooks, Verify with Signatures

The trap of webhook handling begins with the assumption that a webhook must be processed immediately upon arrival. If you write logic trusting an external request — without knowing how many retries the server will receive, when it will arrive, or even who forged the request — data integrity and system stability collapse without a trace.

Signature Verification Is More Than Header Comparison

Many examples verify signatures using HMAC-SHA256, and developers write a single line of code that takes the header value, recomputes it with the same algorithm, and compares. But something is left out here. The first thing to consider in signature verification is the timing-attack problem. Verifying with a simple string comparison (== operator) gives an attacker the opportunity to gradually guess the signature value.

import hmac
import hashlib

def verify_signature(payload, signature_header, secret, tolerance=300):
    # GitHub style: sha256=<signature>
    try:
        sha_name, signature = signature_header.split('=', 1)
    except ValueError:
        return False

    if sha_name != 'sha256':
        return False

    secret_bytes = secret.encode('utf-8')
    payload_bytes = payload.encode('utf-8')
    expected = hmac.new(secret_bytes, payload_bytes, hashlib.sha256).hexdigest()

    if not hmac.compare_digest(expected, signature):
        return False

    # Timestamp verification (optional)
    timestamp = extract_timestamp(payload_bytes)  # implementation required
    if abs(time.time() - timestamp) > tolerance:
        return False

    return True

Adding a time window limit to signature verification is not merely about expiration handling. Even if an attacker cleanly captures a signature in a noise-free environment, that signature becomes worthless as time passes. For services like Stripe or GitHub that include timestamps in headers, this time window must be checked. Setting the window too short (e.g., 60 seconds) means valid requests may be rejected due to normal network latency; setting it too long (e.g., 24 hours) enlarges the attack surface. In practice, 300 seconds (5 minutes) is the safe default, adjusted according to service characteristics.

Without Idempotency Keys, Retries Cause Re-processing

Webhook retries are not exceptions but normal behavior. Stripe attempts up to 3 retries at intervals of up to 3 days for payment failures, and GitHub attempts 5 retries over 3 days. The problem arises when developers react with "I've already processed this?"

{
  "id": "evt_1J...",
  "data": {
    "object": {
      "id": "cs_test_...",
      "amount": 1500,
      "currency": "krw"
    }
  }
}

The same payment event may arrive at 3 AM, midnight, or when the network suddenly recovers. The server will attempt to process the same payment again each time.

To solve this, a webhook ID-based idempotency key must be used. You cannot determine the outcome based on payment amount or order ID alone. Just because the order ID is the same does not mean the same webhook has arrived.

INSERT INTO webhook_event (
  event_id,           -- Webhook unique ID (e.g., evt_1J...)
  processed_at,
  status,
  result_summary    -- Processing result summary (e.g., payment_succeeded)
) VALUES (?, ?, ?, ?)
ON CONFLICT(event_id) DO NOTHING;

The ON CONFLICT DO NOTHING pattern is the simplest way to guarantee idempotency. But there is a trap here: you must consider what processing result to return when a duplicate request arrives. If the initial processing failed (e.g., external API call timed out), the retried request may repeat the same failure. In that case, you need to distinguish between retryable failures and failures that must never be retried.

A payment API call that fails due to timeout is worth retrying via retransmission. But a response like "payment already canceled" will never succeed no matter how many times it is retried, so in those cases, a logic marking permanent failure after 3 attempts is needed.

Retry Wait Time: Process Immediately or Lock the Queue

When a webhook is delayed by 2 minutes or 5 minutes, the processing time is identical from the server's perspective. However, 5 minutes of delay sends a signal to the webhook sender that "this event was not processed normally." Services like GitHub or Stripe accelerate retransmissions upon delay or display warnings on the user console.

Therefore, when the retry wait time is exceeded, a decision to immediately abandon immediate processing and switch to queue locking is needed. The term "queue locking" is intentionally used here. It is not enough to simply put something in a queue — there is a constraint that processing must occur within the queue's retry limit time.

MAX_WAIT_TIME = 3 * 24 * 3600  # 3 days (Stripe retry limit)

async def handle_webhook(request):
    event_id = request.headers.get('X-Webhook-ID')
    timestamp = extract_timestamp(request.body)

    if current_time() - timestamp > MAX_WAIT_TIME:
        # No more retries expected, so lock the queue and process separately
        await queue.enqueue('manual_review', event_id=event_id)
        return HttpResponse(status=200)

With this approach, the item goes not into a queue but into a manual review queue. It is routed to a dashboard or notification system rather than a queue, so an operator can directly decide how to handle it. Webhooks that remain unprocessed beyond the queue lock time should be recognized not as system anomalies but as part of operational procedures.

Without Concurrency Control, Duplication Occurs

Even with signature verification and idempotency keys perfectly implemented, the same event may be processed twice. This stems from concurrency conflicts. When a webhook is retried and a new server instance receives the request while previous processing is still in progress, a race condition occurs.

# Incorrect example: check-then-process pattern
async def process_webhook_event(event):
    existing = await db.fetch_one(
        "SELECT status FROM webhook_event WHERE event_id = ?", (event.id,)
    )
    if existing and existing['status'] == 'completed':
        return  # duplicate prevention
    # ... processing logic ...
    await db.execute("UPDATE webhook_event SET status = 'completed' WHERE event_id = ?", ...)

This code has a time gap: both requests pass the existing check and execute the processing logic. If two requests each perform a SELECT and both determine no result exists, both proceed with processing.

The solution is atomic processing. Database transactions or locking mechanisms must be used.

BEGIN;

SELECT status FROM webhook_event WHERE event_id = ? FOR UPDATE;
-- or Redis SETNX pattern
-- SETNX webhook:lock:evt_1J... "processing" EX 300

-- execute processing logic
-- ...

UPDATE webhook_event SET status = 'completed' WHERE event_id = ?;
COMMIT;

FOR UPDATE locks or Redis SETNX atomically ensure that only one request proceeds through the processing flow. However, if processing takes too long while the lock is held, other requests time out and the retry mechanism creates further problems.

What Happens When Processing Times Out

Most webhook receivers have a response time limit of 5 to 10 seconds. GitHub allows 10 seconds, Stripe 30 seconds, but most proxies or load balancers forcefully terminate the request after 30 to 60 seconds. Exceeding this limit results not in a normal response but in 504 Gateway Timeout or 502 Bad Gateway, which the sender counts as retry attempts.

To address this, the immediate response + asynchronous processing pattern is essential.

Webhook reception (0~100ms)
  ↓
Signature verification + idempotency record
  ↓
200 OK returned immediately
  ↓
Actual processing (asynchronous queue/worker)

Developers commonly make a mistake here: they return 200 quickly but, if the actual processing fails, fall into the delusion that "the error is hidden." A webhook receiver must monitor not just the response code but the processing results themselves.

Retry Failure Logging Is Not Enough with Logs Alone

If a webhook is not retried or a retried request also fails to process, the history must be recorded. Simply logging is insufficient. Logs are scattered across different server instances and are difficult to search.

class WebhookFailureLog:
    event_id: str
    failure_reason: str
    retry_count: int
    last_attempt_at: datetime
    payload_snapshot: str  # for debugging

Keeping a snapshot of the failed webhook payload is not just for simple reprocessing. Sometimes manual adjustment is necessary. For example, if a payment system encounters an amount error but the retried request's payload has been corrected, a developer may need to directly modify and reprocess the request.

Without Failure Handling, Stability Is an Illusion

The stability of a webhook system is not judged by success rate alone. Failure scenarios and corresponding response plans must exist alongside it. Stripe provides a dashboard feature to manually retransmit after giving up, but this is the sender's responsibility, not the receiver's. The receiver must send notifications on retry abandonment or maintain a manual reprocessing interface.

# Failure threshold settings
MAX_RETRIES = 5
failure_counts = {}

def should_alert_or_retry(event_id):
    count = failure_counts.get(event_id, 0)
    if count >= MAX_RETRIES:
        notify_ops_team(event_id)
        return False
    return True

In the end, the core principle of webhook reception is to start from distrust. External requests are always unstable, retries are normal, and forgery attempts are possible. Trusting and processing all of these leads the system to instability at all times.

Verify request authenticity through signatures, suppress duplicates with idempotency keys, judge processing delays using retry time limits, prevent concurrency conflicts through atomic processing. And most importantly, webhooks become trustworthy only when a culture of recording failures and responding to them is established.

Comments

Loading comments.