Payment webhooks done right: signatures, idempotency, and the failures nobody warns you about

Webhook handlers have an unusual property: the broken ones often look fine. They return 200, the logs are quiet, and the bug only shows up as a customer complaining that they paid and nothing arrived — or as an accounting discrepancy nobody can explain.

Four failure modes account for most of it.

1. Parsing before verifying

The signature covers the exact bytes that were sent. If your framework parses JSON into an object and you then re-serialize it to verify, key order or whitespace can differ and the signature will never match. Worse, some teams 'fix' this by disabling verification.

The fix is to capture the raw body before any parsing middleware touches it.

import hmac, hashlib

@app.post("/webhooks/finality")
def finality_webhook():
    # request.data is the raw body. Do NOT use request.get_json() here.
    expected = hmac.new(
        WEBHOOK_SECRET.encode(), request.data, hashlib.sha256
    ).hexdigest()

    sig = request.headers.get("X-Finality-Signature", "")
    if not hmac.compare_digest(expected, sig):
        abort(401)

    evt = request.get_json()   # safe to parse now
    ...

2. Comparing signatures unsafely

A plain string comparison returns as soon as it finds a differing byte, which leaks how much of a guessed signature was correct. Use a constant-time comparison — every language has one.

In Node there is an extra trap: crypto.timingSafeEqual throws a RangeError when the two buffers have different lengths, rather than returning false. An attacker sending a short signature header can therefore crash the endpoint unless you compare lengths first.

3. No deduplication

Delivery is at-least-once, by design. Any system that retries on timeout will occasionally deliver twice — including when your server did the work but its 200 never made it back.

Use the event ID header as an idempotency key and record it in the same transaction as the side effect. Checking a set and then acting is not enough under concurrency; two workers can pass the check simultaneously.

BEGIN;
  -- Unique constraint on event_id makes the second delivery a no-op.
  INSERT INTO processed_webhooks (event_id) VALUES ($1)
  ON CONFLICT (event_id) DO NOTHING;

  -- Only fulfill if this insert actually happened.
  -- If ON CONFLICT fired, rows_affected is 0 and we skip the side effect.
COMMIT;

4. Fulfilling on the wrong event

This is the expensive one. Payment lifecycles emit several events, and only some of them mean the money is actually yours.

Branch explicitly on the event you want. A handler written as 'if the event starts with payment., fulfill' will eventually ship goods for an underpayment or a blocked transfer.

Testing all of this before launch

None of the four failure modes appear in a happy-path test. You need to actually deliver events at your handler, ideally without spending money to do it.

With a sandbox key you can create a payment and force it to settle, which fires a real webhook in the production format. That exercises signature verification, deduplication, and event routing end to end. Deliberately replay the same event to confirm your deduplication works — if the second delivery ships a second order, you found the bug before your customers did.