← Back to documentation

Incident Response

Detect, limit, communicate, and recover from a delivery incident.

3 min read

Use this guide for a broad delivery problem or a problem that affects a critical endpoint.

Purpose#

Use this guide for a standard incident flow:

  • Detect and declare the incident.
  • Define the affected traffic.
  • Limit the impact.
  • Communicate updates.
  • Recover and review the incident.

Before you start#

  • Access to Activity, endpoint controls, and target controls.
  • On-call ownership for the affected endpoints.

Procedure#

1. Detect and declare#

If you see one of these patterns, declare an incident:

  • A sustained increase of DELIVERY_FAILED.
  • Widespread authentication rejections or validation rejections.
  • An interruption of a business-critical flow.

Record the start time of the incident and the affected endpoint IDs.

2. Scope impact#

  1. Filter Activity to the affected endpoints.
  2. Classify the scope:
  • One endpoint
  • One provider or one destination
  • More than one endpoint, or the complete system
  1. Estimate the request volume and the customer impact.

3. Contain quickly#

Select the containment action for the retry behavior of the sender:

  1. If you must keep the events, pause the endpoint first. A webhook request receives 503 Service Unavailable. An inbound email receives a temporary SMTP 451. Make the pause shorter than the retry window of the sender. If Retry-After is present, tell the webhook senders to use it. A paused attempt does not appear in Request activity.
  2. If one destination has a problem, correct that destination in place. You can also use its configured retries or its failover. The other destinations continue independently.
  3. If you intend to drop the traffic, disable the endpoint, regenerate its URL, or disable or remove a destination. Then coordinate the change. A disabled URL and an old URL return 404. An enabled endpoint with no enabled destination that matches can return 202 Accepted and create no delivery.
  4. Roll back the smallest recent configuration change that restores the known-good behavior.

Use Pause and Resume now beside the endpoint enable control for temporary containment. A pause depends on sender retries and on at-least-once delivery. A receiver must deduplicate a repeated event with a stable event key.

4. Communicate updates#

Share these items as a minimum:

  • The failure, with the outcomes and the endpoints
  • The mitigation in progress
  • The time of the next update

For external help, open Support with the timestamps, the endpoint IDs, and the error details.

5. Recover and use controlled traffic#

  1. Use controlled traffic to examine the fix.
  2. If the endpoint is paused, resume it.
  3. Make sure that the success outcomes return, and that the senders sent the temporarily rejected events again.
  4. Keep the monitoring at a high level until the traffic is stable.

6. Post-incident review#

Record these items:

  • The root cause
  • The detection gap
  • The recovery timeline
  • The preventive actions and their owners

Expected result#

  • The impact is limited.
  • Delivery health returns to the previous level.
  • Each follow-up action has an owner.

Common issues and fixes#

  • The scope is too broad: isolate one endpoint first.
  • The incident is closed too early: wait for healthy outcomes for a long period.
  • The evidence is weak: record the outcomes and the errors before you edit.
← Back to documentation