← Back to documentation

Observability

Read Activity charts, filter outcomes, and investigate endpoint problems.

8 min read

Use Request activity to find errors and to examine a fix.

Purpose#

Use this guide to:

  • Read the request volume trends.
  • Investigate the request outcomes and the error reasons.
  • Isolate a problem on one endpoint.

Before you start#

  • Signed-in access to Request activity.
  • One endpoint or more that receives traffic.

Procedure#

1. Open Request activity#

The page has two views:

  • Volume Chart
  • Request Log

The two views support the endpoint filter.

2. Analyze request volume#

In Volume Chart:

  1. Select the endpoint (All endpoints or one endpoint).
  2. Select the window (Month, Week, Day).
  3. Move to the previous window or to the next window.
  4. Read the total, average, and peak counts.

The chart labels and the summary values update when you change the endpoint or the time window.

3. Investigate request outcomes#

In Request Log:

  • Use the outcome filters (All, Completed, Pending, Errors, Test).
  • Read the Outcome column. Expand a row to see its error reason, the method of the attempt, the source IP, and the delivery details when they are available.
  • For a large data set, move through the pages.

Activity shows a readable label for each outcome. The machine code in parentheses appears in the API responses and in the CSV exports. You can also enter it in the Activity error search.

Current outcomes:

  • Completed (ACCEPTED): PayloadRelay accepted the HTTP event, and all the tracked deliveries completed with no error. An accepted event with no enabled destination completes and creates no delivery.
  • Auth Failed (AUTH_FAILED): the inbound authentication failed because of wrong credentials, an expired HMAC signature, or SPF, DKIM, or DMARC enforcement on an email endpoint.
  • Captcha Failed (CAPTCHA_FAILED): the Turnstile or reCAPTCHA token was missing or invalid.
  • CORS Blocked (CORS_BLOCKED): the endpoint CORS configuration does not permit the browser origin.
  • Delivery Failed (DELIVERY_FAILED): PayloadRelay accepted the event, all the tracked deliveries completed, and one destination or more failed.
  • Temporarily Unavailable (DELIVERY_QUEUE_UNAVAILABLE): a temporary service problem stopped the acceptance. Expand this metadata-only row to see the recorded reason, the method of the attempt, and the source IP. It has no delivery log. Send the original request again after the given delay.
  • Email Accepted (EMAIL_INGESTION_ACCEPTED): PayloadRelay accepted the email, and all the tracked deliveries completed with no error. The Completed filter includes this outcome.
  • Disabled (EMAIL_INGESTION_DISABLED): a historical outcome for an email sent to a disabled endpoint. Request activity does not show a current hit on a disabled endpoint.
  • Email Rejected (EMAIL_INGESTION_REJECTED): an email policy rule or configuration rule rejected the email. The causes are an endpoint that is not an email endpoint, or a body-as-payload format that PayloadRelay does not support. An empty body or an HTML-only body can also cause a rejection when the endpoint needs text/plain. An allowlist can reject a null SMTP sender. The errorReason field of the log entry contains the detail.
  • Failover Triggered (FAILOVER_TRIGGERED): a webhook destination used all its retries and activated its failover destination. It can appear as Pending (PENDING) while the failover delivery is in progress.
  • Validation Failed (FIELD_VALIDATION_FAILED): a payload field did not agree with a configured validation rule.
  • IP Blocked (IP_BLOCKED): the source IP is not on the endpoint IP allowlist.
  • Malformed Form Payload (MALFORMED_FORM_PAYLOAD): PayloadRelay did not parse the application/x-www-form-urlencoded body.
  • Malformed JSON Payload (MALFORMED_JSON_PAYLOAD): the application/json body was not valid JSON.
  • Malformed XML Payload (MALFORMED_XML_PAYLOAD): the application/xml or text/xml body was not valid XML. The error reason is Malformed XML body.
  • Malformed Query Param (MALFORMED_QUERY_PARAM): the query parameter parsing failed, for example, because of invalid Base64 in the query parameter mode.
  • Missing Headers (MISSING_REQUIRED_HEADERS): a necessary endpoint header is missing or has the wrong value.
  • Method Not Allowed (METHOD_NOT_ALLOWED): the incoming HTTP method does not match the endpoint method.
  • Paused (PAUSED): a historical outcome for a hit on a paused endpoint. Request activity does not show a current hit on a paused endpoint.
  • Pending (PENDING): PayloadRelay accepted the event, and one regular delivery or failover delivery is in progress. The row moves to a final outcome after all the deliveries complete.
  • Payload Too Large (PAYLOAD_TOO_LARGE): the event is larger than the plan size limit. The limits are 256 KB for Sandbox and Solo, 512 KB for the Team trial and Team, and 1 MB for Scale. For a custom plan, the agreement of the organization sets the limit.
  • Quota Exceeded (QUOTA_EXCEEDED): the monthly accepted-event quota was empty when the event arrived.
  • Endpoint Rate Limited (RATE_LIMITED_ENDPOINT): the request count is more than the per-minute limit of the endpoint. An HTTP request receives 429 with Retry-After. An inbound email receives a temporary SMTP rejection, and the sender can send it again.
  • Body Filter Failed (REQUEST_BODY_FILTER_FAILED): the request body did not agree with a configured body filter.
  • Subscription Blocked (SUBSCRIPTION_BLOCKED): the subscription of the organization is inactive, expired, or blocked from relaying.
  • Test accepted (TEST_ACCEPTED), Test Rejected (TEST_REJECTED), and Test Rate Limited (TEST_RATE_LIMITED): the request-level outcomes of the relay-target Send test button. They are not real traffic. Expand the row for the delivery result of each target.
  • Unsupported Media Type (UNSUPPORTED_MEDIA_TYPE): the incoming Content-Type does not match the endpoint format.
  • Webhook Disabled (WEBHOOK_INGESTION_DISABLED): a webhook request arrived at an enabled endpoint with the webhook ingestion disabled. Request activity does not show a hit on a disabled endpoint.

An email-sourced relay uses the same detailed outcomes when it is possible. A body filter records REQUEST_BODY_FILTER_FAILED. A field validation records FIELD_VALIDATION_FAILED. SPF, DKIM, DMARC, and a sender that is not empty and not permitted record AUTH_FAILED. A different email configuration rejection or policy rejection records EMAIL_INGESTION_REJECTED.

A request that PayloadRelay rejects only because the endpoint is disabled or paused creates no Request activity row.

Additional filters#

The request log has more filters to make the results narrow:

  • Outcome category: select All, Completed, Pending, Errors, or Test.
  • Source IP filter: enter an IP address to show the requests from that source IP.
  • Error search: enter text to filter the logs by error reason, outcome code, or outcome label. The search ignores the case.

The URL query string contains the complete filter state for a deep link.

CSV export#

The Export CSV button is in the header of the Request Log panel. Clear filters appears in the outcome-filter row when a filter is active. The export downloads the log entries that match the filters and the time window.

The exported columns, in order, are: timestamp_utc, endpoint_id, endpoint_name, outcome, http_method, source_ip, error_reason, request_id. The http_method value can be blank for a historical row, or for activity with no specific inbound HTTP method.

Row limit: the export contains a maximum of 100,000 rows. If the filters return more rows, PayloadRelay cuts the file and adds # truncated at 100000 rows; narrow your filters. To make the result smaller, use a shorter date range or fewer outcomes.

PayloadRelay can limit the exports temporarily during high-volume use. If the page shows a retry message, wait before you start a new export.

4. Drill down by endpoint#

  1. Filter to one endpoint.
  2. Compare the outcomes with the endpoint configuration, such as the method, the authentication, the captcha, the validation, and the destinations.
  3. Use the trend to make sure that the endpoint recovered after the change.

Expected result#

  • You can see the difference between an input rejection and a destination delivery error.
  • You can limit an incident to one endpoint, to one provider, or to a larger impact.

Data retention and privacy:

  • PayloadRelay keeps a metadata-only Activity trail and the delivery outcomes for 30 days.
  • PayloadRelay does not store a customer relay request body or a customer message body in its databases, activity logs, object storage, or backups. A body stays only in memory and in the delivery, retry, and dead-letter queues. It stays there only while PayloadRelay delivers the event. If the delivery fails permanently, the message stays in a dead-letter queue for a maximum of 7 days for diagnosis or a new delivery.
  • PayloadRelay can keep operational metadata in the 30-day Activity trail. This metadata contains the SMTP envelope sender addresses, the source IP addresses, the HTTP methods, the endpoint identifiers and the target identifiers, the timestamps, and the delivery outcomes. It never contains request-body content or message-body content.
  • The stored endpoint templates, the destination configuration, and the billing records are separate from the customer relay-event bodies.
  • The delivery is at-least-once. A successful acceptance response does not mean that every destination delivered the event. If PayloadRelay returns a temporary failure, send the original event again. Use a stable event key, so that a receiver can deduplicate a repeated delivery.

Common issues and fixes#

  • The traffic drops to zero: make sure that the sender integration and the endpoint URL are correct.
  • METHOD_NOT_ALLOWED increases: the sender method is different from the endpoint method.
  • PAYLOAD_TOO_LARGE increases: the sender payloads are larger than the endpoint limit for the current plan.
  • MALFORMED_FORM_PAYLOAD increases: the sender sends invalid application/x-www-form-urlencoded data.
  • MALFORMED_XML_PAYLOAD increases: the sender sends invalid XML, or a body that does not match the XML payload format of the endpoint.
  • DELIVERY_FAILED increases: examine the relay target authentication, the URL, the TLS, and the availability of the destination.
← Back to documentation