← Back to documentation

Observability

Read Activity charts, filter outcomes, and investigate endpoint problems.

11 min read

Use Request activity to find errors and to examine a fix.

If one endpoint or more has failed requests in the last 7 UTC days, today included, the Home page shows the card Endpoints with failed requests. The card lists a maximum of 5 endpoints, with the most failed requests first. Select an endpoint to open its failed requests in the Request log.

Purpose#

Use this guide to:

  • Read the request volume trends.
  • Investigate the request outcomes and the error reasons.
  • Isolate a problem on one endpoint.

Before you start#

  • Signed-in access to Request activity.
  • One endpoint or more that receives traffic.

Procedure#

1. Open Request activity#

The page has two views:

  • Volume chart
  • Request log

The two views support the endpoint filter. If two endpoints have the same name, the filter adds the start of the endpoint ID to each of the names.

2. Analyze request volume#

In Volume chart:

  1. Select the endpoint (All endpoints or one endpoint).
  2. Select the window (Month, Week, Day).
  3. To move to the previous window or to the next window, select Back or Next.
  4. Read the Total, Errors, Average per hour or Average per day, and Peak values.

The Volume chart view with the endpoint filter, the window, the summary values, and the daily request bars

The chart labels and the summary values update when you change the endpoint or the time window. The vertical axis shows the request count. The Day window shows Average per hour. The Week and Month windows show Average per day.

If a request in the window failed, the top part of its bar is red. The red part shows the share of the failed requests in that bar. The legend under the chart names the two colors No error and Error. To read the counts of one bar, move the pointer onto the bar.

Each bar of the Day window is one UTC hour. Each bar of the Week and Month windows is one UTC day. The last bar of the current window is the current hour or the current day, so its count can still increase. The current window moves forward with the time. Back and Next move the window by its full length, so two adjacent windows have no gap and no overlap. PayloadRelay keeps the activity for 30 days, so Back is not available when the previous window is older than that. The page then shows this reason under the button.

The log can show a group of repeated errors as one row with a count, for example ×3. The chart adds all the occurrences of that group to the bar of its first occurrence.

3. Investigate request outcomes#

In Request log:

  • Use the outcome filters (All, Completed, Pending, Errors, Test).
  • Read the Outcome column. A closed row shows a short preview of the error. Expand a row to see its full error reason, the method of the attempt, the Originating IP, and the delivery details when they are available. While a request is Pending, the page refreshes the log, and an expanded row loads its delivery details again.
  • For a large data set, move through the pages.

The Request log view with the outcome filters and the request rows

Activity shows a readable label for each outcome. The machine code in parentheses appears in the API responses and in the CSV exports. You can also enter it in the Error contains filter of Activity.

Current outcomes:

  • Completed (ACCEPTED): PayloadRelay accepted the HTTP event, and all the tracked deliveries completed with no error. An accepted event with no enabled destination completes and creates no delivery.
  • Auth Failed (AUTH_FAILED): the inbound authentication failed because of wrong credentials, an expired HMAC signature, or SPF, DKIM, or DMARC enforcement on an email endpoint.
  • Captcha Failed (CAPTCHA_FAILED): the Turnstile or reCAPTCHA token was missing or invalid.
  • CORS Blocked (CORS_BLOCKED): the endpoint CORS configuration does not permit the browser origin.
  • Delivery Failed (DELIVERY_FAILED): PayloadRelay accepted the event, all the tracked deliveries completed, and one destination or more failed.
  • Temporarily Unavailable (DELIVERY_QUEUE_UNAVAILABLE): a temporary service problem stopped the acceptance. Expand this metadata-only row to see the recorded reason, the method of the attempt, and the source IP. It has no delivery log. Send the original request again after the given delay.
  • Email Accepted (EMAIL_INGESTION_ACCEPTED): PayloadRelay accepted the email, and all the tracked deliveries completed with no error. The Completed filter includes this outcome.
  • Email Ingestion Disabled (EMAIL_INGESTION_DISABLED): a historical outcome for an email sent to a disabled endpoint. Request activity does not show a current hit on a disabled endpoint.
  • Email Rejected (EMAIL_INGESTION_REJECTED): an email policy rule or configuration rule rejected the email. The causes are an endpoint that is not an email endpoint, or a body-as-payload format that PayloadRelay does not support. An empty body or an HTML-only body can also cause a rejection when the endpoint needs text/plain. An allowlist can reject a null SMTP sender. The errorReason field of the log entry contains the detail.
  • Failover Triggered (FAILOVER_TRIGGERED): a webhook destination used all its retries and activated its failover destination. It can appear as Pending (PENDING) while the failover delivery is in progress.
  • Validation Failed (FIELD_VALIDATION_FAILED): a payload field did not agree with a configured validation rule.
  • IP Blocked (IP_BLOCKED): the source IP is not on the endpoint IP allowlist, or the Allowed IP version of the endpoint does not permit its IP version.
  • Malformed Form Payload (MALFORMED_FORM_PAYLOAD): PayloadRelay did not parse the application/x-www-form-urlencoded body.
  • Malformed JSON Payload (MALFORMED_JSON_PAYLOAD): the application/json body was not valid JSON.
  • Malformed XML Payload (MALFORMED_XML_PAYLOAD): the application/xml or text/xml body was not valid XML, or it had more than 100 nested element levels. The error reason is Malformed XML body.
  • Malformed Query Param (MALFORMED_QUERY_PARAM): the query parameter parsing failed, for example, because of invalid Base64 in the query parameter mode.
  • Missing Headers (MISSING_REQUIRED_HEADERS): a necessary endpoint header is missing or has the wrong value.
  • Method Not Allowed (METHOD_NOT_ALLOWED): the incoming HTTP method does not match the endpoint method.
  • Paused (PAUSED): a historical outcome for a hit on a paused endpoint. Request activity does not show a current hit on a paused endpoint.
  • Pending (PENDING): PayloadRelay accepted the event, and one regular delivery or failover delivery is in progress. The row moves to a final outcome after all the deliveries complete.
  • Payload Too Large (PAYLOAD_TOO_LARGE): the event is larger than the plan size limit, or the transform rules made it larger than this limit. The limit is 256 KB for Sandbox and Solo, and 512 KB for the Team trial and Team. Scale and a custom plan have a limit of 1 MB.
  • Quota Exceeded (QUOTA_EXCEEDED): the monthly accepted-event quota was empty when the event arrived.
  • Endpoint Rate Limited (RATE_LIMITED_ENDPOINT): the request count is more than the per-minute limit of the endpoint. An HTTP request receives 429 with Retry-After. An inbound email receives a temporary SMTP rejection, and the sender can send it again.
  • Body Filter Failed (REQUEST_BODY_FILTER_FAILED): the request body did not agree with a configured body filter.
  • Subscription Blocked (SUBSCRIPTION_BLOCKED): the payment of the organization is past due, and the recovery period ended. Activity also records this outcome for a closed organization that waits for deletion.
  • Test Accepted (TEST_ACCEPTED), Test Rejected (TEST_REJECTED), and Test Rate Limited (TEST_RATE_LIMITED): the request-level outcomes of the relay-target Send test delivery and Test reachability buttons. They are not real traffic. Expand the row for the delivery result of each target.
  • Unsupported Media Type (UNSUPPORTED_MEDIA_TYPE): the incoming Content-Type does not match the endpoint format.
  • Unsupported Selector (UNSUPPORTED_SELECTOR): a stored XML selector of the endpoint is outside the XPath subset for XML. The error reason names the rule type. Update that selector in the endpoint configuration. This outcome does not count toward auto-pause.
  • Webhook Ingestion Disabled (WEBHOOK_INGESTION_DISABLED): a webhook request arrived at an enabled endpoint with the webhook ingestion disabled. Request activity does not show a hit on a disabled endpoint.

An email-sourced relay uses the same detailed outcomes when it is possible. A body filter records REQUEST_BODY_FILTER_FAILED. A field validation records FIELD_VALIDATION_FAILED. SPF, DKIM, DMARC, and a sender that is not empty and not permitted record AUTH_FAILED. A different email configuration rejection or policy rejection records EMAIL_INGESTION_REJECTED.

A request that PayloadRelay rejects only because the endpoint is disabled or paused creates no Request activity row.

Additional filters#

The request log has more filters to make the results narrow:

  • Outcome category: select All, Completed, Pending, Errors, or Test.
  • Source IP: enter an IP address to show the requests from that source IP.
  • Error contains: enter text to filter the logs by error reason, outcome code, or outcome label. The search ignores the case. The search reads a maximum of the 10,000 newest requests in the time window. If the window has more requests, the log shows a message. To search all the requests, select a shorter window.

The URL query string contains the complete filter state and the selected tab (tab=log for Request log) for a deep link. If you select a different endpoint, the filters stay and the log goes back to the first page.

The outcome counts and the volume chart do not use the Source IP filter and the Error contains filter. When one of these two filters is set, the outcome filters show no counts, and a note under the filters gives the reason.

CSV export#

The Export CSV button is in the header of the Request log panel. Clear filters appears in the outcome-filter row when a filter is active. The export downloads the log entries that match the filters and the time window.

The exported columns, in order, are: timestamp_utc, endpoint_id, endpoint_name, outcome, http_method, source_ip, error_reason, request_id. The http_method value can be blank for a historical row, or for activity with no specific inbound HTTP method.

Row limit: the export contains a maximum of 100,000 rows. If the filters return more rows, PayloadRelay cuts the file and adds the line # truncated at 100000 rows. Narrow your filters.. To make the result smaller, use a shorter date range or fewer outcomes.

PayloadRelay can limit the exports temporarily during high-volume use. If the page shows a retry message, wait before you start a new export.

4. Drill down by endpoint#

  1. Filter to one endpoint. In the All endpoints view, you can also select Show only this endpoint in the expanded row. To open the endpoint details, select the endpoint name in the row.
  2. Compare the outcomes with the endpoint configuration, such as the method, the authentication, the captcha, the validation, and the destinations.
  3. Use the trend to make sure that the endpoint recovered after the change.

Expected result#

  • You can see the difference between an input rejection and a destination delivery error.
  • You can limit an incident to one endpoint, to one provider, or to a larger impact.

Data retention and privacy:

  • PayloadRelay keeps a metadata-only Activity trail and the delivery outcomes for 30 days.
  • PayloadRelay does not store a customer relay request body or a customer message body in its databases, activity logs, object storage, or backups. A body stays only in memory and in the delivery, retry, and dead-letter queues. It stays there only while PayloadRelay delivers the event. If the delivery fails permanently, the message stays in a dead-letter queue for a maximum of 7 days for diagnosis or a new delivery.
  • PayloadRelay can keep operational metadata in the 30-day Activity trail. This metadata contains the SMTP envelope sender addresses, the source IP addresses, the HTTP methods, the endpoint identifiers and the target identifiers, the timestamps, and the delivery outcomes. It never contains request-body content or message-body content.
  • The stored endpoint templates, the destination configuration, and the billing records are separate from the customer relay-event bodies.
  • The delivery is at-least-once. A successful acceptance response does not mean that every destination delivered the event. If PayloadRelay returns a temporary failure, send the original event again. Use a stable event key, so that a receiver can deduplicate a repeated delivery.

Common issues and fixes#

  • The traffic drops to zero: make sure that the sender integration and the endpoint URL are correct.
  • METHOD_NOT_ALLOWED increases: the sender method is different from the endpoint method.
  • PAYLOAD_TOO_LARGE increases: the sender payloads are larger than the endpoint limit for the current plan.
  • MALFORMED_FORM_PAYLOAD increases: the sender sends invalid application/x-www-form-urlencoded data.
  • MALFORMED_XML_PAYLOAD increases: the sender sends invalid XML, or a body that does not match the XML payload format of the endpoint.
  • DELIVERY_FAILED increases: examine the relay target authentication, the URL, the TLS, and the availability of the destination.