Error Handling

This guide covers how to classify errors from the SurfacedBy Data API, when to retry, and how to retry writes safely.

Every error from the Data API is an RFC 7807 problem document served as application/problem+json. The body carries type, title, status, detail, and request_id; the same request id is echoed in the X-Request-Id response header. See Errors for the document shape and the slug catalogue.

Classify errors first

The fragment of the type URL (for example https://docs.surfacedby.com/v1/api/errors#rate_limited) is the stable switch point. Combine it with the HTTP status:

A status without a specific slug falls back to http_<status> (for example #http_404).

Respect Retry-After

Every 429 carries a Retry-After header with the number of seconds to wait, and the problem body's limit_window field says which window you hit (per-minute or per-month). Successful responses carry X-RateLimit-Limit-Minute, X-RateLimit-Remaining-Minute, X-RateLimit-Limit-Month, and X-RateLimit-Remaining-Month, so you can slow down before you reach a limit.

A per-minute limit resets within the minute. A per-month limit resets at the start of the next UTC calendar month, but its Retry-After is a flat 3600 seconds rather than the time until the reset. When limit_window is per-month, stop and alert instead of retrying. Requests sent while you are over the limit still count against the window. See Rate limits for the per-plan quotas.

import random
import time
 
import httpx
 
RETRYABLE = {409, 429, 500, 502, 503, 504}  # codegen-ignore: literal statuses
MAX_WAIT_SECONDS = 90
 
def call_with_retry(method, url, headers, json=None, max_attempts=5):
    for attempt in range(max_attempts):
        resp = httpx.request(method, url, headers=headers, json=json)
        if resp.status_code not in RETRYABLE:
            return resp
        if resp.status_code == 409 and not resp.json().get("type", "").endswith("#idempotency_in_progress"):
            return resp  # idempotency_conflict: retrying will not help
        retry_after = resp.headers.get("Retry-After")
        delay = int(retry_after) if retry_after else min(2 ** attempt, 30) + random.random()
        if resp.status_code == 429 and resp.json().get("limit_window") == "per-month":
            return resp  # monthly quota: surface it instead of retrying
        if delay > MAX_WAIT_SECONDS:
            return resp
        time.sleep(delay)
    return resp

Exponential backoff for 5xx

Retry 5xx responses with exponential backoff, not a tight loop: for example 1, 2, 4, 8, 16 seconds, capped at around 30 seconds, with a small random jitter so many clients do not retry in lockstep. A 503 caused by a slow upstream dependency also carries X-Data-Degraded: true.

Stop after a bounded number of attempts (5 is reasonable). Persistent 5xx errors indicate a real problem; retrying forever hides it. Keep the request_id from each failed response so support can trace it.

Idempotency for writes

Write requests (POST and DELETE) require an Idempotency-Key header; without one the API returns 400 idempotency_required. Set the key to a UUID you generate once per logical operation, and reuse the same key and the same body on every retry of that operation.

curl -X POST https://api.surfacedby.com/api/v1/ext/domains \
  -H "X-API-Key: sk_live_..." \
  -H "Idempotency-Key: 9f8e2b1a-0c4d-4a7e-8b3f-5d1e2a3c6d7f" \
  -H "Content-Type: application/json" \
  -d '{"domain": "yourbrand.com"}'

• If the original request completed, a retry with the same key and body returns the stored response with X-Idempotent-Replay: true instead of running the write again.
• If the original is still running, the retry gets 409 idempotency_in_progress with Retry-After: 5.
• If you reuse a key with a different body while the original succeeded or is still running, you get 409 idempotency_conflict.
• If the original failed, its key is released, so a corrected retry with the same key is processed as a new request.

See Idempotency for details.

Safe retries checklist

• Switch on the type slug and the status. Retry only 429, 5xx, and 409 idempotency_in_progress.
• Honour Retry-After, and do not keep retrying against a monthly quota.
• Use exponential backoff with jitter for 5xx.
• Cap retry attempts.
• Send an Idempotency-Key on every write and keep it stable across retries.
• Log the request_id (body or X-Request-Id header) so you can correlate retries.