Retries
That a client retries 503 and 429, waits what Retry-After says, and stops after a fixed number of tries.
Logic - Testing
Free Chaos API Endpoint
Point a client at /chaos and it gets the failure you picked: a 503 with Retry-After, a 429, an answer that comes late, a proxy's HTML error page, an empty body or JSON labelled as text. Test retries, timeouts and parsing against the failures you otherwise meet only in production.
- No API key
- Any method
- 15 statuses
- Delays up to 10 s
- Nothing stored
Call the free Chaos API endpoint
Put the status you want in the path. The answer comes at once, with any method, and neither the body nor the query string is read, so the client under test can call the way it always does.
curl -s https://aisenseapi.com/services/v1/chaos/503{"error":"Service Unavailable","chaos":{"status":503,"delay_ms":0}}Add a delay in milliseconds as one more segment, up to 10000:
curl -s https://aisenseapi.com/services/v1/chaos/200/3000{"ok":true,"chaos":{"status":200,"delay_ms":3000}}The answer arrives after three seconds. A client with a shorter deadline should give up, which is what the test checks; curl reports 28 for a timeout:
curl -s --max-time 1 https://aisenseapi.com/services/v1/chaos/200/3000; echo "exit $?"Statuses
200, 201, 204, 400, 401, 403, 404, 409, 410, 422, 429, 500, 502, 503 and 504, each with the headers a real service would send with it:
| Status | Answer |
|---|---|
| 200, 201 | {"ok": true, "chaos": {...}} |
| 204 | No body |
| 400, 403, 404, 409, 410, 422 | {"error": "Not Found", "chaos": {...}}, with the status's own reason phrase |
| 401 | The same, with WWW-Authenticate: Bearer realm="chaos" |
| 429, 503 | The same, with Retry-After: 2 |
| 500, 502, 504 | The same |
curl -si https://aisenseapi.com/services/v1/chaos/429HTTP/2 429
retry-after: 2
content-type: application/json
x-chaos: 429
{"error":"Too Many Requests","chaos":{"status":429,"delay_ms":0}}Broken answers
Three answers that break a client which assumes everything is JSON. Each takes a delay the same way, for example /chaos/html/2000.
| Path | Answer | What goes wrong in a naive client |
|---|---|---|
/chaos/html | 502 with an HTML page, as a proxy answers when the service behind it is down | Parsing JSON fails on <html> |
/chaos/empty | 200 labelled application/json, with no body | "Unexpected end of JSON input" |
/chaos/wrongtype | 200 with valid JSON labelled text/plain | A client that trusts the type treats the answer as text |
curl -s https://aisenseapi.com/services/v1/chaos/wrongtype{"ok":true,"chaos":{"event":"wrongtype","delay_ms":0}}A chosen failure, or a real one
Every chosen answer carries X-Chaos with what the path asked for, such as 503 or 200/3000, and a chaos object in the JSON body. A real refusal never has either, and has fix instead.
A delay holds a place while it waits, at most four at a time from one address. When none is free the answer is a real 503 with Retry-After: 1 and no X-Chaos, and nothing waits. An answer without a delay needs no place.
A client test in Python, with nothing to install
import time, urllib.request, urllib.error
BASE = "https://aisenseapi.com/services/v1/chaos"
def get(url, tries=3, timeout=2):
for attempt in range(tries):
try:
with urllib.request.urlopen(url, timeout=timeout) as response:
return response.status
except urllib.error.HTTPError as error:
if error.code in (429, 503) and attempt < tries - 1:
time.sleep(int(error.headers.get("Retry-After", "1")))
continue
raise
start = time.time()
try:
get(BASE + "/503")
raise AssertionError("503 never reached the caller")
except urllib.error.HTTPError as error:
assert error.code == 503 and time.time() - start >= 4, "gave up too early"
try:
get(BASE + "/200/3000")
raise AssertionError("no timeout")
except (TimeoutError, urllib.error.URLError):
pass
print("client handles 503 and slow answers")It passes when the client tries three times with the two-second pause Retry-After asks for, and gives up on an answer slower than its deadline.
Try it
Errors
| Status | When |
|---|---|
| 404 | A path that is not one of the forms, or a status chaos does not answer with. The fix lists the statuses and events |
| 400 | A delay over 10000 milliseconds |
503 without X-Chaos | No place free for a delay, with Retry-After: 1 |
| 429 | The service-wide limit of 5000 requests per IP per 24 hours, which chaos calls count towards |
What to test with it
Timeouts
That a slow answer ends in a timeout the client handles, not a hung job.
Parsers
That an HTML error page, an empty body or a wrong content type gives a clear error, not a crash.
Agents and tools
That an agent reads an error, backs off and reports it, instead of trying the same thing forever.
Privacy and limits
Nothing is written or stored. The body and the query string are not read, but the access log line records the path and any query string, as it does for every request, so keep anything private out of the query. Delays run up to 10000 milliseconds, four at a time from one address, and every call counts towards the service-wide 5000 requests per IP per 24 hours.