Rate limits & retries
A 429 (Rate limit exceeded) means you hit a rate or concurrency limit — shed load and retry as below.
Recommendations
- Lower concurrency and retry frequency; don't hammer one model
- Exponential backoff (1s → 2s → 4s) with random jitter so retries don't align
- Reduce simultaneous long-lived streaming connections
Backoff retry example
python
import random, time
for attempt in range(5):
resp = call_api() # your request function
if resp.status_code != 429:
break
delay = 2 ** attempt + random.random() # 1s -> 2s -> 4s... + jitter
time.sleep(delay)