API Reliability and Rate Limit Troubleshooting
An API integration between two core tools kept failing intermittently. Records went missing under load, usually during the busiest hours, and nobody could say why.
Industry
Direct-to-consumer e-commerce brand (anonymized)
Role
Systems consultant — API diagnostics, integration rebuild, monitoring and documentation
Impact
Records stopped silently disappearing. The data flow now survives traffic spikes, and when something breaks the team hears about it in minutes instead of weeks later at reconciliation.
01 - Problem
The problem
An API integration moved records between the two systems this business ran on. Most days it worked, but every few weeks records simply failed to arrive on the other side — always during peak periods, when missing data hurts most. There was no error, no failed-run alert, nothing in the logs anyone watched. They only found the gaps when a customer or a monthly report surfaced a record that should have existed. By the time I was brought in, they'd stopped trusting the integration and were spot-checking transfers by hand.
The root cause is one I see constantly in workflow automation: the original build assumed the API would always say yes. Under load, the destination API started returning rate-limit responses and the odd server error, and the integration treated anything short of a crash as success. Records weren't vanishing mysteriously — they were being rejected politely, and the automation shrugged and moved on. Not a tooling problem, a structure problem: the integration had no concept of failure, so it could never recover from one.
02 - System Flow
How the system moves
Record event fires
Webhook from source system
Queued, not pushed
Make data store as queue
Batch picked up
Scheduled Make scenario
API call sent
REST API, rate-aware batch size
Response checked
Status code and body validation
Retry on failure
Backoff via error handlers
Alert if stuck
Failure route notifies team
Record confirmed delivered
Status logged for reconciliation
03 - Build
What I built
Diagnosis in Postman before touching anything
I needed proof of what was failing before rebuilding. In Postman I replayed the integration's calls against the destination REST API, one at a time and then in rapid bursts mimicking peak load. The pattern showed fast: past a certain request rate the API returned rate-limit responses, and the existing scenario had no handling for them — it logged the run complete and dropped the record. I documented the real limits and every error shape so the rebuild ran against observed behavior, not the docs' happy path.
A queue-first rebuild in Make
Instead of firing a call the instant a webhook arrived, every incoming record now lands in a Make data store acting as a queue. A scheduled scenario drains it in batches sized to stay under the rate limit, validates each response — status code and body, since this API sometimes returned a 200 with an error inside — and uses Make's error handlers to retry failed calls with increasing delays. A record only leaves the queue once the API confirms it. Spikes just make the queue deeper, not empty.
Alerting, logging, and a handover runbook
Silent failure was the real enemy. Any record that exhausts its retries stays flagged in the queue, and a webhook alerts the team channel with the record ID and last error. Every attempt is logged with its outcome, turning month-end reconciliation into a lookup instead of an investigation. I closed with a short runbook: what each alert means, how to requeue a flagged record, and how to adjust batch size if the API's limits change — so the team runs this integration without me on call.
04 - Impact
Business impact
Records stopped silently disappearing — each one is now either confirmed delivered or visibly flagged, so 'missing data' went from a recurring mystery to a non-event.
Traffic spikes no longer break the data flow; the queue absorbs the burst and drains within the API's limits instead of losing the busiest hour's records.
The team retired manual spot-checking — what was a nervous afternoon cross-checking two systems is now a few minutes reviewing the delivery log.
Failures surface in minutes via alerts instead of weeks later at reconciliation, so problems get fixed while the context is still fresh.
05 - Edge Cases
What had to be handled
Sustained spikes beyond normal volume: the queue holds records rather than dropping them, and an alert fires if queue depth keeps growing instead of draining.
The API occasionally returned a 200 with an error in the body, so the scenario validates the body as well as the status code before marking a record delivered.
Duplicate webhook deliveries are caught by checking the record ID against the queue and delivery log, so retries never create downstream duplicates.
During an extended API outage, retries stop escalating at a ceiling and the queue holds everything until service returns — with an alert saying it's in holding mode, not failing.
06 - Improvements
What I would improve next
Add a lightweight reporting dashboard showing queue depth, retry counts, and error trends, so the team spots the API straining before it becomes an incident.
Move high-volume operations to the platform's bulk endpoints where available, cutting total request count instead of just pacing it.
Schedule a periodic reconciliation job that compares record counts between the two systems and flags any drift automatically — a safety net behind the safety net.
Related capabilities