Recovering Lost Applications: Diagnosing a Memory Crash and Rebuilding the Missing Data
Role: Automations Engineer, Atrium Academy Program: Uniswap Hook Incubator (UHI) Stack: n8n, Typeform, Airtable, GitHub API, Node.js / bash, OpenSSL
Overview
A handful of developer applications were submitted but never made it into our database. My error monitoring flagged that something was failing, but the alerts were nearly empty, so pinning down what had broken (and which applications were lost) took real digging. The n8n workflow that processes applications was crashing partway through on certain submissions, and the crashes silently failed to write their data anywhere. I traced the cause, reconciled exactly which applications had been lost, rebuilt and replayed them through the real pipeline, and capped the resource usage that caused the crashes so it couldn't happen again.
What broke
Each UHI application comes in as a Typeform webhook, gets enriched with the applicant's GitHub activity, runs through LLM review, and lands in Airtable. The GitHub enrichment step was the problem: on developers with very active GitHub profiles, it pulled so much data that the n8n execution hit an out-of-memory error and crashed mid-run.
What made this tricky wasn't a lack of warning, it was a lack of detail. An error-handling workflow I'd built fired a Slack alert whenever an execution failed, so I knew something was breaking. But the alerts came through nearly empty: no usable error details, and n8n had no retrievable history for the failed runs to dig into. The most they hinted was that it might be an out-of-memory issue, with nothing concrete to confirm it. Meanwhile the crash happened after the webhook was received but before the record was written, so the affected submissions never landed in the database and the applicants had no idea. Diagnosing it meant working backward from those vague, detail-less alerts to figure out what was actually failing and which submissions had been lost.
Finding the missing records
I couldn't trust the database to tell me what it was missing, so I reconciled it against the source of truth: I exported the full set of Typeform responses and the unfiltered Airtable database and compared them.
A plain email comparison caught the obvious cases (first-time applicants whose email appeared in Typeform but not in Airtable). But that method has a trap: re-applicants. Someone who had applied before already has their email in the database, so an email-only check would wrongly assume their new submission landed. To catch those, I cross-referenced each Typeform submission's date against the Created Time of the newest matching Airtable record. If the existing record predated the new submission, that submission never processed, even though the email was already there. That step surfaced missed records a broad comparison would have passed right over.
Recovering them
Rather than re-key the lost applications by hand (error-prone, and it skips all the enrichment and scoring the pipeline does), I rebuilt them as real submissions and sent them back through the live workflow.
I reconstructed faithful Typeform webhook payloads for the missing records, using the real field IDs and structure of an actual Typeform payload so the workflow would parse them exactly as it parses live traffic. Then I wrote a small bash script (replay.sh) to POST each payload to the n8n webhook.
The catch was the signature. The webhook doesn't accept just any POST, it validates Typeform's signature: the Typeform-Signature: sha256=... header, which is an HMAC-SHA256 of the raw request body. So a replayed payload only gets through if it's signed exactly the way Typeform signs its live requests. The script computes that HMAC per request with OpenSSL over the exact bytes being sent, with the secret read from an environment variable rather than hard-coded. Each replayed application passed the same security gate a real submission does, then flowed through enrichment, scoring, and into Airtable the way it should have the first time. I ran them one at a time, starting with the lowest-stakes record, to confirm each landed cleanly before sending the rest.
Making sure it couldn't happen again
Recovering the data fixes the symptom. The crash was the disease, and its root cause was the GitHub enrichment pulling unbounded data on high-activity profiles. The fix was to cap it: limit the number of repositories and API calls per applicant, and order them by the most significant repositories (by stars and relevance) rather than just pulling the most recent activity. That keeps the enrichment inside a safe memory budget while still surfacing the signal that actually matters when evaluating a developer (their best work, not their latest commit). With the ceiling capped, the executions stopped crashing, and the silent data loss stopped at the source.
Impact
Confirmed the database was complete again. The real value wasn't the number of records recovered (a small handful). It was moving from "we think we have everyone, but a few may have slipped through" to a verified, reconciled 100%.
Recovered the applicants without manual data entry, and without skipping the enrichment and scoring every other applicant goes through, so the recovered records were first-class, not patched in by hand.
Closed the failure at its source. Capping the GitHub enrichment removed the crash that was dropping data, so the pipeline stopped losing submissions in the first place, rather than depending on me catching and reconstructing them after the fact.
Left a reusable recovery tool behind. The replay script lives on as a terminal tool I can point at any reconstructed payload, so if a gap ever shows up again, backfilling it is a quick, repeatable step rather than a fresh scramble.