Recovering Lost Applications:
Diagnosing a Memory Crash and Rebuilding the Missing Data
Role: Automations Engineer, Atrium Academy
Program: Uniswap Hook Incubator (UHI)
Stack: n8n, Typeform, Airtable, GitHub API, Node.js / bash, OpenSSL
Overview
A handful of developer applications were submitted but never made it into our database. My error monitoring flagged that something was failing, but the alerts were vague, so pinning down what had broken (and which applications were lost) took some digging. The n8n workflow that processes applications was crashing partway through on certain submissions, and the crashes silently failed to write their data anywhere. I traced the cause, reconciled exactly which applications had been lost, rebuilt and replayed them through the real pipeline, and capped the resource usage that caused the crashes so it couldn't happen again.
What broke
I have each UHI application come in as a Typeform webhook, enrich it with the applicant's GitHub activity, run it through an LLM review, then it lands in Airtable. The GitHub enrichment step was the problem: on developers with very active GitHub profiles, it pulled so much data that the n8n execution hit an out-of-memory error and crashed mid-run.
What made this tricky was the lack of detail. An error-handling workflow I'd built fired a Slack alert whenever an execution failed, so I knew something was breaking. But the alerts came through with limited error details, and n8n had no retrievable history for the failed runs to dig into. The most they hinted was that it might be an out-of-memory issue, with nothing concrete to confirm it. Meanwhile the crash happened after the webhook was received but before the record was written, so the affected submissions never landed in the database and the applicants had no idea. Diagnosing it meant working backward from those vague, detail-less alerts to figure out what was actually failing and which submissions had been lost.
One of Atriumbot's many tasks is to report me of errors in any of my workflows.
This error came in several times overnight and in the morning. While it did mention a possible OOM issue and which step it failed on, going to the workflow runs didn't actually return any data.
Finding the missing records
I couldn't trust the database to tell me what it was missing, so I reconciled it against the source of truth: I exported the full set of Typeform responses and the unfiltered Airtable database and compared them.
A plain email comparison caught the obvious cases (first-time applicants whose email appeared in Typeform but not in Airtable). But that method has a trap: re-applicants. Someone who had applied before already has their email in the database, so an email-only check would wrongly assume their new submission landed. To catch those, I cross-referenced each Typeform submission's date against the Created Time of the newest matching Airtable record. If the existing record predated the new submission, that submission never processed, even though the email was already there. That step surfaced missed records a broad comparison would have passed right over.
Recovering them
Rather than re-key the lost applications by hand (error-prone, and it skips all the enrichment and scoring the pipeline does), I rebuilt them as real submissions and sent them back through the live workflow.
I reconstructed faithful Typeform webhook payloads for the missing records, using the real field IDs and structure of an actual Typeform payload so the workflow would parse them exactly as it parses live traffic. Then I built a small bash script (replay.sh) to POST each payload to the n8n webhook.
The catch was the signature. The webhook doesn't accept just any POST, it validates Typeform's signature: the Typeform-Signature: sha256=... header, which is an HMAC-SHA256 of the raw request body. So a replayed payload only gets through if it's signed exactly the way Typeform signs its live requests. The script computes that HMAC per request with OpenSSL over the exact bytes being sent, with the secret read from an environment variable rather than hard-coded. Each replayed application passed the same security gate a real submission does, then flowed through enrichment, scoring, and into Airtable the way it should have the first time. I ran them one at a time, starting with the lowest-stakes record, to confirm each landed cleanly before sending the rest.
#!/usr/bin/env bash
set -euo pipefail
WEBHOOK_URL="${WEBHOOK_URL:-}" TYPEFORM_SECRET="${TYPEFORM_SECRET:-}"
cd "$(dirname "$0")"
sign() {
printf 'sha256=%s' "$(openssl dgst -sha256 -hmac "$TYPEFORM_SECRET" -binary "$1" | openssl base64 -A)"
}
send() {
local f="$1"
local headers=(-H "Content-Type: application/json")
if [[ -n "$TYPEFORM_SECRET" ]];
then headers+=(-H "Typeform-Signature: $(sign "$f")")
fi
echo "→ POST $f"
curl -sS -X POST "$WEBHOOK_URL" "${headers[@]}" \
--data-binary "@$f" \
-w "\n HTTP %{http_code} (%{time_total}s)\n"
echo
}
send "$1"
The script used to send my payload to n8n Typeform webhook.
To use you just needed to export both the Typeform secret, and the n8n webhook URL as variables.
While I could have just disabled the secret verification in the workflow itself, by building a script I could incorporate this into a tool for my colleagues.
Making sure it couldn't happen again
Recovering the data fixes the symptom. The crash was the disease, and its root cause was the GitHub enrichment pulling unbounded data on high-activity profiles. The fix was to cap it: limit the number of repositories and API calls per applicant, and sort by most recently pushed so the capped set is the applicant's most relevant, currently-active work rather than an arbitrary slice. That keeps the enrichment inside a safe memory budget while still surfacing the signal that actually matters when evaluating a developer (what they're building now). With the ceiling capped, the executions stopped crashing, and the silent data loss stopped at the source.
Through the Github API my sort options available were created, updated, pushed and full_name.
Of these, pushed was the most sensible option to limit to a collection of their most recent work.
Impact
Confirmed the database was complete again. The real value wasn't the number of records recovered (a small handful). It was moving from "we think we have everyone, but a few may have slipped through" to a verified, reconciled 100%.
Recovered the applicants without manual data entry, and without skipping the enrichment and scoring every other applicant goes through, so the recovered records were first-class, not patched in by hand.
Closed the failure at its source. Capping the GitHub enrichment removed the crash that was dropping data, so the pipeline stopped losing submissions in the first place, rather than depending on me catching and reconstructing them after the fact.
Left a reusable recovery tool behind. The replay script lives on as a terminal tool I can point at any reconstructed payload, so if a gap ever shows up again, backfilling it is a quick, repeatable step rather than a fresh scramble.