Do interviews earn their keep?
A randomized admissions A/B experiment
with automated rails to keep thing moving
Role: Automations Engineer, Atrium Academy
Client: Atrium Academy + Uniswap
Scale: 108 developers in 1 cohort over 3 months
Stack: n8n, Airtable, Typeform, Brevo, Claude Haiku
Overview
Every UHI applicant used to go through the same steps: apply, get reviewed, then sit in a 20 minute interview before we'd hand out a spot. The interview felt important (it caught the people who looked good on paper but didn't actually fit the technical requirements), but it was the most time consuming step we ran, and it was where the most people dropped off.
Our application review process had gotten so good that most individuals who made it to an interview would end up accepted. It felt silly to continue wasting both applicants time and our employees time with an interview stage just because it was a "sacred cow" in our established admissions processes.
I wanted to run a direct admit A/B test as as an experiment to see what happened if we removed interviews entirely.
The setup
Two tracks: Direct Admit got an offer straight out of a positive application review with no interview, and Interview Track went through the old flow, interview and all. The interesting part wasn't the split itself, it was building it so nobody involved had to change how they worked.
Track assignment ran on its own. When an application came in, an Airtable script counted how many records were already on each track across the whole pool and dropped the new one onto whichever track was behind (random tiebreak when they were even). That holds the split at worst ±1 without me watching it, and since it counts the whole pool instead of backfilling rejected slots, a bad run on one track can't snowball into more people getting sent there. 108 applicants got assigned this way, 54 to each arm. After pulling out test records and duplicates I had 52 Direct Admit and 48 Interview Track to measure outcomes on.
// Guard: only run if track not already assigned
let record = await table.selectRecordAsync(recordId, { fields: [trackField] });
if (record.getCellValue(trackField)) {
console.log("Track already assigned, skipping.");
output.set("track", null);
return;
}
// Count existing assignments across active records (excluding pre-review rejections)
let query = await table.selectRecordsAsync({ fields: [trackField, statusField] });
let activeRecords = query.records.filter(
r => r.getCellValue(statusField)?.name !== "Rejected"
);
let directAdmitCount = activeRecords.filter(
r => r.getCellValue(trackFieldt)?.name === "Direct Admit"
).length;
let interviewCount = activeRecords.filter(
r => r.getCellValue(trackField)?.name === "Interview Track"
).length;
// Assign to whichever has fewer; coin flip on tie
let track;
if (directAdmitCount < interviewCount) {
track = "Direct Admit";
} else if (interviewCount < directAdmitCount) {
track = "Interview Track";
} else {
track = Math.random() < 0.5 ? "Direct Admit" : "Interview Track";
}
Javascript that handles the Airtable sorting via an Airtable automation
From there the application ran through the review pipeline I'd already built: Typeform webhook into n8n, GitHub enrichment, then a Claude pass that scores the applicant, tags a persona, and writes a recommendation. A human still makes the actual call off that recommendation, move forward or reject. The one thing the reviewer couldn't see was the track, hidden from them on purpose.
The reason is a bias that shows up the second you make the track visible. A reviewer looks at a borderline application and thinks "I'll move them forward if they're interview track since the interview will catch it, but not if they're direct admit." Reasonable instinct, but it would have wrecked the experiment, because the two tracks would stop getting the same decisions, skewing the data. So the track stayed invisible and every application got judged on its own merits.
The rest was making the two paths diverge underneath the reviewer while they kept working normally. They batch-move applicants through the same statuses as always (to review, to write personalized line, to send interview). For anyone flagged Direct Admit, an Airtable automation catches them at the personalized-line stage and jumps them straight past the entire interview sub-flow to the acceptance-prep stage. The reviewer never has to know it happened.
That bypass exists because the personalized line is genuinely different between the two. The interview-track line is an invitation to interview ("I'd love to hear more about X"), the direct-admit line is an invitation to enroll ("this part of your application made you a fit"). You can't swap one for the other. So direct admits skip the manually-written interview line and gets one generated by Claude Haiku instead, which pulls the strongest signal from their application (theme-aligned hook idea, then shipped projects, then technical depth, on down a priority list) and attaches it to their record in the enrollment format.
At that point the two tracks re-merge. A direct admit has skipped the interview and the interview-flavored line and arrives at the acceptance stage with their own enrollment line, ready to send like anyone else. The final split is the acceptance email itself: Brevo logic reads the track off the record and routes direct admits down their own branch, so the email they get matches the line they were given.
Skipping the interview also meant skipping everything it did beyond vetting: it was where we got people excited and walked them through the practical stuff (the build theme, when workshops run, timezone, where it's hosted). So the direct admit acceptance email is built bigger on purpose, an expanded version that doubles as an onboarding guide, FAQs and everything that conversation used to cover, delivered in writing instead.
So in the end, the two parties never had to adjust: the application reviewer, who works exactly as before, and the applicant, who never sees the difference. The automations handle the sorting.
The full flow overview in its entirety. Direct Admit bypasses interview stage, and has a separate email.
The Results
Direct Admit roughly doubled enrollment. 61.5% of Direct Admit applicants confirmed a spot, versus 31.2% on the Interview Track. Some of that gap is baked in (skipping the interview removes a place to drop off), but it's still big, the interview was losing us about half the people who'd otherwise have enrolled.
It also resolved twice as fast. Direct Admit went from application to confirmed enrollment in a median of 6 days (mean 8.6). Interview Track took a median of 16 (mean 15.0). Scheduling, sitting, and reviewing an interview adds roughly an extra week per person.
The interviews cost time, and less than you'd think. Of the 54 Interview Track applicants, only 19 actually got interviewed (the rest were filtered at review or dropped before they scheduled), plus one no-show that still took up a slot. That's about 6.7 hours of interviewer time for this one arm. Interviewing every eligible applicant the old way would have run closer to 18, so it compounds fast as the funnel grows. Direct Admit spent zero.
But (the honest catch) the interview stage did increase individual engagement. Among people who enrolled, Interview Track attendees showed up to more of the first four workshops: 65% attendance per head versus 44% for Direct Admit, and only 1 of 15 Interview Track enrollees had skipped everything so far versus 9 of 32 on Direct Admit. That being said, thanks to the higher enrollment numbers the direct admit track was able to pull, Direct Admit still had more overall attendances.
So which one wins?
Here's the thing that made the decision easy: even though Direct Admit enrollees are less engaged per head, there are twice as many of them, so the track still pulls more total attendance overall (about 56 workshop-attendances versus 39).
Running it as a 50/50 split instead of flipping everything at once did two things for us. It hedged this cohort's engagement (half still came through the interview, so the cohort didn't take the full per-head dip while we tested), and it surfaced the one real drawback with real data before we committed. That means we head into full direct admit already knowing the weak spot, and future cohorts can work on closing that gap instead of taking the hit blind.
So we're dropping the interview for good. Every applicant goes direct admit from here, which means the whole admissions flow runs automated end to end, no scheduling, no interview slots, no post-interview review. We trade a bit of per-head engagement for more throughput and a lot less time, and now that the tradeoff has a number on it we can chip away at going forward.