12 calibrated tracks — React to .NET, game engines to DevOps
Arbiter screens engineering candidates in live Firecracker sandboxes — real systems, genuinely broken — and grades the work by running it: SQL counted at the driver, heap and DOM read over Chrome DevTools Protocol, images torn open layer by layer on a real Docker daemon, GC pauses pulled from V8 itself. Never by parsing the code. Forty applicants become three interviews, and your senior engineers only ever talk to people the evidence already backs.
From $399/month · Invoiced, not charged · Answered in a working day
No webcams · No lockdown browser · No keystroke logging — ever
Real defects from the assessment library, replayed. Candidates meet them live, in a real sandbox — and a person reads how they worked, not just whether it passed. Run one yourself, live →
The case
A first-round technical screen costs an hour of someone who could be shipping, plus the context switch on either side of it. Most of that hour confirms what a well-built exercise would have told you before anyone joined a call.
So the funnel filters progressively. Everyone gets a ten-minute fundamentals task, auto-graded, no human in the loop. Only the people who pass reach a deep dive, and only those reach your tech lead — who opens one view holding the code, the grader report and the recruiter’s notes.
Worked example
Engineering hours returned
~30/month
26 of 40 filtered before a tech lead opens anything; 14 deep dives × 75 min ≈ 32 hours returned. Substitute your own numbers — the shape is the point, not our arithmetic. At $399 a month, break-even is under three of those hours at a loaded senior rate; the other twenty-nine are the product.
The pipeline
Every track in the catalogue runs the same shape of funnel with different tasks in it. The point is not the tasks — it is that nobody expensive looks at anybody until something cheaper has already said they are worth it.
Everything starts from a position. Your recruiter opens one, adds candidates into it, reviews fit with the CV and notes attached, and dispatches the assessment from there. They never see code — that boundary is a database policy, not a hidden button.
No engineering time
Routing reads the CV and the role: stack, seniority, domain. Every underlying challenge was proven against a flawed baseline and a reference solution before any candidate saw it — the wording adapts to the person, the bar never moves.
Stack · profile · seniority
A ten-minute screen filters the field; survivors get a 45–55 minute deep dive in a genuinely broken repo. A hidden suite scores correctness; the weighted rubric scores the craft — what they fixed, and how.
Correctness + craftsmanship
After submitting, the candidate can write a short debrief — next steps, doubts, where AI helped and how they checked it. A model reads the diff and those words, and drafts questions for your tech lead, anchored to file and line. Never a score, never a recommendation.
From the real diff
The diff, the grader report, the guide and the recruiter’s notes in one view. Your lead reads, asks, and makes the call — no candidate is ever subject to a decision made solely by software.
GDPR Art. 22 · human decision
The catalogue
12 calibrated tracks, 160 ready-to-dispatch challenges — every one proven against a deliberately flawed baseline and a reference solution before any candidate sees it. No guesswork, no “contact us to find out”: if it is not on this list, we do not sell it as ready.
5 junior5 mid6 seniorcalibrated
4 junior5 mid4 seniorcalibrated
4 junior4 mid4 seniorcalibrated
4 junior5 mid5 seniorcalibrated
3 junior5 mid4 seniorcalibrated
3 junior5 mid4 seniorcalibrated
4 junior5 mid5 seniorcalibrated
6 junior7 mid7 seniorcalibrated
4 junior3 mid4 seniorcalibrated
2 junior1 mid1 seniorcalibrated
1 junior8 mid6 seniorcalibrated
5 junior5 mid7 seniorcalibrated
In the pipeline — labelled honestly
Both runtimes’ sandbox performance is already proven on our live infrastructure — sub-3-second test loops, measured, not estimated. The tracks appear above the moment their first challenges survive calibration, and never a day before: that is the same rule that put every track above on this page.
Scale tier · engineered for you
The public library tells you someone can engineer. Only your own domain tells you they can engineer here. A Custom Calibrated Track is a genuinely broken system that looks like yours — your framework versions, your architecture’s sharp edges, the incident your seniors still talk about — rebuilt by our engineers as a timed, gradable challenge. And it goes through the same harness as every public track: a flawed baseline that predictably fails, a reference solution that scores 100%, proven on live cloud sandboxes before your first candidate sees it.
Exclusive to your workspace
Enforced by a database policy, not a promise. No other customer can see it, dispatch it, or know it exists.
You give us an hour
One scoping call with a tech lead; a sealed, calibrated track in your dashboard in about two weeks. We author, you inform.
Candidates meet your real work
The honest job preview generic quizzes cannot give — and a signal AI assistance does not wash out, because the failure is contextual.
Ruby, Rust, C++, or something stranger — the harness that calibrated the ten public tracks points wherever you hire.
Inside the React track
A React challenge here is a real application misbehaving the way real applications do: a form that double-submits on a slow link, a list that shows the wrong results when responses race, a table that loses an edit when two saves overlap. Five architectural areas, each built around the specific failure a senior engineer is paid to see and fix.
Forms
Multi-step inputs with real-time validation, a submit that disables while a request is in flight, and errors that clear the moment they are fixed. The failure it is built around is the one every slow network reveals: a form that fires twice because nothing disabled the button, or an error that lingers after the field is already valid.
Data & CRUD
Create, read, update, delete with auto-save, an optimistic UI, and local-storage sync. The trap is what happens when a save fails: the edit has to roll back to its last confirmed value, and two overlapping saves must both survive — a slow response for one row cannot silently overwrite another row’s newer edit.
Fetching
Loading and error states, caching, and request de-duplication under real latency. The defect reproduces only for users on a slow link: type fast, and responses land out of order, so the list ends up showing results for a query the box no longer holds. Latest-query-wins, and the stale response is dropped.
Client state
A cart, an authenticated session, a theme — state that lives above a dozen components and is read far below. Context and external stores both appear here, and so does their sharp edge: a provider whose value re-renders the entire subtree on every change, when only one consumer actually moved.
Performance
Debounced search, lazy-loaded routes and components, and virtualisation or infinite scroll for lists that are thousands of rows long. The candidate has to keep the mounted node count bounded as the dataset grows, and cut a keystroke storm of fetches down to a debounced few — measured by what actually re-renders, not by which hooks appear in the diff.
One challenge, your domain
Every candidate meets the identical sealed code and hidden suite — re-skinned into a domain drawn from their own background. The performance challenge arrives as a fintech transaction ledger, a health system’s patient record list, or an e-commerce product feed. A write-up posted online describes an exercise the next candidate never sees.
How the grading actually works
A grader that reads code can be talked into a good score by code that reads well. Ours cannot, because it never reads the code. It runs the candidate’s system against real infrastructure and measures what happens — and that is a signal an AI assistant does not wash out, because the failures are in the behaviour, not the prose.
No AST parsing
We do not grep the source or walk its AST for the “right” keyword or hook — the exact check a copilot defeats in one paste. Every score comes from running the system and watching what it does: the requests it makes, the rows it leaves in the database, the nodes it paints. A correct fix passes however it is written; a plausible-looking one that misbehaves under load does not.
Real Postgres, per attempt
Backend challenges boot a genuine PostgreSQL in the sandbox. An N+1 is caught by counting the queries an endpoint issues and proving the count holds constant as the dataset scales — five hundred rows must cost the same as five. A locking fix is judged by the row state a real BEGIN/COMMIT leaves behind, not by whether the candidate happened to type FOR UPDATE.
Concurrency
Race conditions do not reproduce one request at a time, so we do not test that way. We blast a booking endpoint with fifty simultaneous requests against a capacity of five and prove the database enforced the limit: exactly five succeed, remaining never goes negative, the rest get a clean 409. Overselling, double-charges and lost updates fail here the way they fail in production.
Network rules
Four widgets mount, four requests 401 at once, and a naive interceptor fires four token refreshes against a rotating token — locking the user out. We intercept the network and count: exactly one refresh may go out, and all four original requests must replay to success. Coalescing concurrent refreshes is the line between a resilient client and a self-inflicted outage, and we measure it rather than read for it.
Memory isolation
Session bugs live in memory, so we look there. We prove a signed-out user’s data never survives into the next session — not for a single render frame — by recording every frame the DOM paints, and that a restricted role’s rows are absent from the DOM entirely, not merely hidden with CSS. The token in browser storage, the store in memory, the nodes on the page: read directly.
Chrome DevTools Protocol
Angular and Vue challenges run in headless Chromium and the engine itself is deposed over CDP: Runtime.queryObjects counts the component instances still reachable after garbage collection — fifty navigations must not leave fifty dead panels wired to a live stream — and a MutationObserver counts the DOM nodes one table insert actually cost: one row keyed by identity, 3,001 mutations keyed by index. Sealed call counters catch the fix that only moves the work: debouncing a reactivity storm passes the latency plane and fails the evaluation counter at 205,000 calls.
A real Docker daemon
DevOps challenges grade against dockerd running inside the Firecracker sandbox — real builds, real layers, no egress. The candidate’s image is built, saved, and every blob of the archive scanned: a registry token “deleted” in a later RUN is recovered from the COPY layer that still ships it. Deploys are graded by a traffic generator that holds requests through the cutover and counts every drop, and migrations by pg_stat_activity sampled at 100ms — no query may wait on a lock past half a second while 500,000 rows move under live writes.
V8, asked directly
Game-loop challenges run headless under a hard heap budget while a PerformanceObserver records every collection V8 performs. An object-per-bullet spawn pattern churns 42,380 bytes per thousand shots and spends 289ms inside the collector; a pool churns 4,004 and spends 80ms — and a behaviour digest guarantees the fast version still plays the same game, so recycling a dirty slot scores zero, not full marks. Lockstep determinism is proven by replaying one match in four separate OS processes, snapshot-restore included, and comparing fifty checkpointed hashes.
The kernel’s own memory
Streaming challenges spawn the candidate’s server as its own OS process and read VmHWM from /proc — the kernel’s high-water mark, immune to anything the code reports about itself. A buffering import grows 232MB across a tenfold file and dies inside a literal 64MB heap cap; a streaming one holds flat and finishes row-exact. The scaling law is the criterion: ten times the data may not cost materially more memory, measured on fresh processes because a high-water mark never comes back down.
Why it is a moat
Spinning up a real database per attempt, firing concurrent spikes and inspecting the network and the DOM is a harder platform to build than a code box that shells out to a linter. That is the point: the same difficulty that makes it a moat is what makes its score one you can defend to a candidate, an auditor, or a court.
What it measures
So we stopped grading the artifact. Candidates work copilots-allowed in a live broken system, and what gets measured is the judgment around the code: what they fixed, what they distrusted, and how they verified what a model wrote for them. Inverting a binary tree does not tell you whether someone can debug a production incident — the work you are actually buying is reading unfamiliar code, distrusting a green checkmark, and finding the failure mode nobody wrote a test for.
Every criterion in a rubric is authored as either correctness — does it observably work — or craft — how it is built: the cleanup function, the cache key that can hit, the structure a senior flags in PR review after CI is green. Two subtotals from the same graded criteria, pure arithmetic, no model opinion anywhere.
Because “the tests pass and the code is a liability” and “one test fails and the architecture is exactly right” are opposite hiring conversations — and a single score cannot tell you which one you are about to have.
Sound approach, flawed execution
craft ✓ · correctness ✗
Correct, and built to last
both axes clear
Neither axis cleared
the rubric says why
Works, but brittle
correctness ✓ · craft ✗
The moat
Not a code box that shells out to an evaluator. A container, a database with its own isolation boundary, and a grader that reports structured evidence rather than whatever landed on stdout — because judgment can only be measured against systems real enough to have judgment about.
Execution
Every attempt boots a genuine sandbox with a filesystem, a package manager and a network. Candidates run the suite, read the failure, and iterate the way they would on a Monday — because a challenge that cannot be run is a quiz about code, not a test of engineering.
Isolation
Every tenant’s data is separated by Postgres policies, forced on every table, with no application role able to bypass them.
alter table assessments
force row level security;
create policy tenant_scope
on assessments
using (tenant_id
= app.current_tenant_id());The Debrief
When they submit, the pressure ends and the signal starts: what would you do next, what do you trust least, where did AI help and how did you check it. Voluntary, five minutes, verbatim to your reviewer — the asynchronous version of the best moment of pair programming. Their own words become the opening questions of the interview, which is why bluffing them is pointless.
Our stance
We care how the candidate directed it and verified it. Every workspace seeds a DECISIONS.md the candidate fills in as they work — what was actually wrong, why this fix over the alternative, how they checked it — committed with the code and read by your tech lead beside the diff. An AI can write the fix; it cannot write the candidate’s account of why they trusted it. No webcams, no lockdown browser, no keystroke logging — the reasoning is the signal, not surveillance.
After grading
Hidden suites and a weighted rubric decide the score. A model then reads the diff the candidate wrote — and their own post-submit debrief — and drafts three or four questions for your tech lead, each anchored to a file and line. Questions — never a score, never a recommendation.
Routing
A model extracts a skill vector. Arithmetic — not the model — selects from your verified library, so the same seniority always draws comparable work.
Integrity
The workspace records the shape of the work — test runs and their failures, commands, log reads, paste bursts, focus loss — and the terminal prints a reference after each test run that a rationale can cite, verifiably tying the story to the run it describes. All of it is disclosed up front and shown to reviewers as context. None of it moves a score on its own, because a signal that silently fails someone is one you cannot defend in an appeal.
Leak resistance
High-volume assessment tools leak: candidates post the challenge, and the signal dies. The brute-force answer is ten thousand shallow puzzles. Ours is architecture, in three layers. Routing rotates on live exposure — every dispatch is counted platform-wide, the least-seen calibrated challenge wins the tie, and the pick that tops a forum thread stops being the pick, deterministically, with the count recorded in the audit trail. Every candidate meets a different skin — the domain, the story and the fixtures are re-rendered per person from their own background, while the sealed code and hidden tests stay byte-identical, so a posted write-up describes an exercise the next candidate never sees. And a leaked answer fails differently — a perfect score delivered in a fraction of the calibrated time is the solve pattern of prior knowledge, and it is flagged with the arithmetic attached for a human to weigh. No randomness anywhere: every one of these is a stored calculation you can audit eighteen months later.
Three views
A hiring decision is shared by three people who have always been shown one interface built for none of them. Every Arbiter seat opens onto a view built for exactly its reader — and deliberately no fourth step: your tech lead decides, your executive is briefed, and nobody waits on a second sign-off.
Recruiters
Open a position, add candidates, dispatch, track the funnel. Source code is invisible to this seat — enforced by database policy, so calling the API directly returns exactly what the screen shows.
Foolproof by construction
Tech leads
The rubric, the diff, the candidate’s own debrief, then AI-drafted questions anchored to file and line. Human evidence always outranks generated prose — the machine asks, it never concludes.
Fifteen minutes to a confident interview
Executives
Band, score, criteria passed, time used, and the tech lead’s read — every sentence assembled from structured facts. No model writes your executive summary, so nothing in it can be hallucinated.
Informed, never bottlenecked
Seats
Three seats, matched to the three people who share a hiring decision — and the boundaries between them are row-level policies in the database, not hidden buttons. A recruiter who calls the API directly gets exactly what their screen shows. Seats are unlimited on every plan: the fourth reviewer improves the decision and costs you nothing.
For whoever runs the workspace
Runs the workspace: team, positions, dispatch, and every result. One per workspace.
For your tech leads
Read submitted code, grader reports and rubric detail. Can overrule a grade.
For your talent team
Send and revoke assessment links, and see pass or fail. No access to code.
The obvious question
You can. It costs nothing, takes a minute, and returns a confident paragraph. Here is what you would actually be running — four failures, any one of which is enough:
Their copilot co-wrote the artifact — pasting it into a chat measures frontier model capability with a human’s name attached. So we do not grade the artifact alone: every candidate fills in a DECISIONS.md as they work — what was wrong, why this fix, how they verified it — committed with the code. An AI can write the fix; it cannot write the candidate’s account of how they directed it and why they trusted the result. That is the signal AI assistance does not erase.
Ask twice, get a 6 and an 8 — against no baseline, predicting nothing. LLM judges carry documented biases: sycophancy, verbosity preference, and self-preference — they rate model-written code more highly, which is exactly what candidates now submit. Every Arbiter score comes from a challenge proven to fail a flawed baseline and pass a reference solution.
An AI system that evaluates candidates is an AEDT under NYC Local Law 144 — bias audit, notice — and high-risk under the EU AI Act’s employment annex; a solely automated rejection violates GDPR Article 22. The chat-window version has none of that, and the candidate’s code and CV now sit in a consumer tool’s history with no data processing agreement.
A rejected candidate, an auditor, or a court asks why. Our answer is a signed grader report, a calibrated rubric, a process record, and a named human’s decision. The shortcut’s answer is a chat transcript.
And the part that surprises people
We run frontier models all through Arbiter — reading the CV to route a challenge, personalising the task to a candidate’s background, drafting the interview questions from their real diff. For judgment, never. That is not a capability we lack; it is the product. A chat window is not our competitor — it is a component, and a component does not come with a calibration harness, a disclosure architecture, or someone to hold liable.
Pricing
One flat rate a tech lead can approve alone — priced for adopting this quarter, not for a procurement cycle. The rigour is not discounted with it: every workspace still begins with a challenge calibrated against your stack, proven against a flawed baseline and a reference solution until it separates them. That hour of our time is in the price, not behind a quote.
One hiring team, calibrated challenges, invoiced.
$399/month, invoiced
or $3,990/year — 12 months for the price of 10
Unlimited seats · 5 active positions
Several openings in flight, and a stack of your own.
$699/month, invoiced
or $6,990/year — 12 months for the price of 10
Unlimited seats · unlimited positions
For regulated hiring and central procurement.
Talk to us
Unlimited seats · unlimited positions
Asked for, not yet built
Listed because you will ask, and because a feature list that quietly includes them would be one you could hold us to. Tell us which of these you need and we will tell you when — or that we will not.
Tell us what you are hiring for. An engineer reviews your request, calibrates your first challenge, and replies within one working day with your workspace and the paperwork your finance team will ask for.
Request access