Most hiring processes are not designed. They accumulate. Someone adds a round after a bad hire, someone else keeps a puzzle question they were once asked, and three years later nobody can explain what any stage measures.
The research on this is unusually clear, and unusually ignored. Decades of personnel selection studies have ranked what actually predicts job performance, and the methods most companies rely on sit near the bottom of that list.
Here are the nine errors that do the most damage when assessing software engineers, and what the evidence says to do instead. Rise92 sees these patterns repeatedly across senior engineering hiring.
The short version
| # | Mistake | Cost |
| 1 | Starting before the role is defined | Every later stage measures the wrong thing |
| 2 | Algorithm puzzles as a proxy for the work | Selects for recent practice, not capability |
| 3 | Treating years of experience as capability | One of the weakest known predictors |
| 4 | Unstructured interviews | Roughly half the predictive power of structured ones |
| 5 | Oversized or unpaid take-homes | Strong people decline; your pool self-selects downward |
| 6 | Confusing speed with judgment | Filters out careful engineers |
| 7 | Adding rounds instead of improving them | Senior professionals exit the process |
| 8 | References as a formality | You skip the highest-signal conversation available |
| 9 | Debriefing before independent scoring | The loudest voice becomes the decision |
Mistake 1: Starting before anyone defines the role
The most expensive error happens before the first conversation.
A requisition opens, a job description gets copied from the last one, and sourcing begins. Nobody has written down what problem this person solves, what decisions they own, or what success looks like at 90 days.
Everything downstream inherits that vagueness. Interviewers each invent their own bar. The panel then argues about people rather than criteria, and the argument is unresolvable because no criteria exist.
What to do instead:
- Write the problem the hire owns, not a list of technologies
- Define the three or four things they must do well from day one
- Write the 90-day success definition before sourcing, and use it as the scorecard
- If you cannot write it, you are not ready to hire
Mistake 2: Using algorithm puzzles as a proxy for the work
Timed algorithm questions measure how recently someone practiced algorithm questions. For most product engineering roles, that correlates weakly with the job.
The people who perform best on these are often recent graduates and active job seekers who have drilled the format. The senior engineer running a payments system for six years has not, and will underperform relative to their actual value.
Work sample tests, meaning tasks that resemble the real job, consistently rank among the strongest predictors of job performance in the selection research. The distinction is not “test versus no test.” It is whether the test looks like the work.
What to do instead:
- Build the exercise from a real problem your team solved recently
- Let them use documentation, search, and their normal tools
- Assess the reasoning and the tradeoffs, not just the output
- Reserve algorithm depth for roles where algorithm depth is genuinely the job
Mistake 3: Treating years of experience as capability
This one has the clearest evidence against it and the strongest grip on hiring practice.
Years of experience alone shows a predictive validity of roughly .18, with diminishing returns after about five years. In Sackett’s meta-analysis it ranked behind 22 other selection criteria. Years of education alone performs worse still, at around .10.
To put those numbers in context, a validity of .51 explains roughly 26% of the variance in job performance, while .10 explains about 1%. Screening on experience and credentials uses methods five to 25 times weaker than the best available alternatives.
Stack keywords carry the same problem. A strong engineer moves between languages in weeks. Filtering for exact framework matches removes capable people and keeps people who happened to use your tools.
What to do instead:
- Treat experience as context, not as evidence
- Test transferable depth rather than tool familiarity
- Ask what they would do differently on a past project, and why
Mistake 4: Running an unstructured technical interview process
The unstructured conversation feels productive. Both people talk, rapport develops, and the interviewer leaves with a confident opinion. That confidence is mostly unearned.
Schmidt and Hunter’s 1998 meta-analysis put structured interviews at a validity of .51 against .38 for unstructured. Sackett and colleagues, using more conservative corrections in 2022, put the gap wider still, at .42 versus .19. Google’s internal hiring research found the same pattern across thousands of hires at every level and function.
Structure means the same questions in the same order, scored against defined criteria. This is not bureaucracy. It is the difference between a measurement and an impression.
A well-designed technical interview process also combines signals rather than relying on one. A structured interview paired with a cognitive ability measure produces a composite validity of around .63, materially stronger than either alone.
What to do instead:
- Write the questions and the rubric before the first interview
- Ask every person the same core set
- Score against anchored criteria, not overall feeling
- Train interviewers on the rubric, once, properly
Mistake 5: Oversized or unpaid take-home assignments
A four-hour unpaid assignment quietly filters your pool by availability rather than ability.
The people who complete it are those with spare evenings and low current demand. The senior engineer with a demanding role and two children declines and takes the offer from the company that respected their time. You never learn they existed.
What to do instead:
- Cap it at two hours, and mean it
- Pay for it, at a real rate
- Offer a live pairing session as an alternative for people who prefer it
- Review it properly, and give specific feedback either way
If the exercise is worth doing, it is worth paying for. If it is not worth paying for, it is not worth doing.
Mistake 6: Confusing speed with engineering judgment
Live coding under time pressure measures performance under time pressure. That is a real skill in incident response and almost nowhere else.
Many strong engineers are deliberately slow in the first minutes of a problem. They ask about edge cases, question the requirement, and check assumptions before writing anything. Under a timer with someone watching, that behavior reads as hesitation. In production, it is the behavior you are paying for.
What to do instead:
- Separate “can they think” from “can they type fast”
- Explicitly tell them that asking clarifying questions is being assessed positively
- Weight the quality of their questions as heavily as the solution
- Watch for whether they notice the problem’s ambiguity at all
Mistake 7: Adding rounds instead of improving them
Every bad hire produces institutional pressure to add a stage. Nobody ever removes one.
The result is a five or six round process that takes a month. Startups already average around 15 interviews per hire, rising to 18 for technical roles, consuming roughly 21 interviewer-hours per technical hire at the smallest companies.
Senior professionals do not wait through that. They are usually not searching actively in the first place, and they exit the moment the process signals disorganization. You lose them silently, and you attribute the loss to the market.
More rounds also do not mean more signal. Four unstructured conversations produce four correlated impressions, not four independent data points.
What to do instead:
- Three rounds maximum at senior level
- Each round tests something the others do not
- Run stages in parallel where the sequence does not matter
- Decide within a week of the final conversation
Speed here is not a compromise on rigor. It is a consequence of rigor, because a structured process needs fewer stages to reach confidence.
Mistake 8: Treating references as a formality
Most reference calls happen after the decision, with a list the person chose, asking questions that cannot produce a negative answer.
This wastes the single richest source of evidence available. References are the only stage where you hear from people who have actually watched this person work for years, under pressure, on real problems, with real stakes.
What to do instead:
- Call before the final decision, not after
- Ask for a peer and a direct report, not only a manager
- Ask behavioral questions: how they handled disagreement, what they needed support with
- Seek at least one conversation with someone who has no stake in the outcome
- Ask directly what the person is not good at, and listen to the pause
This is where off-market sourcing has a structural advantage. When someone arrives through a trusted network rather than an application, longitudinal evidence about how they work is already available. That is the basis of a proper narrative dossier introduction rather than a resume and a hope.
Mistake 9: Letting the room debrief before anyone scores
The panel gathers, the most senior or most confident person speaks first, and everyone else calibrates to that opinion. The meeting then produces consensus that looks like agreement but is actually anchoring.
The related failure is the late override, where a committee discards a structured assessment based on an informal impression formed in the final conversation. That single move reintroduces exactly the affinity bias the structure existed to remove, and it is the most common way rigorous developer skills assessment collapses at the last step.
What to do instead:
- Every interviewer submits a written score before the debrief
- Scores are visible only once all are in
- Discuss disagreements specifically, since disagreement is information
- No overriding a structured assessment on the basis of “feel”
- Track outcome patterns over time for demographic disparities
What assessing software engineers badly actually costs
The numbers are worse than most teams assume.
| Failure | Documented cost |
| Bad senior hire | Commonly estimated at around 200% of salary |
| New hires failing within 18 months | 46%, with 89% attributable to attitude rather than skills |
| Interviewer time per technical hire | ~21 hours at companies under 25 people |
| Weak screening methods | Five to 25 times less predictive than the strongest alternatives |
Two things follow. First, the dominant cost in software engineer hiring is not the fee paid to find someone, it is the cost of choosing wrong. Second, most of that risk sits in attitude and working style rather than technical capability, which is precisely the area unstructured processes measure worst.
This is also why comparing sourcing options on headline fee alone is a mistake. An ongoing percentage markup compounds monthly, while a one-time fee does not, and both are dwarfed by the replacement cost of a mis-hire. The full comparison is laid out in transparent fee structures.
The pattern underneath all nine
Every mistake here is the same mistake wearing different clothes: substituting a proxy for the thing you actually want to know.
Years of experience proxies for capability. Algorithm puzzles proxy for engineering. Rapport proxies for fit. Round count proxies for rigor. Each proxy is easier to measure than the real quality, and each is substantially less predictive.
The fix is not complicated, though it does require discipline. Define what good looks like before you start. Test work that resembles the work. Ask everyone the same questions. Score before you discuss. Call references who will tell you something true.
Companies doing this well are not running harder processes. They are running shorter ones that measure fewer things more accurately, which is what makes them fast enough to win senior people.
Frequently asked questions
Starting before the role is defined. Every subsequent stage inherits that ambiguity, interviewers apply inconsistent standards, and the panel ends up debating people rather than criteria. Writing the 90-day success definition before sourcing is the highest-leverage 30 minutes in the entire process.
Yes, but structure determines their value. Structured interviews show roughly twice the predictive validity of unstructured ones across multiple meta-analyses, and combining a structured interview with another signal such as a work sample produces stronger prediction than either alone. The problem is the freeform conversation, not the interview itself.
Three is generally right for senior roles, with each round testing something the others do not. Adding rounds tends to produce correlated impressions rather than independent signal, while extending the timeline enough that senior professionals withdraw. Depth per stage beats number of stages.
Yes, and they should be capped at around two hours. Long unpaid assignments filter by available free time rather than ability, which systematically removes employed senior engineers and people with caring responsibilities. Paying also signals that you take their time seriously, which matters in a small market where reputation travels.
What to take from this
The choice is narrower than the marketing suggests. Establish whether the person is an employee or a genuine contractor, establish whether you own a local entity, and five of the six options usually eliminate themselves.
Then model total cost per country rather than comparing provider fees, because statutory variation dwarfs fee variation.
And keep the sourcing decision separate from the employment decision, because conflating them is how companies end up paying an ongoing percentage for something a one-time arrangement would have covered.
To work through which structure fits your situation, get in touch.
Fixing this is cheaper than replacing a bad hire
You do not need a new applicant tracking system or an assessment vendor. You need a written definition of the role, the same questions for everyone, scores recorded before discussion, and reference calls that ask real questions.
That is a week of work, and it changes the quality of every hire that follows.
If your constraint is not the process but access, meaning the engineers you want are employed and not applying anywhere, that is a different problem and it needs a different solution. To talk through which one you are actually facing, get in touch.
