Recruiters remain terrible at assessing skills (from reading a resume)
Many years ago when Andrew Banks was still a recruitment agency executive I attended an industry event where he was the guest speaker.
He said something that was so obviously true I wondered why the concept he explained had not occurred to me (which tells you all you need to know about the difference in thinking between me and a man who was, for many years, on the BRW Rich List, now the Financial Review Rich List, an annual survey of the 200 wealthiest people resident in Australia, published by the Australian Financial Review each May).
Banks said words to the effect of
“Looking at the resume of a candidate who’s obviously wrong for a job and deciding they are wrong is easy.
Looking at the resume of a candidate who’s obviously right for a job and deciding they are right is also easy.
What’s highly valuable is viewing a resume that appears right for a job and identifying how the candidate is wrong.
Viewing a resume that appears wrong for a job and identifying how the candidate is right is also highly valuable.
That’s how you differentiate yourself as a top recruiter.”
I know from many experiences across 14 years as an agency recruiter that there’s nothing quite like the deep satisfaction of placing a ‘wrong’ candidate into a job that is right for them and then hearing some months, or years, later how well they are doing.
Despite the many years since I heard Andrew Banks utter those words, it appears a recruiter’s core skill of accurately assessing a resume remains problematic if recent research is anything to go by.
A study published last Wednesday by interviewing.io asked 76 technical recruiters (both agency and in-house) to review and judge 30 engineers’ resumes each (actual engineers currently on the job market who gave their permission to the researchers for their resumes to be used in the study) just as they would in their current roles.
The recruiters answered two questions per resume:
#1 Would you interview this candidate? (Yes or No)
#2 What is the likelihood this candidate will pass the technical interview (as a percentage)?
(Note: In most hiring processes, the technical interview follows the recruiter call and determines whether candidates proceed to the onsite. Being able to accurately predict which candidates will succeed at this stage is important and should inform the decision about whether to interview the candidate or not.)
Nearly 2,200 evaluations of over 1,000 resumes were collected and analysed.
The recruiters’ evaluations of the resumes were compared to how those engineers actually performed on interviewing.io: skills scores, feedback from interviewers, and ultimately, whether they passed or failed their mock interviews.
As it turned out recruiters’ resume judgments are only slightly better than tossing a coin.
The outcome for question #1 was recruiters (aggregated) in the study recommended 62% of candidates for an interview. The researchers calculated recruiter accuracy by treating each candidate’s first interview (pass/fail) as the truth, and recruiters’ decision to interview as a prediction. It turns out recruiters chose correctly only 55% of the time, an alarmingly low score.
Coin toss odds ultimately looked great compared to the recruiters’ much worse collective ability to assess the technical capability of engineering candidates based on a resume.
The outcome for question #2 showed when recruiters predicted the lowest probability of passing (0-5%), those candidates actually passed the technical interview with a 47% probability and when recruiters predicted the highest probability of passing (95-100%), those candidates actually passed with a 64% probability.
Recruiters’ predictions below 40% underestimate these candidates by an average of 23 percentage points. Above 60%, they overestimated by an average of 20 percentage points.
The researchers acknowledged there’s lots of noise in resume evaluations so they explored the question: Were recruiters’ noisy judgments at least consistent when reviewing the same resumes?
Nearly 500 resumes were evaluated by more than one recruiter. Based on a random selection of two evaluations per resume, the overall likelihood of two recruiters agreeing to either interview or not interview a given candidate was 64%.
Since recruiters also guess the probability a candidate will pass the technical interview, a comparison can be made as to how different these guesses are for a given candidate.
The average differential between two randomly selected recruiters’ evaluations of the same resume was 41 percentage points, i.e. if one recruiter predicts a 30% probability the candidate would pass; another recruiter evaluating the same resume would predict, on average, a 71% probability of passing.
The researchers concluded, after looking at the standard deviations for across-candidate evaluations and same-candidate evaluations, that when two recruiters are asked to judge the same candidate, their level of disagreement is nearly the same as if they evaluated two completely different candidates!
Each recruiter was asked to identify why they had passed on a candidate.
In aggregate, the top four cited reasons were (in order):
- Missing skill (easily #1- about 2x greater than #2)
- Unclear what they worked on
- No top firm
- Too many short stints
However, the researchers analysed the rejected resumes and looked at common traits to identify the actual causes of inaccurate recruiter assessment. The top four reasons technically competent candidates missed out on an interview were (in order):
- No top firm (as the candidate’s current or former employer)
- No MBA
- No top school (as the candidate’s tertiary alma mater)
- No Masters degree
Although the recruiters stated that, overwhelmingly, they rejected a candidate on a skill-based reason the reality showed the top four rejection reasons are all based on a candidate’s background, not their skill set.
The researchers were keen to find out what improves recruiter decision-making.
The answer was simple – time.
The more time a recruiter spends assessing a resume the more likely they are to make an accurate assessment of the candidate’s technical match to the job’s skill requirements
The median time spent on resume evaluations was just 31 seconds. Broken down further by Question #1 — whether or not the recruiter would interview them — the median time spent was:
- 25 seconds for those advanced to a technical interview
- 44 seconds for those placed in the reject pile
It turns out that spending more time on resume evaluations, notably more than 45 seconds, is associated with more accurate predictions — just spending 15 more seconds appears to increase accuracy by 34%
As Michael Lewis comprehensively proved in his book Moneyball (better known as a movie starting Brad Pitt), predicting job performance (in his case, of professional baseball players) remains little more than a coin toss by over-confident recruiters (talent scouts) who inaccurately weigh easily seen, but poorly predictive of performance, aspects of a potential hire (and under-value hard-to-see but highly predictive of performance, aspects of a potential hire).
Nearly three decades later Andrew Banks remains unerringly right about one of the key skills of top recruiters
Note: Just in case you were wondering how seriously the 30 participating recruiters took this exercise – each participant was paid a base rate and then received a bonus for every accurate prediction they made.
Related blogs
Why crap interviewers don’t know they are crap
Gut feeling, evidence and the Moneyball effect: Lessons from ATC 2012
Avoid hiring turkeys, even if you don’t recruit a single eagle