Finding rockstar coders is tough
Finding new developers to join a team in the tech industry can be very challenging. It is difficult to gauge a candidate’s technical skills, to ascertain whether they will fit in well with the team socially, or to identify (and avoid!) individuals who will be hunting for a better deal within a month of joining the team. Initially, we believed our recruitment process was solid, but it didn’t take long for us to realize just how far off the mark we really were.
Initially, we operated with optimism. We trusted recommendations from our network and conducted generic interviews. We welcomed candidates who appeared enthusiastic and competent during interviews. Yet, it was soon revealed that many lacked necessary technical skills or struggled to integrate well into our team.
For instance:
We had a promising junior join the team, but he didn’t stay because mentorship from a senior developer was a deal breaker for him. We couldn’t provide it at the time (we now do! ), but if we knew it was a deal-breaker for the junior, we may have favored another candidate over him.
In another case, we took on a graduate who did a very impressive job on a technical evaluation “homework assignment,” only to discover later that he did not even fully grasp the coding concepts of “variable” and “function.” We tried to fill the gaps through some mentorship and coaching but… we are not a web development college. We are a dev house, and we ultimately could not justify spending dozens of hours teaching our employees the basics. So we parted ways with this individual also.
The overall process that works for us
Through experiences such as these, it became starkly clear that we need to refine our hiring process.
Through a few iterations and valuable conversations with Jaco van Vuuren from Credens People Solutions, we implemented a three-part interview process:
- Hello Interview
- Technical Assessment
- Structured Interview
The Hello Interview serves as a role-fit, an introductory session, where we meet candidates to learn about their backgrounds and what they seek in their careers as developers. This stage is essential for identifying candidates who are enthusiastic and eager to learn and grow. However, enthusiasm alone does not guarantee success. That’s why the Hello Interview is just the first step.
Candidates are not usually disqualified after the hello interview, but sometimes they are. This could be because, for example:
- We learn that our interests and theirs don’t align well. For example, we might want someone who can come into the office twice a week, but the candidate lives too far away or has no practical means of transport. Or we learn that they are specifically interested in working in FinTech, which is something we can’t guarantee they will be able to do for long with us.
- They make a bad impression by arriving late, not presenting themselves in a professional manner, or in some way showing that they are not really enthusiastic about joining our team.
After a successful Hello interview, we invite candidates to a Technical Assessment (more on this later). The technical assessment is where most candidates have fallen off so far: only about 1 in 5 pass.
Those who successfully navigate the technical assessment are then invited to the final round: the Structured Interview. In this round, we dive deeper into whether the candidate’s values align with ours, discuss the specifics of the role, and assess their likelihood of staying beyond the initial months.
To date we have not disqualified anyone yet on the structured interview, but it has happened that multiple candidates make it through all 3 rounds, and then we have to make a choice between more than one viable candidate, and information from all 3 rounds is then considered.
Especially for more senior positions, we also let candidates who pass all 3 rounds do a 16PF psychometric evaluation, again through Credens People Solutions, who then report back to us. Through this evaluation, we learn more about the candidate’s strengths and weaknesses and how well their personality aligns with the role that we are considering them for.
Ever since we have adopted the above process, all our hires have been quite successful, and we have not faced disappointments like before.
The technical assessment that didn’t work
Our initial approach to technical assessments was to task candidates with mini-projects. We would assign them a project and then ask them to submit it as soon as it was complete. The projects were of such complexity that they should take a good graduate about 4 to 8 hours to complete.
When candidates asked us how much time they have to complete it, we simply asked them to submit it as soon as possible. The intent was to evaluate not only their capability to solve complex problems but also their level of excellence and eagerness.
However, we soon discovered a significant flaw—this method didn’t allow us to observe the process candidates used to reach their final solution or verify if they completed the tasks themselves. This limitation meant candidates could potentially be dishonest about their work and, by extension, their technical skills.
But… Do you see the flaw here? We soon found candidates submitting impressive homework projects with elegant solutions going even beyond what was asked. However, after said candidates were on the job for a week or two, it became clear that there was no way that they completed the technical assessment project on their own.
It systematically happened that candidates with extremely poor technical skills passed this technical assessment and were appointed. These candidates were then unable to perform even basic tasks given to them, and it caused frustration with both the team and with customers.
The technical assessment that does its job
In order to protect ourselves from appointing team members who are unable to perform, we opted for real-time technical assessments where we could observe a candidate solving problems presented to them.
*Below is an image of a technical assessment in progress. Our interviewer has just posted a question for the interviewee, and now the interviewing panel is observing the interviewee tackle the problem.*
The assessment consists of a couple of basic coding questions, which they can complete in a programming language of their choice. Candidates are allowed to Google for help, as long as we can observe them doing this. However, they are not allowed to use chatGPT or similar LLM tools (even though we are happy for them to use these once on the job).
They get 1 hour to complete the questions, which are given to them one by one while a DrakkenTech developer and HR manager observe them over screen share.
Some example questions we use (or I should say “used,” now that the cat’s out the bag):
Check if a string contains the word “florida man” and then return the rest of the string after “florida man”.
Create a list that contains ten randomized coordinate tuples. Their values should be between 0 and 10. Now, filter the list to contain only coordinates with x>5 and y<5.
Check if the string is palindrome and return True if it is. (A palindrome is a letter sequence that reads the same backwards as forwards, e.g. tenet.)
Once the hour is over, we thank the candidate and tell them we will be in contact about the outcome in the days to come. The DrakkenTech developer and HR manager then dive into an internal debrief to discuss the evaluation and ultimately decide whether they can recommend that the candidate proceed to the Structured Interview or to disqualify them at that stage.
One of the most striking revelations from this process is that only about 1 in 5 candidates who pass the Hello Interview perform well enough in the Technical Assessment to be considered further.
These assessments can be nerve-wracking
A robust assessment process matters
Good new team members can bring so much to the team: new energy, new ideas, the joy of growth, and even just some good old additional elbow grease. By contrast: unqualified or ill-fitting team members can harm operational success, reputation, and team morale.
A good appointment process underpinned by a solid technical evaluation is therefore critical to a young dev house like DrakkenTech and most likely to most companies.
We’d love to hear from you. Do you agree that false negatives should be accepted if it protects you from false positives? What has been your team’s experience with technical assessments or appointment processes in general? Have you refined your processes, and how do they look now?
