About the author

Tyler Mann is the CTO of PullRequest and has interviewed hundreds of software engineers for roles ranging from mobile, backend, and front-end engineers.

PullRequest offers a service for helping administer and evaluate code challenges: Qualified by PullRequest

Using Code Challenges

One of the more popular ways to assess the technical skills of a software engineering candidate during the interview process is to have them solve a coding challenge. There are many different types of coding challenges ranging from coding on a whiteboard, doing an online automated challenge, or doing a take-home coding assignment. The benefit of all of these approaches is that you are able to look beyond the resume of an applicant and see how they interact with a challenge closer to the requirements for the role.

The approaches described below apply to assessing any software engineer, from a Python or Golang backend engineers to web frontend React or Angular developers to Kotlin or Swift mobile app developers.

Not All Code Challenges Are the Same

Any software engineer that has interviewed at several different companies can probably tell you that the types of technical interviews given can vary significantly. Some companies are infamous for asking certain types of questions that can come across as a brain teasers or “trick problems.” These questions can be useful for assessing certain ways the individual may be creative or their ability to solve certain types of problems, but they often will bias towards a different skill set than is required for most software engineers. In other words, many technical interview problems aren’t focused on assessing the most important things.

A common misconception is that a challenge should be complicated, or hard to solve, in order to evaluate the limit of the candidate’s skills. When a technical problem is designed to focus on how advanced the developer is, useful signals for understanding their strengths can be lost or ignored. Further, this reinforces a culture of “gaming” the technical interview process. Interviewees will invest time in studying similar challenges. Many challenges are overly complicated, relying on advanced recursion or dynamic programming techniques that are not commonly used in day-to-day development.

Give Candidates an Opportunity to Excel

Instead of reaching for a highly complex problem, try to use something that more closely resembles a problem that the candidate would face in an on-the-job environment. This will activate the creative muscles they’ll likely be using day-to-day and provide a more accurate representation of the skills that matter most. This approach also highlights important subtleties that help represent how experienced the individual is. For example, both a very junior and very senior developer may be able to solve the problem in a way that satisfies the requirements, but the way in which they go about it, the things they do or do not do, speak volumes for how experienced they are. These include things like:

• Best practices in the language/framework
• Error handling
• Writing modular code that is easy to understand
• Unit testing
• Attention to detail

How the candidate handles these things will help you understand their abilities when it comes to building scalable, reliable, and high-quality software much more than trying to see if they catch a “trick” in overly complicated code challenge.

Skills to Evaluate

The best skills to evaluate for a coding challenge are those used by engineers on a daily basis in the role being hired for. Avoid primarily testing skills that are not frequently used for the role in question. Concepts like dynamic programming, complex recursion, rotating buffers, bit shifting, tree/graph traversal are overly represented in challenges compared to their use in the roles being hired for.

This can vary slightly depending on the role, however these are some common, general engineering skills that are relevant for the assessment of almost any developer candidate:

• Essential data structures (hashmaps/lists/sets)
• Basic algorithms (aggregating data or translating data structures)
• Language familiarity and experience
• Error handling/attention to detail
• Performance (generally more important for backend roles)
• Concurrency (important for specific languages and more senior roles)
• Security (more important for specific roles, but good to touch on)

Start with Real Problems

Software engineering candidates that are interviewing will not usually have time or be willing to spend days working on an assignment. So coding challenges often must be slightly simplified from real-world engineering challenges to fit within time constraints. However, taking some simplified components from a real-world problem can often lead to a more comfortable challenge that feels closer to a normal day of development. Using a real-world problem also allows you to frame the problem with a backstory that helps provide context for assumptions that the developer may want to make.

Sometimes the best technical interview problems come from the problems that are right in front of you. Start with tasks that your company has had to solve recently and see if they can be simplified. Here are some examples:

• If your company develops a mobile photo sharing app, then you could have the candidate create a simple app that lets you take a photo and view photos in your library.

• If your company lets users buy and sell cryptocurrencies, then you could have the candidate build an API that supports a simplified version of trading stock symbols.

Either of these examples above could be implemented by most developers to some extent, but a senior developer would likely give a much more impressive solution on some of the categories mentioned above.

Bring Your Own Tools

A common barrier that can prevent an engineering candidate from performing at their best on a coding challenge is if they have to complete the challenge in a new and unfamiliar environment. Unless the candidate has a lot of experience with it, one of the most uncomfortable environments is writing code on a whiteboard. Many candidates succeed despite this, but it is not necessarily true that they will be better at the role than someone who is not comfortable whiteboarding.

This also applies to the tools the candidate is required to use to complete a code challenge. Unfamiliar editors in assessment web applications can have similar issues. Your candidate’s response will be inherently polluted by the work they had to do to learn the tool(s) while they were solving the problem. Even if the online tool has syntax highlighting and the ability to run your code, it will feel very foreign to most developers to use the new tool. Some of the best engineers use very customized code editing techniques and tools like Vim, or another favorite IDE with lots of tab completion and things to help them develop. If you remove these tools productivity will often go down significantly and they will be somewhat distracted by the environment.

The ideal approach for coding challenges to accurately assess a candidate is to have them use their own tools, especially if they are expected to write a significant amount of code. This often means their own computer and IDE.

Normalize Evaluations for Accurate Grading

It is important that a candidate’s solution for a code challenge can be accurately and fairly graded. The ideal case is that this is being done by another engineer who is able to detect all of the subtleties mentioned above. Some of the best candidates may occasionally be confused by an instruction and develop a solution that doesn’t match automated grading criteria, but is impressive upon manual inspection.

The best way to ensure that you can properly evaluate a candidate is to have a frame of reference from already seeing several different solutions for the same problem. Though it may be easy to spot a very poor solution from a truly exceptional one on the first submission, comparing a good vs a great solution can be surprisingly difficult without the proper context. Many high-performing engineering recruiting programs will have newer members of the team shadow 5+ interviews before they are responsible for evaluating the candidate on their own.

If you are introducing a new challenge, it is good to test it out with some friends or coworkers before you give it to a candidate. This will give you a barometer for what to expect since the relative skill of these individuals is already known. It can be helpful to do this in an anonymized fashion as well so that you don’t know exactly which engineer’s solution you are reviewing until after you have graded it. This “trial run” will also help you gather feedback from trusted sources to refine the requirements and context before the problem is used in a real interview.

Code Challenge Services

If you want help in evaluating candidates, whether it is pre-screening or getting a second opinion on a developer, there are several types of code challenge services you can use.

We offer one of these services ourselves, Qualified by PullRequest. With Qualified, you will get access to our extensive list of code challenges for almost every language/skill set. These challenges have already been developed to fit the ideal criteria mentioned above. The candidate develops their solution using their own tools. The solution is then reviewed and assessed by engineers from the PullRequest network of expert reviewers without knowledge of the candidate’s resume. Our reviewers see many of these challenges daily and are highly experienced at evaluating candidates and grading on a normalized scale.

If you are looking for something more automated, services like HackerRank and CoderByte are commonly used. Just note that using one of these services will often bias towards engineers that have a lot of experience with these sorts of algorithmic code challenges which often do not directly correlate with the role being hired for. This can be known to cause both false positives and false negatives.

Image by Caique Gomes