Interviewing Researchers with Passion and Ability
I share my interviewing experience as a hiring manager when leading a research team at a large crowdsourcing platform.
Having a research job in the industry is great! The pay is nicer, the job itself is exciting, and besides the usual benchmarks, our advances are judged by the market. But how do we hire the best people for this job? In this post, I share my interviewing experience as a hiring manager when leading a research team. I interviewed, hired, and managed people who were paid for doing research as their daily jobs.
When I worked at a large data labeling company, I spent almost two years as the head of research before being promoted to managing multiple teams additionally to my original team. I was responsible for leading the research agenda, managing research projects, communicating with other teams, and, of course, hiring and managing people. I defined two goals for my team. First, to demonstrate exciting evidence how crowdsourcing helps solving very difficult problems in Artificial Intelligence. This was needed for marketing purposes, mostly. Second, to help internal teams leverage recent advances in research for boosting the data annotation quality and throughput. This was needed to safely handle external and internal challenges that no one can address, but us. Both goals were ambitious and required a great team, so I had to learn and act quickly.
Who are the Researchers?
What roles exist, and what is the difference between people called researchers? Let us start with the academia. Academic research is mainly done by research fellows, postdocs, and graduate students, who perform similar duties and are dreaming to become professors one day. Not everyone would, but there is no strict separation of responsibilities. Due to the limited funding and specificity of the incentives, everyone is doing everything to succeed with publications, grants, theses, tenures, and so on.
The industry is somewhat different. It has role separation, and I would roughly distinguish three main tracks of individual contributors. First, research scientists, who are involved in catching the latest problems and producing disseminated research on some topic, similarly to postdocs, but without a permanent struggle to find their funding. Second, applied scientists, who are laser-focused on product-related research and producing actionable insights for the product teams. Last, but not least, research engineers, who implement the new methods and efficiently reproduce the existing ones. Although my experience is limited to Machine Learning and Artificial Intelligence fields, I believe it might work for other disciplines, too.
Note that your mileage may vary. Some companies have a different separation, and there are often specialized positions like software engineers in distributed training or inference, for which this interview process does not apply.
Interviewing
Interviewing is a complex multi-step process. Companies usually have specific interviews for the mathematical, programming, or behavioral skills of the candidate. Depending on the organization, there are usually dedicated interviews for these. Being good at math, coding, and talking does not make one a good researcher. So I organized a specific research experience interview, in which I was eager to evaluate two vital traits of the candidate researcher: passion and ability. And I believe that this approach might be useful in both industrial and academic settings.
So, a research experience interview. Instead of throwing mathematical puzzles or LeetCode problems, read in advance the candidate’s resumé and skim through their publication list, if available. This is needed to discuss the candidate’s experience in delivering high-quality research output. I structured this interview as a one-hour-long dialogue between two professionals, the candidate and you.
Passion
By passion, I meant the ability to define and lead the research agenda, i.e., whether the candidate can apply their experience and knowledge. I like asking questions like these:
Why does this company need researchers?
Tell me about the recent paper you enjoyed.
What made you want to leave academia? (If applicable.)
What are your research interests?
What scientific breakthroughs will make our business obsolete?
At job interviews, people are trying to be nice and show a good impression, especially when asked open-ended questions. That is inevitable. Moreover, receiving social-desirable responses is perfectly fine. I would not focus on the questions that feel more like behavioural interview questions—they are just short ice-breakers to warm up the conversation. But I would dive deep into what drives the candidate. Some people are extremely good communicators, but no innate charm and communication skills can overcome the lack of understanding and participation in meaningful research.
How does one understand if the candidate is going well? By diving deep and expecting clear, specific answers. The stronger the candidate, the deeper they can dive while giving reasonable responses and explaining the clear rationale for their decisions.
For instance, if they are interested in Machine Learning, what specific aspect of this wide discipline makes them interested? Any examples of how they can apply it in the new company domain? What methods might be useful? Overly generic ideas like “take all the company data and train an awesome model” are not good; it’s highly likely that your team has already tried and rejected such ideas. Specific and well-articulated ideas like “apply pre-trained models to reduce the amount of human effort for a data labeling task with a prevalent response” are good. Even if the ideas are oddly enough, that’s fine as the candidate might be a non-expert in your particular fine-grained domain. And diving deep into the details greatly helps—explaining why is more important than explaining what.
Ability
By ability, I mean the presence of a research track record, i.e., whether the candidate gets things done and how. I prefer asking questions like these:
What was your role in your last paper?
Can you explain what your study focused on exactly?
Have you implemented a method described in a recent paper?
Have you organized a research conference or similar activity?
How would you approach a new research problem?
Having dove deep into this conversation, it is easy to read the areas of the candidate’s expertise. Were they the first author of a nice paper? Why? Was it because they drove the entire study? Were they the last author? Again, why? Maybe they managed to provide funding or just invented the idea. Some people love to invent trailblazing methods. Some people are great at designing rigorous evaluation protocols. Some people are so good at writing that even a mediocre paper becomes strongly accepted at a decent venue. How do they do it?
Sometimes, weird things happen. For instance, one candidate was not able to find their publications and admitted that they did exactly what their advisor told them to do, however being the first author of a conference paper.
It is important to keep an eye on the parts of the research process the candidate finds most exciting. Especially if these parts are rare, like evaluation design, statistical analysis, or downstream applications. My experience shows that most candidates like just inventing methods, playing with latent representations, or prompting language models. It requires a certain discipline to seek people who think and see not like you, but it allows attracting talents who bring a more holistic perspective to everything your team produces. Different points of view are important to deliver cohesive results.
Final Steps
Now you have a brief understanding of how deep the candidate’s passion is and what they have achieved so far as a researcher. If the candidate is passionate, but cannot provide specific details, it might be a poor fit for the research or applied scientist job. Do they prefer writing code? Maybe the research engineer position is better. Are they good at catching hype without going beyond fairly superficial things? That’s a good trait for a developer advocate, which is an amazing job, too. If the candidate feels stellar but does not have a strong publication record for a research scientist, can they be an intern or take a junior position? They would require a mentor, which is beneficial for both if the team’s capacity allows that.
What happens if the candidate’s research profile does not exactly match the one of your team? Well, you might know this in advance by reading the resumé. During the interview, you should steer the discussion to let the candidate show their passion and ability in the domain that is new for them.
Depending on the candidate’s background, they might already be able to apply some of their knowledge to your field. If the candidate comes from a more applied field, you need to be open-minded and have an interdisciplinary agenda in your strategy. Sometimes, it is a good match, e.g., Human-Computer Interaction or Computational Physics, but not limited to it. Sometimes, it is not.
Right after the interview, you make a decision, Hire or No Hire. If you would be happy to work together with this person and you obtained a crystal clear understanding of how exactly they will contribute to your team’s agenda, it’s a Hire. The exact level depends on their experience and ability to dive deep while keeping the reasoning meaningful.
If you are in doubt or confused, I recommend following a Joel Spolsky’s rule: “if you can’t tell, that means No Hire.” It is better to say sad news sooner than to wait until they become a huge problem not for just you but for the entire team or even the entire organization.