I just got off a one of the most surreal video calls of my life - a live video call with a candidate who was interviewing to become an L3 software engineer at Kapwing, our online video editing startup. The interview started extremely well - the candidate seemed like a clear fit in terms of background and experience - until halfway through when they suddenly were unable to speak any further about any of their technical experiences. When I pressed a bit further, they eventually admitted to having prepared for the interview using AI, and we ended the interview. In this post, I'd like to share about the experience in more detail, as well as the steps that led me to discover that this candidate was not being entirely truthful.

Preparing for the interview

Our interview process at Kapwing starts with internally reviewing submitted resumes, and if a candidate seems to have relevant experiences, we invite them to a 30 minute phone screen with a member of our team. Sam (not their real name) seemed like a great fit for our L3 software engineering role. They were a master's student finishing their program at a reputable school, and their resume listed relevant industry experience in three different startups. To prepare for the interview, I reviewed their resume and was ready to ask a few questions about their experiences.

While the resume did seem a bit optimized for keywords, the companies that were listed were in fact real companies and the timelines seemed to match up with their time in school. In addition, this candidate had a public LinkedIn profile with a real picture which also listed similar experiences. It seemed like a good fit, so we decided to schedule an initial phone screen.

Our phone screen process

During phone screens with prospective software engineering candidates, our main goal is not to assess engineering ability, but rather get to know the candidate - their career goals and professional experiences - and determine whether it could be a good fit on both sides. The phone screen starts with mutual introductions, and then behavioral questions about their professional experiences. Finally, we save time to answer any questions they may have about Kapwing.

We've also been the target of hiring scams in the past, so one policy we have is to only conduct "phone screens" on live video calls with the camera turned on. When candidates don't turn on the video camera, we politely ask them to or ask to reschedule the interview. This helps ensure that we are speaking with a real person who matches their profile. Sam (not their real name) was no exception - they showed up on time with their camera turned on.

My interview with Sam

I usually start these initial video calls with a short introduction about myself and Kapwing, and the first thing I usually ask is, "Tell me a little about yourself and what you're looking for in your next role".

Sam's answer was highly relevant. They had experiences working in startups similar to ours in size, which is a plus since it can sometimes be an adjustment to work in a smaller team. They also had worked with relevant web technologies like those we use on Kapwing - React, Node, and building and managing servers on GCP. And finally, Sam seemed like a great fit culturally - they had experience as a video creator themselves and could fully emphasize with many of the painful video editing experiences that we at Kapwing have been working tirelessly to solve.

I was excited to continue the interview, and asked, "Tell me about a difficult technical challenge that you faced in one of your recent roles, and how you were able to solve it".

Sam's answer was also relatively strong. They started by describing their recent experience working at a smaller startup building an app for day cares to manage relationships between parents, care providers, and students. They worked specifically on a part of the app that allowed students to send a notification to their parents to indicate that they were ready for pickup at the end of the day.

Sam described how, during a client visit while demo testing the app, they noticed that there was a spike in the server load caused by the application triggering too many SMS notifications. To solve this issue, they implemented "rate limiting on the backend, as well as a lazy loading solution for the frontend". They also added "pagination for Dynamo DB" and "retry mechanisms" for the app.

On the surface this experience is highly relevant for our work at Kapwing. We often have spikey traffic due to various video trends, and rate limiting requests during periods of high load, as well as lazy loading or paginating videos within a users' video library, have been essential for the performance and stability of our website. I was excited to ask Sam a bit more about their implementation.

Where things didn't add up

While the projects they described seemed relevant, I wanted to know if they understood how their work fit into the greater picture of the business. I asked Sam about the spike in server load, and what might have been causing it.

Sam started to explain that that the day care might have had a class of thirty or so students, but if the teacher would send SMS notifications to all the parents at once, then the Twilio API might rate limit the request. To solve this, they said they decided to "batch the outgoing message requests".

I started to feel like something was off. It didn't make sense that the Twilio API would not be able to handle sending 30 SMS messages at once - this seemed like a scaling issue that would be easily resolved through upgrading the plan. Even if that was an issue, batching the outgoing messages seemed like the wrong solution - batching messages would only make sending a lot of messages at once even more common. Finally, I had some questions about the application itself: When would a teacher want to send 30 SMS notifications at once to each of the students' parents to coordinate pickup? Was this an issue with smaller day cares with 5-10 students?

Sam replied that I might have been right about the batching and they might have been misremembering the batching solution. But they knew that the Twilio rate-limits were an issue that they worked around on the backend.

At this point, I decided to circle back on one of the other things that they had mentioned in case they were just forgetful due to time and nervousness. I remembered Sam had mentioned (and written in their resume) that they had added pagination to Dynamo DB while working on this application. I was curious about this, so I asked, "Why did you decide to implement pagination? What data were you paginating?".

It was at this point that Sam answered, "Hm, let me think about that for a second." And then they just became relatively silent, only vocalizing filler words like "hm".

Discovery and admission

Two minutes passed as we essentially looked at each other in silence on the video call, and then I asked, "How do you not know what data you were paginating?".

At this point it had become clear that they were not telling the full truth about their experiences. I think we were both in shock to a certain extent - they didn't seem prepared that I would ask such follow ups on their experiences, and I had a surreal feeling that a candidate was actually brazenly doing this on a live video call.

After some time, I broke the silence and just asked, "Can you just tell me the truth? What did you actually work on?". To their credit, Sam came clean.

They admitted to having prepared for the interview using AI. They said that they had indeed worked briefly on the day care application, but it had been some time ago, and they never worked on any of the features that they had previously mentioned. They said that they indeed had experience working with a frontend in React, but it had been some time since touching the backend systems that they had described.

Ending the call

Usually at Kapwing, we end the interview and send a follow up note over email - whether we end up rejecting or moving forwards with a candidate, we try to stay transparent throughout the process. In this case, I told Sam that we wouldn't be moving forward directly during our call.

I ended by saying that the software community is smaller than it seems, and integrity and reputation goes a long way. I told them that I feel that its important to be honest with their experiences, and they thanked me for the advice before we hung up the video call.

Reflections on interviewing in the age of AI

This experience was especially surreal because of how far the candidate was able to get using AI. In previous hiring scams, we've seen candidates make a relatively fake looking resume, or be unwilling to turn on their video camera. But in this case, it was a real person matching a real LinkedIn profile simply preparing their experiences with AI. It was scary in a way, because I believe that I am a fairly good judge of character, and within the first 10-15 minutes of the interview, this person seemed like a very strong candidate that I would have been excited about to join our team.

The next stage of our interview process, had this candidate moved forward, is to implement a take-home project that we have specifically designed for prospective candidates to complete. Had we moved this candidate forward, I have no doubt that they would have been able to use AI to pass the take home project with flying colors.

Upon reflection, I think there are a few learning lessons from this situation that I wanted to share them in case other founders or hiring managers find themselves in something similar:

1. Even in behavioral interviews, ask detailed situational questions. In this case, the AI preparation helped this candidate talk about the company, their experience, and their project, but it wasn't able to connect the dots between those separate things. For example, they spoke about their experience working with people at a day care, and also their project of paginating database calls, but when I asked specifically what kind of collection of data was so large that it needed to be paginated, they weren't able to say. It may be harder for an LLM to quickly come up with an answer that makes sense within the context of that lived experience.
2. Ask questions that have to do with the human experience. Another area where the candidate struggled to answer was how exactly their technical solutions mapped to human problems. For example, they mentioned that they added lazy loading, but they weren't able to specifically describe the UI that necessitated it, and what the end customers of the product where experiencing while using that UI. Most human engineers shouldn't be implementing blindly, and should be able to explain clearly why their changes were impactful, not just to the metrics, but to the downstream users as well.
3. Insist on camera ON phone screens. Some candidates can potentially outsource phone screens, or even go as far to use an AI avatar or voice during a video call, so having a live video call can go a long way to ensure that a person's claimed identity matches up. If a candidate isn't in a convenient location to turn on their camera, our policy is to politely request to reschedule the interview.
4. Always make a reference check before hiring. It's helpful to verify what a candidate claims, and while it might be possible for one person to get away without telling the full truth, it can be much harder to convince a second work colleague to do the same. I believe that in the age of AI, a human reference check is essential before making offers as a final way to assess a candidate's work experience.
5. Conduct yourself with professionalism and empathy no matter what the situation. Even though the candidate used AI to prepare for the interview, I felt that they were still a human being doing everything they could to try to find a job. To their credit, they did come clean and told the truth about using AI to prepare for the interview. I believe that people are not trying to lie but rather do so when they feel like they don't have another choice. Reputation goes both ways, and its unhelpful to berate a candidate or accuse them of wasting time. Rather, it's best to end the call with professionalism and take each meeting as a learning experience.

Conclusion

I'm sharing this post with not only on our blog but also our internal team for training purposes. I feel it's important to raise awareness that there are candidates who may choose to exaggerate their experiences with the help of AI, and this can only be found through more specific lines of questioning.

For prospective job candidates, my advice is still that "the truth will set you free". Even if you do get the job offer in the end, your experience at the company may be short lived if your work doesn't meet expectations.

I hope that this post helps both hiring managers and job seekers alike. I would love to hear more about your similar experiences as well!