Beyond memory tests: Designing interviews for assessing competence

I chose the old photo banner art (Photo by Debby Hudson on unsplash) because most of my recent interviews have felt more like a memory test than a competency test.

“Tell me about a time a family member gave you some advice, what was it and how did your life change as a result? Did you send them a thank you card? What was on the card?”

Yikes, I don’t remember!

… Ironically, I also do remember - but I remember lots of advice over many years, all fairly vaguely, but all contributed to who I am and what I do today… which is the best one to try and dig up the details for?

Inevitably, I just reach for the most recent example, because I remember the most about it. It might not be a very relevant one though. Oh dear

“Hi Paul, just some feedback from the interview - the team didn’t think your answer to the family advice question was what they were looking for. They think maybe you haven’t been close enough to your family compared to other candidates.”

… the analogy is getting a bit stretched. 😃

Candidates could, of course, prep for a selection of potential likely questions. Here are more than 60 common questions that don’t even touch on your specialism, I’d argue you could find a factor of 10 more for specialism related questions, which is an unrealistic amount of prep - even assuming we could remember enough details of examples from years ago while prepping.

This isn’t just about struggling in the interview process, I’ve also experienced interviews where I was lucky enough to have recently worked very closely with the topics and questions posed, so got the job. I was aware at the time that if the questions had focussed on some other technical aspect that I hadn’t worked so closely on so recently, that I would have had a much harder time recalling the information confidently.

So let’s see if we can tackle this…

The Challenge

I’ll be focussed mostly on so called ‘soft skill’ assessment in this post. Technical capability assessment is a post for another time.

The Anti-Pattern:

• Interviews reward time-boxed memory recollection skills.

• Candidates are rewarded for clarity under pressure, story polish, and recall of anecdotes over deep expertise.

• People who tend to think deeply before speaking, or who use techniques to assist memory that aren’t available to them in interviews, can find themselves penalised.

• We optimise for “smooth talk” over “thoughtful analysis”.

Desired state:

• A process that measures competence, judgement, self-awareness, and learning habits.

• One that values thoughtfulness and humility as evidence of deep understanding.

• One that compensates for memory degradation by changing the format and time horizon of the evaluation.

Root Causes

Redesigned Interview Play Book

Framing Interview Exchanges

1. Pre-Interview Reflection Prompts

Give candidates a small set of “themes” or prompts beforehand (e.g., “Tell me about improving delivery effectiveness” or “Describe a complex cross-team dependency you managed”). They can prepare notes or examples.

➡️ This shifts the evaluation from memory recall to insight quality.

Benefits:

• Tests reasoning and synthesis.

• Reduces cognitive load and anxiety.

• Promotes depth over speed.

2. Scenario-Based Discussions Instead of “Tell Me About a Time”

Pose a realistic scenario (ideally aligned with your actual domain, perhaps one your experiencing now) and let the candidate work with you on it. For example:

“Imagine your platform’s test suite has slowed from 15 to 90 minutes. You’re managing a team under release pressure. What would you do first?”

You’re now evaluating judgement, prioritisation, and clarity of reasoning — not memory.

Bonus: If the candidate says “it depends,” follow up with: “Talk me through the key aspects you’d consider.”

3. Asynchronous or Written Components

Give a written problem or prompt prior to the interview. Let the candidate think, structure, and write a short response or diagram. Discuss it live.

➡️ Measures depth, coherence, and reasoning.

Also mimics real work: few people make critical decisions within an hour without references.

4. Behavioural Calibration Training for Interviewers

Train interviewers to spot false confidence and reward measured humility.

Examples:

• Experienced candidates often say ‘it depends’ — follow up to see if they can outline the variables clearly.

• Don’t penalise pauses; consider collaborating on the problem, like they are a colleague already.

• Ask them to draw trade-offs, not just final answers.

This shifts interviewer culture from performance judging to reasoning elicitation.

5. Narrative Reconstruction Interviews

Instead of asking for an isolated memory, reconstruct a timeline together:

“Let’s look at a project that spanned several months. What was happening at each stage?”

Provide a whiteboard or prompt sheet.

This helps trigger memory cues and lets candidates build a coherent narrative with your support.

Augmenting The Process

Interviews are a bit like Democracy in my mind - The worst system we have, apart from all the others.

Here is a list of some additional material I always look for when considering and assessing candidates:

• Linkedin Recommendations

Just like any review system, the more there are, the greater the trustworthiness. Look for themes in the recommendation comments, perhaps use an LLM to help you convert the recommendations into a cheat sheet of personalised areas to dig into or verify.

Do the people recommending them also have recommendations? We’re digging for likelihood of mutual respect between capable people.

Do they recommend others? What kind of things do they appreciate about others? Does that fit with your culture and aims for the role?

Are the recommendations coming from different places in the org structure (their manager, their reports, other departments)? This can give you a signal on level of outreach this person has.

My feeling on this is an interview process wont tell you as much as a careers worth of personal recommendations.

I understand that not everyone prioritised this or even has a linkedin page. It can only augment a candidate’s application, it can’t be used to eliminate.

• Community Contributions

Look out for Personal websites, Github presence, Blogs, Open source contributions, or anything else that’s relevant to the role. Getting a feel for what drives someone enough to put effort in and publish publicly can be a strong indicator for personal motivations.

If you’re a candidate - promote these things on your CV!

Best of luck and I hope that is of some help or has spawned some new ideas.