We scan new podcasts and send you the top 5 insights daily.
As generative models make static take-home work and essays impossible to trust, assessment must shift toward real-time observation. Educators and hiring managers test true competency by having candidates explain decisions aloud or defending their output via post-hoc oral interrogation.
Standard benchmarks fall short for multi-turn AI agents. A new approach is the 'job interview eval,' where an agent is given an underspecified problem. It is then graded not just on the solution, but on its ability to ask clarifying questions and handle changing requirements, mimicking how a human developer is evaluated.
To determine if an employee critically engaged with AI-generated content, bypass reading the lengthy document. Instead, directly question them on its substance. Their ability to confidently defend, elaborate on, and explain the material is the true test of their understanding and ownership of the work.
With LLMs making remote coding tests unreliable, the new standard is face-to-face interviews focused on practical problems. Instead of abstract algorithms, candidates are asked to fix failing tests or debug code, assessing their real-world problem-solving skills which are much harder to fake.
AI can easily generate high-quality written reports, making them an unreliable measure of student understanding. Oral examinations and project defenses are becoming critical to verify a student's actual comprehension and problem-solving skills, rather than their ability to prompt an AI.
In response to AI making take-home assignments unreliable, universities are reverting to "old-school" assessment methods like in-class blue book exams, spontaneous writing sessions, and oral exams to ensure student work is authentic.
To assess a candidate's ability to use AI as a thinking partner, have them solve a problem with an LLM. The key is observing their follow-up prompts and their ability to guide the AI step-by-step, rather than just accepting the initial output.
An internal MIT report states that LLMs can effectively complete essays, problem sets, and coding assignments, making traditional homework an unreliable measure of knowledge. The university now recommends a pivot to assessments that cannot be easily completed by AI.
The proliferation of AI-polished communication has created an "authenticity crisis," eroding trust in written and virtual interactions. As a result, employers and universities are reverting to in-person formats for interviews and exams to verify that a candidate's or student's thoughts are genuinely their own.
Since AI assistants make it easy for candidates to complete take-home coding exercises, simply evaluating the final product is no longer an effective screening method. The new best practice is to require candidates to build with AI and then explain their thought process, revealing their true engineering and problem-solving skills.
Traditional hiring methods are becoming obsolete. To hire PMs, Mercor uses a single test for AI fluency, then immediately shifts to whiteboarding. This assesses fundamental skills like judgment, experimental design, and statistics—capabilities that cannot be outsourced to AI and reveal a candidate's core thinking.