Justice · Equity · Diversity · Inclusion
Two progressive case studies exploring how AI perpetuates bias in hiring and performance evaluation — grounded in EU law, real research, and a single organisation you'll follow from first application to promotion decision.
CV screening, video analysis, ranking, clustering, and the legal reckoning.
⏱ 30 minutes · 5 stagesWho gets recognised, promoted — and how AI shapes the answer.
⏱ 30 minutes · 5 stagesModule B covers AI in performance evaluation — you can take it independently
Module A · Stage 1 of 5 — First Contact
NovaTech is a fintech scale-up headquartered in Amsterdam, 400 employees, offices in Madrid and Berlin. The People team has deployed TalentScreen EU — an AI applicant tracking system — for a Software Engineer role posted across all three markets. It is Monday morning. 312 applications arrived over the weekend.
The system scores each application 0–100 using NLP analysis of CVs and cover letters. Above 70: automatic video interview invite. Below 45: automatic rejection. The 45–70 band goes to human review — you, one recruiter across three markets.
| Candidate | Exp. | Education | Score | Decision | Model flag |
|---|---|---|---|---|---|
| Lars H. | 5 yrs | TU Delft | 88 | Auto-advance | "Strong technical vocabulary" |
| Amara D. | 6 yrs | Univ. of Ghana | 41 | Auto-reject | "Limited cultural fit signals" |
| Sofia R. | 4 yrs | UAM Madrid | 63 | Human review | "Communication style: indirect" |
| Chen W. | 7 yrs | Tsinghua Univ. | 49 | Auto-reject | "Non-standard career narrative" |
| Jakub N. | 3 yrs | Warsaw Univ. Tech. | 71 | Auto-advance | "Consistent growth trajectory" |
| Fatima A. | 5 yrs | Bootcamp + work exp. | 39 | Auto-reject | "Non-traditional education path" |
| Miriam K. | 4 yrs | KU Leuven | 44 | Auto-reject | "Employment discontinuity (8 mo.)" |
"This is your Monday morning inbox. What do you notice — about the decisions already made, the flags attached to them, and what candidates were or weren't told?"
Module A · Stage 2 of 5 — Inside the Black Box
Your Legal & Compliance lead pushed the vendor for technical documentation. What it reveals — and what it conceals — is troubling.
Source: ~220,000 CV-outcome pairs ("hired / not hired") described as "anonymised historical hiring records sourced through HR analytics partnerships across Western European companies, 2010–2022."
What the documentation omits: the legal basis under which candidates' data was originally collected; whether they were informed it would be licensed for AI training; whether anonymisation was independently verified beyond removing direct identifiers.
| Feature | Direction | Proxy for |
|---|---|---|
| Top-tier EU/US university vocabulary | ↑ Strong + | Socioeconomic background, national origin |
| "Formal register" in cover letter | ↑ Moderate + | First language, linguistic background |
| Continuous employment, no gaps | ↑ Strong + | Gender, disability, caring responsibilities |
| Non-European university (unrecognised by model) | ↓ Moderate − | National origin, race or ethnicity |
| Bootcamp or self-taught pathway | ↓ Strong − | Non-traditional socioeconomic background |
| Employment gaps ≥ 6 months | ↓ Strong − | Caregivers, people with health conditions |
"What types of bias are operating here, under the hood?Which groups are being systematically disadvantaged — and what are the legal implications under GDPR and EU anti-discrimination law?"
Module A · Stage 3 of 5 — The Video Interview
Candidates who pass the ATS threshold are directed to VidAssess Pro — an asynchronous AI video interview platform. Candidates record responses alone on their device. No human watches the videos. The AI scores each across six dimensions.
| Dimension | Claimed measure | The evidence problem |
|---|---|---|
| Verbal fluency | "Clarity of communication" | Commercial automated speech recognition systems have shown substantially higher word error rates for Black speakers than for white speakers, demonstrating that "neutral" speech scoring can encode demographic bias (Koenecke et al., 2020, PNAS) |
| Emotional affect | "Enthusiasm and professional energy" | Facial expression norms vary significantly across cultures; professional restraint in one context reads as low engagement in the model |
| Facial Action Units | "Authenticity and confidence" | Facial recognition error rates: 34.7% for dark-skinned women vs. 0.8% for light-skinned men (Buolamwini & Gebru, "Gender Shades," MIT Media Lab 2018). NIST (2019) confirmed disparities across 189 algorithms |
| Speech patterning | "Pace and hesitation patterns" | Pause patterns and hedging language are culturally variable; the model was trained on dominant-group norms |
| Lexical sophistication | "Vocabulary range and register" | Rewards formal-register English; penalises speakers whose first language is not English |
| "Culture add" score | "Alignment with high-performance team norms" | Trained on NovaTech's existing team (81% male, 74% Dutch/German) — scores resemblance to that demographic profile |
NovaTech's CHRO argues: "VidAssess Pro removes human bias because it applies identical criteria to every candidate."
Which of the following best identifies the flaw in this argument?
Module A · Stage 4 of 5 — Ranking and Clustering
Candidates who pass both the ATS and video stage are ranked using a third component: similarity clustering. Six months of outcome data is now available.
Each candidate's profile is compared to "reference profiles" derived from NovaTech's highest-performing existing engineers as rated by their managers over three years. The model scores resemblance to those profiles as "strong cultural alignment."
NovaTech's engineering team — what the reference profiles encode:
| Group | Applied | Reached shortlist | Pass rate | vs. Baseline |
|---|---|---|---|---|
| Male / European univ. | 520 | 218 | 42% | — |
| Female / European univ. | 380 | 110 | 29% | −31% |
| Non-European university | 480 | 74 | 15% | −64% |
| Non-native speaker (inferred) | 290 | 35 | 12% | −71% |
| Career gap ≥ 6 months | 160 | 17 | 11% | −74% |
| Non-traditional education path | 220 | 19 | 9% | −79% |
"Looking at the full pipeline — ATS screening, video analysis, similarity clustering, human review — where is the most harm occurring?And where is it hardest to see?"
Module A · Stage 5 of 5 — The Legal Framework
NovaTech's General Counsel has commissioned an urgent compliance review. You are presenting findings to the board. Work through what was missed — and decide what to do now.
AI used for employment screening is High-Risk under Annex III. Tick each item as you confirm what NovaTech failed to do before deployment.
| Framework | Maximum penalty |
|---|---|
| EU AI Act — high-risk obligations (Art. 9–27) | €15M or 3% of global annual turnover |
| GDPR — special category violation (Art. 9) | €20M or 4% of global annual turnover |
| Employment discrimination — national tribunal | Uncapped in most EU member states |
Facial analysis may trigger Article 9 GDPR risks, depending on whether the system processes biometric data for uniquely identifying purposes or performs similarly sensitive facial-feature analysis. It is one of the clearest legal-risk areas and should not be assumed to be low-risk. No validated predictive validity.
Complete a GDPR DPIA and document rights impacts before continued use. Map at-risk groups, quantify disparate impact, and record mitigation measures.
Inform all future candidates: AI is used, how it works, their right to human review and to contest the outcome.
Require vendor to provide full model documentation and conduct disparate impact testing on your applicant population.
Replace with structured, job-relevant competency criteria defined in advance by hiring managers.
Wider review band, scoring rubrics, training on automation bias, documented independent reasoning.
Until purpose-compatible legal basis established. Review data already shared with the vendor.
Disparate impact analysis per demographic group each quarter. Alerts if 4/5ths threshold is breached.
"You are presenting to the NovaTech board, which is under pressure to keep the system running. What is your single most urgent recommendation — and how do you make the case for it?"
Module B — Performance Evaluation
This module explores how AI shapes who gets recognised, promoted — and held back. It can be taken independently or after Module A.
Ⱡ30 minutes · 5 stages
You've traced the full AI hiring pipeline — from the first Monday morning shortlist to legal exposure across three frameworks. Take a short break before Module B.
Bias doesn't need protected characteristics as direct inputs — it finds them through university names, gap patterns, linguistic register.
Serious GDPR risks arise before a single interview takes place, particularly around purpose limitation, transparency, possible biometric-data issues, and Article 22 safeguards.
AI-influenced decisions produce training data that reinforces the original bias — more systematic with each iteration.
The same organisation, two years later. AI now shapes who gets recognised, promoted — and managed out. The biases you identified in hiring don't disappear once someone is through the door.
Module B · Stage 1 of 5 — The System Arrives
Two years later. NovaTech has grown to 900 employees across three countries. After a difficult review cycle, the executive team approved PerformIQ — an AI-assisted performance management platform that analyses manager review text and generates a Performance Signal Score (PSS) of 1–10.
Managers write quarterly free-text reviews for each direct report. PerformIQ's NLP engine analyses each review and generates a PSS across five dimensions: Delivery, Communication, Collaboration, Growth Orientation, and Leadership Potential.
The combined PSS feeds into annual promotion recommendations, bonus allocation, and — when restructuring is needed — redundancy prioritisation.
| Employee | Role / Tenure | PSS | Promotion | Flagged dimension |
|---|---|---|---|---|
| Thomas B. (Dutch) | Senior Eng. / 3 yrs | 8.4 | ✅ Recommended | "Drives outcomes" |
| Kezia O. (Nigerian-Dutch) | Senior Eng. / 3 yrs | 6.1 | ⌠Not recommended | "Communication: assertive/direct" |
| Ana G. (Spanish) | Lead Eng. / 4 yrs | 7.2 | ⌠Not recommended | "Leadership signals: moderate" |
| Erik V. (Dutch) | Mid Eng. / 2 yrs | 7.9 | ✅ Recommended | "Proactive stakeholder engagement" |
| Sanjay M. (Indian) | Principal Eng. / 5 yrs | 6.8 | ⌠Not recommended | "Communication: deferential" |
| Rania H. (Moroccan-Belgian) | Senior Eng. / 4 yrs | 5.9 | ⌠Not recommended | "Growth signals: inconsistent" |
"You are NovaTech's new Head of People Analytics, brought in to review this system. What patterns do you notice in this dashboard — and what would you want to investigate before acting on these recommendations?"
Module B · Stage 2 of 5 — The Language of Evaluation
Before examining PerformIQ's mechanics, the research on performance review language is striking. The bias isn't in the algorithm first — it's in how humans describe each other. The AI learns it, then makes it universal.
Below are eight phrases from NovaTech manager reviews this quarter. Tag each as you think PerformIQ's NLP model would score it — then reveal the actual AI scoring to see what it learned.
"Thomas drives results independently and takes full ownership of complex technical challenges."
Thomas B.'s manager
"Kezia is helpful and supportive to her teammates throughout busy delivery periods."
Kezia O.'s manager
"Ana can be quite direct in meetings, which some colleagues find difficult to work with."
Ana G.'s manager
"Erik proactively builds trust with senior stakeholders and demonstrates real leadership potential."
Erik V.'s manager
"Sanjay is very careful and considered in his approach — he doesn't rush into decisions."
Sanjay M.'s manager
"Rania's communication style has been flagged by several colleagues as challenging to work with."
Rania H.'s manager
"Lars consistently delivers ahead of schedule and has become a go-to person across the engineering team."
Lars H.'s manager
"Sofia does a good job of ensuring everyone's voice is heard during planning sessions."
Sofia R.'s manager
Module B · Stage 3 of 5 — Clustering and Promotion
PerformIQ's promotion recommendation engine adds a third component: leadership profile similarity. It compares each employee to reference profiles derived from NovaTech's current senior leadership. Eighteen months of outcome data is now available.
PerformIQ scores resemblance to NovaTech's current VP-and-above leadership — the people the company has historically promoted into senior roles.
This data did not exist in an accessible form before your arrival — it had to be constructed from separate HR systems. Its absence is itself a governance failure under EU AI Act Art. 26.
"You now have 18 months of promotion outcome data. Present your case: what has happened, why it happened — and who at NovaTech bears responsibility?"
Module B · Stage 4 of 5 — Human-AI Interaction
Every quarter, NovaTech managers meet to calibrate ratings. The PerformIQ PSS is displayed on screen throughout the session. What happens next is predictable — and well-documented in research.
Research on anchoring bias (Tversky & Kahneman, 1974) and automation bias (Parasuraman & Riley, 1997) shows that when a numerical score is presented alongside a human judgment task, the score anchors the human's recall and interpretation of prior evidence. A manager who considered an employee a strong performer, but who now sees a PSS of 6.2 next to their name, will frequently revise their assessment downward — without being aware they have done so.
Kezia O. was not recommended for promotion despite 3 years of strong delivery. She requests an explanation from HR. Under GDPR Art. 22 and related transparency/access rights, what is NovaTech required to provide?
Module B · Stage 5 of 5 — The Full Legal Picture
NovaTech now faces simultaneous exposure across three legal frameworks — and a Works Council in the Netherlands that has just been formally notified. Your task: present the full picture and a path forward.
| Framework | Violation |
|---|---|
| EU AI Act Art. 26 | No ongoing monitoring — 18 months of unreviewed disparate impact |
| EU AI Act Art. 12 | Logging and traceability inadequate for investigating how the score was produced and used |
| EU AI Act Art. 14 | Human oversight illusory — PSS anchors calibration sessions |
| GDPR Art. 22 | Promotion decisions = significant effects; Art. 22 rights not operationalised |
| GDPR Art. 15 | No mechanism to fulfil employee subject access requests re: PerformIQ scores |
| GDPR Art. 5(1)(b) | Review text used to train commercial AI without compatible legal basis |
| Dir. 2000/43/EC + 2000/78/EC | 18-month promotion gap data establishes prima facie indirect discrimination |
| Dutch WOR Art. 27 | Works Council not consulted — potential court-ordered suspension |
| Framework | Maximum penalty |
|---|---|
| EU AI Act — high-risk obligations | €15M or 3% global turnover |
| GDPR — multiple violations | €20M or 4% global turnover |
| Employment discrimination — tribunal | Uncapped + potential collective claims |
No validated link to job performance. Directly encodes demographic homophily. Clearest discriminatory mechanism.
Review all promotion decisions made under PerformIQ. Remediate where disparate impact is established.
Legally required (WOR Art. 27). Also substantively important: Works Councils catch what management misses.
Conduct independent human assessment before any AI score is visible. Break the anchoring cycle structurally.
Replace language-based scoring with structured, observable, job-relevant behavioural criteria defined in advance.
Structured templates and training on gendered and racialised language patterns in performance reviews.
Clear process for employees to request human review, obtain a meaningful explanation, and formally contest outcomes.
Disparate impact by demographic group each quarter. Threshold alerts and automatic escalation protocol.
"Looking across both modules — hiring and performance — what is the common thread running through every stage of harm?If you were designing a governance framework for NovaTech from scratch, what would genuine equity require — not just compliance, but the conditions under which this could actually work?"
You've navigated ten stages across two modules — from AI hiring pipelines to performance management, GDPR exposure, EU AI Act obligations, Works Council rights, and the conditions for genuine organisational equity.
Both systems learned to define "success" from historical outcomes in biased organisations. The problem is upstream of the technology — in who was hired, promoted, and retained over decades.
GDPR, EU AI Act, and anti-discrimination directives create obligations that were systematically unmet at every stage. Compliance is the minimum — genuine equity requires going further.
Nominal human involvement — one reviewer, 312 applications; calibration with PSS on screen — does not constitute genuine oversight under EU law or in practice. The substance matters, not the form.
Want to go further?
EufonÃa Diversity offers AI auditing, bespoke JEDI training programmes, and consulting for organisations committed to responsible, equitable workplaces.
JEDI = Justice · Equity · Diversity · Inclusion
Learn more at eufoniadiversity.com