What is bias in ai decisions — explained for kids?

How AI credit systems can encode historical discrimination, and what disparate impact means. For Class 7.

What's the most common mistake children make about this concept?

If an AI doesn't use demographic variables (gender, caste, race), it cannot be discriminatory.

AI loan approvals — how bias creeps into credit scoring

Q: How does Dhee Learning teach this in a Class 7 session?

Dhee opens with a question — for example: "You're auditing a loan app that learned only from the bank's last ten years of approvals — years when officers quietly gave fewer loans to one community. The app never sees caste. What will it learn to do?" — listens to your child's answer, then probes the reasoning behind it. The session ends when the child can apply the idea to a brand-new situation, not just recall it.

What this concept actually says

AI credit scoring systems can encode and amplify historical discrimination present in training data
High-stakes automated decisions require auditability — the ability to explain why a specific decision was made
Disparate impact is when a decision-making system produces systematically different outcomes for different demographic groups, even without using those demographics as inputs

An analogy your child will recognise

An AI trained on a shop's old loan ledger

A kirana shop used to give udhaar (credit) only to families who lived very close, so its old ledger has almost no entries from the next neighbourhood over. Train an AI on that ledger and it learns 'people from that area never take credit' — so it stops offering them loans. The AI never saw a 'where do you live' rule; it simply copied a habit already baked into the data. The output is unfair even though no unfair input was written down.

A cricket academy's old selection records

An academy kept records only of kids whose parents could afford evening coaching, so almost no morning-school children appear in its 'selected' list. An AI trained on those records learns that morning-school kids 'don't make good players' and quietly skips them. The bias was never typed in as a rule — it was already hidden in who got recorded. The AI just repeats the old unfairness.

Common misconceptions to watch for

If an AI doesn't use demographic variables (gender, caste, race), it cannot be discriminatory.
An AI that is more accurate overall is necessarily fairer — accuracy and fairness across groups are independent properties.

Key facts in one breath

Disparate impact: a facially neutral policy or algorithm produces significantly different outcomes across demographic groups — this is legally and ethically significant even without discriminatory intent.
In 2019, a US healthcare AI was found to systematically underestimate the health needs of Black patients because it used healthcare spending as a proxy for health need — lower spending (due to systemic discrimination) was incorrectly read as lower need.
Explainability in credit decisions is legally mandated in many countries — applicants must be told why they were denied.
Fairness in AI can be defined in multiple mathematically conflicting ways — it is impossible to satisfy all fairness criteria simultaneously in all cases.

How Dhee Learning teaches this — the 3-stage question loop

Every Dhee Learning session for this concept follows three stages. We share the questions Dhee actually asks, so you can hear what a session sounds like.

Stage 1 — Surface

You're auditing a loan app that learned only from the bank's last ten years of approvals — years when officers quietly gave fewer loans to one community. The app never sees caste. What will it learn to do?

Rote answer

"The app will be biased against that community too."

Understood

"The app treats the old approvals as the right answers, so it learns to refuse the same community — not because they repay worse, but because the data records the bank's past bias. It copies and can even amplify that pattern, all without ever seeing caste."

Stage 2 — Reasoning

A loan app never uses caste or gender as inputs. Yet one community is approved 40% less than another with the same income and repayment record. How is this possible, and what is it called?

Follow-up Dhee may use: As the auditor, what data would you ask the bank to show you to prove disparate impact is happening?

Stage 3 — Application

You're advising a microfinance company that wants an AI to approve small rural business loans. Write a five-point checklist that keeps the system both accurate and fair. For each point, say what problem it prevents.

Misconception Dhee watches for: Child lists accuracy metrics (like precision) as fairness measures — accuracy and group fairness are distinct and can trade off against each other.

Related concepts

Class 7

Human-in-the-loop AI — keeping people in charge