The Hidden AI in Your Life: Systems Deciding About You Without Telling You

Back to Blog

Most people can name the moment a human turned them down. Far fewer can name the moment software did — and that's the point. Automated decision systems are now embedded in hiring, housing, lending, insurance, and healthcare, and they are typically invisible by design: no notification, no explanation, no obvious place to object. You just get a rejection that reads like a form letter, because it is one. The good news, and the reason this article exists, is that the paper trail you're entitled to is stronger than most people realize — and the systems are built on the assumption that you'll never ask for it.

Where These Systems Actually Sit

This isn't a forecast. Each of the following is in production right now, at scale, in ordinary consumer life:

• Hiring. Applicant tracking and screening tools rank and filter candidates before any person reads a résumé. In the ongoing Mobley v. Workday litigation, a federal court conditionally certified a nationwide class of older workers, noting that the platform's screening may have touched more than a billion applications [3].
• Housing. Tenant screening scores decide who gets shown an apartment. SafeRent settled for $2.3 million over an algorithm that allegedly scored Black and Hispanic applicants and housing-voucher holders lower — partly because it leaned on credit data while ignoring that a voucher guarantees much of the rent [2].
• Health coverage. The suit over UnitedHealth's nH Predict tool alleges it estimated how long patients "should" need post-acute care, and that coverage was cut on that estimate rather than the treating physician's judgment [1].
• Credit and insurance. Scoring models set approvals and prices, increasingly using behavioral inputs well beyond payment history.
• Government services. Eligibility, fraud flags, and benefits determinations increasingly run through risk models first.

The Design Assumption: You Won't Appeal

The single most revealing number in any of this is 0.2%. That's the share of denied claims the UnitedHealth complaint says were appealed — and the same filing alleges roughly nine in ten denials that were challenged got overturned on appeal or before a judge [1]. Read those two figures together and a business model appears: a tool can be wrong most of the time and still work perfectly, provided almost nobody pushes back.

That asymmetry is the actual subject of this article. These systems aren't primarily dangerous because they're biased, though some demonstrably are. They're dangerous because they're quiet, and quiet decisions don't get contested. Every rejection that arrives with no explanation is betting on your exhaustion.

The Rights You Actually Have

Here's where it gets more useful than most coverage admits. You have real, specific entitlements — they're just rarely advertised.

What to Actually Do

If a decision goes against you and you suspect software drove it, this sequence costs a stamp and puts everything in writing:

• Ask, in writing, whether an automated system was used. One sentence. The written record matters more than the answer, and vague replies are themselves informative.
• Request the specific reasons and the data used. Not "you didn't meet our criteria" — which criteria, and which data points about you.
• Get the underlying report and check it. Screening files carry mismatched names, stale addresses, and other people's records with startling regularity.
• Explicitly request human review. Use those words. In the EU/UK it's a legal right; in the US it increasingly triggers an internal process that exists mainly because regulators asked for it.
• Appeal even when it feels pointless. The 0.2% figure is the whole ballgame. Appeals are where these systems' error rates become visible.
• Escalate to a regulator. CFPB for credit, HUD for housing, EEOC for employment, your state insurance commissioner, or the relevant data protection authority in Europe.

Back to Blog