When Sources Disagree: The COVID Guidance Problem
This is Part 4 of our 16-week series on Context Degradation—the hidden failure modes that break AI systems before anyone notices. The Mask Guidance Chaos
Remember early 2020?
January: WHO advises masks only for healthcare workers February: CDC says healthy people don't need masks March: Some Asian countries report success with universal masking April: CDC reverses—now recommends cloth face coverings July: WHO finally recommends masks in some settings
For humans, this was confusing. For AI systems, it was catastrophic.
Any retrieval system pulling CDC and WHO documents from 2020-2021 faced an impossible task: the sources didn't just disagree—they disagreed with themselves across time.
The Source Conflict Problem
Most RAG systems are built on an assumption: retrieved sources are complementary. You gather information from multiple documents, synthesize them, and produce a coherent answer.
But what happens when sources legitimately conflict?
Source A says X Source B says not-X Both sources are authoritative Both sources are relevant to the query
This isn't a retrieval failure. The system retrieved correctly. This isn't a generation failure. The model works as designed.
This is a CLASH—a fundamental conflict in the source material that no amount of model capability can resolve.
Real Examples Beyond COVID
Source conflicts aren't unique to pandemic guidance. They appear everywhere:
Legal jurisdictions: California law says one thing, Texas law says another. Both are "correct."
Medical guidelines: American Heart Association and European Society of Cardiology have different recommendations for the same conditions.
Financial regulations: SEC guidance versus FINRA guidance versus state-level requirements. All authoritative. All different.
Technical documentation: Official docs say X, but the widely-used library fork changed that behavior three versions ago.
Evolving science: Yesterday's meta-analysis versus today's new study. Both peer-reviewed. Opposite conclusions.
How Current Systems Fail
The Averaging Problem
When faced with conflicting sources, most LLMs do something reasonable-sounding but wrong: they average.
"Some experts recommend X, while others suggest Y. Consider both approaches."
This sounds balanced. It's also useless—and potentially dangerous when one answer is clearly more current, more authoritative, or more applicable to the user's situation.
The Recency Illusion
Some systems prefer recent sources. But newer isn't always better:
A recent blog post isn't more authoritative than an older peer-reviewed study Today's hot take isn't more reliable than yesterday's consensus The latest documentation might have bugs the previous version didn't
The Authority Paradox
Preferring "authoritative" sources fails when authorities disagree. During COVID, the CDC and WHO were both authoritative. Preferring one arbitrarily isn't a solution.
The Confidence Collapse
Some models, when facing contradiction, become appropriately uncertain. But they signal this by hedging everything—including the parts that aren't actually disputed.
CLASH: Source Variance Without Resolution
In our framework, CLASH is high variance in the S (Superfluous) component—specifically, variance that represents genuine disagreement rather than mere irrelevance.
The signature is distinctive:
Multiple sources retrieved High inter-source variance in claims No clear resolution signal User query can't be answered without taking a position
CLASH is different from CONFUSION (noise + bloat) because all sources might be individually valid. The problem isn't that some sources are garbage. The problem is that valid sources disagree.
Why This Matters for Enterprise AI
In regulated industries, CLASH failures are particularly dangerous:
Healthcare AI: A diagnostic assistant that averages conflicting guidelines might recommend something that violates your hospital's specific protocols.
Financial AI: An advisor that blends SEC and FINRA guidance without distinguishing which applies might give compliance-violating recommendations.
Legal AI: A contract assistant that merges jurisdictional requirements might create documents that satisfy neither jurisdiction.
The failure mode isn't "wrong answer." It's "confident synthesis of irreconcilable positions."
What COVID Taught Us
The pandemic was a stress test for information systems. We learned:
1. Temporal context matters Guidance from March 2020 and March 2021 shouldn't be weighted equally. But retrieval systems don't naturally understand that.
2. Authority is contextual CDC is authoritative for US guidance. WHO is authoritative for global guidance. Neither is universally "more right."
3. Users need to know about conflicts The worst outcome isn't "I don't know." It's "here's a confident answer" when the sources fundamentally disagree.
4. Synthesis isn't always the right answer Sometimes the correct response is "these sources conflict—here's what each says."
What a Certificate Would Have Caught
A Context…