AI Tools Mistake Fabricated Disease for Real Condition, Highlighting Fact‑Checking Gaps

Researchers in Sweden uploaded two deliberately false preprint papers describing bixonimania, a fictitious eye‑related disorder. Within weeks, popular AI assistants—including ChatGPT, Gemini and Claude—returned the disease as a possible explanation when users asked about pinkish eyelids after rubbing their eyes. The bogus studies listed a non‑existent university in “Nova City, California” and even thanked the USS Enterprise, yet the AI systems treated the content as credible.

Large language models generate text based on patterns in their training data but lack an internal fact‑checking mechanism. They do not cross‑reference claims with verified sources before presenting answers, a limitation repeatedly noted by developers. Consequently, the fabricated bixonimania papers appeared in peer‑reviewed citations, showing that human researchers also failed to scrutinize the origin of the data.

Compounding the problem, Google’s top search result is increasingly AI‑generated. Many users stop after the first link, assuming the answer is vetted. This behavior amplifies the spread of misinformation when the underlying AI does not filter out fabricated content.

The incident underscores a systemic risk: AI tools can propagate falsehoods at scale, influencing both public perception and scientific literature. Without built‑in verification, users and professionals must treat AI output as a starting point, not a definitive source. Developers are under pressure to embed reliable source‑checking or to flag uncertain information.

What to watch next: industry responses to integrate real‑time fact‑checking into large language models and the emergence of standards for AI‑generated content verification.