Recently, according to Gizmodo, research by a team of scientists led by Pattie Maes at the Massachusetts Institute of Technology shows that AI-generated content and deepfake technology are successfully distorting human memory, even implanting entirely fabricated false memories in participants' brains.
Experiments show that test subjects exposed to AI-modified photos were not only highly prone to misremembering the original images, but also displayed extremely high confidence in their false memories. This phenomenon is especially pronounced in scenarios such as political scandals, where it not only reduces the public's willingness to vote but also continues to influence political decision-making even after information has been corrected.
As generative AI is deeply deployed across various industries, the industry has generally pinned its hopes on advanced experts in 'human-machine collaboration' to review output results. However, the research points out that human users not only struggle to reliably identify AI hallucinations, but their own judgment is also highly susceptible to being misled by erroneous information.
Previously, in the incident where the US military launched an airstrike on an elementary school in Iran causing major casualties, an internal Department of Defense investigation showed that part of the cause was excessive reliance on the Palantir AI system driven by outdated data. This series of risks highlights that, in the current absence of foolproof error-correction mechanisms, blindly trusting and deploying AI systems is having profound and potentially catastrophic impacts on social truth and safety.