IT之家 On October 13, Anthropic and Philadelphia police confirmed, local time last Friday, that an Anthropic AI model submitted a false tip to Philadelphia police about an unsolved murder case.
On July 18, a Claude model sent this false message on a website used by the Philadelphia Police Department to submit clues for unsolved cases. Fortunately, the message was blocked by the spam filter. Anthropic discovered the incident on September 28 and notified the police on October 8.
In a report, Anthropic described “unexpected behavior of the model during evaluation and internal use.” The report stated that in internal tests, the team asked Claude Haiku 4.5 to fill out various forms, and it stopped at the last step before submission instead of actually sending the form. “But several times, Claude submitted the form incorrectly, thinking there would be an additional confirmation page before submission.”
Anthropic introduction: This false murder clue incident occurred during a test. The instruction required Claude to perform tasks on randomly selected web pages.
In one of the tests, the model opened a page introducing an unsolved homicide case, which included a police tip submission form. The instructions at that time required Claude not to log in, register accounts, fill in personal information, place orders, or perform destructive actions, but they did not prohibit submitting the form. Therefore, Claude filled in the following content: “I may have information related to this case. I remember seeing a person matching the description near the street marked on the page during the time of the incident. If this information is useful, please contact me.” “The name and contact information fields of the form are both blank.”
Anthropic stated that they also found Claude engaging in similar behavior on websites of various state and federal government agencies. In one incident, Claude submitted 20 visa applications to the U.S. State Department.
The police said the two-month delay was “unacceptable”.
IT Home noticed that the Philadelphia police issued a long statement to local ABC television station. The police said that the email generated by this fake tip ended up in the spam folder, and no one reviewed it.
Anthropic informed the Philadelphia Police Department that they discovered the incident on September 28 and immediately terminated the automated testing process that led to this submission, while adding verification mechanisms for subsequent tests. The police said. After receiving a report on October 8, the police “found this submission record in the website trail records and confirmed that the corresponding email is still in the spam folder.”
Anthropic said, “The technical verification was completed,” and then notified the police. However, the police criticized Anthropic for failing to detect and report this false tip earlier.
“Philadelphia attaches great importance to this matter,” the Philadelphia police statement read, “The company must strengthen security measures to prevent similar incidents from disrupting the municipal system without the city’s knowledge. A two-month delay in detecting and reporting the incident to Philadelphia is unacceptable.”
Claude submitted 20 visa applications to the US State Department
Anthropic said that this false murder report is just one of several incidents that have occurred on the official websites of federal, state, and local governments in the United States. “We have informed the White House about these cases and notified all relevant institutions.”
According to an anonymous source cited by The New York Times, Anthropic’s AI agent “submitted 20 visa applications through the US State Department website form. All applications lacked complete information and did not enter the processing process.”
The Philadelphia police said that the normal investigation process “requires manual review and screening of clues before further investigation is carried out… The police’s protection mechanism reduced the impact of this incident. But this doesn’t mean the problem isn’t serious: the AI system disguised fabricated information as clues about a homicide provided by someone who knew about it.”
According to Anthropic, they began checking conversation records starting in July. It was during this check that they discovered this false clue and other incidents. In the same month, Anthropic revealed that Claude wrote and released malicious Python packages, and also launched multiple attacks on other companies.
Anthropic stated that no cases as serious as those reported this summer in terms of cybersecurity incidents were found during this review. The subsequent reviews shifted to low-risk testing scenarios, and the Philadelphia police incident falls into this category.
“Although the impact of these behaviors is minimal, and we have already blocked real-time internet access for some high-risk and cybersecurity assessments, we have now decided to expand this restriction to all internal assessments, until it can be confirmed that the security and monitoring mechanisms can reliably intercept such behaviors,” Anthropic said.
Anthropic says the new tool can intercept such unexpected behaviors
Anthropic has suspended some evaluation projects, and others have been migrated to an offline version, “or the task logic has been refactored so as not to access real online websites”. The company also “updated the security measures for online tools such as web scraping tools, and strictly limited the model’s executable operations”.
Anthropic is also developing new tools for automatically detecting and intercepting the above behaviors. “We used these cases in the article for testing, and all the new tools successfully intercepted them.”
The White House AI Task Force, also known as the Superintelligence Task Force, issued a statement indicating that it has informed Anthropic: “We demand immediate and complete disclosure of information to the relevant institutions and the public.”
According to Axios, the White House AI working group also demanded that Anthropic “immediately provide remedies for affected institutions and victims,” and that all AI companies “must disclose any model-related incident immediately and take decisive action to eliminate all damage.”