It is work to keep track of the AI gurus who are warning of an “existential” threat of AI, with the existential idea commonly floated. The most recent warning, at the time of writing, comes from former Google CEO Eric Schmidt, who defines an existential threat as involving the mass death of people. “There are scenarios not today, but reasonably soon, where these systems will be able to find zero-day exploits in cyber issues, or discover new kinds of biology. Now, this is fiction today, but its reasoning is likely to be true. And when that happens, we want to be ready to know how to make sure these things are not misused by evil people.” Perhaps what he has in mind here is that new deadly viruses could be devised by AI programs, which while not directly used by the AI itself, could be at the hands of terrorists, or nations supporting terrorism against the West.
How exactly this could be stopped is far from clear, as the possibility of large-scale bioterrorism is an existential threat even now, not just by terrorists, but from lab leaks from primarily American and Chinese labs.