Last month, I reported on the widening web of connections between the effective altruism (EA) movement and AI security policy circles — from top AI startups like Anthropic to DC think tanks like RAND Corporation. These are linking EA, with its laser-focus on preventing what its adherents say are catastrophic risks to humanity from future AGI, to a wide swath of DC think tanks, government agencies and congressional staff.
Ethical Considerations Beyond the Present
While critics of the EA movement argue that the focus on long-term, existential risks overlooks the pressing issues of bias, misinformation, and traditional cybersecurity, the influence of EA on AI security efforts cannot be ignored. The movement, initially aimed at ‘doing good better,’ is now heavily funded by tech billionaires who consider preventing an AI-related catastrophe their number one priority.
“The threat of opportunistic criminals, terrorist groups, or highly-resourced nation-state operations accessing the weights of the most sophisticated and powerful LLMs is alarming because ‘if an attacker got access to the entire file, that’s the entire neural network.'” – Jason Clinton, CISO of Anthropic
Researchers from RAND Corporation, one of the think tanks with close ties to the EA community, also express concern over the security of AI models. They warn of potential risks, such as malicious actors misusing AI models for biological weapon development.
Different Perspectives on EA Influence
Not all AI and policy leaders share the same viewpoint on the influence of effective altruism. For example, Nick Frosst, co-founder of Cohere, a competitor to Anthropic, does not believe that large language models pose an existential threat. While Cohere prioritizes the protection of its model weights, Frosst sees the business risk associated with unauthorized access as the main concern.
“A big flaw in effective altruism is to ‘assume that you can look at the good you’re doing and assign a number and know exactly how effective this is. It ends up in weird places like, hey, we should make as much money as possible. And we should put it all [towards combating] the existential risk of AI.'” – Nick Frosst, Co-founder of Cohere
Similarly, Yoav Shoham, co-founder of AI21 labs, emphasizes that while his company keeps its model weights secret, he does not believe they are necessarily the key enabler for bad actors. He highlights the importance of policy measures and task-specific models to mitigate potential risks.
The Debate within Policy Circles
Within policy circles, there is a growing awareness of the influence of effective altruism on AI security. While some experts view EA as a valuable perspective that should be considered, others approach it with caution.
Marek Posard, a researcher at RAND, acknowledges that there are real concerns in the field of AI but believes that the current focus on AI philosophical debates, including effective altruism and effective accelerationism, can be a distraction. He emphasizes the importance of diversity of thought in policy discussions.
Other policy experts, like Mark Beall from Gladstone AI, advocate for common sense safeguards and responsible AI policy driven by technical realities. They argue that effective altruists are not the sole drivers of AI risk mitigation efforts within government agencies.
“At DoD, we established responsible AI policy and invested heavily in AI safety. Our mission has always been to accelerate responsibly. And for those on the fringes who think that US officials haven’t been independently tracking AI risks — or that they are somehow being duped — are wrong.” – Mark Beall, CEO of Gladstone AI
While some experts, like Dan deBeaubien from the SANS Institute, point out that they are more focused on present-day security risks rather than hypothetical existential threats, others, like Ian Bremmer from Eurasia Group, believe that there is a legitimate debate about securing model weights and the impact of AI on civil society.
“If effective altruism’s focus is really altruism, we need to make sure that AI isn’t aligning with business models that undermine civil society – that means testing models not just for misuse but also to see how normal expected use impacts social behavior (and the development of children).” – Ian Bremmer, President of Eurasia Group
The influence of effective altruism on AI security policy is a complex topic that sparks varied perspectives within the AI and policy communities. While the focus on long-term existential risks has drawn criticism, the movement continues to drive important discussions on the implications and safeguards of AI technology.
