Why has Cognition fallen behind the industry standard for AI Safety?

Amid fierce debates surrounding AI safety, it’s easy to forget that most disagreements concern what particular risks we face and how to address them. Very few people will try to argue that there are no risks or that serious caution isn’t warranted.

In light of this, there is an emerging consensus among policy experts (and the AI industry itself) that risk evaluation is critical. This consensus has informed some of the state-of-the-art safety policies released by OpenAI, Anthropic, and Google — as well as the commitments that 16 leading AI companies made just last month.

These AI companies agree that, as we develop increasingly capable AI systems, we must rigorously evaluate them for potential dangers.

These risks include near-term threats, such as the proliferation of misinformation or the potential to produce biased and discriminatory outputs, as well as speculative future threats, such as the ability to aid malicious actors in developing computer malware or bioweapons.

And this consensus isn’t just industry reputation-washing:

1. Independent AI safety experts in academia and civil society have widely endorsed this model as a first step toward responsible self-governance for AI developers.
2. World governments have supported it — the governments of the United Kingdom and South Korea, for example, spearheaded the 16 aforementioned corporate risk-evaluation commitments.
3. According to recent polling conduced by the AIPI, the majority of the US population supports such policies.

It’s no surprise that this emerging industry standard is so popular. It’s basically the equivalent of saying: “We are, at a minimum, going to test new AI products to see if they pose significant threats. If they do, we won’t release them — at least not before implementing significant safeguards to protect against misuse.”

But still, only some people are on board. Following an investigation from The Midas Project, it appears one startup has chosen to eschew risk evaluation policies in favor of rushing to the market.

Cognition is an AI startup developing Devin — an autonomous AI agent that can write and execute code, and otherwise act independently to solve real-world problems. Granting such autonomy to AI systems has long been theorized as a major risk factor for dangerous AI behavior. In fact, this could be why Cognition hasn’t released any public safety policies. Do they think that if they publicly commit to conducting risk evaluation, the results won’t look good for them?

Cognition may have been hoping the public wouldn’t notice how far behind the industry standard they’ve fallen. That’s why The Midas Project has begun a public awareness campaign concerning Cognition’s lack of publicly detailed industry-standard safety practices. As a member of the public, it's your right to know when companies are putting their users at risk.

If you want to support our campaign, consider signing our petition asking Cognition to release an industry-standard evaluation-based safety policy, or sharing this article to help spread the word.

AI has the potential to make our world so much better — including the products that Cognition is developing — but the public must demand that development be conducted responsibly and carefully.