Exclusive: AI security company Gray Swan relaunches its red-teaming Arena contest for the era of AI agents
Source: Fortune ·
Gray Swan's new Arena includes $300,000 in prize money for developing tactics to jailbreak models from leading AI companies
From the source
Gray Swan, an AI security startup that helps frontier AI companies including Anthropic, OpenAI, and Google DeepMind stress-test their models before release, is overhauling its public red-teaming platform to keep it relevant for the era of AI agents. The Pittsburgh-based company on Tuesday launched a redesigned version of its Arena, an online platform where participants compete for prize money by trying to trick AI models into violating their guardrails. Alongside the relaunch, which Gray Swan CEO Matt Fredrikson refers to as Arena 2.0, the company is running its largest competition yet: a challenge called Indirect Prompt Injection (IPI) Black, with $300,000 in prizes sponsored by Anthropic, Meta, Google DeepMind, and Reflection AI. New attack scenarios will be released each week through Nov. 2. In an indirect prompt injection, an attacker hides instructions in content an AI agent will come across while doing a task—a web page or a forum post, for instance—hoping to hijack the agent.…