If you give Copilot the reins, don't be surprised when it spills your secrets

'All of the defaults are insecure' Zenity CTO claims

Black Hat One hopes widely used enterprise software is secure enough. Get ready for those hopes to be dashed again, as Zenity CTO Michael Bargury today revealed his Microsoft Copilot exploits at Black Hat.

"It's actually very difficult to create a [Copilot Studio] bot that is safe," Bargury told The Register in an interview ahead of conference talks, "because all of the defaults are insecure." 

Bargury is speaking twice about security failings with Microsoft Copilot at Black Hat in Las Vegas this week. His first talk focused on the aforementioned Copilot Studio, Microsoft's no-code tool for building custom enterprise Copilot bots, its defaults, and other aspects. The second covered all the nasty things an attacker can do with Copilot itself if they manage to break into the IT environment of an organization that uses the tech, and how Copilot can help someone gain entry to that environment.

Zenity, for what it's worth, offers among other things security controls for Copilot and similar enterprise-level assistants. Bear that in mind. It warns of the risks of using Microsoft's AI services here.

Your Copilot bots are quite chatty

If you don't have much exposure to Copilot Studio, it's a tool for non-technical people to create simple conversational bots, using Microsoft's Copilot AI, that can answer people's questions using internal business documents and data. This is made possible by what's called retrieval-augmented generation, or RAG.

It's Microsoft's way "to extend [Copilot's] tentacles into other business areas, such as CRM and ERP," as we wrote here. Companies can create customer and/or employee-facing bots that provide a natural-language interface to internal information.

Unfortunately for all the Copilot Studio customers out there, we're told the default settings in the platform at least were entirely insufficient. Combine those with what Zenity marketing chief Andrew Silberman told us is nearly 3,000 Copilot Studio bots in the average large enterprise – we're talking Fortune 500-level companies here – along with research indicating that 63 percent of those are discoverable online, and you have a recipe for potential data exfiltration.

If these bots are accessible to the public, and we're told a good number of them are, they can be potentially tricked into handing over, or simply hand over by design, information to people that should not have been volunteered during conversations, it's claimed.

As Copilot bots frequently have access to internal company data and sensitive documents, it's a matter of figuring out how to fool or prompt them into disclosing that data, we're told. Bargury said he was able to do that by configuring ChatGPT to fuzz Copilot bots with automated, malformed prompts.

"We scanned the internet and found tens of thousands of these bots," Bargury said. He blamed the high online availability of these agents on default Copilot Studio settings that published them to the web without any need to authenticate to access them - an oversight Microsoft has since fixed after the Zenity team brought it to their attention. 

Unfortunately, the new settings that keep Copilot Studio bots off the public internet by default currently only apply to new installations, Bargury said, so users of the suite who deployed bots before now should check their settings to be sure.

Bargury and his team have released a tool to "scan for publicly accessible Copilot Studio bots and extract information from them," which folks are welcome to use to test their environments' security. Dubbed CopilotHunter, it's available as a module in PowerPwn, a software toolkit Zenity released at Black Hat last year for probing Microsoft 365 guest accounts. 

In short, if you've deployed Copilot Studio bots, check to see how available they are to the world.

While Bargury told The Reg he may have overextended himself by planning two Black Hat talks this year, his second shows no less effort than the first. 

Copilot, Bargury demonstrated this week, is susceptible to indirect prompt injection attacks, which he argues is on a par with remote code execution (RCE) in terms of severity.

"An RCE is simply, from a remote location, being able to execute code that does something on your machine," Bargury said. "Indirect prompt injection that makes an AI do something on your behalf is the exact same thing with the same impact." 

With access to a compromised environment, Bargury said he can instruct that Microsoft customer's Copilot "to automate spear phishing for all of your victim’s collaborators," use the technology to lure internal users to phishing pages, access "sensitive content without leaving a trace," and more.

To top it all off, it's claimed Copilot can be tricked into granting someone initial access to that environment, and conduct other malicious activities, with nothing but an email, direct message, calendar invite, or other common phishing tactic, and that this can work without the user needing to interact with it or click a link because of how Copilot scans messages. The key thing here is the indirect prompt injection: Crafting messages that the AI assistant picks up and parses to be helpful, and as a result does things it shouldn't.

"Microsoft Copilot is built on the enterprise graph," Bargury explained. Once a message, email or invite is sent it hits the graph, Copilot scans it, "and that's a path for me to start with prompt injection."

In one example, Bargury demonstrated how he was able to change banking information to intercept a bank transfer between a company and client "just by sending an email to the person." 

An AI bot feature

Bargury explained to us that he sees these discoveries as indicative of the industry still being in the very early days of artificial intelligence in the enterprise, and having to face the fact that AI is changing our relationship with data. 

"There's a fundamental issue here," he said. "When you give AI access to data, that data is now an attack surface for prompt injection." 

When you give AI access to data, that data is now an attack surface for prompt injection

If that's true, Copilot bots are by their very nature insecure since many are publicly accessible, they're tied closely to enterprise data, and are ready to spill secrets with a bit of hidden HTML or a ChatGPT-powered fuzzing bot. 

"It's kind of funny in a way - if you have a bot that's useful, then it's vulnerable. If it's not vulnerable, it's not useful," Bargury said. 

The Zenity CTO noted that Microsoft has been incredibly responsive to his reports, and said several of the faults he found have been addressed, albeit within limits.

"[AI] apps are basically changing in production because AI chooses to do what it wants, so you can't expect to have a platform that's just secure and that's it," Bargury said. "That's not going to happen because these platforms have to be flexible, otherwise they're not useful." 

Bargury believes that securing AI software like Copilot requires real-time monitoring, inspecting conversations with the bots, and tracking potential prompt-injection RCEs. The bottom line is that businesses are the guinea pigs testing an experimental drug called artificial intelligence, and we're not at a point where we know how to make it safe yet.

Finally, Bargury and team released another testing kit called LOLCopilot for organizations that want to test their setups for vulnerability to his exploits.

"Copilot has great skills. It can search, it can enable your employees to find data they have access to but didn't know they did … those things are important," Bargury told us. "But that's not as important as preventing remote code execution."

We're seeking a response from Microsoft direct about Zenity's findings, and will let you know if we hear back from the Windows giant. ®

Updated to add

Spokespeople for Microsoft have been in touch to tell us the Azure giant believes it can keep its Copilot users protected, though it appreciated Zenity's reports:

We appreciate the work of Michael Bargury in identifying and responsibly reporting these techniques through a coordinated disclosure. We are investigating these reports and are continuously improving our systems to proactively identify and mitigate these types of threats and help keep customers protected.

Microsoft Security provides a robust suite of protection that customers can use to address these risks, and we’re committed to continuing to improve our safety mechanisms as this technology continues to evolve.

To recap, Bargury and his team disclosed various ways to meddle with Copilot at Black Hat, some that Microsoft fixed (such as those insecure defaults), and some that require compromising a victim's environment, and some that involve prompt injection. Redmond acknowledged as much, telling us: "Similar to other post-compromise techniques, these methods require prior compromise of a system or social engineering."

In short, check through Bargury's resources linked to above to make sure your bots aren't unexpectedly facing the public, and also are hardened against other forms of attacks.