OpenAI bots smashed in their first clash against human Dota 2 pros

AI can react faster than humans, but don't play well enough to beat the masters yet

The International In the past hour, OpenAI's artificially intelligent bots lost their first match against professional players at smash-hit computer game Dota 2 at The International – the video game's annual championship tournament.

It's the first bout in a best-of-three competition between human professional players versus OpenAI's code, the other two rounds will take place over the next two days, each day a different human team.

Thousands of hardcore Dota fans lit up by glowing bracelets sat down at the Rogers Arena in Vancouver, Canada, to watch pros battle against a machine running OpenAI's software in this first round. The humans – dubbed Team paiN – were five players from Brazil, while OpenAI Five is made up five long-short-term memory neural-network-based agents.

Dota 2 is a popular battle strategy game played online. The goal is to gain take over bases and destroy the enemy’s “Ancient”, a sacred structure. Teams will often clash in sporadic bursts of battle, where the screen is ablaze with all players casting all sorts of glittery spells and powers. Sorta like Command and Conquer meets Tolkien.

It was a close match that lasted more than 45 minutes. The OpenAI bots seemed pretty disorientated right at the start. A group of three walked around in circles seemingly confused, while the human team explored the map. The software agents quickly snapped to their senses, and it was a pretty even match at the beginning. The first blood was drawn by Team paiN, and the kills were pretty tit-for-tat afterwards.

However, a third of the way in, the bots began to dominate in fights and collected more gold. Humans began to catch up a little after to take more towers, and the game seemed pretty neck and neck – despite OpenAI Five racking up more kills. The bots’ towers gradually fell, however, and in the final battle the AI players were wiped out and left their Ancient exposed. The human team quickly demolished it to win.

“We were definitely less confident about winning this time round, because the pros are a lot harder to face. It’s disappointing to lose, but I’m happy that we could stand against a top team in such a long match,” Filip Wolski, a member of technical staff at OpenAI, told The Register.

OpenAI Five showed some strange behavior such as lingering outside the lair of Roshan, a powerful enemy that requires teamwork to defeat. Slaying Roshan is a crucial step as it rewards players gold and an item that allows a hero to respawn quickly after death, however being killed by Roshan sucks up time, effort and resources. All the meanwhile, the map is open for the opposite team to obliterate towers. Any hesitation is therefore not good.

The bots also appeared to cast powerful spells during battles at odd times, sometimes when there were few or no enemies nearby.

Restricted play

Dota 2 is a complicated game. There are more than a hundred playable characters known as heroes each with their own strengths and weaknesses. Some have the ability to cast particularly powerful magical spells, some can travel quickly, some are very strong, and one even comes equipped with baby spiders. All of this, combined with the ability to collect gold to buy a plethora of items to strengthen heroes, makes the game incredibly complex.

The match of Team paiN against OpenAI Five, however, doesn’t capture all of this. There are simply too many combinations and possibilities for a computer to master at the moment. OpenAI has been steadily lifting restrictions on its software agents during gameplay as its bots improve – effectively taking off the training wheels. These changes include:

• The number of AI heroes playable in each match has increased from five to 18.
• The characters in both teams are agreed upon by OpenAI engineers and the human team to make sure they’re balanced.
• Reaction time for the agents has increased from 80 milliseconds to 200 milliseconds to give the humans players a chance.
• More items, such as Bottle, which restores health points, can now be used in the game.
• Only one killable courier instead of five invincible ones so that only one hero can use the items carried by the courier at any one time and the items can be lost.

The decision to pre-select the heroes for both teams was made by OpenAI and the organizers of The International to save time, we're told.

Pro players practice for hours trying to improve reaction times, their teamwork, and their various strategies. Luckily, OpenAI Five has also had a lot of time to practice. In fact, it can play more games in a single day than someone can in their entire human lifetime. Each bot acquires a whopping 180 years of experience every day during the training process, so that’s 900 years per day for the entire team.

Unlike humans, bots can be cloned. OpenAI uses a technology called Rapid, which is a reinforcement-learning framework that allows multiple agents to play a large number of games in parallel. These agents accumulate knowledge in batches which is then used to train all five of the OpenAI Five bots using a proximal policy pptimization (PPO) algorithm, a general-purpose reinforcement learning algorithm. They’re always playing against themselves, too, a method known as self-play.

All of this training guzzles up a heck of a lot of GPUs and CPUs. OpenAI didn’t tell us exactly how many were used to gear up for The International after its practice match in August. However, the short answer is a lot – it slurped 128,000 CPU cores and 256 Nvidia P100 GPUs on Google Cloud when it was trained to play a series of informal matches earlier this year.

Two more matches to go

The bots also have other advantages besides more experience. They can see the whole map and state of the game at once. All the information – such as the health and positions of the heroes, and items in their inventory – is fed to the neural networks every four video frames. “It’s like playing with your eyes closed and opening them every four frames,” Greg Brockman, cofounder and OpenAI CTO, previously explained to The Register.

OpenAI Five therefore gets to glimpse the whole map at once, whereas humans have to manually move their heroes around to see different parts of the map. So, both teams do have access to the same information but not at the same time.

Although the gameplay reaction time has increased from about 80 milliseconds to 200 milliseconds, so that the bots resemble similar reaction times for human players, the AI players still have the upper hand here. When it comes to menial tasks, clicking the right buttons is pretty much muscle memory, however, for critical moments, such as knowing when to strike with certain spells, a human will often pause to deliberate, and is likely to take about a second or two. The computer is still working in the order of hundreds of milliseconds.

And yet, the computer lost seemingly because it didn't have the long-term strategy honed by the humans.

"What we proved today is we're right there at the edge of human ability, and the test is whether we can beat a pro team," said Brockman in the past few minutes. "We're planning to play other pro teams at The International this week, and we'll have more to share tomorrow and Friday. The test for us is whether we can play at a high enough level to win against a professional team this week."

OpenAI have been steadily working on beating Dota 2 for a while. Last year, it quietly crushed Dendi, a pro player at The International in a restricted game of one-v-one in a mirror match, where each side plays with the same selection of heroes.

This year, it increased this to five-v-five matches, and these were initially mirror matches too until a public game against semi pros in a public match earlier this month.

Focus is often on OpenAI Five's victories, but losses are what drive our progress. On Aug 9th, we were crushed in a best-of-3 against @teamsecret (#4 team in pro circuit)—but won game 1 as Secret adjusted to Five's playstyle. Last week lost to several non-pro teams. Tomorrow...?

— Greg Brockman (@gdb) August 21, 2018

OpenAI Five may have lost the first match, but it’s not over yet. It’s the best out of three matches, and they will face more pro teams on Thursday and Friday. Stay tuned for more. ®