OpenAI’s machine learning bots have beaten another team of semi-professionals in Dota 2, in their second public match in the traditional five-versus-five settings.
The human team – made up of popular Twitch streamers and former professionals ranked in the 99.95th percentile – hunkered down to play against the bots known as OpenAI Five in San Francisco on Sunday. OpenAI Five smashed its opponents, winning comfortably in two out of three games.
It did lose one game, however, after spectators watching the match live and on Twitch were allowed to pick the pool of heroes – the playable characters in the game. Each hero comes with its own strengths and weaknesses and picking a balanced combination is paramount to winning. If you have too many characters for the same role, the other team will steamroll you. The crowd-picked team was imbalanced. It was only under these circumstances that the humans managed to get their own back against the machine.
What does it take for an OpenAI bot to best Dota 2 heroes? 128,000 CPU cores, 256 Nvidia GPUsREAD MORE
Dota 2 is a difficult game to master; it requires teamwork, strategic thinking and lightning-fast reaction times during battles. OpenAI has been working on cracking Dota 2 for a while. Last year, one of its bots won against Dendi, a pro player, under very limited settings that did not represent realistic competitions.
But this year, after cranking up the amount of hardware and engineering required to scale up its machine learning system, OpenAI has lifted a lot of restrictions to level out the playing field between humans and machines.
OpenAI Five is made up five identical long short-term memory networks, each about the size of an ant’s brain, apparently. It learned to play Dota 2 by playing in tonnes of matches against itself, racking up a whopping 180 years’ worth of experience every single day during training. That’s 900 years of collected experience for the five bots – something that human teams can never achieve.
“It has been training pretty non-stop since June and eats up a CPU for every single game,” Jie Tang, a member of OpenAI’s technical staff told The Register.
Now, the bots can play in the traditional five-versus-five settings. In the first match in June, OpenAI won comfortably against a team of slightly weaker semi-pros ranked in the 93rd to 99th percentile in mirror matches - where the bot and human teams play with the exact same heros: Necrophos, Sniper, Viper, Crystal Maiden, and Lich.
It was trained on Proximal Policy Optimization, a reinforcement learning algorithm. For the match in June, the system guzzled a mind-boggling 128,000 CPU cores and 256 Nvidia P100 GPUs on Google Cloud.
Tang said that training for the match this time around required even more CPUs and GPUs. The amount of compute needed for training increased from 40 petaflop/s-days for the system used in the first semi-pro match, to 190 petaflop/s-days for the latest practice match.
All that heavy number-crunching is needed as some of the previous restrictions have been lifted. These include:
- The number of heroes playable in a single match went up from five to 18
- No mirror matches, both teams can choose freely from the pool of 18 heroes
- Reaction time has increased from 80 milliseconds to 200 milliseconds
- No scan – meaning players cannot scan the minimap to scout out unseen local enemies
- More items were allowed to make the game more complex. For example, items such as Bottle, which restores health points, can now be used by players
But OpenAI Five has still retained some of its major advantages. It can still see the whole map at once every few frames, making it easier to calculate the range of its attacks. Although its reaction time has been increased and is closer to the speed of a human’s, the machines still have the advantage.
Stephen Merity, an ex-Salesforce AI researcher, pointed out that humans can be just as quick at clicking a button, but only when it comes to those moves where it's likely to be a less important and more down to muscle memory. But for the crucial decisions that require more strategy, such as checking the enemy’s inventory and remembering what spells have been cast before making a move during a battle takes longer than 200 milliseconds to execute for humans.
A human reaction to a stimulus may be possible in 215ms - but that's likely an uninformed muscle memory response. OAI5 - and essentially any other automated decision maker - can perform a far richer analysis and make a far better decision in the same time.— Smerity (@Smerity) August 5, 2018
Adding more heroes makes the game much more complex as there is more information and there are possibilities to take into account. In the post-match interview, a panel of OpenAI staff said that it increases the number of different possible game combinations to over 11 million.
It’s impossible to play all these games, but OpenAI Five thrashed the human team anyway. It stuck to close to its strategy of playing aggressively right at the beginning, making a beeline to destroy enemy towers, and killing its opponents.
In the first game, it calculated that its chance of winning was 99 per cent within the first five minutes. And after 21 minutes, the human team lost after the bots took over the map. The humans did slightly better in the second match and held on for 25 minutes.
William Lee – aka “Blitz” – an ex-pro Dota 2 player and coach, said it felt “pretty weird” playing against the bots. “We didn’t know what to expect. It felt like playing against a really solid team. They respond well, and have good teamwork”.
All previous matches have been practice and used to benchmark the machine’s progress in time for its ultimate match at The International, the annual Dota 2 tournament, held in Vancouver, Canada, with the total prize pool of $24m at the end of this month – though the bots won't be able to partake in this.
OpenAI Five will play against a team of professionals under the same restrictions, which limit the full complexity of the game, which has over 100 heroes. Lee was asked whether he expected OpenAI Five to win this time around. After a little hesitation, he said: “I’m going to have some faith in it and say yes, it’ll beat the pro team.” ®