Game over, machines: Humans defeat OpenAI bots once again at video games Olympics

Computer software not ready to beat living Dota 2 pros yet

OpenAI’s bots were knocked out of The International – the Dota 2 computer game's annual Olympics – on Thursday after they lost to human pros 2-0 in a best-of-three contest.

Dota 2 is a hugely popular online battle strategy game (think Command and Conquer meets Tolkien) played by several hundred thousand people worldwide. The aim is to storm a map, take over bases by destroying towers, and take the ultimate prize: the enemy’s “Ancient.”

Teams are made up of five players known as heroes. There is a pool of more than a hundred different hero types all with their own strengths and weaknesses. Players can also collect gold, by farming or killing enemies, to buy items such as healing potions and special abilities for their hero characters.

The OpenAI team, dubbed OpenAI Five, played against a team of former Chinese Dota 2 professionals on Thursday in its second round against humans. The machine-learning software agents looked promising at the start: OpenAI Five was less jittery and more aggressive than in the first round on Wednesday. The agents banded together to chase down enemies, and made a beeline for the towers within the first two minutes of the game.

They also looked pretty solid in the battles, and were up 9-4 in terms of kills at the start. The humans came back to narrow the kill gap and had collected gold, despite having fewer towers standing. Then nearly two thirds of the way in the match, OpenAI Five lost two towers as it fought against Roshan, a monster that drops a powerful item that allows players to respawn quickly if it is slain.

At 23 minutes, both the human and AI teams were pretty much even with similar number of kills and towers. As the game wore on, however, the humans overpowered the bots in violent clashes, and had an excessive lead on gold to buy more powerful items to make their hero characters stronger. Eventually, all of OpenAI Five’s towers were destroyed, and it was only a matter of time before the humans eventually tore down the agents' Ancient to win.

It was a particularly nerve-racking ordeal for OpenAI as its software lost its first match against humans, a group called Team paiN, the day before.

“We’re not really shocked,” Filip Wolski, a member of technical staff at OpenAI, told The Register. “We came into the competition with a high level of uncertainty. We had no idea about certain things like who we would play even on the day of the matches.”

OpenAI has been trying to crack Dota 2 for a while, slowly ratcheting up the difficulty of the game conditions to challenge their machine-learning algorithms during training and development. The computer code can’t play to the same level of complexity as humans can, and the matches contain some restrictions such as a limited choice of heroes, items, and abilities. The bots had some advantages, such as lighting fast reaction times, and an ability to see the whole map at once in the game.

All-nighters

Engineers worked tirelessly day and night to tweak the OpenAI Five code. They decided to lift the restriction from using five invincible couriers, a unit that can ferry items to players during the match, to a single killable courier to make the games more realistic. The change was implemented only a few days before OpenAI Five’s appearance at The International.

After it suffered its first defeat on Wednesday, the programmers rejigged the AI's reward system in an attempt to maximize the chances of a victory. OpenAI Five is trained using reinforcement learning, where bots are programmed to chase rewards.

“The biggest reward is given when the bots win the game,” Wolski explained to The Register. "Intermediate rewards are for behaviors such as killing enemy heroes or gaining gold. We decided to lower the amount for intermediate rewards in the hopes that it would focus more on winning the game compared to other tasks."

OpenAI Five were formidable opponents for the humans. The bots had surgical precision when it came down to lobbing spells in battles, however, a lack of strategic planning was the software's downfall.

“There were a lot of sleepless nights working on this project,” Wolski said. "We’ll go home, rest, and probably obsess over whether we uploaded the right hyper-parameters for the bots. But we’ll continue working on Dota 2 so we can be better at playing the game with fewer and fewer restrictions." ®