This article is more than 1 year old

Like the Architect in the Matrix... but not: Nvidia trains neural network to generate cityscapes

GPU biz also open sources PhysX and teases beefy graphics card

In conjunction with opening of AI conference NeurIPS, until recently known as NIPS, chip biz Nvidia has demonstrated a system that generates 3D environments using neural networks, open sourced its PhysX physics engine, and announced its Titan RTX GPU.

Nvidia created a driving game for NeurIPS attendees that allows players to navigate through an AI-generated environment. The simulated world comes from videos of the real one. As described in a research paper released earlier this year and presented at the conference, Nvidia boffins trained their neural network models on videos captured from moving cars and used these models to generate somewhat realistic-looking 2K resolution videos.

Good for games

Relying on generated, rather than captured, street scenes has potential advantages for game and simulation makers in terms of production costs and time. It also affords more post-production control. "For example, a user can easily replace all the buildings with trees in a street view video," Nvidia's researchers explain in their paper.

Nvidia credits its Tensor Core GPUs for making the project possible. The training regime involved Nvidia Tesla V100 GPUs on a DGX-1 deep learning system, supported by cuDNN-accelerated PyTorch and fed on thousands of videos from the Cityscapes, and Apolloscapes datasets.

The system still needs some fine-tuning. The researchers observe in their paper that their model doesn't handle cars making turns very well. They suggests additional 3D data, such as depth maps, could do the trick. They also observe that their model doesn't assure consistency of color throughout a video. Using object tracking info to enforce consistency may help, they say.

Lastly, they acknowledge that manipulating shapes – replacing buildings with trees, for example – can leave visible artifacts. More expansive semantic labeling could help, they say, by making the trained model less sensitive to label shapes.

The GPU maker also said its PhysX SDK 4.0 physics engine will be available on December 20, 2018, under the open source 3-Clause BSD license. The revised license scheme is being extended backwards to version 3.4.

PhysX is used for handling physics in gaming and simulations. It's the default physics engine for both Unity3D and Unreal Engine, but the new license only applies to some of the supported platforms – iOS, macOS, Google Android ARM (2.2+), Linux, and Windows XP+, but not XBox One, Playstation 4, or Switch which remains under the existing Nvidia EULA.

Version 4 comes with a handful of additional capabilities, including a Temporal Gauss-Seidel Solver for making the joint-based animation work better. "TGS dynamically re-computes constraints with each iteration, based on bodies’ relative motion," Nvidia explains.

The Nvidia Titan RTX is "the fastest PC graphics card ever built," Nvidia insists. When it actually ships later this month, it could be, though the company hasn't offered test results to support that claim.

For $2,500, it had better be fairly speedy. According to the company, it can deliver 130 Tensor TFLOPs for deep learning applications and 11 GigaRays of ray-tracing performance using its 576 tensor cores and 24 GB of GDDR6 memory. Its recommended power supply is 650 watts. ®

More about


Send us news

Other stories you might like