Microsoft will upgrade its HoloLens gizmo with a mysterious chip to handle machine-learning on the space-age goggles.
Harry Shum, EVP of Microsoft's Artificial Intelligence and Research Group, announced yesterday at the annual CVPR computer vision conference in Honolulu, Hawaii, that the second-generation HoloLens will include a custom AI coprocessor for implementing deep neural networks.
Details are scant but it's pretty safe to say that the coprocessor will be used for performing inference on trained models loaded into the headsets. In other words, this magic custom chip will take a trained neural network model and use it to carry out real-time predictions and decision making. Inference doesn't draw anywhere as near as much power as training requires – which typically involves beefy chips from the likes of Intel and Nvidia.
It's entirely possible to do inference from battery-powered silicon. You don't want to be performing inference outside the device, anyway, such as in the cloud, because the latency will wreck the immersive experience HoloLens tries to offer.
Don't just take our word for it. Bloomberg, citing anonymous sources, reported this May that Apple is now testing iPhones with special chips for processing AI.
The main processor on the all-in-one, first-gen HoloLens processes info from the headset's on-board sensors. The new co-processor will apparently allow deep neural networks to crunch that incoming data and produce predictions from it.
A Microsoft spokesperson referred The Register to a blog post by the HoloLens director of science, Marc Pollefeys, about the announcement, noting: "We have nothing more to share at this time."
The chip – which "supports a wide variety of layer types" for deep neural nets that are "fully programmable" – will be able to run "continuously, off the HoloLens battery," according to Pollefeys. In Hawaii, Shum apparently demonstrated hand segmentation live.
"The blog doesn't actually give enough detail to form any impression of how much difference the AI coprocessor will make," said Professor Stephen Furber, a computer engineer at the UK's University of Manchester who studies human-brain-inspired neuromorphic computing.
Researchers have already shown that you can do a "reasonable job" of implementing the computer vision task of simultaneous localization and mapping (aka SLAM) on mobile devices, he said – for example, this study indicates that 3D mapping and tracking could be done on an embedded device with a 1W power budget.
More hardware assistance with deep networks could potentially allow for higher frame rates, better accuracy of tracking and better object recognition.
But he pointed out that Google's 700MHz Tensor Processing Unit, a large eight-bit integer matrix multiplier designed for neural network applications, is too power-hungry for mobile device use. It consumes about 40W when running, according to a Google blog post.
"I would guess that this is similar, though smaller, less powerful and less power-hungry," he added. "But who knows?"
Microsoft has not officially announced a release date for the HoloLens 2, but some reports say it might not be until 2019. ®