Only Kinect: Microsoft boffins build Minority Report-style tools

Big, expensive goggles no longer compulsory for AR


In the two years since Microsoft launched Kinect, there’s been huge interest among enthusiasts in adapting the hands-free controller to new settings beyond gaming.

Techies and geeks of open- and closed-source shades have hacked the hardware, used the tools or adapted Kinect to steer computerised, wheeled armchairs.

Among those doing the fiddling is Microsoft Research in Cambridge, the Microsoft division whose work on real-time human pose recognition and depth-sensing algorithms (here and here) helped make Kinect a reality in the first place.

These engineers have been using Kinect in different experiments in an attempt to construct Star Trek-style holodecks and Minority Report-style 3D touch-displays.

They have combined feeds from multiple Kinect cameras positioned around a room, stitching together a 3D map of the room and then combining the map with moving pico projectors. The result has been to combine Kinect’s depth sensing with the pico data so you can see and touch the computer-generated pixels without the need for special glasses, headsets or gloves to operate. Among the achievements so far is a pixel-generated car driving around the floor.

The Kinect sensor has also been combined with mirror and web-camera tracking to produce a holodesk and Kinect sensors used as hand-held units to scan objects in 3D.

Microsoft Research 3D car

Augmented reality car using Kinect, from Microsoft Research

Shahram Izadi, who jointly leads Microsoft Research Cambridge’s interactive 3D technologies group, reckons augmented reality – or what he also calls “looking at pixels in the air without a screen” - will be the “next big thing” thanks to Kinect and other industry efforts. On Izadi’s team is Andrew Fitzgibbon who helped author that original paper on human skeletal 3D recognition for the Kinect.

“We are starting to move towards this notion of augment reality. It’s about augmentation with the instrumentation of the environment, instead of augmentation of the user,” he said on a recent visit to Microsoft in Cambridge.

“We have been working a lot with the Kinect and know the potential,” Izadi told The Reg.

Izadi points to a number of innovations in a relatively short period of time, including Kinect. We've also had multi-touch motion gaming and augmented reality in smartphones from Apple and Google.

“Companies like Microsoft will be looking to exploit it,” Izadi predicted.

Izadi reckons on reason augmented reality is about to step from science fiction to fact is the relatively ubiquity of the Kinect. The Kinect sold so many units during its first 60 says (eight million) that it set a Guinness World record. The Kinect's companion Xbox 360 game unit – it has since been made available for the PC, too – is also now the biggest selling games console in the US.

Another factor on the Kinect’s side is its price point: £129 from the Microsoft online store in the UK. This makes getting your hands on the Kinect doable - which is not the case with most augmented-realty kit. “Kinect is just a starting point,” Izadi says.

“The reason I’m so excited is two years ago there was nothing like this on the market and it cost tends of thousands to buy [the equipment]. Now we are talking about natural language user interface and augmenter reality as if they are almost upon us, and two years ago we were far away form that vision.”

Izadi’s team has been capitalising on Kinect’s depth-sensing technology to meet the vision. The depth-sensing camera can tell the system how far an object is from the sensor. The Kinect is a rich source of 3D data and it is mostly this information that is being interpreted. An image is smaller than one you’ll find in most digital cameras, 640 x 480 pixels, but the Kinect manages to track 300,000 points at a rate of 30 frames per second.

But there are niggles, hence the work’s still at the prototype stage.

One problem is that the various Kinects interfere with each other when set up and running at the same time, a fact that creates visual noise on the display. This is something Microsoft Research realised could be solved for now with the addition of a little vibration to each unit; the vibes are supplied by a small motor fixed to the top of each Kinect in the holodeck idea (here and here).

Also, sensor readings can be incorrect, with 3m being read as 2.9 meters for example, or they can fluctuate. In other scenarios depth data can’t be scanned, for example in the case of the Dell logo on a Dell PC box.

Izadi also reckons the next challenge is not input - although issues still remain over the input received via the Kinect – but rather the lag in output technologies, for the moment filled by pico projectors.

When will holodecks and holodesks become a reality?

Despite his enthusiasm, Izadi points to the fact the work at Microsoft Research tends doesn’t pay off immediately – it’s generally a good decade before something gets turned into a product. Kinect was quicker, of course. It was first discussed in 2007 and officially unveiled by Microsoft at E3 in June 2009 under the codename Project Natal.

In the meantime, the pace is being pushed on tablets and smartphones thanks to companies like Layar and Qualcomm, which lets you point your phone at a real-world object and receive information, with the phone acting as a head-up-display.

“Many companies are now developing augmented reality or immersive technologies,” Izadi says, "and that will accelerate a lot of what we are seeing. But to add the disclaimer: we are a search group and the work we are doing is five to 10 years out on what could feasibly be a product... [however] there is this real acceleration effect in we are seeing." ®

Similar topics


Other stories you might like

  • Prisons transcribe private phone calls with inmates using speech-to-text AI

    Plus: A drug designed by machine learning algorithms to treat liver disease reaches human clinical trials and more

    In brief Prisons around the US are installing AI speech-to-text models to automatically transcribe conversations with inmates during their phone calls.

    A series of contracts and emails from eight different states revealed how Verus, an AI application developed by LEO Technologies and based on a speech-to-text system offered by Amazon, was used to eavesdrop on prisoners’ phone calls.

    In a sales pitch, LEO’s CEO James Sexton told officials working for a jail in Cook County, Illinois, that one of its customers in Calhoun County, Alabama, uses the software to protect prisons from getting sued, according to an investigation by the Thomson Reuters Foundation.

    Continue reading
  • Battlefield 2042: Please don't be the death knell of the franchise, please don't be the death knell of the franchise

    Another terrible launch, but DICE is already working on improvements

    The RPG Greetings, traveller, and welcome back to The Register Plays Games, our monthly gaming column. Since the last edition on New World, we hit level cap and the "endgame". Around this time, item duping exploits became rife and every attempt Amazon Games made to fix it just broke something else. The post-level 60 "watermark" system for gear drops is also infuriating and tedious, but not something we were able to address in the column. So bear these things in mind if you were ever tempted. On that note, it's time to look at another newly released shit show – Battlefield 2042.

    I wanted to love Battlefield 2042, I really did. After the bum note of the first-person shooter (FPS) franchise's return to Second World War theatres with Battlefield V (2018), I stupidly assumed the next entry from EA-owned Swedish developer DICE would be a return to form. I was wrong.

    The multiplayer military FPS market is dominated by two forces: Activision's Call of Duty (COD) series and EA's Battlefield. Fans of each franchise are loyal to the point of zealotry with little crossover between player bases. Here's where I stand: COD jumped the shark with Modern Warfare 2 in 2009. It's flip-flopped from WW2 to present-day combat and back again, tried sci-fi, and even the Battle Royale trend with the free-to-play Call of Duty: Warzone (2020), which has been thoroughly ruined by hackers and developer inaction.

    Continue reading
  • American diplomats' iPhones reportedly compromised by NSO Group intrusion software

    Reuters claims nine State Department employees outside the US had their devices hacked

    The Apple iPhones of at least nine US State Department officials were compromised by an unidentified entity using NSO Group's Pegasus spyware, according to a report published Friday by Reuters.

    NSO Group in an email to The Register said it has blocked an unnamed customers' access to its system upon receiving an inquiry about the incident but has yet to confirm whether its software was involved.

    "Once the inquiry was received, and before any investigation under our compliance policy, we have decided to immediately terminate relevant customers’ access to the system, due to the severity of the allegations," an NSO spokesperson told The Register in an email. "To this point, we haven’t received any information nor the phone numbers, nor any indication that NSO’s tools were used in this case."

    Continue reading

Biting the hand that feeds IT © 1998–2021