Augmented reality dreamers Magic Leap has finally begun shipping its hardware – along with a long series of ludicrous security requirements.
Developers interested in making content for the untested product have to agree to a long list of restrictions over what they are allowed to say and do with the company's kit – including keeping it in a locked safe when not in operation.
Even for a company obsessed with secrecy, the idea of requiring software folk to lock away a headset like it's a dangerous substance is laughable. It has already resulted in some developers deciding it's simply not worth the trouble.
But the safe requirement raises the question: is this just security theater aimed at building buzz, or is there something more interesting behind it?
We have been following Magic Leap closely – including attending its first public session at the Games Developer Conference in San Francisco last week – and have come up with our own theory.
Based on the extremely limited demos that the company has shown – all of which come from a stationary camera position and feature very simple room layouts – we think Magic Leap is unable to deal with movement of the headset, particularly when it comes to correlating it with the movement of virtual objects in the same space – something called "inside-out tracking."
Magic Leap's entire promise is that you can wear the wireless headset anywhere and its system will recognize your environment in real-time using only sensors on the device. The system can then adjust the position and shape of virtual objects as you move around to give you a palpable sense of them existing in your space.
That is an extremely difficult thing to achieve. The first generation of VR/AR devices used external sensors to locate you within a virtual space. But placing sensors everywhere is both expensive and complicated. The "magic leap" will be when your headset is capable of locating everything around you from your position – effectively seeing the same way we do with our eyes.
There are quite a few companies trying to perfect this technology – and they are finding it extremely difficult. When we tested one such system last year from Eonite Perception, we had strong suspicions that the company had cheated by inputting the dimensions of the room and its main objects into its system beforehand to give the perception of live inside-out tracking.
In a demo that Magic Leap showed last week, a virtual robot was shown hiding behind a sofa – but at one point as it emerged from the side of the sofa, the image jumped – which is the exact same thing we have experienced with other inside-out tracking systems.
The system tries to provide a smooth experience as you move around but there is a lag in the actual movement of headset sensors and the images you see while wearing the headset. Sooner or later, if you move too much in one direction, the system has to catch up – and you experience that as a jump in position of objects in the room.
Magic Leap's augmented reality approach – where you see the real world through the headset rather than a pure virtual world inside a headset – would make any such discrepancies all the more obvious and disorientating.
Inside-out tracking also gets harder the further away something is for two reasons: one, a small jump becomes more significant as the object itself is much smaller; and two, the slightest movement of your head would lead to a much larger movement of an object further away.
As a result, the further away an object is, the more precise and accurate the tracking needs to be. This could well explain why all of Magic Leap's videos only show things close to the wearer.
Even a much-lauded Star Wars video – which the company had months to perfect – shows spaceships going into hyperdrive a short distance from the viewer. The closer something is, the easier it is to render realistically.
Here's our theory: Magic Leap's hardware works. In that its sensors work and the headset is capable of both recognizing physical objects in the room and accurately representing them. The hardware also works in that it is capable of producing rapidly moving virtual objects in front of your eyes that can be controlled with sufficient accuracy to give a real perception of depth.
We also think its software works. In that it is possible to create virtual objects that can then be displayed accurately with the system and in the headset.
But we think the problem is that the software linking the headset to the images does not work to an acceptable degree, causing image-jumps and movement-lags.
The problem should be solvable at some point with software: which would explain why Magic Leap remains persistently optimistic that it can pull off its promise of a next generation headset. And why it has started shipping hardware.
It would also explain why the company's actual software engineers were remarkably vague in their hour-long presentation last week, using the conditional tense every time they spoke about what was possible.
"This is something we could do…," said one. "If a human tilted their head [in response to your behavior] that would bond you," said another. Literally every mention of the actual experience was constructed this way.
When the team spoke about how a virtual character would respond to a real-world event – the example given being a dog running into the room – their eyes lit up. Because that is the promise of Magic Leap – an experience that fits with reality rather than producing its own reality.
But it doesn't work. The software works and the hardware works but the connection between them doesn't.
Which would explain why Magic Leap has this ludicrous requirement that developers lock away their headsets and sign NDAs not to disclose their experience of the system.
Our bet is that Magic Leap is very proud of the virtual imaging it has built into the headset and believes that the missing software connection to make it work with inside-out tracking is just a matter of time.
Companies have tried many different ways to produce virtual objects so they appear to be three-dimensional to the viewer. The most common solution – as you may have experienced in the cinema – is to overlay two different images and rely on our eyes to create a 3D image out of them.
But in that situation, the screen stays in the same place. As do you, mostly. Augmented reality is basically the screen moving. And what Magic Leap promises is both the screen and you moving.
To make things seem real while maintaining an illusion of depth while you and they are moving requires a complex combination of new hardware and software. It is believed that Magic Leap's technology is an interesting combination of placing virtual objects within the lens you look through – as opposed to projecting an image onto the lens - combined with depth-perception achieved by blurring images as your gaze moves.
The eyes have it
Last week at GDC, Tobii demonstrated a remarkable system that combined virtual reality with its eye-tracking hardware and produced a much more realistic experience. Once you know where an eye is looking, it is possible to bring whatever is being looked at into focus and blur everything else. Not only does this cut down on processing requirements it also provides a much more human experience since this is how our eyes actually work.
If Magic Leap has cracked that approach, it could well explain why it remains confident that it has something unique, and why organizations keep piling money into a company that consistently overpromised and under-delivered.
But if a rival company were able to get hold of Magic Leap's hardware and reverse-engineer that virtual imaging, it is very possible that their engineers could figure out the missing inside-out-tracking piece before Magic Leap was able to.
And then Magic Leap's sole market differentiator – its reasons for existing in the market – would vanish in a puff of smoke, along with the billions of dollars that have been invested in it.
From a legal standpoint, a software developer could grant access to Magic Leap's hardware to a rival company at its premises without either party breaking a contract or any laws.
But if there is a requirement that the hardware be locked in a safe when not in use, then both companies would break contractual terms and/or the law if they were to gain access to the hardware.
That is our theory as to the latest nonsense from Magic Leap. We would love to discuss this theory with any current or former Magic Leap employees, or anyone in the AR/VR space. ®