Augmented reality: Like it or not, only Apple's ready for the data-vomit gush

It's all about AR... and iPhone has the X factor

This month's release by Apple of the iPhone X with FaceID begins the first wave of consumer products designed from the ground up for continuous awareness of space, place and face - crowning a half a century of research in augmented reality destined to fuse our rising sea of data onto the real world.

Over the last few years, Augmented Reality (AR) travelled a hype-curve rollercoaster from the crashing failure of Google's Glass to the soaring heights of Pokémon Go, accelerating into open competition between APIs from Facebook, Apple, Alphabet/Google and Microsoft (hereafter known as FAAM).

Yet the story of AR goes back much further, all the way to the dawn of modern computing. Ivan Sutherland, the Turing Award-winning genius behind Sketchpad - the first truly interactive computer program - followed that up with another modest innovation - a “head mounted three dimensional display.”

Known as the Sword of Damocles because of the way the mechanical tracking apparatus hung down from the ceiling over the user's head, Sutherland's invention used dual CRTs, focused and and reflected through half-silvered mirrors to overlay a wireframe 3D image onto the real world.

Here's where the histories get things wrong: this was never a demo of virtual reality. Sutherland invented augmented reality - virtual reality meant simply running the system with the room lights off.

AR predates VR, because AR had a real-world use case: Sutherland's paymasters at the Defense Advanced Research Projects Agency (DARPA) needed visualisation tools for jet fighter pilots, something to help them manage the overwhelming flood of data generated by those very fastest and most responsive of machines.

Jet fighters have had heads-up displays for nearly 30 years, a now-indispensable element of flight instrumentation. They begin with Sutherland - who also simultaneously invented real-time 3D computer graphics, to have something to render to within the Sword of Damocles. That's a hell of a beginning

Outside the military, AR stagnated, because computers of the time weren't up to the job. NASA's earliest VR systems eschewed real-world integration in favour of easier to generate fully-synthetic worlds. Bringing the real world into the virtual world requires a sophisticated capacity to model the real world, in real-time.

Twenty years ago, aircraft giant Boeing spearheaded one of the the first non-military deployments of AR, at their massive 777 assembly facility just outside Seattle, where technicians used AR to keep track of the countless separate component assembly operations required to fabricate a completed aircraft. Boeing's AR system took what would have been a 10,000+ page manual and projected it into a display headset overlaying the body of aircraft with the necessary technical information for the assembly process, making the process both faster and less error-prone.

Google's Glass wasn't much of an advance on Boeing's work. Like so much else in this world, Glass leveraged smartphone mass-manufacturing to cram smartphone-like components into a different form factor - all of it running a version of Android. Yet it did feature one of the hallmarks of augmented reality - continuous surveillance.

They weren't Glassholes, they were ahead of their time

One of the key points of difference between AR and VR is that AR needs to be fed a continuously updated view of the world around. Without that, AR can't know what to augment, or where to augment it. Google's Glass featured a rather Cylon-like forward facing camera with a bright red “recording” LED that made almost anyone facing it feel as though they'd been scanned by Skynet. Hence the phenomenon of the "glasshole", banned from the local pub and shunned by society. Glass died - not because it wasn’t useful, but because of the "Segway Effect": no one else wanted to be near anyone using one.

After Glass gasped its last (recently reborn as an enterprise tool closer to Boeing's use case than Sergey's) the Great Leap Forward in AR began - spearheaded by Microsoft. Adapting some of the tech from Xbox Kinect into a headset, Microsoft's Hololens represented the first viable commercial implementation of an AR platform, built around “Windows Holographic” - now a core Win10 API renamed “Windows Mixed Reality”, spanning both AR and VR devices.

In Hololens, Moore's Law had finally caught up with Ivan Sutherland: the computing power necessary to create and maintain augmented reality could fit into a bulky-but-not-uncomfortable headset. All those cycles were needed not to create the images, but to manage the core technology of modern AR - Simultaneous Localisation And Mapping, or SLAM.


Why Microsoft's 3D HoloLens goggles aren't for Google Glassholes


SLAM blends an assemblage of computer vision and inertial sensor technologies to create a 3D map of a space. Hololens uses a "depth camera" - mounted above the eyes - to create a "depth map". That's essentially the same technique Kinect used to map a game controller onto a human body, but pointed outward, rather than toward the user. That 3D map can then be used to situate "holograms" or other forms of 3D augmentation within real space. Look away, then look back, and the augmentation remains, because the map is the territory.

There's more than one way to SLAM. Back in 2014, Google launched "Project Tango" developing its own library of smartphone-compatible SLAM techniques that eschewed an expensive depth camera, instead focusing on compute-intensive (but cheaper) computer vision techniques. As a camera scans a space, the difference between two captured frames, combined with data from the inertial sensors on the smartphone, generates the requisite 3D map.

Tango found its way into only two devices in 2016, and as a result attracted only a handful of Android app developers - like "hologram" purveyors 8i .

In July 2016, everything changed. Niantic's tie-up with Nintendo produced Pokémon Go, the fastest selling mobile game of all time, and - in just its first month - US $200,000,000 in revenues, as users greedily purchased in-game content. With that kind of market validation, augmented reality broke through - quickly surpassing a resurgent virtual reality as the new shiny in tech.

With that, the race was on between the FAAM firms to dominate the next new frontier - the real world, augmented.

Zuckerberg led with the charge with a twenty-minute discussion of the future of AR during this year's F8 keynote, touting the smartphone camera as "the new interface", and promising a rich library of AR tools for Facebook's developers, starting with AR Studio.

Like every other major player, Facebook has its own SLAM technology. SLAM is the major barrier to entry for AR, so each of the majors touts their SLAM solution as better than their competitors. Each have different strengths and weaknesses. Facebook has to be cross-platform, working across a range of smartphone cameras and computing capacities, while device manufacturers like Apple and Microsoft can optimise their experiences.

As for Apple, they've been in the AR game for years, purchasing Kinect developers PrimeSense in 2013, and, two years later, AR developer Metaio - presumably for its SLAM software. Although much had been rumoured about a pair of Apple “spectacles” - the mythical sunglass-like idealised interface for augmented reality - very little hardware had been announced to support AR until just a month ago.

Project Tango handset

Project Tango: Google's all-ringing, all-dancing 3D-sensing smartphone


At this year's World Wide Developers Conference (WWDC), Apple unveiled ARKit, an iOS library seamlessly providing SLAM and integration with other key iOS services, such as CoreLocation. Suddenly, everything hard about AR became as easy as flipping a few switches in Xcode and copying a few code examples. Apple fanbois noted Apple's SLAM implementation was incredibly good, and very easy to use. A series of eye-popping demos surfaced on YouTube - including, most memorably, a “portal” into a jungle-like dimension from the middle of a city street.

Apple followed that up with the iPhone X and its TrueDepth camera - essentially a Kinect in about one-thousandth the volume, capable of accurately mapping a human face, both for identification purposes, and to produce a stream of real-time motion capture data that can be mapped onto a poo emoji.

The TrueDepth camera is the first real bit of AR technology that hundreds of millions will own - and it's clearly the direction Apple will be taking AR on iOS, with expectations it will show up across the product line within the next few years.

Alphabet ran a search-and-replace over Project Tango (perhaps not quite literally, but fairly obviously), announcing its own Android-based ARCore. It currently works only on Google's Pixels and Samsung's Galaxy S8, while Apple's ARCore works on all iOS devices back to the iPhone 6S. There are a few ARCore demos out, but nothing like the avalanche of ARKit apps that came with the release of iOS 11. Hundreds of ARKit apps were released in the six weeks following the launch of iOS 11, giving Apple an early advantage in the ARwars, as Google tries to spread ARCore to other Android devices.

Delaying a next-generation Hololens until early 2019, Microsoft promises a "3rd generation device" closer to the ideal of nearly weightless sunglasses. Meanwhile, Windows Mixed Reality remains a developer's toolkit primarily geared toward virtual reality apps. At well more than $3,000, Microsoft has sold only a few tens of thousands of its Hololens, not nearly enough to kickstart an ecosystem of AR developers.

Facebook seems to be stuck somewhere between the obvious-to-all-but-itself collapse of its VR efforts, as Oculus struggles to reach even a half million units in sales, and the launch of its AR Studio applications. No one doubts Facebook can push AR capabilities to its billions of mobile users - but when?

If, as Zuck and Tim Cook and Satya Nadella all suggest, AR is the next great computing platform, there will be another generation of winners and losers. Microsoft lost the smartphone wars, came out strong with Hololens, but seems to be lost once again. Google appears hamstrung by platform fragmentation. Only Apple has shown both the technical and marketing chops to make augmented reality an everyday experience in 2017. Right now it's all about iOS 11, ARKit and the half billion devices that support both. That’s a hell of a dent in the universe. ®

Similar topics

Other stories you might like

  • Lonestar plans to put datacenters in the Moon's lava tubes
    How? Founder tells The Register 'Robots… lots of robots'

    Imagine a future where racks of computer servers hum quietly in darkness below the surface of the Moon.

    Here is where some of the most important data is stored, to be left untouched for as long as can be. The idea sounds like something from science-fiction, but one startup that recently emerged from stealth is trying to turn it into a reality. Lonestar Data Holdings has a unique mission unlike any other cloud provider: to build datacenters on the Moon backing up the world's data.

    "It's inconceivable to me that we are keeping our most precious assets, our knowledge and our data, on Earth, where we're setting off bombs and burning things," Christopher Stott, founder and CEO of Lonestar, told The Register. "We need to put our assets in place off our planet, where we can keep it safe."

    Continue reading
  • Conti: Russian-backed rulers of Costa Rican hacktocracy?
    Also, Chinese IT admin jailed for deleting database, and the NSA promises no more backdoors

    In brief The notorious Russian-aligned Conti ransomware gang has upped the ante in its attack against Costa Rica, threatening to overthrow the government if it doesn't pay a $20 million ransom. 

    Costa Rican president Rodrigo Chaves said that the country is effectively at war with the gang, who in April infiltrated the government's computer systems, gaining a foothold in 27 agencies at various government levels. The US State Department has offered a $15 million reward leading to the capture of Conti's leaders, who it said have made more than $150 million from 1,000+ victims.

    Conti claimed this week that it has insiders in the Costa Rican government, the AP reported, warning that "We are determined to overthrow the government by means of a cyber attack, we have already shown you all the strength and power, you have introduced an emergency." 

    Continue reading
  • China-linked Twisted Panda caught spying on Russian defense R&D
    Because Beijing isn't above covert ops to accomplish its five-year goals

    Chinese cyberspies targeted two Russian defense institutes and possibly another research facility in Belarus, according to Check Point Research.

    The new campaign, dubbed Twisted Panda, is part of a larger, state-sponsored espionage operation that has been ongoing for several months, if not nearly a year, according to the security shop.

    In a technical analysis, the researchers detail the various malicious stages and payloads of the campaign that used sanctions-related phishing emails to attack Russian entities, which are part of the state-owned defense conglomerate Rostec Corporation.

    Continue reading
  • FTC signals crackdown on ed-tech harvesting kid's data
    Trade watchdog, and President, reminds that COPPA can ban ya

    The US Federal Trade Commission on Thursday said it intends to take action against educational technology companies that unlawfully collect data from children using online educational services.

    In a policy statement, the agency said, "Children should not have to needlessly hand over their data and forfeit their privacy in order to do their schoolwork or participate in remote learning, especially given the wide and increasing adoption of ed tech tools."

    The agency says it will scrutinize educational service providers to ensure that they are meeting their legal obligations under COPPA, the Children's Online Privacy Protection Act.

    Continue reading
  • Mysterious firm seeks to buy majority stake in Arm China
    Chinese joint venture's ousted CEO tries to hang on - who will get control?

    The saga surrounding Arm's joint venture in China just took another intriguing turn: a mysterious firm named Lotcap Group claims it has signed a letter of intent to buy a 51 percent stake in Arm China from existing investors in the country.

    In a Chinese-language press release posted Wednesday, Lotcap said it has formed a subsidiary, Lotcap Fund, to buy a majority stake in the joint venture. However, reporting by one newspaper suggested that the investment firm still needs the approval of one significant investor to gain 51 percent control of Arm China.

    The development comes a couple of weeks after Arm China said that its former CEO, Allen Wu, was refusing once again to step down from his position, despite the company's board voting in late April to replace Wu with two co-chief executives. SoftBank Group, which owns 49 percent of the Chinese venture, has been trying to unentangle Arm China from Wu as the Japanese tech investment giant plans for an initial public offering of the British parent company.

    Continue reading
  • SmartNICs power the cloud, are enterprise datacenters next?
    High pricing, lack of software make smartNICs a tough sell, despite offload potential

    SmartNICs have the potential to accelerate enterprise workloads, but don't expect to see them bring hyperscale-class efficiency to most datacenters anytime soon, ZK Research's Zeus Kerravala told The Register.

    SmartNICs are widely deployed in cloud and hyperscale datacenters as a means to offload input/output (I/O) intensive network, security, and storage operations from the CPU, freeing it up to run revenue generating tenant workloads. Some more advanced chips even offload the hypervisor to further separate the infrastructure management layer from the rest of the server.

    Despite relative success in the cloud and a flurry of innovation from the still-limited vendor SmartNIC ecosystem, including Mellanox (Nvidia), Intel, Marvell, and Xilinx (AMD), Kerravala argues that the use cases for enterprise datacenters are unlikely to resemble those of the major hyperscalers, at least in the near term.

    Continue reading

Biting the hand that feeds IT © 1998–2022