Face, face, face! Apple, TrueDepth and a nose-driven iPhone X game

Selfie-satisfied to augment reality with their mugs

On the November 3, 2017, Brad Dwyer set to work unearthing the mysteries of Apple’s released-that-day iPhone X and its strange new TrueDepth camera. The engineer and entrepreneur wanted to create an app to leverage that new forward-facing face-scanning camera - to build one of the in a first generation of "face-driven games - but how?

Although Apple had touted TrueDepth as the key hardware component of its “unlock with your face” FaceID system, very little had come to light about exactly how TrueDepth worked, or the kind of access developers would have to the device.

screenshot from Apple's app store of the noze Zone app

Apple's App Store preview of the Noze Zone app

A decade ago TrueDepths gestation began in Israeli firm PrimeSense’s first-of-its-kind “depth camera.” Adding a Z depth value to the planar X-by-Y pixel image, every pixel carried metadata - its distance from the camera. The PrimeSense device projected a pattern of dots (in the infrared, so invisible to the human eye), reading those reflected points of light with a paired infrared camera. That scan - plus quite a bit of software post-processing - generated the depth map fast enough to allow for real-time depth mapping.

Depth mapping wasn’t unique to PrimeSense, but had always suffered from either very low resolution - like the ultrasonic rangefinders pioneered by Polaroid’s SX-70 camera - or had required incredibly expensive, LIDAR-like solutions. The combination of simple silicon plus software gave PrimeSense a 100-to-1 price advantage over its nearest competitors.

That innovation caught the eye of Microsoft, looking for a peripheral that might help Xbox 360 meet or surpass the capabilities of Sony’s EyeToy. An early example of camera-based real-time image processing and gesture recognition, EyeToy never really caught on with the console crowd, but it pointed to a future where the console controller might eventually disappear, replaced by a sensor and game-specific gestures.

PrimeSense had exactly what Microsoft needed to kick things up a notch: an inexpensive, high-quality, reasonably high-resolution depth camera, perfect for integrating the gamer’s body (or pretty much anything else in the real world) into the game.

Only Kinect and the Microsoft crowd

Microsoft licensed the tech and revealed its work on "Project Natal" at E3 in 2009. It took another year to bring the "Kinect" to market - where it was greeted with incredible hype and phenomenally lacklustre sales.

The technology worked, it was certainly interesting, but it suffered from the same chicken-and-egg problem that hamstrings most console peripherals – game developers won’t develop for a peripheral unless it has ubiquitous availability, and gamers won’t buy a peripheral unless it’s supported ubiquitously. Gored on the horns of that dilemma, Kinect never gained the forward motion it needed to succeed in the highly competitive – and highly price-sensitive – console market.

Kinect did find an interesting afterlife as a PC peripheral, after Microsoft released a set of Windows drivers for Kinect, adding depth camera capabilities to garden-variety PCs. Although always little more than a sub-market within a sub-market, Kinect for PC became a core part of the Microsoft story going forward, with glitzy demos crafted by Microsoft Research: office spaces mapped out by multiple Kinects, a work environment with pervasive depth sensing and gestural controls. It made for some great videos - but no sales.

It made no PrimeSense

Well, there was one major sale: that of PrimeSense itself. Not, as one might expect, to Microsoft. Instead, at the end of 2013, PrimeSense was gobbled up by Microsoft’s biggest frenemy – Apple.

What did Apple want with PrimeSense? The Colossus of Cupertino gave nothing away. PrimeSense went silent, even as it staffed up from 300 employees to well over 600. What was it working on? Rumours abounded, generally in connection with Apple’s widely rumoured efforts to create fabled, mythical AR "spectacles". All AR requires some form of depth sensing – whether by camera, with Google’s Project Tango, or something closer to Kinect, as within Microsoft’s Hololens. It seemed obvious that PrimeSense would be a key element of Apple’s move into augmented reality.

So it eventually proved – but not in the way anyone had predicted. Face-scanning had already showed up on smartphones, unsuccessfully. Samsung’s own goal with a Galaxy S8 face scanner that could seemingly be fooled with a photograph made the problem seem a bit ridiculous. Could a face – so unique – be stolen so easily? Not if you focused on the contours that, like fingerprints, make faces unique. Measuring contours calls for a depth camera. Although Apple may not have had this use case in mind when purchasing PrimeSense, the marriage of depth camera and facial identification makes perfect sense, while facial identification without a depth map isn't credible.

At last year's iPhone launch, Apple extolled the virtues of FaceID and the TrueDepth camera that made it happen. As a technical marvel, TrueDepth scores high marks, squeezing the rough equivalent of the Kinect 1 into approximately one-thousandth the volume and one-hundredth the power budget. That’s Moore’s Law squared - and demonstrates what a motivated half-trillion dollar technology firm can achieve. That reach came with some costs – iPhone X shipped in lower numbers than expected because its TrueDepth component proved difficult to manufacture at scale, keeping yields low and iPhone X in short supply until late December.

Promo from apple of iPhone X's FaceID using TrueDepth tech

Apple says its Face ID is enabled by the TrueDepth camera and ... "projects and analyses more than 30,000 invisible dots to create a precise depth map of your face"

Then I saw your face... now I'm a believer

When Brad Dwyer sat down to have a play with his new iPhone X, he knew it had all the makings of a Kinect. But could it do what the Kinect could? One of the Apple APIs gave Dwyer direct access to the depth map. “It’s noisy,” Dwyer told The Register. “You can point it at something and see depth map data, but it’s really messy. Hard to make anything out. But when you point it at a face, then it’s completely different.”

In true Apple fashion, the TrueDepth camera has been very carefully tuned to be very good at one thing – faces. “It’s hard to tell where the hardware ends and the software begins. But you can tell it’s not just using the data from the TrueDepth camera to map faces. There’s a lot more going on.”

Dwyer has a suspicion about what that might be. “I don’t think it’s an accident that Apple introduced ARKit at the same time they introduced CoreML. I think they’re tied together around the TrueDepth camera.”

It’s a classically Apple approach - careful hardware and software integration: noisy sensor data captured by the TrueDepth camera passes through CoreML, which detects the contours of the face, cancelling the noise and making that data clean and smooth - ready for a programmer.

After a weekend of learning, Dwyer sat down with his staff to brainstorm some ideas for TrueDepth apps. “In an hour we’d come up with at least 50 of them.”

A week later Dwyer submitted his first app – Noze Zone – for Apple’s approval. Noze Zone turns your face into a controller for a simple stare-em-down-and-blow-em-up video game. A toe in the water, something to establish Dwyer’s firm as innovators in this newest of interfaces. That’s when things got complicated. “It took about a month for Apple to approve NozeZone, because of an article in the Washington Post.”

On 30 November the Bezos-owned broadsheet published an article asking pointed questions about how the facial data gathered via TrueDepth + ARKit + CoreML would be used. Would it be shared? Would it be uploaded? Would apps be able to sneak a look at your face and read your emotions? Certainly, a firm like Facebook would be very interested in that - more food for their own algorithms. In the wake of the article, Apple - which publicly prides itself on how it engineers products to protect customer privacy - developed a set of guidelines that any TrueDepth apps would need to adhere to. Or face rejection.

“Apple asked us what we were doing with the face data, where it was going, that sort of thing. When they were satisfied we were keeping all of that private, they approved Noze Zone - a month after we’d submitted the app.” Out on the cutting edge, sometimes a small developer has to wait for a behemoth to catch up with itself.

All indications point to a next-generation iPad equipped with a TrueDepth camera - a sign the technology will spread pervasively across Apple’s iOS product line. Whether these cameras remain user-facing is a bigger question. “Right now,” muses Dwyer, “ARKit can detect flat surfaces like walls and floors. With a TrueDepth camera pointed outward, ARKit would be able to detect everything within a space.”

That may happen, but given the close pairing of ARKit and CoreML required to recognise faces, it may be a big ask for a hardware/software system that identifies the shapes of everything. It’s more likely faces are a first step in a graduated approach that will rapidly grow to encompass our bodies - detect faces, then arms and torsos and legs. That’s quite a bit right there, allowing TrueDepth to realise the same expressive capabilities as the original Kinect.

What about inanimate objects? Will Apple need to teach this noisy sensor to recognise dinner plates or trees or automobiles? Here TrueDepth starts to reveal itself as the leading edge of an Apple strategy in augmented reality, building a base of hardware and software techniques that begin to nibble away at one of the biggest technical challenges facing augmented reality - recognising the world.

Dwyer is excited. “We’re spending time working on what we can build now with TrueDepth - a photorealistic mapping app very much like a high-quality version of Apple’s own animojis. But we’re looking forward to the AR spectacles.” ®

Other stories you might like

  • Assange can go to UK Supreme Court (again) to fend off US extradition bid

    Top Brit judges may consider whether an American prison is just too much

    Julian Assange has won a technical victory in his ongoing battle against extradition from the UK to the United States, buying him a few more months in the relative safety of Her Majesty's Prison Belmarsh.

    Today at London's High Court, the Lord Chief Justice of England and Wales Lord Burnett approved a question on a technical point of law, having refused Assange immediate permission to appeal to the UK Supreme Court. The WikiLeaker's lawyers had asked for formal permission to pose this legal conundrum about Assange's likely treatment in US prisons to the Supreme Court:

    Continue reading
  • They see us Cinnamon Rolling, they're rating: GeckoLinux incorporates kernel 5.16 with familiar installation experience

    A nice, clean community distro that works well

    Most distros haven't got to 5.15 yet, but openSUSE's downstream project GeckoLinux boasts 5.16 of the Linux kernel and the latest Cinnamon desktop environment.

    Some of the big-name distros have lots of downstream projects. Debian has been around for decades so has umpteen, including Ubuntu, which has dozens of its own, including Linux Mint, which is arguably more popular a desktop than its parent. Some have only a few, such as Fedora. As far as we know, openSUSE has just the one – GeckoLinux.

    The SUSE-sponsored community distro has two main editions, the stable Leap, which has a slow-moving release cycle synched with the commercial SUSE Linux Enterprise; and Tumbleweed, its rolling-release distro, which gets substantial updates pretty much every day. GeckoLinux does its own editions of both: its remix of Leap is called "GeckoLinux Static", and its remix of Tumbleweed is called "GeckoLinux Rolling".

    Continue reading
  • Running Windows 10? Microsoft is preparing to fire up the update engines

    Winter Windows Is Coming

    It's coming. Microsoft is preparing to start shoveling the latest version of Windows 10 down the throats of refuseniks still clinging to older incarnations.

    The Windows Update team gave the heads-up through its Twitter orifice last week. Windows 10 2004 was already on its last gasp, have had support terminated in December. 20H2, on the other hand, should be good to go until May this year.

    Continue reading
  • Throw away your Ethernet cables* because MediaTek says Wi-Fi 7 will replace them

    *Don't do this

    MediaTek claims to have given the world's first live demo of Wi-Fi 7, and said that the upcoming wireless technology will be able to challenge wired Ethernet for high-bandwidth applications, once available.

    The fabless Taiwanese chip firm said it is currently showcasing two Wi-Fi 7 demos to key customers and industry collaborators, in order to demonstrate the technology's super-fast speeds and low latency transmission.

    Based on the IEEE 802.11be standard, the draft version of which was published last year, Wi-Fi 7 is expected to provide speeds several times faster than Wi-Fi 6 kit, offering connections of at least 30Gbps and possibly up to 40Gbps.

    Continue reading
  • Windows box won't boot? SystemRescue 9 may help

    An ISO image you can burn or drop onto a USB key

    The latest version of an old friend of the jobbing support bod has delivered a new kernel to help with fixing Microsoft's finest.

    It used to be called the System Rescue CD, but who uses CDs any more? Enter SystemRescue, an ISO image that you can burn, or just drop onto your Ventoy USB key, and which may help you to fix a borked Windows box. Or a borked Linux box, come to that.

    SystemRescue 9 includes Linux kernel 5.15 and a minimal Xfce 4.16 desktop (which isn't loaded by default). There is a modest selection of GUI tools: Firefox, VNC and RDP clients and servers, and various connectivity tools – SSH, FTP, IRC. There's also some security-related stuff such as Yubikey setup, KeePass, token management, and so on. The main course is a bunch of the usual Linux tools for partitioning, formatting, copying, and imaging disks. You can check SMART status, mount LVM volumes, rsync files, and other handy stuff.

    Continue reading

Biting the hand that feeds IT © 1998–2022