Proof-of-concept open-source app can cut'n'paste from reality straight into Photoshop using a neural network

Code available if you want to toy with similar camera-grabbing projects


Video We've written a lot about academic research, startups, and internet giants making use of artificial intelligence. Sometimes source code is shared, and sometimes it isn't, which can be frustrating – we feel that pain.

For those of you thinking about toying with machine-learning in a practical sense, how about this interesting open-source proof-of-concept application as an inspiration or a starting guide?

A programmer has built an Android smartphone app in JavaScript and TypeScript, and corresponding back-end server in Python, that allows you to snap a photo of a real object and then transfer that image over the air almost instantly into Adobe Photoshop, with the background automatically removed by a neural network. Support for other imaging editing programs in the works, we're told.

The end result is the ability to cut'n'paste stuff from what you can see through your phone straight into Photoshop. If that doesn’t sound exciting then, well, perhaps you have to see it to really appreciate it:

To make it work, Cyril Diagne, a self-proclaimed digital interaction artist, implemented BASNet, a neural network that extracts the boundary of an object in an image. This is the magic that makes the automatic cut-out work. This approach is supposed to produce smoother results than traditional image-processing algorithms that detect the outlines of objects.

BASNet, for what it's worth, works by transforming an image into a saliency map, which is a fancy term for a heat map that blocks out the background of a photo to highlight the pictured object itself. That is then passed onto a refinement module that sharpens the boundaries to reduce any effects of noise.

The end result of this project is an Android app that passes, via a REST API, camera photos to a back-end server that does the actual AI processing. The server also connects to your installation of Photoshop via a remote-control connection to paste the snapped object into the open document. It's a neat way to bring stuff straight into Photoshop to place and mess around with.

The code for the copy-and-paste augmented reality prototype can be found here. The Register has asked Diagne for comment. ®


Keep Reading

Twitter: Our image-cropping AI seems to give certain peeps preferential treatment. Solution: Use less AI

Let's just go back to human-selected cropping, eh?

AI in the Enterprise: How can we make analytics and stats sound less scary? Let's call it AI!

Register Debate New names for old recipes

Got a problem with trust in AI? Just add blockchain, Forrester urges. Then bust out the holographic meetings. Welcome to the future

It takes 'grit' to send in a holograph to meetings instead of struggling with mute buttons yourself...

AI in the enterprise: AI may as well stand for automatic idiot – but that doesn't mean all machine learning is bad

Register Debate Is AI just a rebrand of yesterday's dumb algorithms? We present the argument against this motion – and don't forget to vote

Microsoft builds image-to-caption AI so that your visually impaired coworkers can truly comprehend your boss's PowerPoint abominations

Better-than-before code to make Office more accessible

AI snatches jobs from DJs and warehouse workers, plus OpenAI and PyTorch sittin' in a tree, AI, AI, AI for you and me

Roundup January's other AI news summarized for you... by a human... honest

Samsung combines 5G, AI, drones and cloud in conspiracy ... to ease network maintenance costs

To save telco workers from climbing the greasy pole as networks get denser

NHS England offers £15m to AI firms for software that helps with stroke victims' treatment as COVID-19 stretches service

Raising inevitable concerns over protection of patient data

Biting the hand that feeds IT © 1998–2020