This article is more than 1 year old

Proof-of-concept open-source app can cut'n'paste from reality straight into Photoshop using a neural network

Code available if you want to toy with similar camera-grabbing projects

Video We've written a lot about academic research, startups, and internet giants making use of artificial intelligence. Sometimes source code is shared, and sometimes it isn't, which can be frustrating – we feel that pain.

For those of you thinking about toying with machine-learning in a practical sense, how about this interesting open-source proof-of-concept application as an inspiration or a starting guide?

A programmer has built an Android smartphone app in JavaScript and TypeScript, and corresponding back-end server in Python, that allows you to snap a photo of a real object and then transfer that image over the air almost instantly into Adobe Photoshop, with the background automatically removed by a neural network. Support for other imaging editing programs in the works, we're told.

The end result is the ability to cut'n'paste stuff from what you can see through your phone straight into Photoshop. If that doesn’t sound exciting then, well, perhaps you have to see it to really appreciate it:

To make it work, Cyril Diagne, a self-proclaimed digital interaction artist, implemented BASNet, a neural network that extracts the boundary of an object in an image. This is the magic that makes the automatic cut-out work. This approach is supposed to produce smoother results than traditional image-processing algorithms that detect the outlines of objects.

BASNet, for what it's worth, works by transforming an image into a saliency map, which is a fancy term for a heat map that blocks out the background of a photo to highlight the pictured object itself. That is then passed onto a refinement module that sharpens the boundaries to reduce any effects of noise.

The end result of this project is an Android app that passes, via a REST API, camera photos to a back-end server that does the actual AI processing. The server also connects to your installation of Photoshop via a remote-control connection to paste the snapped object into the open document. It's a neat way to bring stuff straight into Photoshop to place and mess around with.

The code for the copy-and-paste augmented reality prototype can be found here. The Register has asked Diagne for comment. ®

More about


Send us news

Other stories you might like