This article is more than 1 year old
Proof-of-concept open-source app can cut'n'paste from reality straight into Photoshop using a neural network
Code available if you want to toy with similar camera-grabbing projects
Video We've written a lot about academic research, startups, and internet giants making use of artificial intelligence. Sometimes source code is shared, and sometimes it isn't, which can be frustrating – we feel that pain.
For those of you thinking about toying with machine-learning in a practical sense, how about this interesting open-source proof-of-concept application as an inspiration or a starting guide?
The end result is the ability to cut'n'paste stuff from what you can see through your phone straight into Photoshop. If that doesn’t sound exciting then, well, perhaps you have to see it to really appreciate it:
4/10 - Cut & paste your surroundings to Photoshop— Cyril Diagne (@cyrildiagne) May 3, 2020
Garment: SS17 by @thekarentopacio
Type: Sainte Colombe by @MinetYoann @ProductionType
Technical Insights: ↓#ML #AR #AI #AIUX #Adobe #Photoshop pic.twitter.com/LkTBe0t0rF
To make it work, Cyril Diagne, a self-proclaimed digital interaction artist, implemented BASNet, a neural network that extracts the boundary of an object in an image. This is the magic that makes the automatic cut-out work. This approach is supposed to produce smoother results than traditional image-processing algorithms that detect the outlines of objects.
BASNet, for what it's worth, works by transforming an image into a saliency map, which is a fancy term for a heat map that blocks out the background of a photo to highlight the pictured object itself. That is then passed onto a refinement module that sharpens the boundaries to reduce any effects of noise.
The end result of this project is an Android app that passes, via a REST API, camera photos to a back-end server that does the actual AI processing. The server also connects to your installation of Photoshop via a remote-control connection to paste the snapped object into the open document. It's a neat way to bring stuff straight into Photoshop to place and mess around with.
The code for the copy-and-paste augmented reality prototype can be found here. The Register has asked Diagne for comment. ®