Video Cameras on drones have helped make aerial photography more accessible – and with machine-learning software, they may be able to assist budding filmmakers in the future, too.
A team of researchers from Carnegie Mellon University (CMU) in the US, and Yamaha Motors in Japan, have published a paper showing how drones can be used for simple cinematography (or, you know, surveillance).
“Autonomous aerial cinematography has the potential to enable automatic capture of aesthetically pleasing videos without requiring human intervention, empowering individuals with the capability of high-end film studios,” they wrote in a paper on arXiv late last month.
The scheme requires careful control to keep the drone flying in the right positions to capture the scene from certain angles for prolonged periods of time - all of which needs to be done autonomously.
The researchers trained a DJI M210 drone, a consumer unit commonly used for drone photography, to autonomously film a scene by analyzing images from the camera feed. The drone uses a convolutional neural network to identify objects it has been trained on – in this case it’s only humans, cars, and bicycles – and places a bounding box around them.
The ground-based gizmo calculates the optimal distance from its target to keep the images in focus, and learns to shifts the drone's trajectory and avoid obstacles in order to keep the center of the bounding box in the desired position on screen. In effect, it tracks people, cars, and bikes.
All of the software takes about 300 milliseconds to run on a portable Nvidia Jetson TX2 device featuring a Pascal GPU with 8GB memory and 59.7GB/s of memory bandwidth. The drone travels slowly at 7.5 metres per second, and can film for about 25 to 30 minutes before its batteries drain.
Here’s some example footage it shot...
Okay, it’s not quite Mission Impossible. At the moment, it can only focus on single objects and it’ll be awhile before drones can capture epic car chases or deal with particularly noisy scenes with complicated backgrounds.
The ideas the team are working on including multiple people in one scene, different types of artistic shots, focusing on the motions of actors, and even controlling multiple drones, Rogerio Bonatti, co-author of the paper and a PhD student at CMU, told The Register.
“[We’ve done more] recent experiments involving multiple people in one scene, and using only vision input to localize the actor. What usually happens is that initially the drone will use a person or car or bicycle detection algorithm (depending on the type of actor we expect), and focus on the actor that it can localize with higher confidence.
“After the initial detection the drone tracks the image template over new frames, until confidence falls below a threshold, for example when it loses sight of the actor. At this point we go back to the detection phase. In the future we plan to explore algorithms that allow the drone to make more complex decisions taking into account multiple actors in the scene.”
Fly and spy
Using drones to film action packed scenes for thrillers or nature documentaries is all fine and dandy, but it can also be potentially used for more nasty reasons. What if the drone is trained to track a particular person?
Boffins build smallest drone to fly itself with AIREAD MORE
“I can imagine a huge number of malicious – as well as constructive – uses for such a technology,” said Jack Clark, coauthor of a paper exploring the potential dangers of AI, and strategy and communications director at OpenAI, who first highlighted the CMU-Yamaha work.
"I think the primary dangerous cases here are things like surveillance applications or potentially weaponizing surveillance platforms to also carry out payload delivery – though note this requires a variety of other bits of research," he told us.
Bonatti also recognised this and said that universities and the government should be regulate such technology.
“Just as any other technology, AI is a tool that can be used for both noble and evil purposes," he said.
"As researchers, we focus on the aspects of technology that can improve people's lives. I believe that both universities and the government should up to a certain degree regulate the uses of technology, making sure that projects are being developed for the good of society.
"Exactly how the regulation should happen is a tricky point, because too many restrictions can deter innovation in a country." ®