This article is more than 1 year old

You looking for an AI project? You love Lego? Look no further than this Reg reader's machine-learning Lego sorter

All you need is tens of thousands of Lego bricks, a Raspberry Pi, and a laptop GPU

An engineer has built something that is sure to be the envy of any self-respecting Lego fan: an AI-powered Lego sorting machine.

Daniel West, a software engineer from Australia, built the device, which is also made out of Lego, of course, over two years. The impressive gizmo stands about 80 centimetres tall, contains over 10,000 individual Lego parts, and 15 motors to move pieces that need to be sorted along a series of conveyor belts. West claims it can process a bucket of jumbled up pieces, grouping 2,927 types of Lego into 18 different bins at a rate of one brick every two seconds. Not bad, eh?

You can watch the machine chugging away below.

Youtube Video

Here’s how it works: First, a giant fistful’s worth of random Lego parts are poured down a chute at the top of the machine. The multicoloured river of Lego is processed and shaken by a vibrational feeder to filter it into individual pieces that pass under a Raspberry Pi computer.

The computer contains a camera and snaps a series of pictures of the passing brick and sends them to a laptop running ResNet-50, a popular convolutional neural network, to categorize. The result is then sent back to the machine, opening the right combination of gates to allow the particular piece to fall into the right bucket.

Sorting Lego may be a trivial and tedious task for humans, but it’s actually pretty difficult for machine learning models. Lego parts come in thousands of distinct types, in multiple colours, and combined with the fact that they all look different when viewed at various angles, it’s no wonder that collecting the right combination of training datasets was the hardest part.

West told The Register this week he, at first, tried to generate fake simulations of the Lego bricks. He took 3D models of the pieces from LDraw Part Library, an open source program that allows enthusiasts to build Lego virtually, and rendered them in Blender, free animation software.

The 3D model bricks could then be simulated at different rotations and colors. These individual images were collected into a synthetic dataset containing over a whopping 25 million pictures. But to West’s frustration, his AI Lego sorter trained on the fake images failed to identify real parts.

“I came to a point where I almost gave up on the project entirely,” he said. “I spent months implementing a complex method to transform synthetic images into real-looking images, with little success.” So, he turned to another popular technique often used in robotics to help reduce the simulation-to-reality gap: domain randomisation.

World's first Universal Lego Sorting Machine?

Domain randomisation trains a model to recognize even more variations in data. Instead of just different rotations and colors of Lego bricks, the system learned to account for various lighting effects, texture, and noise. To boost the performance of the Lego sorter further, West also incorporated a smaller dataset containing snaps of real Lego pieces.

“The real dataset only contains parts from my personal collection which ends up being only 544 different types of parts,” he said. These pieces are more common types found among Lego collections, whereas the synthetic dataset features rarer types.

angry lego minifig man turns on anxious lego minifig man

How do we stop filling the oceans with Lego? By being a BaaS-tard, toy maker suggests


By training on both datasets, the camera on West’s Lego sorter should, in theory, be able to recognize pieces that it hasn’t actually seen before. In other words, the machine can categorize bricks in the synthetic dataset that isn’t in West’s personal collection. The performance varies, however, it’s 93 per cent accurate at identifying parts that are in the real dataset compared to 74 per cent for ones that are only in the synthetic dataset.

West trained his ResNet-50 using approximately 800 CPU cores rented on AWS over a day. The Lego sorter then runs on Nvidia’s GeForce RTX 2060 graphics card commonly found in gaming laptops. West believes that it would also work reasonably fast with weaker GPUs too. He is planning to release the code in the future.

It’s not the world’s first Lego sorter. West was inspired by other gizmos, some of which also used AI algorithms. He claimed, however, that it was the world's first Universal Lego Sorting Machine. “I call it 'Universal' because, thanks to the use of cutting-edge AI, it's capable of recognizing and sorting any Lego part that has ever been produced," he said.

“As a lifelong Lego fan, ever since I saw Akiyuki's original Lego sorting machine way back in 2011, I'd had the idea of building a similar machine in the back of my mind," West continued. "About three years ago, I became interested in computer vision.

"AI and CNN's were booming and I began studying them intensely. I also had a lot of experience with computer graphics as during this time, I had just released my video game 'Airscape: The Fall of Gravity'. I realized that I would be able to combine my skills in a unique way to finally realize my lifelong dream of building a real working Lego sorting machine, while at the same time gaining real-world experience in designing and deploying AI systems.” ®

Similar topics


Send us news

Other stories you might like