Mandala Club

A few months after our successful proof of concept, we were offered an opportunity for Tessa to perform at the Mandala Club in Singapore. Since this was going to be for a live audience, we wanted to work with a much better robot arm so that we could paint on larger canvases. Additionally, we wanted to paint with a canvas upright on an easel so that the brush strokes and resulting artwork would be more realistic to the way artists paint.

The natural choice for the robot arm was a Universal Robots UR3, which is a compact collaborative robot, also known as a co-bot, around the same size as a human arm. Universal Robots has been building award winning innovative co-bots for over a decade, and my collaborator Richard also has previous experience working with these amazing machines.

Thanks to Daryl Lim, who is co-founder of Augmentus, which is building the future of industrial robotics automation, I was introduced to Ix Lee and his team at Universal Robots based in Singapore. I explained our ideas and aspirations to Ix and Marvin, and they graciously agreed to support us.

Not only did we need the robot arm, but also a gripper for the robot to be able to pick up the bushes and paint. Ix introduced me to Niels Soerensen at OnRobot, who produces top of the line grippers and end effectors. Niels was also excited by our idea and agreed to support us.

It was a really great experience meeting everyone through this process, and we wanted to give special thanks and call out to Sam Tan at Universal Robots for his great support and advice throughout the whole project!

And so, for Tessa’s first public performance, we needed to get her ready for the stage, which meant a complete transformation to how we originally painted.

Using the OnRobot Gripper on the UR3 robot arm, we wanted to make this faster and more reliable than our original proof of concept.

Picking up the brushes from the side of the brush handle would be slow, so to get the painting apparatus ready, so we created modified paint brushes by chopping off most of the handle, and affixing a 3D printed cube on top so that the robot gripper could reliably pick up the brushes.

Using a similar concept as our first proof of concept, we created an improved version of the paint cap tops that mount around the brush handle. These caps allow the brush to soak up paint when they are sitting in the paint containers, and keeps the brushes upright and in the same place for the robot to pick up.

We also designed and 3D printed paint container holders to hold 7 paint containers to provide a variety of colours.

In order to easily transport the UR3, we fabricated a machined aluminium adapter to be able to mount the UR3e on a portable tripod style stand.

So, we finally got the full set up ready to go! Now with all of the physical pieces of the puzzle assembled, it was time to move on to the software, the music processing, the AI agent development, and painting.

Speaking of painting, between Richard and I, the understanding of how to paint was the part we were missing the most! Through my good friend Jay MacCarthy, I met the artist Ellis Liu, who then in turn introduced me to Louisa Raj and Anita Kaur, all three of these talented artists agreed to help us on our mission, which I’m forever grateful for!

Anita, Louisa and Ellis worked with us to plan the paintings, to prepare the canvases, to produce the colour palettes, and mixing the paints to produce the right colours. It has been fantastic working with them and we are continuing our collaboration for some future events too, so watch this space :)

And so with my living room literally turned into a robot painting musical AI development lab, we planned, developed and coded to get ready for the event at Mandala Club.

Once everything was up and running, the Thursday evening before the performance at Mandala, my artist collaborators Ellis, Louisa and Anit came over, and we had a test run. On the music side, both myself and my good friend and amazing keyboardist Eivind Lodemel were there to provide the live music improvisation.

The results of the tests were very successful and we all felt ready for the show!

The final system we created for the show consisted of:

An AI agent running in Python, that listens to the music, performs multidimensional analysis and produces brush stroke patterns in two dimensions.

An audio processing system that acquires stereo audio from the musician’s live performance

A robot control system, which handles the configuration and setup of the robot, and sends the two dimensional brush stroke patterns to the robot coming from the AI agent. All of the control for the robot arm and gripper is performed by this control system, with an ethernet-based messaging system wired directly to the robot and the gripper as well.

Along with the canvases, easel, robot and tripod, the paints and paint brushes were placed on a pedestal to the side of the robot, which allows us to assist the robot with the paints, filling the containers up as the painting progresses.

Saturday morning came, and it was time to get everything over to Mandala Club and set up for the show later that evening. Performing any robot actions is complex and requires precise alignment. In our case we have both the alignment of the robot and the canvas, along with the robot and the paint brushes on the pedestal. In both cases the system has some tolerance built in. So the first step in getting ready for the show was to get everything aligned.

After that, we set up all the audio equipment and the interface between the music and the robot. And now it was time for the show!

We held the event in a room at the Mandala Club, with a guest audience mostly consisting of the Mandala Club’s NFT members. The event was a great success, and we painted two pieces, with each painting running for around 45 minutes each.

Here’s some of the footage from the event, along with the music being played during each painting session, so you can see how the robot reacted differently to the different styles, speed and structure of the music.

The two final paintings are shown here. They came out great and were very well received by all of the attendees!

The two paintings are on display at the Mandala Club, and are planned to be sold at auction later this year.

We learned a lot through the journey of this process.

We were able to test for the first time how fast and precise the Universal Robot UR3 was, and it performed amazingly well along with the OnRobot Gripper.

We learned a lot about how long the painting would take, giving us a lot of ideas about how to make it even faster.

We saw for the first time what Tessa could do with a larger canvas and greater choice of paint colours.

We learned a lot about the complexities of alignment, and have plans to make this a lot simpler in the future, for both the brushes and the canvas.

In the end, Tessa’s first public performance went very well. We had a few hiccups at the very start of it’s attempts to paint, but we were able to fix that pretty quickly and continue on to produce a couple very cool and different paintings in the process!

Thank you again to everyone at Universal Robots, OnRobot, and our collaborating artists Ellis, Louisa, Anit and Eivind for all of their tremendous contributions.