Friday, July 2, 2021


RSS 2021: Robotics x Arts Workshop

Peter Schaldenbrand ( and Jean Oh

As impressive as style transfer machine learning techniques have become at styling photographs as paintings, it does not match the fidelity of true paint on paper. Technologies that modify and generate images have improved in quality as machine learning methods progress, but still lack the quality that viewers expect from human artistry due to low resolution digital output of the generative models. One method to increase the quality of the generated imagery is to bring it into the physical world by rendering it via drawing or painting. Traditional animation, as seen in the 2017 film Loving Vincent, in which every frame is hand painted by an artist, requires an extreme amount of resources and effort. In order to lower the resources necessary for hand animation, we introduce the first fully autonomous painting animation agent: AniPainter. Similar to rotoscoping, a robot arm paints a given video frame by frame. The video can optionally have its style be transferred prior to painting to a style which may be more suitable for transferring to paint and canvas. Although AniPainter is designed to paint as truthfully to the original image as possible, the constraints from the brush sizes, number of paint colors, and inaccuracies of this inexpensive robot arm do not allow for a perfect recreation of the image. The robot’s imperfections in recreating the given image create abstractions that draw the eye in and create questions on the intentionality of the brush stroke planning. AniPainter provides artists with a new tool for expressing visual media. AniPainter’s painting algorithm explores the question, “What if a printer had harsh constraints?” with the constraints being that a brush and paint are the robot’s only tools for rendering. In the robot’s pursuit to render imagery while overcoming these constraints, is the robot considered creative? If not, what do human painters add that AniPainter does not contribute to the painting process?

Artist Bio

Peter Schaldenbrand is a technical staff member at Carnegie Mellon University. He will begin his PhD studies at CMU’s Robotics Institute in the Fall of 2021. Peter is interested in researching how artificial intelligence can augment human creative expression and communication. This involves creating tools to facilitate creativity as well as measuring the potentially creative abilities of machine learning models. Recently, he has been focusing on a robotic artistic painting project.

All info credited to the artists listed above.