The Thirty-Ninth Annual Conference on Neural Information Processing Systems

Dec 2nd - 7th, 2025

San Diego, California, USA

Overview: This paper introduces Synthaesthetic Art, a collaborative framework that combines live perception, generative AI, and robotic execution into a unified creative process. Rather than replacing the artist, this system positions AI and robotics as collaborators that amplify human intent, challenging traditional boundaries of authorship and aesthetic form.

The Technical Pipeline: The researchers present a fully realized system capable of capturing a subject's image, stylizing it, and physically painting it in real-time.

Capture & Preprocessing: The system captures facial images and utilizes Meta’s Segment Anything Model v2 (SAM2) to remove backgrounds, focusing strictly on facial features.

Generative AI: Visual features are extracted using Microsoft’s Florence-2 and rephrased by a LLaMA-based model to create descriptive prompts. These drive a Stable Diffusion model fine-tuned on artist Ivy Chen’s caricature style using custom Low-Rank Adaptation (LoRA) modules.

Robotic Execution: The diffusion output is converted into scalable vector paths (SVG) and executed stroke-by-stroke by an xArm robot using acrylic paints.

Core Principles The framework is grounded in six guiding principles:

1. Machine-Augmented Creativity: The machine acts as an extension of human expression.

2. Hybrid Intelligence: Decisions are shaped by both human intuition and computational inference.

3. Temporal and Spatial Fluidity: The art embraces adaptive, real-time creation.

4. Sensory Fusion: Blends sound, vision, and gesture into unified compositions.

5. Algorithmic Aesthetics: Explores emergent patterns and computational beauty.

6. Decentralised Authorship: Questions the notion of a singular artistic origin.

Redefining Authorship: A key focus of the work is preserving artist agency. The "Artist-in-Residence," Ivy Chen, retains authorship by training the model on her specific hand-drawn data, curating the generated outputs, and manipulating the robot's brushstrokes in real-time via OSC controls. This creates a "Hybrid Intelligence" where the robot is an expressive instrument rather than an autonomous painter.