Visual Instrument

Audience

Visual Artists, Musicians, and VJs

Team

Maxwell Crabill, Peter Franko

Contributions

Concept and Proof
Research
Coding and Testing

Design 481, UW, Winter 2017
Personal, Winter 2020 Onward

Empowering visual artists to improvise alongside musicians.

For a coding exercise, I sought to inject an additional layer of artistry and control in to music visualization. The Visual Instrument gives digital artists the power to live-paint visual landscapes with brushes which react to music.

I took the traditional concept of the visual music synthesizer and combined it with a generative painting program. The program features three unique brushes, and two separate canvas clearing functions that each play differently with the brush algorithms.

All code up up to 2020 written in processing. All prototypes built after 2020 are built with code in TouchDesigner.

How might we use code to allow visual artists and illustrators to synthesize their work with music as its being played?

The latest iteration of the visual instrument, a series of explorations combining live performance, painting, analog footage, and, most recently, AI. No audio reactivity is used, all inputs are performed by a human (that's me!). Read on to see every step along the way.

Background

01
—

Tools and Constraints

Processing, a visual coding language, was to be our only tool. With this exercise, we intended to familiarize ourselves with the processes, constraints, and demands of coding.

02
—

Problem Area & Research

Despite their close associations, marrying illustration with music is challenging, thanks to music's temporal, reactive, and improvisational nature contrasting with the drawn out, process-to-payoff nature of traditional artwork.

To prepare for my solution, I investigated existing ways to create dynamic visuals:

Footage by STVinMotion

—
‍

Resolume

An incredibly popular and affordable VJ choice. Enables the rapid swapping of video clips and the application of preset effects. High fidelity, low potential for detail sculpting and improvisational content.

Performance by Cecily Hoyt.

—
‍

Analog Liquid Light Shows

100% analog solution. High fidelity, abstract images created by mixing dyes and chemicals over projectors. Highly reactive and improvisational. Low potential for detail and sculpting, as hands and instruments block the image capture.

Footage captured by Geoffrey Lawrence.

—
‍

Particle Illusion

A particle emitter software which, when cleverly rigged, enables artists to "paint" custom particle effects onto the canvas. High potential for individual expression, requires immense amounts of preparation.

Development

03
—

Brush Concepts

Four hypothetical generative brushes concepts, with approximations of their parameters and what inputs would control them

Once I confirmed that the use and control of multiple brushes was indeed possible within Processing, I drafted ideas based off of what I knew could be accomplished from other code I had seen. I wanted to create a wide variety of textures for the artist to use to recreate the feel of a wide variety of songs and sounds.

04
—

Concept Art

I created multiple sketches of potential variations on the concept. The first sketch features a GUI featured which allows users to switch brush symmetry on and off and select brushes on screen. A synthesizer runs in the background, which operates on the same unified color palette as the chosen brush. Other variations included a user-controlled procession of background images and videos, taking a cue from Resolume, allowing for more storytelling capabilities.

First Iteration

05
—

Product Function

The program has three generative brushes. Brushes are swapped with the 1, 2, and 3 number keys on the keyboard.

—

Generative Brush A

This brush generates a colorful web while adding the paths it draws to the application memory. Pen pressure affects the web size, and tilt determines the hue. Music adds additional influence over hue and brightness.

—

Generative Brush B

Functions similarly to Brush A, but does not add new paths to the application memory, and therefor can only interact with paths drawn by Brush A. Compared to Brush A, colors are inverted and shift less erratically with music, allowing for a greater control.

—

Atmospheric Brush

Creates complex, lingering baubles and lines that can be used to add gentle atmosphere or to create intense visuals.

Additionally, two distinct functions exist for clearing the canvas, activated with z or c.

—

Total Clear (z)

Clears canvas and wipes brush memory clean. Good for removing clutter and giving the artist a fresh start.

—

Visual Clear (c)

Clears canvas, but preserves brush memory. Can be used in conjunction with Brush B to rapidly iterate complex drawings on paths created by Brush A.

Further Iterations (2020 and beyond)

Above is an abbreviated slice of a prototype performance made in TouchDesigner. The full performance is choreographed and paced to highlight features of the prototype in synchronization the music. If you have the patience, I encourage you to click the button below to see the full performance.

full performance

06
—

Moving to TouchDesigner

When I began further development of this idea as a personal project, I moved development to TouchDesigner. TouchDesigner's node-based programming environment allows me to ideate and iterate quickly, building out ideas in sketches and "blocks" that can be quickly meshed together.

07
—

Product Function

This iteration was built to better emulate painting and analog liquid light visuals, with the added control of a wacom stylus. This time, there is only one brush, and the artist uses additional inputs to control the behavior of the "paint" itself.

—

Brush

The brush is a circular mask that expands and contracts with pressure, much like a traditional painting program. The mask allows the values from a moving noise pattern to bleed onto the canvas.

—

Paint

Once noise has bled onto the canvas, it enters a feedback loop which causes it to propagate like watercolor. The artist controls the direction and speed at which it bleeds. Rapid expansion creates a "zoom in" common in music visuals. The artist can also reverse the bleed, causing the image to contract and shrink. This technique replaces the "clear canvas" function.

—

Canvas / Plates

The patterns can be additionally controlled by adjusting the speed at which the canvas rotates, mimicing the spinning plates of analog liquid light artists.

08
—

Exploring Audio Reactivity

Above is an exploration of audio reactivity in TouchDesigner.

The initial TouchDesigner prototype had succeeded in adding more artist control, so much so that it might have been hard to notice it wasn't naturally audio-reactive. While it was nice to discover audio-reactivity wasn't a necessity, I wanted to explore it: audio-reactivity is a feature that computers can offer that analog visuals cannot.

—

In the initial prototype, a spontaneously generated noise texture was used as the base "canvas" for the paint.

—

In this iteration, the computer is holding replicated 3D circles in its memory and oscillating them to the rhythm of the music. It trains a camera on the result, and uses the resulting image as the basis for the visuals.

—

This exploration is deliberately focused on exploring audio-reactivity. It is entirely automated and features no live artist control, unlike the prior iteration.

09
—

Integrating footage and AI

—

In order to create a more organic, random, and rewarding visual feeling, and to overcome the "sameiness" of visuals based on Perlin Noise, I used footage of analog liquid lights to alter the noise pattern.

—

In addition, I incorporated streamDiffusion—a lightweight stableDiffusion analog, capable of quickly outputting low resolution images at a pace suitable for live performance. A slider was added to control the influence of the AI. It acts mostly as a video filter, slightly altering the shapes laid down by the painting engine beneath it into organic-looking hallucinations.

—

I incorporated various functions to allow the original paint underneath the AI to bleed through in areas of low contrast or dark values, in order to allow high-resolution, quick-reacting visuals to interplay with the choppier, slower AI filter output.

Epilogue

09
—

Further Development

Exploring dance and shadow puppetry

Lets collab, bro

I'm grateful to currently have artistic collaborators helping to bring this dream into fruition. Our goal is to enhance analog liquid light (after admitting to ourselves that nature can render visuals with infinite times the efficiency of computers) with digital flourishes and control methods, some of which are explored in this case study. I've chosen to keep the most exciting innovations under wraps for now, out of respect for my partners.

Visual Instrument

Audience

Team

Contributions

Background

01—

Tools and Constraints

02—

Problem Area & Research

—‍

Resolume

—‍

Analog Liquid Light Shows

—‍

Particle Illusion

Development

03—

Brush Concepts

04—

Concept Art

First Iteration

05—

Product Function

—

Generative Brush A

—

Generative Brush B

—

Atmospheric Brush

—

Total Clear (z)

—

Visual Clear (c)

Further Iterations (2020 and beyond)

06—

Moving to TouchDesigner

07—

Product Function

—

Brush

—

Paint

—

Canvas / Plates

08—

Exploring Audio Reactivity

—

—

—

09—

Integrating footage and AI

—

—

—

Epilogue

09—

Further Development

Oh no! This portfolio isn't ready for mobile layouts yet.

It's been recently renovated. Log on with your desktop or laptop computer, or come back on Wednesday, August 25th. It'll be done before then!

See you on the other side <3

01
—

02
—

—
‍

—
‍

—
‍

03
—

04
—

05
—

06
—

07
—

08
—

09
—

09
—