Meaning and Inspiration

Creating "Elemental" was a project that perfectly combined three of my greatest passions: music, art and development. As someone who has always loved music, I've always been fascinated by the way that sound can evoke emotion, tell a story, and create an immersive experience. Similarly, as a designer with many years of experience, I've always been drawn to the visual side of things, using color, form, and texture to create beautiful and compelling visuals.

The idea to combine these two passions into a generative audiovisual project was a natural one for me. I wanted to explore the way that music and visuals could work together to create something that was greater than the sum of its parts. I wanted to create an experience that would be both beautiful and immersive.

The inspiration for the composition came from a few different sources. The first, was probably Batu's latest single called Built on Sand. The album art fits the emotion/feel of the track so perfectly and it was released right as I was part way through my Three.js Journey.

The second was this piece that sits in my living room. It was a gift from my wife's late aunt, and I've always loved the caustic effect it creates in the space.

Behind The Scenes

I first began crafting the "element". This was a very explorative process for me, as I was still getting a sense of my workflow with the new libraries and everything.As I experimented with different materials and effects, I discovered that by inserting a refractive material inside of a distorted transmissive material, it created a mesmerizing effect that would become the foundation for the entire piece. (A huge thank you to the pmndrs folks and their incredible tooling)

This discovery was a pivotal moment in the project, as it gave me a starting point from which I could build and layer other visual and audio elements. It was incredibly exciting to see the piece take shape, and to witness the different elements come to life in a way that was both engaging and immersive.

I created a crystal gltf in Blender which works as an excellent geometry piece for the refraction. Using a variety of scales and positions, I've included many of these inside of each orb. The coloration is applied within a range of hues based on the edition's element, with "sacred" editions having more variation of hue and lightness.

Most colors in the piece are in relation to something I called the `baseHue`, which provided not only the coloration for the crystals, background, cursor, etc., but also the base frequency of the sound. I take the baseHue value and convert the hue into a base frequency using Tone.js's Tone.Frequency() primitive which can provide me with a base note to create my scales.

After I got the base note and my scales, it was a matter of creating the various different melodies and harmonies for the different instrumentations. Here's an adapted version of the code to showcase how the "magical" synth melodies are created for each piece:

const baseFreq = Tone.Frequency(baseHue).toNote().charAt(0)
const baseNote = `${baseFreq}${octave}`

let notes = [];
for (let o = octave; o < octave + octaves; o++) {
  const scale = Tonal.Scale.notes(`${baseNote.charAt(0)} major`);
  if (!sacred) {
    shuffleArray(scale, seed);
  }
  scale.forEach((note) => {
    if (random_bool(0.5)) {
      notes.push(`${note}${o}`);
    }
    if (random_bool(0.1)) {
      notes.push(null);
    }
  });
}

Tonal.js was also a big help here, as it's Tonal.Scale primitive provides an array of notes in a particular scale. After that, I randomly select notes from the scale in each octave, and unless the piece is "sacred", I shuffle the array to create a more dynamic sound for each piece. It also emphasizes the "ascending" feeling for the sacred pieces which I feel helps them stand out.

If you've got keen ears, you may have also noticed that each element has its own foley sound. Each foley sound was carefully chosen to compliment the element's visual and the generative melody. For example, you'll hear a crackling flame for the fire element, and a slow drip for water. Each one of these sounds was edited and has custom effects to fit the piece.

The Future of Elemental

I've already got some ideas of future extensions for Elemental that I'm excited to start working on soon. It's very early on, so I don't want to box myself in on a direction yet, but I'll just say that I would love to explore various "unlocks" that each different element could potentially provide.

If you've gotten this far. Thanks so much for reading this article. I really appreciate your interest in the piece!