Stravinsky, Floppy Disk and Neutone Morpho — a dawn of new nano sampling technology
<This is an article written for the Neutone Blog on the launch of Neutone Morpho—a real-time tone morphing audio plugin>
Even if you can’t immediately imagine the sound when you hear the phrase, if you listen to a few examples, you should think, “Oh, that’s the sound.” It’s the sound of 1980s pop, used by many artists like Michael Jackson, Duran Duran, and YES.
Actually, this sound, if traced back to its origin, leads to Stravinsky’s “Firebird,” a fact that might not be widely known.
The world’s first digital sampler, the Fairlight CMI, which was released in 1979, came with a library of ready-to-use sounds on an 8-inch floppy disk. Among them, the sound files named ORCH1, ORCH2… were the origin of this sound. Moreover, it is known that one of the developers of the Fairlight CMI personally sampled this sound from the beginning of a Stravinsky record he owned (for more details on this history, see this YouTube video).
As the name suggests, the charm of a sampler is that you can freely sample sounds, so isn’t it against the philosophy to use the library that comes with it as a bonus? Why did the sounds on the floppy disk that came with the sampler become widely used? Of course, the appeal of the distinctive sound itself is a factor, but another reason should not be overlooked: for most users (musicians), the concept of sampling was still too novel at the time, and they were unable to sample effectively on their own or not motivated to do so.
Africa Bambaata’s Planet Rock is arguably one of the most notable songs that used Fairlight CMI and the orchestra hit. Arthur Baker, the producer of the song, admitted he couldn’t sample by himself.
“You couldn’t sample on the Fairlight, it was all pre-sampled sounds, so we used an explosion, the handclaps, and the orchestra.” [1]
In fact, the developer envisioned the primary purpose of a sampler to be a cost-effective replacement for expensive instruments like grand pianos and violins [2]. As such, it was natural to distribute numerous floppy disks containing a ready-to-use pre-recorded sound library with the sampler (in this sense, today’s expensive sampling libraries that come with multiple DVDs may be considered a legitimate way to use a sampler).
The developers of the sampler probably did not anticipate that artists would record sounds like breaking glass or the sound of a McDonald’s paper bag and use them in songs, let alone sample records of others. This “misuse” of tools by artists led to new musical expressions and genres, including Hip Hop.
Fast forward to today.
In July 2023, We founded a company called Neutone with the goal of pioneering new musical expressions in collaboration with artists using the power of AI.
Then, Neutone Morpho, our first commercial product, is finally out today. Our proprietary AI model, the Morpho algorithm, performs so-called “timbre transfer.” You can transfer any input sound into a specific, different timbre in real-time.
You can imagine it as something like an mp3 compression algorithm specialized in a specific sound. If a compression algorithm were specifically designed for piano sounds, any input, including a violin sound or your voice, would be altered to sound like a piano after compression and decompression. An important point here is that the pitch, dynamics, and noise of the input sound are reflected in the output.
For the Morpho plugin, in addition to traditional instruments such as violins, we have prepared diverse models, including percussion such as djembes, Chinese and Middle Eastern instruments, and choirs. Additional models can also be purchased within the plugin.
When it comes to timbre transfer, the focus tends to be on how faithfully and naturally the sound of existing instruments can be reproduced. Yes, the same as the goal of the pioneers of the sampling technology. However, we believe that this timbre transfer or tone-morphing algorithm has countless possibilities beyond imitating the sound of existing instruments.
One concrete example of such ideas is “textural” models trained on the sound of a ranch at dawn, the emergency call of 911, or a specific song (for example, my own album). These are all intended to reproduce, in a sense, sample and reuse the atmosphere and texture of each sound.
I personally consider Neutone Morpho a new sampling technology. It’s a new kind of sampler that utilizes sound rather than using MIDI as a trigger. Instead of playing back the recorded sounds as they were, it reconstructs sounds based on statistical “sampling” from the distribution of the training data. It could be said that the sound is sampled twice: once when creating the learning data, and again when re-synthesizing the input sound.
I myself use the term “Nano Sampling” to describe this new way of sampling, but when I asked ChatGPT if it had a good name, it suggested “Quantum Sampling.” Indeed, in the sense that you are not quite sure what you get until you feed the input, Quantum Sampling may not be so far off the mark.
The current Neutone Morpho models available in the plugin, or those available for purchase, are akin to a pre-sampled sound library on a floppy disk. Of course, this limits the range of sounds you can explore. To fully utilize the new sampler, you should be able to sample freely.
That’s where Neutone Cocoon, set to be launched soon, comes in. Neutone Cocoon allows you to easily train Neutone Morpho models just by uploading your collected training data to our server. The trained model automatically becomes available for use in your Morpho plugin. The models trained by users are exclusive to them and are owned only by them. Only with this training service can the vision of Neutone Morpho be realized!
RAVE, proposed in 2019 by researchers Antoine Caillon and Philippe Esling at IRCAM, was a groundbreaking algorithm that enabled high-quality real-time timbre transfer for the first time. Their decision to open-source the RAVE architecture led to the inception of many projects, including the original Neutone plugin. Their contributions are invaluable.
We are 100% sure artists around the world will find new ways to use (or misuse) Morpho and Cocoon that is far beyond our imagination.
However, training the RAVE model requires several days (4–6 days) of training on an expensive GPU, which is not suitable for a service like Cocoon. In addition, although RAVE is open source, its commercial use is prohibited. This posed a practical obstacle for us in relying on RAVE and continuing development, as we don’t have any financial backing.
In developing Morpho, we focused on reducing the size of the model itself and the time and necessary training data, while maintaining sound quality. As a result, Morpho, a model smaller than RAVE, was developed. It can be trained in approximately a day using about an hour of training data.
As we prepare for the launch, the Neutone development team, including myself, is genuinely excited. Will a new orchestra hit for the 21st century be born from the library provided by Neutone Morpho? We are 100% sure artists around the world will find new ways to use (or misuse) Morpho and Cocoon that is far beyond our imagination.
Neutone Morpho is more than just a plugin. It is our first step in challenging the status quo of AI creative tools and offering better alternatives for artists and musicians:
- Using others’ works as training data without permission
- Artists cannot easily train their own models for their specific needs
- Relying on generic AI models trained by someone else
- Surprisingly troublesome to integrate into a familiar creative workflow
- Text input is the only viable option to control the output
- Difficult to operate in real-time on a local PC
Neutone’s challenge has just begun. While still resolving issues such as improving Morpho’s sound quality and stability, we hope to contribute to the creation of new musical expressions and genres in the AI era. If there are artists who resonate with our challenge, please feel free to contact us at contact(at)neutona.ai and join our discord channel!
Apr 2, 2024
Nao Tokui
@naotokui_en
References
[1] Buskin, R. 2008. Classic Tracks: Afrika Bambaataa & The Soulsonic Force ‘Planet Rock’. Sound on Sound. November. pp. 78–84.
[2] Paul Harkins “Digital Sampling — The Design and Use of Music Technologies”
