How music works, using Ruby

by Thijs Cadier

In his presentation at RubyConf 2022, Thijs Cadier explores the intricate relationship between music and programming by demonstrating how sound can be understood and manipulated through Ruby. The talk emphasizes how technology has fundamentally changed music, particularly with the evolution from live performances to recorded audio. Cadier outlines several key components that define music and highlights how Ruby can replicate industry-standard audio processing techniques.

Key Points Discussed:

- Understanding Music:

- Music as audio involves sound waves traveling through air, interpreted by our brain to create musical experiences. Cadier introduces fundamental musical concepts:

- Pitch: Frequency of sound waves; lower pitches (like bass) and higher pitches (like violins).

- Timbre: The unique quality of sound, illustrated by instrument varieties that contribute to musical depth.

- Tempo: The rhythm created from silence and sound interspersing.

Historical Context:
- Cadier provides a brief timeline leading to modern digital audio, starting with Edison’s wax cylinders, Berliner’s discs, and the transition to magnetic tapes; ultimately culminating in digital music with CDs and computer technology.
Using Ruby for Audio Processing:
- He introduces the wave file gem, a Ruby library for reading and writing WAVE files, serving as a tool for sound manipulation.
- Audio Amplification: Demonstrates amplifying sound samples and warns about clipping, which creates harsh distortion when signal peaks exceed allowed levels.
- Mixing: Merging audio tracks to enrich soundscapes, illustrated with an example combining two tracks.
- Compression: Reducing sound peaks for a balanced audio mix, common in radio and popular music.
Sound Synthesis:
- Cadier discusses synthesizer sounds and the generation of noise, square waves, sine waves, and their combinations to create complex auditory textures and chords.
- Fourier Transforms: Encouraged for recognizing sound patterns, proving valuable for understanding complex sounds.
- Final Remarks: Cadier concludes with an example showcasing how basic waveforms can result in intricate musical pieces, and shares that he has created a cover of a song using samples generated during the talk.

Overall, the presentation effectively merges music theory with programming, empowering participants to grasp sound production techniques using Ruby.

00:00:00.000 Ready for takeoff.

00:00:16.940 Well, welcome to RubyConf! I'm really happy to be here.

00:00:19.740 It's nice to be in person. Today, I'm going to talk about music, specifically recorded music.

00:00:22.560 I'll discuss the kinds of tricks that producers and musicians use to make the sound good, and I'll explain that with some Ruby code.

00:00:35.640 What I like to do is use programming to understand the world. The world is a very complicated place, and as programmers, we have this amazing tool to actually comprehend things.

00:00:39.719 When you write code, you have to understand it; otherwise, you can't write it. This compels you to learn it, and that's what I did with music.

00:00:47.340 So today, I want to take you through a process where you don't just understand it but also try to write some code.

00:00:51.180 After a while, you start to grasp it, and you end up with hopefully some simple code and examples that I'll share with you.

00:00:59.160 Let’s begin with a very brief history of the technology behind music that has developed over the last several years, which will lead us to digital audio.

00:01:05.440 This is where we can start using Ruby to actually see what's going on.

00:01:10.680 There are several code examples accompanying this talk. You can find those here, and I will also post the slides after the talk.

00:01:15.900 If you want to follow along, you can quickly go to this link.

00:01:20.700 So, what is music? It is described by a few key aspects.

00:01:23.520 Audio, like what you are hearing right now, is a waveform moving through the air. Air molecules bounce against each other, and our ears perceive these vibrations.

00:01:32.520 The easiest way to visualize this is to compare it to waves in a pool of water, but instead of water molecules, we have air molecules.

00:01:38.520 These waves travel through the air from a sound source to your ears, where tiny hair cells called follicles vibrate in response to these sound waves.

00:01:45.720 Your brain then processes this signal to create the illusion of speech or music. So, there isn't anything inherently musical about these physical phenomena.

00:01:55.440 Your brain generates models to make sense of the incoming vibrations. Neuroscientists don't fully understand how this works yet, so there is a bit of magic involved.

00:02:05.280 Let’s explore the basic patterns that the brain uses to map these sounds.

00:02:10.680 The first important concept is pitch.

00:02:15.200 Pitch refers to the frequency of oscillations per second. A lower pitch corresponds to fewer oscillations, while a higher pitch corresponds to more.

00:02:21.879 For example, a bass guitar produces lower pitches, while a violin produces higher pitches.

00:02:28.200 Let’s listen to a basic pitch that sounds like this.

00:02:35.400 [Pitch Example]

00:02:37.860 We're all familiar with that sound.

00:02:41.760 The next element is timbre, which refers to the unique quality or color of a sound.

00:02:46.540 Timbre encompasses all the variations within pitches.

00:02:48.960 An example of timbre can be heard in a classic Beatles song from the 1960s, where different instruments contribute to the overall depth of the sound.

00:02:56.280 Finally, we have tempo, which is established through the presence of silence between sound waves.

00:03:02.400 This silence contributes to the perception of rhythm and can create a loop.

00:03:05.520 If we combine pitch, timbre, and tempo, we create a musical piece that sounds something like this.

00:03:13.280 Historically, music was always live; there was no recorded music until technology evolved.

00:03:20.760 This technological advancement changed the world in many ways, leading to the advent of recorded music.

00:03:27.480 Let’s quickly review the steps that led us to digital audio.

00:03:29.640 It all started with Thomas Edison, who developed a device that could etch sound onto a wax cylinder using vibrations from sound.

00:03:37.560 However, due to Edison's patent, this technology did not gain traction.

00:03:41.760 A different inventor, Berliner, proposed using discs instead of cylinders, which became the dominant format for recorded music.

00:03:49.860 After discovering magnetic tape technology, audio recording quality significantly improved, which facilitated multi-channel recording and transformed the music industry.

00:03:57.060 In the 1980s, music became digital, marking a shift from analog grooves to CDs and modern computer technology.

00:04:05.880 Now, we can use software to replicate tools that were once available only in large studios, making powerful audio production accessible.

00:04:14.640 Next, we’ll dive into what digital audio actually is and replicate some popular tools using Ruby to better grasp their functions.

00:04:20.760 Digital audio consists of samples; when we take a waveform, we can measure the intensity of the signal at specific points.

00:04:27.120 By using these samples, we can recreate and play back the sound.

00:04:35.880 Often, we process this data as a sequence of numbers, which allows for manipulation and playback.

00:04:42.060 In this presentation, I’ve chosen the wave file gem, a Ruby library for reading and writing WAVE files.

00:04:48.120 This gem enables us to read a WAVE file into a large array of samples, simplifying audio processing.

00:04:56.640 Let’s visualize this with an audio sample.

00:05:04.200 [Sample Visualization] Now that we have a representation of our audio data, we can start using some Ruby functions to analyze and modify these samples.

00:05:12.960 In fact, our first step will be to amplify the audio using Ruby.

00:05:20.640 By multiplying each sample by a factor, we can increase their amplitude and create a higher volume.

00:05:28.440 Let’s try this with a sound sample and listen to how the amplification affects the sound.

00:05:39.600 [Amplification Example] You can hear the increased volume, represented by peaks that rise higher than before.

00:05:47.160 However, if we amplify too much, we risk clipping, which distorts the sound and is generally undesirable.

00:05:57.960 Clipping occurs when the peaks exceed the limits of the waveform, leading to a harsh distortion.

00:06:03.420 Let’s listen to how clipping sounds.

00:06:10.380 [Clipping Example] This distortion is often used in genres like heavy metal to achieve a specific sound quality.

00:06:18.900 Next up, we’ll discuss mixing. Mixing involves merging audio tracks together to create more complex soundscapes.

00:06:27.840 When mixing, we can take several audio samples and sum them up to produce a new waveform that combines elements from each sample.

00:06:37.740 Let’s see how this works when we mix two audio samples.

00:06:44.040 [Mixing Example] You can hear how the combination of sounds builds a richer auditory experience.

00:06:52.200 Following mixing, we will cover compression, a technique used by audio engineers to enhance the overall loudness of a sound.

00:07:00.360 Compression targets the peaks of sounds to bring them down, which allows for an overall louder mix without distortion.

00:07:06.960 In radio or popular music, compression is employed extensively. Let’s see how we can achieve compression using Ruby.

00:07:14.520 The process involves defining a threshold that determines when a signal is too loud, and reducing the volume of those peaks.

00:07:26.340 After lowering the peaks, we can apply a makeup gain to raise the overall volumes while avoiding clipping.

00:07:35.520 Let’s listen to an example of this compression process.

00:07:46.800 [Compression Example] You can notice the difference in how much more balanced this sound is compared to the uncompressed version...

00:07:56.760 Now, let’s switch gears to sound synthesis. Many contemporary music pieces use synthesizers to generate sounds.

00:08:06.720 For example, we’ll start with generating noise through Ruby, which we can do using random number functions.

00:08:12.600 The output will appear chaotic, representing the randomness associated with noise.

00:08:19.560 Let’s visualize this noise sample.

00:08:26.640 [Noise Sample Visualization] You'll see how noise can form the basis of percussive sounds in electronic music.

00:08:33.840 Next, we’ll explore square waves, which represent the simplest oscillating sound.

00:08:39.360 Let’s create a square wave using Ruby and listen to its output.

00:08:47.760 [Square Wave Example] Although it produces a 'blunt' sound, it’s a fundamental element within electronic compositions.

00:08:54.480 Now, we’ll delve into sine waves, the core building blocks of many sounds.

00:09:03.180 Generating a sine wave involves using a mathematical function to plot samples in a circular motion.

00:09:10.440 Let’s visualize and listen to what a sine wave sounds like.

00:09:16.440 [Sine Wave Example] This smooth waveform is commonly used across various music genres.

00:09:24.000 Building on that, we can combine multiple oscillators at different frequencies to create chords.

00:09:32.520 Mixing these oscillators produces complex harmonic sounds that lend richness to our music.

00:09:39.420 Let’s hear how this combination sounds.

00:09:48.240 [Chord Example] This is how we can produce rich harmonic textures in our compositions.

00:09:55.920 If you’re interested in further exploration, I encourage you to look into Fourier transforms, where you can deconstruct complex sounds into their sine wave components.

00:10:02.640 This is valuable for recognizing patterns in sound and has applications in graph theory and data analysis.

00:10:09.840 Finally, let’s talk about creating tambre through the combination of different waveforms.

00:10:14.760 By mixing sine waves and square waves, we can generate unique wave shapes that sound richer and fuller.

00:10:22.920 Let’s listen to this final sound example that illustrates this mixture.

00:10:32.880 [Final Hybrid Wave Example] It’s fascinating how from simple building blocks, we can create intricate musical landscapes.

00:10:44.520 That brings us to the end of my presentation.

00:10:51.660 I appreciate your engagement and enthusiasm for this exploration of how music works.

00:10:55.740 As a fun note, I’ve even created a cover of my favorite song using only the samples generated here today.

00:11:06.000 [Musical Outro]