Music ControlNet: Multiple Time-varying Controls for Music Generation (2024)

We use denoisingdiffusionprobabilisticmodels(DDPMs) [17, 18] as our underlying generative modeling approach for music audio.DDPMs are a class of latent generative variable model. A DDPM generates data $\bm{x}^{(0)}\in\mathcal{X}$ from Gaussian noise $\bm{x}^{(M)}\in\mathcal{X}$ through a denoising Markov process that produces intermediate latents $\bm{x}^{(M-1)},\bm{x}^{(M-2)},\dots,\bm{x}^{(1)}\in\mathcal{X}$, where $\mathcal{X}$ is the data space.DDPMs can be formulated as the task of modeling the joint probability distribution of the desired output data $\bm{x}^{(0)}$ and all intermediate latent variables, i.e.,

where $\theta$ denotes the set of parameters to be learned, and $p(\bm{x}^{(M)})\vcentcolon=\mathcal{N}(\bm{0},\bm{I})$is a fixed noise prior.

To create training examples, a forward diffusion process $q(\bm{x}^{(0)},\dots,\bm{x}^{(M)})$ is used to gradually corrupt clean data examples $\bm{x}^{(0)}$ via a Markov chain that iteratively adds noise:

where $q(\bm{x}^{(0)})$ is the true data distribution, and $\beta_{1},\dots,\beta_{M}$ are a sequence of parameters that define the noise level within the forward diffusion process, also known as the noise schedule.

By definition of $q(\bm{x}^{(m)}|\bm{x}^{(m-1)})$, it follows that the noised data $\bm{x}^{(m)}$ at any noise level $m\in\{1,\dots,M\}$ can be sampled in one step via:

where $\bar{\alpha}_{m}\vcentcolon=\prod_{m^{\prime}=1}^{m}(1-\beta_{m^{\prime}})$, $\bm{\epsilon}\sim\mathcal{N}(\bm{0},\bm{I})$, and $M$ is the total number of noise levels or steps during training.It was shown by Ho et al.[18] that we can optimize the variational lower bound[19] of the data likelihood, i.e., $p_{\theta}(\bm{x}^{(0)})$,by training a function approximator, e.g., a neural network, ${f_{\theta}(\bm{x}^{(m)},m):\mathcal{X}\times\mathbb{N}\rightarrow\mathcal{X}}$ to recover the noise $\bm{\epsilon}$ added via(3).More specifically, $f_{\theta}(\bm{x}^{(m)},m)$ can be trained by minimizing the mean squared error, i.e.,

Our approach to music generation is rooted in methodology developed primarily for generative modeling of images.When applying diffusion modeling for image generation, the function $f_{\theta}$is often a large UNet[18, 21].The UNet architecture consists of two halves, an encoder and a decoder, that typically input and output image-like feature mapsin the pixel space[22] or some learned latent space[23].The encoder progressively downsamples the input to learn useful features at different resolution levels,while the decoder, which has a mirroring architecture to the encoder and accepts features from corresponding encoder layers through skip connections, progressively upsamples the features to eventually get back to the input dimension.For practical use,diffusion-based image generation models are often text-conditioned,which requires augmenting the network $f_{\theta}$ to accept a text description ${\bm{c}_{\text{text}}\in\mathcal{T}}$, where $\mathcal{T}$ is the set of all text descriptions.This leads to the following function signature:

which, via the process outlined in Sec.II-A, models the desired probability distribution $p_{\theta}(\bm{x}^{(0)}\,|\,\bm{c}_{\text{text}})$.The text condition $\bm{c}_{\text{text}}$ is typically a sequence of embeddings from a large language model (LLM) or one or more embeddings from a learned embedding layer for class-conditional control. In either case, the conditioning signals $m$ (i.e., the diffusion time step) and $\bm{c}_{\text{text}}$ are usually incorporated in the UNet hidden layers via additive sinusoidal embeddings[18] and/or cross-attention[23].

ControlNet[14] proposed an effective method to add pixel-level (i.e., spatial) controls to large-scale pretrained text-to-image diffusion models.Let the diffusion model input/output space be images, i.e., $\mathcal{X}\vcentcolon=\mathbb{R}^{W\times H\times D}$, where $W,H,D$ are respectively the width, height, and depth (for RGB images, $D=3$) of an image,we denote the set of $N$ pixel-level controls as:

where $D_{n}$ is the depth specific to each $\bm{c}^{(n)}$.For each condition signal $\bm{c}^{(n)}$,every pixel $\bm{c}^{(n)}_{i,j}\in\mathbb{R}^{D_{n}}$, where $i\in\{1,\dots,W\}$ and $j\in\{1,\dots,H\}$, asserts an attribute on the corresponding pixel $\bm{x}^{(0)}_{i,j}$ in the output image. For example, “$\bm{x}^{(0)}_{i,j}$ is (not) part of an edge” or “the perceptual depth of $\bm{x}^{(0)}_{i,j}$”.Naturally, the function to be learned, $f_{\theta}$, should be revised again as:

where $\mathcal{C}$ denotes the set of all possible sets of control signals.The updated $f_{\theta}$ hence implicity models $p_{\theta}(\bm{x}^{(0)}\,|\,\bm{c}_{\text{text}},\bm{C})$.

Our overall goal is to learn a conditional generative model$p(\bm{w}\,|\,\bm{c}_{\text{text}},\bm{C})$over audio waveforms $\bm{w}$, given a global (i.e., time-independent) text control $\bm{c}_{\text{text}}$, and a set of time-varying controls $\bm{C}$. Due to our dataset, we limit $\bm{c}_{\text{text}}$ to musical genre and moods tags.Waveforms $\bm{w}$ are vectors in $\mathbb{R}^{\mathrm{T}\mathrm{f_{s}}}$, where $\mathrm{T}$ is the length of audio in seconds and $\mathrm{f_{s}}$ is the sampling rate (i.e., number of samples per second).As $\mathrm{f_{s}}$ is large (typically between 16kHz and 48kHz),it is empirically difficult to directly model $p(\bm{w}\,|\,\cdot)$.Hence, we adopt a common hierarchical approach of using spectrograms as an intermediary.A spectrogram $\bm{s}\in\mathbb{R}^{\mathrm{T}\mathrm{f_{k}}\times B\times D}$ is an image-like representation for audio signals, obtained through Fourier Transform on $\bm{w}$,where $\mathrm{f_{k}}$ is the frame rate (usually 50$\sim$100 per second), $B$ is the number of frequency bins, and $D=1$ for mono-channel audio.With $\bm{s}$ as the intermediary, we instead model the joint distribution${p(\bm{w},\bm{s}\,|\,\bm{c}_{\text{text}},\bm{C})}$, which can be factorized as:

where $\phi$ and $\theta$ are sets of parameters to be learned.Note that this factorization assumes conditional independence between waveform $\bm{w}$ and all control signals $\bm{c}_{\text{text}}$ and $\bm{C}$ given spectrogram $\bm{s}$,which is reasonable if the time-varying controls in $\bm{C}$ vary at a rate no faster than $\mathrm{f_{k}}$ by nature.

First, the first two dimensions in a spectrogram $\bm{s}$ have different semantic meanings, one being time and the other being frequency, as opposed to both being spatial in an image.Second, the time-varying controls useful to creators are closely coupled with time, but could have a much more relaxed relationship with frequency such that the second dimension of(6) cannot be restricted to $B$.For example,an intuitive control over ‘musical dynamics’ may involve defining volume over time, not over frequency.A high dynamics value for one frame can mean a number of different profiles over the $B$ frequency bins for the corresponding spectrogram frame, e.g., a powerful bass playing a single pitch, or a rich harmony of multiple pitches, which the model has freedom to decide.Therefore, we relax the definition for the set of $N$ control signals to become:

where $B_{n}$is the number of classesspecific to each control $\bm{c}^{(n)}$, which is not bound to $B$.With this updated definition, the correspondence between control signals $\bm{C}$ and the output spectrogram $\bm{x}$ naturally becomes frame-wise.For example, suppose $\bm{c}^{(n)}$ represents dynamics control, a frame for the control ${\bm{c}^{(n)}_{t}\in\mathbb{R}^{1\times 1}}$, where $t\in\{1,\dots,\mathrm{T}\mathrm{f_{k}}\}$, then describes “the musical dynamics (intensity) of the spectrogram frame$\bm{s}_{t}$”.

We propose a strategy to learn the mapping between input controls and the frequency axis of output spectrograms, marking an update from the ControlNet[14] method for image modeling.As mentioned inSectionII-D, ControlNet clones the encoder half of the pretrained UNet for text-to-image generation as the adaptor branch, which uses newly attached zero convolution layers to enable pixel-level control.Let $\tilde{f}^{(l)}(\bm{x}^{(m,l-1)},m,\bm{c}_{\text{text}},\bm{C})$ denote the $l^{\text{th}}$ block of the adaptor branch,where $m$ is the diffusion time step, $\bm{x}^{(m,l-1)}$ contains the features of the noised image after $l-1$ blocks, and $\bm{c}_{\text{text}}$, $\bm{C}$ are the text and pixel-level controls respectively.Considering the case $\bm{C}\vcentcolon=\{\bm{c}^{(1)}\}$ which is consistent with past work[14], the pixel-level control is incorporated via:

where $\mathcal{Z}_{\mathrm{in}}$ and $\mathcal{Z}_{\mathrm{out}}$ are the newly attached zero convolution layers, and $f^{(l)}$ is initialized from the $l^{\text{th}}$ encoder block of the pretrained text-conditioned UNet.

In Music ControlNet, we revamp the control process to be:

where $\mathcal{M}$ is an additional 1-hidden-layer MLP that transforms $B_{1}$, the number of classes for the control $\bm{c}^{(1)}$ following(10), to match the number of frequency bins $B$, and simultaneously learns the relationship between control classes and frequency bins.In cases with multiple controls, i.e., $\bm{C}=\{\bm{c}^{(n)}\}_{n=1}^{N}$, each control is processed with its individual MLP, i.e., $\mathcal{M}^{(n)}$, and then concatenated along the depth dimension, i.e., $D_{n}$, before entering the shared zero-convolution layer $\mathcal{Z}_{\mathrm{in}}$.

To give creators the freedom to input any subset of the $N$ controls, Uni-ControlNet[15] proposed a CFG-like training strategy to drop out each of the control signals $\bm{c}^{(n)}$ randomly during training. We follow the same strategy and further assign a higher probability to keep or drop all controls[15] as we found that this leads to perceptually better generations.In more detail, we let the index set of control signals be $\mathcal{I}=\{1,\dots,N\}$.At each training step, we then select a subset $\mathcal{I}^{\prime}\subseteq\mathcal{I}$ that will be set to zero or dropped.We then directly apply the index subset to the control signals via:

Doing so induces $f_{\theta}(\bm{x}^{(m)},m,\bm{c}_{\text{text}},\bm{C})$ to learn the correspondence between any subset of the $N$ control signals and the output spectrogram.

In Music ControlNet, we further desire a model that allows the given subset of controls to be partially-specified in time.Therefore, we devise a new scheme that partially masks the active controls (i.e., those indexed by $\mathcal{I}\setminus\mathcal{I}^{\prime}$)Specifically, we randomly sample a pair $(t_{n,a},t_{n,b})\in\{1,\dots,\mathrm{T}\mathrm{f_{k}}\}^{2}$, where $t_{n,a}<t_{n,b}$, for each of the active controls, and mask them as:

Fig.2 displays example instantiations of the two masking schemes detailed above.At each training step, after selecting $\mathcal{I}^{\prime}$ (i.e., determining the dropped controls) we choose one of the two masking schemes uniformly at random, and then sample the timestamp pairs (i.e., $(t_{n,a},t_{n,b})$’s) when needed. In this way, we further employ a CFG-like training strategy to enable partially-specified controls in a unified manner.

• •

  Melody ($\bm{c}_{\mathrm{mel}}\in\mathbb{R}^{\mathrm{T}\mathrm{f_{k}}\times 12\times 1}$):  Following[5], we adopt a variation of the chromagram[26] to encode the most prominent musical tone over time.To do so, we compute a linear spectrogram and then rearrange the energy across the $B$ frequency bins into $12$ pitch classes (or semitones, i.e., C, C-sharp, …, B-flat, B) in a frame-wise manner, i.e., independently for each $t\in\{1,\dots,\mathrm{T}\mathrm{f_{k}}\}$, via the Librosa Chroma function[27].To form a better proxy for melody from the raw chromagram, only the most prominent pitch class is preserved by applying an $\mathrm{argmax}$ operation to make the chromagram frame-wise one-hot.Additionally, we apply a Biquadratic high-pass filter[28] with a cut-off at Middle C, or 261.2 Hz) before chromagram computation to avoid bass dominance, i.e., the resulting one-hot chromagram encodes the bass notes, rather than the desired melody notes.At test time, the melody control can be created by recording a simple melody, or simply drawing the pitch contour.A desirable model should be able to turn the simple created melody control into rich, high-quality multitrack music.

• •

  Dynamics ($\bm{c}_{\mathrm{dyn}}\in\mathbb{R}^{\mathrm{T}\mathrm{f_{k}}\times 1\times 1}$):  The dynamics control is obtained by summing the energy across frequency bins per time frame of a linear spectrogram, and mapping the resulting values to the decibel (dB) scale, which is closely linked to loudness perceived by humans[27].To mitigate rapid fluctuations of the raw dynamic values due to note or percussion onsets, and also to bring our dynamics control closer to the perceived musical intensity, we apply a smoothing filter with one second context window over the frame-wise values (i.e., a Savitzky-Golay filter[29]). The dynamics control not only characterizes the loudness of notes, but also is strongly correlated with important musical intensity-related attributes like instrumentation, harmonic texture, and rhythmic density thanks to the natural correlation between loudness and the aforementioned attributes in human-composed music.During inference, creators can simply draw a line/curve of how they want the musical intensity to vary over time as the created dynamics control.

• •

  Rhythm ($\bm{c}_{\mathrm{rhy}}\in\mathbb{R}^{\mathrm{T}\mathrm{f_{k}}\times 2\times 1}$):  For rhythm control, we employ an in-house implementation of an RNN-based beat detector[30] that is trained on a different internal dataset to predict whether a frame is situated on a beat, a downbeat, or neither. We then use the frame-wise beat and downbeat probabilities for control, resulting in $2$ classes per frame.The advantages of our time-varying beat/downbeat control over just inputting a global tempo (i.e, beats per minute) are:(i) it allows creators to precisely synchronize beats/downbeats with, for example, video scene cuts or other moments of interest in the content to be paired with generated music.(ii) it encodes some nuanced information of rhythmic feeling, e.g., whether the music sounds more harmonic or rhythmic, and whether the rhythmic pattern is clear/simple, or complex, on which experienced music creators may want to influence in the generative process.At inference, the rhythm control can be created by time-stretching the beat/downbeat probability curves extracted from existing songs to match the desired tempo.Also, creators can obtain precise beat/downbeat timestamps by feeding the beat/downbeat curves to a Hidden Markov Model (HMM) based post-filter[31, 32], and use the timestamps to shift the curves along the time axis for synchronization purposes mentioned above. We also tried to manually draw spiked curves as the created rhythm control, but the performance of this was worse than our final hand-drawn (i.e., created) dynamics control.

For our Created Controls dataset, we created example melodies, dynamics annotations, and rhythm presets that we envision creators would use during music co-creation via:

• •

  Melody: We record our piano play of 10 well-known classical public domain music melodies (30 seconds long each) composed by Bach, Vivaldi, Mozart, Beethoven, Schubert, Mendelssohn, Bizet, and Tchaikovsky, and crop two 6-second chunks, minimizing repeated musical content as possible, resulting in a 20-example melody controls.

• •

  Dynamics: To simulate a creator-drawn dynamics curves, we draw out 6-second long dynamics curves as $\{\mathrm{Linear,Tanh,Cosine}\}$ functions, either vertically flipped or not, with scaled dynamics ranges of $\{\pm 6,\pm 9,\pm 12,\pm 15\}$ decibels from the mean value of all training examples.This leads to 3$\times$2$\times$4$\,=\,$24 created dynamics controls.

• •

  Rhythm: We create “rhythm presets” via selecting four songs from our in-domain test set with different rhythmic strengths and feelings, extract their rhythm control signals, and time-stretch them using PyTorch interpolation with factors $\{0.8,0.9,1.0,1.1,1.2\}$ to create 20 rhythm controls.

Each set of created controls is then cross-producted with 10 genres $\times$ 10 moods to form the final dataset of 2.0K, 2.4K and 2.0K samples. Our created controls are distinct from controls that are directly extracted from mixture data during training.

We use the following metrics to evaluate time-varying controllability, adherence to global text (i.e., mood & genre tags) control, and overall audio realism.

• •

  Melody accuracy examines whether the frame-wise pitch classes (C, C#,…, B; 12 in total) match between the input melody control and that extracted from the generation.

• •

  Dynamics correlation is the Pearson’s correlation between the frame-wise input dynamics values to the values computed from the generation.We compute two types of correlation, which we call micro and macro correlation respectively.Micro computes $r$’s separately for each generation, while macro collects input/generation dynamics values from all generations, and then computes a single $r$.The micro correlation examines whether relative dynamics control values within a generation is respected, while the macro one checks the same property across many generations.

• •

  Rhythm F1follows the standard evaluation methodology for beat/downbeat detection[37, 38].It quantifies the alignment between the beat/downbeat timestamps estimated from the input rhythm control, and those from the generation.The timestamps are estimated by applying an HMM post-filter[31] on the frame-wise (down)beat probabilities (i.e., the rhythm control signal).Following[38], a pair of input and generated (down)beat timestamps are considered aligned if they differ by $<70$ milliseconds.

• •

  CLAP score[39, 40] evaluate text control adherence via computing the pair-wise cosine similarity of text and audio embeddings extracted from CLAP.CLAP is a dual-encoder foundation model where the encoders respectively receive a text input and an audio input.The text and audio embedding spaces are learned via a contrastive objective[41].To obtain the embeddings for evaluation, we feed the generated audio to the CLAP audio encoder, andset the CLAP text encoder input to “An audio of [mood] [genre] music” to accommodate our tag-based control on global musical style.

• •

  FAD is the Fréchet distance between the distribution of embeddings from a set of reference audios and that from generated audios[42].It measures ‘how realistic the set of generated audios are’, taking both quality and diversity into account.To ensure comparable FAD scores, we utilize the Google Research FAD package[42], which employs a VGGish[43] model trained on audio classification[44] to extract embeddings from audios.Unless otherwise specified, the reference audios for FAD are our in-domain test dataset.