DDSP: Differentiable Digital Signal Processing

DDSP: Дифференцируемая цифровая обработка сигналов
Chenjie Gu, Adam Paul Roberts, Jesse Engel, Roberts, Adam, Lamtharn Hantrakul, Gu, Chenjie
2020-01-14

Differentiable Digital Signal Processingaudio synthesisblind dereverberationdeep learningneural vocoders
Most generative models of audio directly generate samples in one of two domains: time or frequency. While sufficient to express any signal, these representations are inefficient, as they do not utilize existing knowledge of how sound is generated and perceived. A third approach (vocoders/synthesizers) successfully incorporates strong domain knowledge of signal processing and perception, but has been less actively researched due to limited expressivity and difficulty integrating with modern auto-differentiation-based machine learning methods. In this paper, we introduce the Differentiable Digital Signal Processing (DDSP) library, which enables direct integration of classic signal processing elements with deep learning methods. Focusing on audio synthesis, we achieve high-fidelity generation without the need for large autoregressive models or adversarial losses, demonstrating that DDSP enables utilizing strong inductive biases without losing the expressive power of neural networks. Further, we show that combining interpretable modules permits manipulation of each separate model component, with applications such as independent control of pitch and loudness, realistic extrapolation to pitches not seen during training, blind dereverberation of room acoustics, transfer of extracted room acoustics to new environments, and transformation of timbre between disparate sources. In short, DDSP enables an interpretable and modular approach to generative modeling, without sacrificing the benefits of deep learning. The library is publicly available at https://github.com/magenta/ddsp and we welcome further contributions from the community and domain experts.
1
DDSP achieves high-fidelity audio generation without large autoregressive models or adversarial losses, while retaining neural networks’ expressive power.
2
DDSP supports blind room dereverberation, room-acoustics transfer across environments, and timbre transformation between disparate sources.
3
Interpretable modular components enable independent pitch and loudness control, including realistic extrapolation to pitches absent from training.
4
Strong signal-processing inductive biases improve generative modeling without sacrificing flexibility or expressivity.
5
The DDSP library integrates classic differentiable signal-processing elements directly with deep learning models for audio synthesis.

DDSP-based differentiable audio synthesis models integrating classic digital signal processing elements with deep learning

Expressive, high-fidelity, interpretable, and modular audio generation and controllable manipulation of pitch, loudness, room acoustics, and timbre

Publication Details
Publication Date
2020-01-14
Journal
Publisher
ISSN
Cited by
373
Access Type
Author Information
Authors
Chenjie Gu
Adam Paul Roberts
Jesse Engel
Roberts, Adam
Lamtharn Hantrakul
Gu, Chenjie
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%