We Are The Robots: Developing the automatic sound engineer

WE ARE THE ROBOTS !Developing the automa0c sound engineer

Brecht De Man Centre for Digital Music Queen Mary University of London

2

Introduc0on

Approaches

CIRMMT experiment

Future

The state of the music industry •  Changing paradigms – More content being produced and consumed than ever –  Diminishing budgets for produc0on, diminishing revenues

•  Need for a faster and cheaper produc0on process – More 0me to focus on the crea0ve aspects of music –  Further democra0sa0on of music technology

3 Brecht De Man

Bridging the gap MIXING!RECORDING! DISTRIBUTING!

4 Brecht De Man

Autonomous, intelligent, automa0c, (cross-‐)adap0ve mixing

•  Autonomous: without human interven0on •  Intelligent: perceiving, learning, reasoning and ac0ng intelligently

•  Automa0c: using control systems

•  (Cross-‐)adap0ve: manipula0ng an object’s parameters in func0on of measured features of the same object (or of other objects)

5 Brecht De Man

Photography analogy

Automa'c focus, flash, stabilizer, face detec'on…

Amateur no exper0se required

Professional speed, focus on crea0ve aspects

Automa'c faders, panning, EQ, compression, reverb…

Amateur no exper0se required

Professional speed, focus on crea0ve aspects

Brecht De Man 6

Photography analogy

Instagram

Emulate a photographic style, camera, era

Instamix?

Emulate a genre, evoke a mixing style, era, mixing engineer’s sound, or any other ‘seman0c’ preset

Brecht De Man 7

8

Introduc0on

Approaches

CIRMMT experiment

Future

Approaches to autonomous mixing

•  Machine learning

•  Grounded theory

•  Knowledge engineering

9 Brecht De Man

Machine learning

“Maybe the system can figure it out on its own” •  Systems that can learn from data

•  Train on a learning dataset, perform on unseen content

•  Availability of training data? (<> MIR)

•  e.g. Sco_ & Kim 2011 on 48 sets of ‘stems’

10 Brecht De Man

Machine learning (2)

Sco_ & Kim 2011

11 Brecht De Man

Grounded theory

“Maybe we can find out the rules through studies” •  Systema0c genera0on of theory from data

•  Psychoacous0c studies and perceptual evalua0on •  e.g. Pestana 2012-‐2013: assump0on “HPFs should be used on any track without significant LF content”, tested through listening tests and expert interviews

•  Very resource intensive

12 Brecht De Man

Knowledge engineering

“Maybe the rules are already known” •  Integra0ng established knowledge into the system’s constraints

•  Tradi0onal approach to intelligent systems design

•  Best prac0ces generally not known •  e.g. De Man & Reiss 2013: seman0c mixing system

13 Brecht De Man

C4DM June experiment

•  Rule base extracted from prac0cal audio engineering literature

•  Sta0c, ‘deaf’ processing •  Shootout with exis0ng signal-‐dependent system, human engineers and sum

14 Brecht De Man

SNEAK PREVIEW


•  Many EQ and compressor sekngs examples for numerous instruments

•  Some rules of thumb on panning

•  Only few, vague remarks on fader and reverb sekngs

15 Brecht De Man

SNEAK PREVIEW


•  5 songs •  5 x 5 mixes

•  15 subjects •  4 bars

16 Brecht De Man

SNEAK PREVIEW


17 Brecht De Man

SNEAK PREVIEW


18 Brecht De Man

SNEAK PREVIEW


19 Brecht De Man

SNEAK PREVIEW


•  Test audio was professionally recorded •  Need for some degree of adap0vity to input audio features

•  Formalisa0on and valida0on of rule base

•  Add reverb •  Best of both worlds: hybrid system

•  More audio: brechtdeman.com/research.html

Brecht De Man 20

SNEAK PREVIEW

21

Introduc0on

Approaches

CIRMMT experiment

Future

CIRMMT September experiment

•  Research project with George Massenburg and Richard King

•  Mixing assignment for MMus Sound Recording students

•  Year 1 students assess year 2 students’ mixes (and vice versa), including the original mix and… a robot mix

22 Brecht De Man

CIRMMT experiment: the Robot

•  Mixgenius ‘suite’, including automa0c levels, panning, EQ, compression, limi0ng

•  Minimal human interac0on: – overall ‘mastering EQ curve’ (‘jazz’ and ‘rock’) – one-‐size-‐fits-‐all reverb on everything –  lead vocal fixed level boost and pinned centre – manual but objec0ve loudness correc0ons based on newer loudness model

23 Brecht De Man

Mul0track VST plugins

•  VST plugins (C++), real-‐0me •  Mostly based on ‘best prac0ces’(knowledge engineering), validated by listening tests

•  Centre for Digital (Queen Mary University of London) and Mixgenius Inc.

24 Brecht De Man

Mixgenius

•  Founded 2012 •  Montréal based

•  Spun out of Queen Mary University of London

•  www.mixgenius.com

25 Brecht De Man

Automa0c faders

26 Brecht De Man

•  Equal loudness •  Manual boost or a_enua0on •  Ac0vity gate

S. Mansbridge, S. Finn, and J. D. Reiss, “Implementa0on and evalua0on of autonomous mul0-‐track fader control,” 132nd Conven0on of the Audio Engineering Society, April 2012.

Automa0c panning

27 Brecht De Man

•  Spectral and spa0al balancing •  Low frequency sounds centered •  Manual override possible

S. Mansbridge, S. Finn, and J. D. Reiss, “An autonomous system for mul0-‐track stereo pan posi0oning,” 133rd Conven0on of the Audio Engineering Society, October 2012.

Automa0c EQ

28 Brecht De Man

•  Demasking •  Demuddying (auto HPF) •  Denoiser (separate) •  DeEsser (side-‐chain compression) •  Emphasis on correc0ve processing

S. Hafezi and J. D. Reiss, “Autonomous mul0-‐track equaliser (de-‐masking approach),” MSc project report, 2013.

Automa0c compressor

29 Brecht De Man

•  Automa0on of compression parameters •  Cross-‐adap0vity

D. Giannoulis, M. Massberg and J. D. Reiss, “Parameter automa0on in a dynamic range compressor,” Journal of the Audio Engineering Society, 2012.

J. A. Maddams, S. Finn and J. D. Reiss, “An autonomous method for mul0-‐track dynamic range compression,” Proc. of the 15th Int. Conf. on Digital Audio Effects (DAFx-‐12), September 2012.

Automa0c master compressor

30 Brecht De Man

•  Same principle, only stereo track •  Limi0ng to full scale (-‐.3 dB)

D. Giannoulis, M. Massberg and J. D. Reiss, “Parameter automa0on in a dynamic range compressor,” Journal of the Audio Engineering Society, 2012.

J. A. Maddams, S. Finn, J. D. Reiss, “An autonomous method for mul0-‐track dynamic range compression,” Proc. of the 15th Int. Conf. on Digital Audio Effects (DAFx-‐12), September 2012.

MasterEQ

31 Brecht De Man

•  Force audio to predetermined spectrum •  Spectrum may be harvested from exis0ng songs

P. D. Pestana, Z. Ma, J. D. Reiss and D. A. A. Black, “Spectral characteris0cs of popular commercial recordings 1950-‐2010,” 135th Conven0on of the Audio Engineering Society, October 2013.

32 Brecht De Man

µ 10.3089 76.1867 53.8789 46.1100 71.0644 19.6589 70.3533 57.2789 40.4889 53.2356

σ 10.2564 13.5686 11.6063 23.9168 12.6646 9.2183 21.0103 18.1015 12.3709 17.9622

µ 27.5056 46.1433 49.3522 39.0800 77.4644 51.6556 69.9867 62.0067 48.6100 51.4456

σ 19.3799 12.6639 12.4960 19.0994 16.6067 21.5061 14.0059 16.8658 15.7646 19.6447

33 Brecht De Man

µ 36.8778 75.6933 64.6911 77.4489 55.4200 34.6033 73.6022 45.7544 48.1944 50.1611

σ 13.8394 14.9650 13.7291 17.0130 16.0141 14.8073 9.0548 18.8942 15.0073 18.4069

34 Brecht De Man

µ 45.1333 32.5556 77.4089 49.2833 63.6200 62.6678 68.9478 43.8922 43.6289 65.5167

σ 25.2020 13.0335 15.2261 15.2266 18.4768 15.1728 13.2034 20.2594 15.6637 15.9088

35 Brecht De Man

CIRMMT experiment: remarks

•  Preliminary representa0on of data, much processing and inves0ga0on ahead

•  Original, professional mix not always at the top…

•  Assessment very subjec0ve (genera0onal preference?)

•  Robot’s performance modest, but not outlier

36 Brecht De Man

Winners

•  SR 1 My Funny Valen0ne •  SR 1 Lead Me

•  SR 2 High Blood Pressure •  SR 2 No Prize

37 Brecht De Man

Winners



38 Brecht De Man

Pawel Leskiewicz

Winners



39 Brecht De Man

Pawel Leskiewicz

Stuart Bremner

Winners



40 Brecht De Man

Pawel Leskiewicz

Stuart Bremner

Brandon Wells

Winners



41 Brecht De Man

Pawel Leskiewicz

Stuart Bremner

Brandon Wells

Kevin Fallis

42

Introduc0on

Approaches

CIRMMT experiment

Future

CIRMMT: Experiment sequel

43 Brecht De Man

Can you keep up with the machine?

•  Next experiment in January

•  4 months of development 0me

•  Robot feeding on your data [cue evil laugh]

Embracing the technology

44 Brecht De Man

•  Opportunity to focus on more crea0ve aspects •  Higher produc0vity •  Lower threshold for musicians

•  Faster soundcheck

What’s in store

•  Mixgenius: research and product development •  C4DM QMUL and elsewhere: – bleed reduc0on – seman0c approaches to autonomous mixing – cross-‐adap0ve autonomous compression – new loudness models for autonomous mixing

– automa0c reverb – dereverbera0on

45 Brecht De Man

Where to go from here

•  Joint effort of machine learning, signal processing, knowledge engineering experts

•  Research: lack of data –  realis0c mul0track material – example mixes

– example sekngs

46 Brecht De Man

Open mul0track testbed

•  Server and portal accessible to everyone •  Well-‐structured database of mul0track material, mixes, project files, ground truth (song structure and instrument IDs)

•  Contribu0ons from schools, research ins0tu0ons and individuals [email protected]

47 Brecht De Man

Q&A

[email protected]

brechtdeman.com

c4dm.eecs.qmul.ac.uk

Brecht De Man 48

We Are The Robots: Developing the automatic sound engineer

Documents

Transcript of We Are The Robots: Developing the automatic sound engineer