Content-Based Music Information Retrieval: Current Directions and Future Challenges

29 Pages·2008·2.97 MB·English

by  Michael Casey

INVITED
PAPER
Content-Based Music
Information Retrieval:
Current Directions and
Future Challenges
Current retrieval systems can handle tens-of-thousands of music tracks
but new systems need to aim at huge online music collections that
contain tens-of-millions of tracks.
ByMichael A. Casey, Member IEEE, Remco Veltkamp, Masataka Goto, Marc Leman,
Christophe Rhodes, and Malcolm Slaney,
Senior Member IEEE
ABSTRACT|ThesteepriseinmusicdownloadingoverCDsales fingerprinting, content-based music retrieval) and other cues
has created a major shift in the music industry away from (e.g., music notation and symbolic representation), and
physical media formats and towards online products and identifiessomeofthemajorchallengesforthecomingyears.
services. Music is one of the most popular types of online
information and there are now hundreds of music streaming KEYWORDS | Audio signal processing; content-based music
and download services operating on the World-Wide Web. informationretrieval;symbolicprocessing;userinterfaces
Some of the music collections available are approaching the
scaleoftenmilliontracksandthishasposedamajorchallenge
I. INTRODUCTION
for searching, retrieving, and organizing music content.
Researcheffortsinmusicinformationretrievalhaveinvolved Music is now so readily accessible in digital form that
experts from music perception, cognition, musicology, engi- personal collections can easily exceed the practical limits
neering,andcomputerscienceengagedintrulyinterdisciplin- on the time we have to listen to them: ten thousand
aryactivitythathasresultedinmanyproposedalgorithmicand music tracks on a personal music device have a total
methodologicalsolutionstomusicsearchusingcontent-based duration of approximately 30 days of continuous audio.
methods. This paper outlines the problems of content-based Distribution of new music recordings has become easier,
music information retrieval and explores the state-of-the-art prompting a huge increase in the amount of new music
methods using audio cues (e.g., query by humming, audio thatisavailable.In2005,therewasathree-foldgrowthin
legal music downloads and mobile phone ring tones,
worth$1.1billionworldwide,offsettingtheglobaldecline
in CD sales; and in 2007, music downloads in the U.K.
ManuscriptreceivedSeptember3,2007;revisedDecember5,2007.Thiswork
wassupportedinpartbytheU.K.EngineeringandPhysicalSciencesResearch reached new highs [1]–[3].
CouncilunderGrantsEPSRCEP/E02274X/1andEPSRCGR/S84750/01. Traditionalwaysoflisteningtomusic,andmethodsfor
M.A.CaseyandC.RhodesarewiththeDepartmentofComputing,Goldsmiths
College,UniversityofLondon,SE146NWLondon,U.K.(e-mail:[email protected]; discovering music, such as radio broadcasts and record
[email protected]). stores,arebeingreplacedbypersonalizedwaystohearand
R.VeltkampiswiththeDepartmentofInformationandComputingSciences,Utrecht
University,3508TBUtrecht,TheNetherlands(e-mail:[email protected]). learn about music. For example, the advent of social net-
M.GotoiswiththeNationalInstituteofAdvancedIndustrialScienceandTechnology working Web sites, such as those reported in [4] and [5],
(AIST),Ibaraki305-8568,Japan(e-mail:[email protected]).
M.LemaniswiththeDepartmentofArt,Music,andTheaterSciences, has prompted a rapid uptake of new channels of music
GhentUniversity,9000Ghent,Belgium(e-mail:[email protected]). discoveryamongonlinecommunities,changingthenature
M.SlaneyiswithYahoo!ResearchInc.,SantaClara,CA95054USA
(e-mail:[email protected]). ofmusicdisseminationandforcingthemajorrecordlabels
DigitalObjectIdentifier:10.1109/JPROC.2008.916370 to rethink their strategies.
668 Proceedings of the IEEE|Vol.96,No.4,April2008 0018-9219/$25.00 (cid:1)2008 IEEE
Casey etal.:Content-Based Music Information Retrieval:Current Directions and Future Challenges
It is not only music consumers who have expectations approximately 50 person-years to enter the metadata for
of searchable music collections; along with the rise of one million tracks.
consumeractivityindigitalmusic,therearenewopportu- Social media web services address the limitations of
nities for research into using large music collections for centralized metadata by opening the task of describing
discovering trends and patterns in music. Systems for contenttopubliccommunitiesofusersandleveragingthe
trend spotting in online music sales are in commercial power of groups to exchange information about content.
development [6], as are systems to support musicological ThisisthehallmarkofWeb2.0.Withmillionsofusersof
research into music evolution over the corpus of Western portals such as MySpace, Flickr, and YouTube, group be-
classicalmusicscoresandavailableclassicalrecordings[7], havior means that users naturally gravitate towards those
[8].Musicologyresearchaimstoanswerquestionssuchas: partsoftheportalVcategoriesorgroupsVwithwhichthey
which musical works and performances have been shareanaffinity;sotheyarelikelytofinditemsofinterest
historically the most influential? indexedbyuserswithsimilartastes.However,theactivity
Strategiesforenablingaccesstomusiccollections,both on social networking portals is not uniform across the
newandhistorical,needtobedevelopedinordertokeep interests of society and culture at large, being predomi-
up with expectations of search and browse functionality. nantly occupied by technologically sophisticated users,
These strategies are collectively called music information therefore social media is essentially a type of editorial
retrieval (MIR) and have been the subject of intensive metadata process.
researchbyanever-increasingcommunityofacademicand In addition to metadata-based systems, information
industrial research laboratories, archives, and libraries. about the content of music canbe used tohelp users find
There are three main audiences that are identified as the music. Content-based music description identifies what
beneficiaries of MIR: industry bodies engaged in record- the user is seeking even when he does not know speci-
ing, aggregating and disseminating music; end users who fically what he is looking for. For example, the Shazam
want to find music and use it in a personalized way; and system (shazam.com), described in [10], can identify a
professionals: music performers, teachers, musicologists, particular recording from a sample taken on a mobile
copyright lawyers, and music producers. phone in a dance club or crowded bar and deliver the
At present, the most common method of accessing artist,album,andtracktitlealongwithnearbylocationsto
music is through textual metadata. Metadata can be rich purchase the recording or a link for direct online pur-
and expressive so there are many scenarios where this chasing and downloading. Users with a melody but no
approachissufficient.Mostmusicdownloadservicescur- other information can turn to the online music service
rently use metadata-only approaches and have reached a Nayio (nayio.com) which allows one to sing a query and
degree of success with them. However, when catalogues attempts to identify the work.
become very large (greater than a hundred thousand In the recording industry, companies have used sys-
tracks) it is extremely difficult to maintain consistent tems based on symbolic information about musical con-
expressive metadata descriptions because many people tent,suchasmelody,chords,rhythm,andlyrics,toanalyze
created the descriptions and variation in concept encod- thepotentialimpactofaworkinthemarketplace.Services
ings impacts search performance. Furthermore, the such as Polyphonic HMI’s Hit Song Science and Platinum
descriptions represent opinions, so editorial supervision BlueMusicIntelligenceusesuchsymbolicinformationwith
of the metadata is paramount [9]. techniques from Artificial Intelligence to make consulta-
An example of a commercial metadata-driven music tive recommendations about new releases.
systemispandora.comwheretheuserispresentedwiththe Because there are so many different aspects to music
instructionBtypeinthenameofyourfavoriteartistorsong information retrieval and different uses for it, we cannot
and we’ll create a radio station featuring that music and addressallofthemhere.Thispaperaddressessomeofthe
more like it.[The system usesmetadata to estimate artist recent developments in content-based analysis and re-
similarity and track similarity; then, it retrieves tracksthat trievalofmusic,payingparticularattentiontothemethods
theusermightwanttohearinapersonalizedradiostation. by which important information about music signals and
Whereasthequerywassimpletopose,findingtheanswer symbols can be automatically extracted and processed for
iscostly.Thesystemworksusingdetailedhuman-entered usewithmusicinformationretrievalsystems.Weconsider
track-level metadata enumerating musical-cultural prop- bothaudiorecordingsandmusicalscores,asitisbeneficial
erties for each of several hundred thousand tracks. It is tolookatboth,whentheyareavailable,tofindcluesabout
estimated that it takes about 20–30 minutes per track of what a user is looking for.
one expert’s time to enter the metadata.1 The cost is The structure of this paper is as follows: Section II
thereforeenormousinthetimetakentoprepareadatabase introducesthetypesoftasks,methods, and approachesto
to contain all the information necessary to perform evaluation of content-based MIR systems; Section III
similarity-based search. In this case, it would take presents methods for high-level audio music analysis;
Section IV discusses audio similarity-based retrieval;
1http://www.pandora.com/corporate. symbolic music analysis and retrieval are presented in
Vol.96,No.4,April2008|Proceedings of the IEEE 669
Casey etal.:Content-Based Music Information Retrieval:Current Directions and Future Challenges
of music, such as genre, have low specificity: that is, a
search given a query track will return tracks having little
contentdirectlyincommonwiththequery,butwithsome
global characteristics that match. Hence, we divide
specificity into three broad categories: high-specificity
systems match instances of audio signal content; mid-
specificity systems match high-level music features, such
as melody, but do not match audio content; and low-
specificity systems match global (statistical) properties of
thequery.Table1enumeratessomeofthe MIRusecases
and their specificities (high, mid, or low). A more
comprehensive list of tasks and their specificities is given
in [11].
There are three basic strategies for solving MIR use
cases.Eachstrategyissuitedtoagivenspecificity.Thefirst
is based on conceptual metadata, information that is
encoded and searched like text and is suited to low-
specificity queries; the second approach uses high-level
descriptionsofmusiccontentcorrespondingwithintuitive
or expert knowledge about how a piece of music is
constructed. This approach is suited to mid specificity
Fig.1.Flowchartofcanonicalcontent-basedquerysystem. queries. The third strategy is based on low-level signal-
based properties which are used for all specificities. We
outline each of these three approaches in the following
sections.
SectionV;SectionVIpresentsanoverviewofadvancesin
music visualization and browsing; Section VII discusses
systems for music information retrieval; and we conclude Table1ExamplesofMIRTasksandTheirSpecificities
in Section VIII with a discussion of challenges and future
directions for the field.
II. USES CASES AND APPROACHES
A. Use Cases: Specificities and Query Types
Content-basedMIRisengagedinintelligent,automat-
ed processing of music. The goal is to make music, or
information about music, easier to find. To support this
goal, most MIR research has been focused on automatic
musicdescriptionandevaluationoftheproposedmethods.
Thefieldisorganizedaroundusecaseswhichdefineatype
of query, the sense of match, and the form of the output.
Queriesandoutputcanbetextualinformation(metadata),
music fragments, recordings, scores, or music features.
The match can be exact, retrieving music with specific
content, or approximate, retrieving near neighbors in a
musicalspacewhereproximityencodesmusicalsimilarity,
for example.
The main components of an MIR system are detailed
in Fig. 1. These are query formation, description extrac-
tion,matchingand,finally,musicdocumentretrieval.The
scope of an MIR system can be situated on a scale of
specificityforwhichthequerytypeandchoiceofexactor
approximatematchingdefinethecharacteristicspecificity
ofthesystem.Thosesystemsthatidentifyexactcontentof
individual recordings, for example, are called high-
specificity systems; those employing broad descriptions
670 Proceedings of the IEEE|Vol.96,No.4,April2008
Casey etal.:Content-Based Music Information Retrieval:Current Directions and Future Challenges
B. Approaches systemscurrentlyrelyheavilyonmetadatabutarenotable
to easily provide their users with search capabilities for
1) Metadata: MetadataisthedriverofMIRsystems.As finding music they do not already know about, or do not
such, many services exist simply to provide reliable know how to search for. This gap is one of the oppor-
metadata for existing collections of music, either for end tunities for content-based methods, which hold the pro-
users or for large commercial music collections. Most miseofbeingabletocomplementmetadata-basedmethods
musiclistenersusemetadataintheirhomelisteningenvi- and give users access to new music via processes of self-
ronments. A common MIR task is to seek metadata from directed discovery and musical search that scales to the
theinternetfordigitalmusicthatwasBripped[fromcom- totalityofavailablemusictracks.Fortheremainderofthis
pact disc to a computer. This is the core functionality of paper, we focus primarily on content-based music
automaticWeb-basedservicessuchasGracenote(gracenote. description rather than factual or culturally determined
com) and MusicBrainz (musicbrainz.org)Vboth of these parameters. However, content-based methods are consid-
metadata repositories rely on user-contributed content to ered not replacements but enhancements for metadata-
scaletrack-levelmusicdescriptiontomillionsofentries. based methods.
These services provide both factual metadata, namely
objectivetruthsaboutatrack,andculturalmetadata,which 2) High-Level Music Content Description: An intuitive
contains subjective concepts. For a metadata system to starting point for content-based music information
work its descriptions of music must be accurate and the retrieval is to use musical concepts such as melody or
meaning of the metadata vocabulary widely understood. harmonytodescribethecontentofthemusic.Intheearly
Web2.0providesapartialsolutioninthatcommunitiesof days of MIR, many query-by-humming systems were
users can vote on a track’s metadata. This democratic proposed that sought to extract melodic content from
process at least ensures that the metadata for a track is polyphonic audio signals (those with multiple simulta-
consistent with the usage of one, or more, communities. neousmusicallines)sothatausercouldsearchformusic
Problems associated with factual information, artist, by singing or humming part of the melody; such systems
album, year of publication, track title, and duration, can are now being deployed as commercial services; see, for
severely limit the utility of metadata. Ensuring the example, naiyo.com. A survey of sung-query methods was
generality of the associated text fields, for example, conducted by Hu and Dannenberg in [13].
consistencies of spelling, capitalization, international High-levelintuitiveinformationaboutmusicembodies
characters, special characters and order of proper names, thetypesofknowledgethatasophisticatedlistenerwould
is essential to useful functioning [9]. haveaboutapieceofmusic,whetherornottheyknowthey
In addition to factual metadata, subjective, culturally have that knowledge:
determined information at the level of the whole track is
often used to retrieve tracks. Common classes of such BItismelodythatenablesustodistinguishonework
metadataaremood,emotion,genre,style,andsoforth.Most from another. It is melody that human beings are
current music services use a combination of factual and innatelyabletoreproducebysinging,humming,and
cultural metadata. There has also been much interest in whistling.Itismelodythatmakesmusicmemorable:
automatic methods for assigning cultural, and factual, we are likely to recall a tune long after we have
metadata to music. Some services collect user preference forgotten its text.[ [14]
data, such as the number of times particular tracks have
been played, and use the information to make new music Even though it is an intuitive approach, melody
recommendations to users based on the user community. extraction from polyphonic recordings, i.e., multiple
For example, Whitman and Rifkin [12] used music instruments playing different lines simultaneously, re-
descriptions generated from community metadata; they mainsextremelydifficulttoachieve.Surprisingly,itisnot
achieved Internet-wide description by using data mining only difficult to extract melody from audio but also from
andinformationretrievaltechniques.Theirextracteddata symbolic representations such as MIDI files. The same is
was time awareVreflecting changes both in the artists’ true of many other high-level music concepts such as
styleandinthepublic’sperceptionoftheartists.Thedata rhythm, timbre, and harmony. Therefore, extraction of
wascollectedweekly,andlanguageanalysiswasperformed high-level music content descriptions is a subgoal of MIR
to associate noun and verb phrases with musical features and the subject of intensive research. Common high-level
extracted from audio of each artist. descriptorsareidentifiedinTable2.Thegoalofsuchtasks
The textual approach to MIR is a very promising new is to encode music into a schema that conforms to
direction in the field: a comprehensive review of the traditional Westernmusicconceptsthat can thenbe used
methods and results of such research is beyond the scope to make queries and search music.
of the current paper. Automaticextractionoffactual,cultural,andhigh-level
Forallitsutility,metadatacannotsolvetheentiretyof musicdescriptionshavebeenasubjectofintensestudyin
MIR due to the complexities outlined above. Commercial the MIREX music information retrieval experimental
Vol.96,No.4,April2008|Proceedings of the IEEE 671
Casey etal.:Content-Based Music Information Retrieval:Current Directions and Future Challenges
Table2High-LevelMusicFeatures(HardtoExtract) audio matching methods rather than extracting a high-
levelmelodyfeaturefromaudio[18].Theauthorssuggest
that low-level audio methods outperform symbolic meth-
ods even when clean symbolic information is available as
in this task.
Because there is a great number of music recordings
available that can be used as a first stage input to a high-
level music description system, this motivates work on
extracting high-level music features from low-level audio
content. The MIREX community extends the range of
tasks that are evaluated each year, allowing for valuable
knowledgetobegainedonthelimitsofcurrentalgorithms
and techniques.
3) Low-Level Audio Features: The third strategy for
content-based music description isto use the information
inthedigitalaudio.Low-levelaudiofeaturesaremeasure-
ments of audio signals that contain information about a
musical work and music performance. They also contain
extraneous information due to the difficulty of precisely
measuring just a single aspect of music, so there is a
tradeoffbetweenthesignal-leveldescriptionandthehigh-
level music concept that is encoded.
In general, low-level audio features are segmented in
threedifferentways:framebasedsegmentations(periodic
sampling at 10 ms-1000 ms intervals), beat-synchronous
exchange. MIREX provides a framework for formal segmentations (features are aligned to musical beat
evaluation of MIR systems using centralized tasks, boundaries), and statistical measures that construct
datasets, platforms, and evaluation methods [15], [16]. probability distributions out of features (bag of features
Assuch,MIREXhasbecomeaveryimportantindicatorof models). Many low-level audio features are based on the
the state of the art for many subtasks within the field of short-time spectrum of the audio signal. Fig. 2 illustrates
MIR. A summary of the results of the 2007 high-level howsomeofthemostwidelyusedlow-levelaudiofeatures
musictasksisgiveninTable3;thequery-by-hummingtask are extracted from a digital audio music signal using a
has a particularly high score. It is interesting to note that windowed fast Fourier transform (FFT) as the spectral
the best-performing system on this task used low-level extractionstep.Bothframe-basedandbeat-basedwindows
are evident in the figure.
a) Short-Time Magnitude Spectrum: Many low-level
Table3SummaryofResultsofBest-PerformingClassificationand
audio features use the magnitude spectrum as a first step
RecognitionSystemsinMIREX2007Exchange
forfeatureextractionbecausethephaseofthespectrumis
not as perceptually salient for music as the magnitude.
Thisisgenerallytrueexceptinthedetectionofonsetsand
in phase continuation for sinusoidal components.
b)Constant-Q/MelSpectrum:Theear’sresponsetoan
acoustic signal is logarithmic in frequency and uses
nonuniform frequency bands, known as critical bands, to
resolve close frequencies into a single band of a given
center frequency. Many systems represent the constant
bandwidth critical bands using a constant-Q transform,
where the Q is the ratio of bandwidth to frequency [17],
[19].Itistypicaltousesomedivisionofthemusicaloctave
for the frequency bands, such as a twelfth, corresponding
toonesemitoneinWesternmusic,butitisalsocommonto
use more perceptually motivated frequency spacing for
bandcenters.Fig.3showsthealignmentbetweenasetof
linearly spaced frequency band edges and the corre-
spondinglogarithmicallyspacedtwelfth-octavebands.The
672 Proceedings of the IEEE|Vol.96,No.4,April2008
Casey etal.:Content-Based Music Information Retrieval:Current Directions and Future Challenges
Fig.2.Schematicofcommonaudiolow-levelfeatureextractionprocesses.Fromleft-to-right:log-frequencychromagram,
Mel-frequencycepstralcoefficients,linear-frequencychromagram,andbeattracking.Insomecases,thebeattrackingprocessis
usedtomakethefeaturesbeatsynchronous,otherwisesegmentationusesfixed-lengthwindows.
Melfrequencyscalehaslinearlyspacedfiltersinthelower such as 24, 36, or 48 bands. Tuning systems that use
frequency range and logarithmically spaced filters above equally spaced pitch classes are called equal temperament.
1300 Hz. Both logarithmic and Mel frequency scales are Recently, some studies have explored extracting features
used. The Mel or Constant-Q spectrum can be obtained for tuning systems that do not use equally spaced pitch
from a linear spectrum by summing the powers in adja- classes: a necessary extension for application to non-
centfrequencybands.Thisapproachhastheadvantageof Western music.
being able to employ the efficient FFT to compute the d) Onset Detection: Musicaleventsaredelineatedby
spectrum. onsets;anotehasanattackfollowedbysustainanddecay
c) Pitch-Class Profile (Chromagram): Another com- portions. Notes that occur simultaneously in music are
mon type of frequency folding is used to represent the often actually scattered in time and the percept is inte-
energy due to each pitch class in twelfth-octave bands grated by the ear-brain system. Onset detection is con-
called a pitch-class profile (PCP) [20]–[22]. This feature cernedwithmarkingjustthebeginningsofnotes.Thereare
integratestheenergyinalloctavesofonepitchclassintoa several approaches to onset detection, employing spectral
single band. There are 12 equally spaced pitch classes in differences in the magnitude spectrum of adjacent time
Western tonal music, independent of pitch height, so points, or phase differences in adjacent time points, or
there are typically 12 bands in a chromagram represen- some combination of the two (complex number onset
tation. Sometimes, for finer resolution of pitch informa- detection) [23]–[25]. Onset detection is one of the tasks
tion, the octave is divided into an integer multiple of 12 studiedintheMIREXframeworkofMIRevaluationshown
inTable3.
e) Mel/Log-Frequency Cepstral Coefficients: Mel-
frequency cepstral coefficients (MFCC) take the loga-
rithmoftheMelmagnitudespectrumanddecorrelatethe
resulting values using a Discrete Cosine Transform. This
is a real-valued implementation of the complex cepstrum
in signal processing [19]. The effect of MFCCs is to
organize sinusoidal modulation of spectral magnitudes by
Fig.3.Foldingofasetoflinearlyspacedfrequencybands increasing modulation frequency in a real-valued array.
(lowergraph)ontoasetoflogarithmicallyspacedfrequencybands
Values at the start of the array correspond to long wave
(uppergraph).Thex-axisshowsfrequencyofbands,upperlines
spectralmodulationandthereforerepresenttheprojection
arelabeledbytheirWesternmusicnotationpitchclassandlowerlines
bytheirFFTbinnumber(for16384binswith44.1kHzsamplerate). of the log magnitude spectrum onto a basis of formant
Vol.96,No.4,April2008|Proceedings of the IEEE 673
Casey etal.:Content-Based Music Information Retrieval:Current Directions and Future Challenges
peaks. Values at the end of the MFCC array are the features provides a measurement for every beat interval,
projectioncoefficientsofthelogmagnitudespectrumonto ratherthanattheframelevel,sothelow-levelfeaturesare
short wavelength spectral modulation, corresponding to segmented by musically salient content. This has recently
harmonic components in the spectrum. It is usual to use proven to be exceptionally useful for mid-specificity MIR
about 20 MFCC coefficients, therefore representing the taskssuchascoversongsandversionsidentification.Low-
formant peaksofaspectrum;thisextraction corresponds, level audio features in themselves cannot tell us much
inpart,tomusicaltimbreVthewaytheaudiosoundsother about music; they encode information at too fine a tem-
than its pitch and rhythm. poralscaletorepresentperceptuallysalientinformation.It
f)SpectralFlux:The spectral flux of a musical signal isusualinMIRresearchtocollectaudioframesintooneof
estimates the fine spectral-temporal structure in different several aggregate representations. Table 4 describes this
frequency bands by measuring the modulation ampli- second-stage processing of low-level audio features which
tudes in mid-to-high spectral bands [26], [27]. The encodes more information than individual audio frames.
resulting feature is a two-dimensional matrix, with fre- An aggregate feature is ready for similarity measurement
quency bands in the rows and modulation frequency in whereasindividuallow-levelaudiofeatureframesarenot.
columns, representing the rate of change of power in The chosen time scale for aggregate features depends on
each spectral band. the specificity and temporal acuity of the task.
g) Decibel Scale (Log Power): The decibel scale is These low-level audio features and their aggregate
employed for representing power in spectral bands be- representations are used as the first stage in bottom-up
cause the scale closely represents the ear’s response. The processing strategies. The task is often to obtain a high-
decibel scale is calculated as ten times the base-10 loga- level representation of music as a next step in the
rithm of power. processing of music content. The following section gives
h) Tempo/Beat/Meter Tracking: As shown in Fig. 2, a summary of some of the approaches to bridging the
beatextractionfollowsfromonsetdetection,anditisoften gap between low-level and high-level music tasks such
used to align the other low-level features. Alignment of as these.
Fig.4.Beattrackingamusicfilewithbeatroottool.Lowerportionofgraphshowsaudiopowerandupperportionshowsspectrogram.
Darkverticallinesarebeatpositions.Numbersrunningalongtopofgraphareinter-beatintervalsinmilliseconds.
674 Proceedings of the IEEE|Vol.96,No.4,April2008
Casey etal.:Content-Based Music Information Retrieval:Current Directions and Future Challenges
Fig.5.Predominantf0trajectoryofCharlieHadenjazztrackshowninSonicVisualiser.Darkerregionsaremaximaofthistimeversus
logarithmicfrequencyspectrogram.Verticalaxisislaidoutasapianokeyboardformusicalpitchreference.
III. AUDIO ANALYSIS (in time), and vertically (in frequency). Furthermore, the
information in music is constructed with hierarchical
IncontrasttospeechrecognitionandtextIRsystems,most
schemas. Systems for analyzing and searching music
musichasseveralstreamsofrelatedinformationoccurring
content must seek to represent many of these viewpoints
in parallel. As such, music is organized both horizontally
simultaneously to be effective.
As discussed in Section II-B, one of the intermediate
Table4FrameAggregationMethodsforLow-LevelAudioFeatures
goals of MIR is to extract high-level music content
descriptionsfromlow-levelaudioprocesses.Thefollowing
sections describe research into extracting such high-level
descriptions with a view to transforming musical audio
contentintorepresentationsthatareintuitiveforhumans
to manipulate and search. We begin the discussion with
the related high-level music description problems of beat
tracking, tempo estimation, and meter tracking.
A. Beat Tracking
Automatic estimation of the temporal structure of
music,suchasmusicalbeat,tempo,rhythm,andmeter,is
notonlyessentialforthecomputationalmodelingofmusic
understanding but also useful for MIR. Temporal pro-
perties estimated from a musical piece can be used for
content-based querying and retrieval, automatic classi-
fication,musicrecommendation,andplaylistgeneration.If
thetempoofamusicalpiececanbeestimated,forexample,
Vol.96,No.4,April2008|Proceedings of the IEEE 675
Casey etal.:Content-Based Music Information Retrieval:Current Directions and Future Challenges
it is easy to find musical pieces having a similar tempo possible period, to the time-varying degree of musical
without using any metadata. Once the musical beats are accentuation[31].
estimated, we can use them as the temporal unit for Foraudio-basedbeattracking,itisessentialtosplitthe
high-level beat-based computation instead of low-level full frequency band of the input audio signal into several
frame-based computation. This facilitates the estimation frequency subbands and calculate periodicities in each
of other musical descriptions, such as the music struc- subband. Goto [30] proposed a method where the beat-
ture and chorus sections [22]. Since the estimated beats period analysis is first performed within seven logarith-
can also be used for normalizing the temporal axis of micallyequallyspacedsubbandsandthoseresultsarethen
musical pieces, the beat-based time alignment facilitates combined across the subbands by using a weighted sum.
time-scale invariant music identification or cover song Scheirer[31]alsousedtheideaofthissubband-basedbeat
identification [28], [29]. tracking and applied aset ofcomb-filterresonatorstothe
Here, we define beat tracking (including measure or degreesofmusicalaccentuationofsixsubbandstofindthe
bar-line estimation) as the process of organizing musical most resonant period. To locate periodicity in subband
audio signals into a hierarchical beat structure [30]. The signals, Sethares and Staley [32] used a periodicity
typical beat structure comprises the quarter-note level transform instead of using comb-filter resonators.
(the tactus level represented as almost regularly spaced After estimating the beat period, the phase should be
beat times) and the measure level (bar lines). The basic estimated. When onset times are used to estimate the
natureoftrackingthequarter-notelevelisrepresentedby period, a windowed cross-correlation function is applied
two parameters. betweenanonset-timesequenceandatentativebeat-time
Period: The period is the temporal difference between sequence whose interval is the estimated period. The
the times of two successive beats. The tempo (beats per result can be used to predict the next beat in a real-time
minute) is inversely proportional to the period. beat-trackingsystem.Ontheotherhand,whenthedegrees
Phase: The phase corresponds to actual beat positions ofmusicalaccentuationare used,the internal stateofthe
and equals zero at beat times. delaysofcomb-filterresonatorsthathavelatticesofdelay-
Ontheotherhand,themeasurelevelisdefinedonbeat and-hold stages can be used to determine the phase [31].
timesbecausethebeatstructureishierarchical:thebegin- Toestimatetheperiodandphasesimultaneously,thereare
nings of measures (bar-line positions) coincide with beat other approaches using adaptive oscillators [33].
times. The difficulty of beat tracking depends on how
explicitly the beat structure is expressed in the target 2) Dealing With Ambiguity: The intrinsic reason that
music: it depends on temporal properties such as tempo beattrackingisdifficultisduetotheproblemofinferring
changes and deviations, rhythmic complexity, and the an original beat structure that is not expressed explicitly.
presence of drum sounds. This causes various ambiguous situations, such as those
where different periods seem plausible and where several
1)TrackingMusicalBeats(Quarter-NoteLevel):Thebasic onset times obtained by frequency analysis may corre-
approachofestimatingtheperiodandphaseofthequarter- spond to a beat.
note(tactus)levelistodetectonsettimesandusethemas There are variations in how ambiguous situations in
cues. Many methods assume that a frequently occurring determiningthebeatstructurearemanaged.Atraditional
inter-onset interval (IOI), the temporal difference be- approachistomaintainmultiplehypotheses,eachhavinga
tween two onset times, is likely to be the beat period and different possible set of period and phase. A beam search
that onset times tend to coincide with beat times (i.e., technique or multiple-agent architectures [30], [34] have
sounds are likely to occur on beats). beenproposed tomaintain hypotheses. Amore advanced,
To estimate the beat period, a simple technique is to computationally intensive approach for examining multi-
calculatethehistogramofIOIsbetweentwoadjacentonset ple hypotheses is to use probabilistic generative models
timesorclustertheIOIsandpickoutthemaximumpeakor and estimate their parameters. Probabilistic approaches
thetoprankedclusterwithinanappropriatetemporange. with maximum likelihood estimation, MAP estimation,
This does not necessarily correspond to the beat period, and Bayes estimation could maintain distributions of all
though. A more sophisticated technique is to calculate a parameters, such as the beat period and phase, and find
windowed autocorrelation function of an onset-time se- the best hypothesis as if all possible pairs of the period
quence,powerenvelope,orspectralfluxoftheinputsignal, and phase were evaluated simultaneously. For example,
or continuous onset representation with peaks at onset Hainsworth and Macleod [35] explored the use of par-
positions, and pick out peaks in the result. This can be ticle filters for audio-based beat tracking on the basis of
considered an extended version of the IOI histogram be- MIDI-based methods by Cemgil and Kappen [36]. They
cause it naturally takes into account various temporal dif- made use of Markov chain Monte Carlo (MCMC) algo-
ferences such as those between adjacent, alternate, and rithms and sequential Monte Carlo algorithms (particle
everythirdonsettimes.Anothersophisticatedtechniqueis filters) to estimate model parameters, such as the period
to apply a set of comb-filter resonators, each tuned to a and phase.
676 Proceedings of the IEEE|Vol.96,No.4,April2008
Casey etal.:Content-Based Music Information Retrieval:Current Directions and Future Challenges
3) Estimating Beginnings of Measures (Measure Level): becausestereosignalscanbeeasilyconvertedtomonaural
Higher level processing using musical knowledge is ne- signals. While a method depending on stereo information
cessarytodeterminethemeasurelevelofthehierarchical cannotbeappliedtomonauralsignals,amethodassuming
beat structure. Musical knowledge is also useful for monaural signals can be applied to stereo signals and can
selecting the best hypothesis in the above ambiguous be considered essential to music understanding, since
situations. human listeners have no difficulty understanding melody
Variouskindsofmusicalknowledgehavebeenstudied. and bass lines even from monaural signals.
Goto [30] used musical knowledge concerning chord Here, melody and bass lines are represented as a con-
changesanddrumpatternswithafocusonpopularmusic. tinuous temporal-trajectory representation of fundamen-
By estimating the chord changes using signal processing tal frequency (F0, perceived as pitch) or a series of
technique,forexample,thebeginningsofmeasurescanbe musical notes. It is difficult to estimate the F0 of melody
determined so that chords are more likely to change on and bass lines in monaural polyphonic sound mixtures
thosepositions.Ontheotherhand,Klapurietal.[38]used containing simultaneous sounds of various instruments,
musical knowledge concerning temporal relationship because in the time-frequency domain the frequency
among different levels of the hierarchical beat structure components of one sound often overlap the frequency
and encoded this prior knowledge in HMMs that could componentsofsimultaneoussounds.Evenstate-of-the-art
jointlyestimateperiodsatdifferenthierarchicallevelsand technologies cannot fully separate sound sources and
then separately estimate their phases. transcribe musical scores from complex polyphonic
mixtures. Most melody and bass estimation methods
4) Conclusion: The topic of beat tracking still attracts therefore do not rely on separated sounds or transcribed
many researchers because it includes fundamental issues scores but directly estimate the target melody and bass
inunderstandingtemporalaspectsofmusic,contributesto lines from music that has distinct melody and bass lines,
a number of practical applications, and it is difficult to such as popular songs.
achieve perfect beat tracking for various kinds of music.
Therefore, new approaches are proposed every year, 1) Estimating Melody and Bass Lines by Finding the
including holistic beat tracking [39] where information Predominant F0 Trajectory: Since the melody line tends to
aboutmusicstructureestimatedbeforebeattrackinghelps havethemostpredominantharmonicstructureinmiddle-
totrackbeatsbyaddingaconstraintthatsimilarsegments andhigh-frequencyregionsandthebasslinetendstohave
of music should have corresponding beat structure. the most predominant harmonic structure in a low-
frequencyregion,thefirstclassicideaofestimatingmelody
B. Melody and Bass Estimation andbasslinesistofindthemostpredominantF0insound
Automatic estimation of melody and bass lines is mixtures with appropriate frequency-range limitation. In
important because the melody forms the core of Western 1999, Goto [43], [44] proposed a real-time method called
musicandisastrongindicatorfortheidentityofamusical PreFEst (Predominant-F0 Estimation method) which
piece,seeSectionII-B,whilethebassiscloselyrelatedto detects the melody and bass lines in monaural sound
the harmony. These lines are fundamental to the percep- mixtures. Unlike most previous F0 estimation methods,
tionofmusicandusefulinMIRapplications.Forexample, PreFEstdoesnotassumethenumberofsoundsources.
theestimatedmelody(vocal)linefacilitatessongretrieval PreFEst basically estimates the F0 of the most predo-
based on similar singing voice timbres [41], music minant harmonic structureVthe most predominant F0
retrieval/classification based on melodic similarities, and corresponding to the melody or bass lineVwithin an in-
music indexing for query by humming which enables a tentionally limited frequency range of the input sound
usertoretrieveamusicalpiecebyhummingorsingingits mixture. It simultaneously takes into consideration all
melody. Moreover, for songs with vocal melody, once the possibilitiesfortheF0andtreatstheinputmixtureasifit
singingvoiceisextractedfrompolyphonicsoundmixtures, contains all possible harmonic structures with different
the lyrics can be automatically synchronized with the weights (amplitudes). It regards a probability density
singing voice by using a speech alignment technique and function (PDF) of the input frequency components as a
can be displayed with the phrase currently being sung weighted mixture of harmonic-structure tone models
highlighted during song playback, like the Karaoke (represented by PDFs) of all possible F0s and simulta-
display [42]. neouslyestimatesboththeirweightscorrespondingtothe
The difficulty of estimating melody and bass lines relative dominance of every possible harmonic structure
depends on the number of channels: the estimation for andtheshapeofthetonemodelsbymaximumaposteriori
stereo audio signals is easier than the estimation for probability (MAP) estimation considering their prior
monaural audio signals because the sounds of those lines distribution.Itthenconsidersthemaximum-weightmodel
tend to be panned to the center of stereo recordings and as the most predominant harmonic structure and obtains
the localization information can help the estimation. In its F0. The method also considers the F0’s temporal
general, most methods deal with monaural audio signals continuity by using a multiple-agent architecture.
Vol.96,No.4,April2008|Proceedings of the IEEE 677