Table Of Content

Visual Perception of Music Notation: On-Line and Off-Line Recognition Susan E. George University of South Australia, Australia IDEA GROUP PUBLISHING VVVVViiiiisssssuuuuuaaaaalllll PPPPPeeeeerrrrrccccceeeeeppppptttttiiiiiooooonnnnn ooooofffff MMMMMuuuuusssssiiiiiccccc NNNNNoooootttttaaaaatttttiiiiiooooonnnnn::::: OOOOOnnnnn-----LLLLLiiiiinnnnneeeee aaaaannnnnddddd OOOOOffffffffff-----LLLLLiiiiinnnnneeeee RRRRReeeeecccccooooogggggnnnnniiiiitttttiiiiiooooonnnnn Susan E. George University of South Australia, Australia IRM Press Publisher of innovative scholarly and professional information technology titles in the cyberage Hershey • London (cid:127) Melbourne (cid:127) Singapore Acquisitions Editor: Mehdi Khosrow-Pour Senior Managing Editor: Jan Travers Managing Editor: Amanda Appicello Development Editor: Michele Rossi Copy Editor: Michelle Wilgenburg Typesetter: Amanda Appicello Cover Design: Lisa Tosheff Printed at: Integrated Book Technology Published in the United States of America by IRM Press (an imprint of Idea Group Inc.) 701 E. Chocolate Avenue, Suite 200 Hershey PA 17033-1240 Tel: 717-533-8845 Fax: 717-533-8661 E-mail: [email protected] Web site: http://www.irm-press.com and in the United Kingdom by IRM Press (an imprint of Idea Group Inc.) 3 Henrietta Street Covent Garden London WC2E 8LU Tel: 44 20 7240 0856 Fax: 44 20 7379 3313 Web site: http://www.eurospan.co.uk Copyright © 2005 by IRM Press. All rights reserved. No part of this book may be reproduced in any form or by any means, electronic or mechanical, including photocopying, without written permission from the publisher. Library of Congress Cataloging-in-Publication Data George, Susan Ella. Visual perception of music notation : on-line and off-line recognition / Susan Ella George. p. cm. Includes bibliographical references and index. ISBN 1-931777-94-2 (pbk.) -- ISBN 1-931777-95-0 (ebook) 1. Musical notation--Data processing. 2. Artificial intelligence--Musical applications. I. Title. ML73.G46 2005 780'.1'48028564--dc21 2003008875 ISBN 1-59140-298-0 (h/c) British Cataloguing in Publication Data A Cataloguing in Publication record for this book is available from the British Library. All work contributed to this book is new, previously-unpublished material. The views expressed in this book are those of the authors, but not necessarily of the publisher. New Releases from IRM Press • Visual Perception of Music Notation: On-Line and Off-Line Recognition Susan Ella George ISBN: 1-931777-94-2; eISBN: 1-931777-95-0 / © 2004 (cid:127) 3D Modeling and Animation: Synthesis and Analysis Techniques for the Human Body Nikos Sarris & Michael G. Strintzis ISBN: 1-931777-98-5; eISBN: 1-931777-99-3 / © 2004 (cid:127) Innovations of Knowledge Management Bonnie Montano ISBN: 1-59140-229-8; eISBN: 1-59140-230-1 / © 2004 (cid:127) e-Collaborations and Virtual Organizations Michelle W. L. Fong ISBN: 1-59140-231-X; eISBN: 1-59140-232-8 / © 2004 (cid:127) Information Security and Ethics: Social and Organizational Issues Marian Quigley ISBN: 1-59140-233-6; eISBN: 1-59140-234-4 / © 2004 (cid:127) Issues of Human Computer Interaction Anabela Sarmento ISBN: 1-59140-235-2; eISBN: 1-59140-236-0 / © 2004 (cid:127) Instructional Technologies: Cognitive Aspects of Online Programs Paul Darbyshire ISBN: 1-59140-237-9; eISBN: 1-59140-238-7 / © 2004 (cid:127) E-Commerce and M-Commerce Technologies P. Candace Deans ISBN: 1-59140-239-5; eISBN: 1-59140-240-9 / © 2004 Excellent additions to your institution’s library! Recommend these titles to your Librarian! To receive a copy of the IRM Press catalog, please contact 1/717-533-8845, fax 1/717-533-8661, or visit the IRM Press Online Bookstore at: [http://www.irm-press.com]! Note: All IRM Press books are also available as ebooks on netlibrary.com as well as other ebook sources. Contact Ms. Carrie Skovrinskie at [[email protected]] to receive a complete list of sources where you can obtain ebook information or IRM Press titles. VVVVViiiiisssssuuuuuaaaaalllll PPPPPeeeeerrrrrccccceeeeeppppptttttiiiiiooooonnnnn ooooofffff MMMMMuuuuusssssiiiiiccccc NNNNNoooootttttaaaaatttttiiiiiooooonnnnn::::: OOOOOnnnnn-----LLLLLiiiiinnnnneeeee aaaaannnnnddddd OOOOOffffffffff-----LLLLLiiiiinnnnneeeee RRRRReeeeecccccooooogggggnnnnniiiiitttttiiiiiooooonnnnn TTTTTaaaaabbbbbllllleeeee ooooofffff CCCCCooooonnnnnttttteeeeennnnntttttsssss Preface.......................................................................................vi Susan E. George, University of South Australia, Australia SSSSSeeeeeccccctttttiiiiiooooonnnnn 11111::::: OOOOOffffffffff-----LLLLLiiiiinnnnneeeee MMMMMuuuuusssssiiiiiccccc PPPPPrrrrroooooccccceeeeessssssssssiiiiinnnnnggggg Chapter 1 Staff Detection and Removal .......................................................1 Ichiro Fujinaga, McGill University, Canada Chapter 2 An Off-Line Optical Music Sheet Recognition ...............................40 Pierfrancesco Bellini, University of Florence, Italy Ivan Bruno, University of Florence, Italy Paolo Nesi, University of Florence, Italy Chapter 3 Wavelets for Dealing with Super-Imposed Objects in Recognition of Music Notation.....................................................................................78 Susan E. George, University of South Australia, Australia SSSSSeeeeeccccctttttiiiiiooooonnnnn 22222::::: HHHHHaaaaannnnndddddwwwwwrrrrriiiiitttttttttteeeeennnnn MMMMMuuuuusssssiiiiiccccc RRRRReeeeecccccooooogggggnnnnniiiiitttttiiiiiooooonnnnn Chapter 4 Optical Music Analysis for Printed Music Score and Handwritten Music Manuscript..................................................................................108 Kia Ng, University of Leeds, United Kingdom Chapter 5 Pen-Based Input for On-Line Handwritten Music Notation ............128 Susan E. George, University of South Australia, Australia SSSSSeeeeeccccctttttiiiiiooooonnnnn 33333::::: LLLLLyyyyyrrrrriiiiiccccc RRRRReeeeecccccooooogggggnnnnniiiiitttttiiiiiooooonnnnn Chapter 6 Multilingual Lyric Modeling and Management...............................162 Pierfrancesco Bellini, University of Florence, Italy Ivan Bruno, University of Florence, Italy Paolo Nesi, University of Florence, Italy Chapter 7 Lyric Recognition and Christian Music ..........................................198 Susan E. George, University of South Australia, Australia SSSSSeeeeeccccctttttiiiiiooooonnnnn 44444::::: MMMMMuuuuusssssiiiiiccccc DDDDDeeeeessssscccccrrrrriiiiippppptttttiiiiiooooonnnnn aaaaannnnnddddd iiiiitttttsssss AAAAAppppppppppllllliiiiicccccaaaaatttttiiiiiooooonnnnnsssss Chapter 8 Towards Constructing Emotional Landscapes with Music ................227 Dave Billinge, University of Portsmouth, United Kingdom Tom Addis, University of Portsmouth, United Kingdom and University of Bath, United Kingdom Chapter 9 Modeling Music Notation in the Internet Multimedia Age.............272 Pierfrancesco Bellini, University of Florence, Italy Paolo Nesi, University of Florence, Italy SSSSSeeeeeccccctttttiiiiiooooonnnnn 55555::::: EEEEEvvvvvaaaaallllluuuuuaaaaatttttiiiiiooooonnnnn Chapter 10 Evaluation in the Visual Perception of Music.................................304 Susan E. George, University of South Australia, Australia About the Editor .........................................................................350 About the Authors.......................................................................351 Index..........................................................................................354 vi PPPPPrrrrreeeeefffffaaaaaccccceeeee Overview of Subject Matter and Topic Context The computer recognition of music notation, its interpretation and use within various applications, raises many challenges and questions with regards to the appropriate algorithms, techniques and methods with which to automatically understand music notation. Modern day music notation is one of the most widely recognised international languages of all time. It has developed over many years as requirements of consistency and precision led to the development of both music theory and representation. Graphic forms of notation are first known from the 7th Century, with the modern system for notes developed in Europe during the 14th Century. This volume consolidates the successes, challenges and questions raised by the computer perception of this music notation language. The computer perception of music notation began with the field of Optical Music Recognition (OMR) as researchers tackled the problem of recognising and interpreting the symbols of printed music notation from a scanned image. More recently, interest in automatic perception has extended to all components of song lyric, melody and other symbols, even broadening to multi-lingual handwritten components. With the advent of pen-based input systems, automatic recognition of notation has also extended into the on-line context — moving away from processing static scanned images, to recognising dynamically constructed pen strokes. New applications, including concert- planning systems sensitive to the emotional content of music, have placed new demands upon description, representation and recognition. Summary of Sections and Chapters This special volume consists of both invited chapters and open-solicited chapters written by leading researchers in the field. All papers were peer reviewed by at least two recognised reviewers. This book contains 10 chapters divided into five sections: vii Section 1 is concerned with the processing of music images, or Optical Music Recognition (OMR). A focus is made upon recognising printed typeset music from a scanned image of music score. Section 2 extends the recognition of music notation to handwritten rather than printed typeset music, and also moves into the on-line context with a consideration of dynamic pen-based input. Section 3 focuses upon lyric recognition and the identification and representation of conventional lyric text combined with the symbols of music notation. Section 4 considers the importance of music description languages with emerging applications, including the context of Web-based multi-media and concert planning systems sensitive to the emotional content of music. Finally, Section 5 considers the difficulty of evaluating automatic perceptive systems, discussing the issues and providing some benchmark test data. Section 1: Off-Line Music Processing • Chapter 1: Staff Detection and Removal, Ichiro Fujinaga • Chapter 2: An Off-line Optical Music Sheet Recognition, Pierfrancesco Bellini, Ivan Bruno, Paolo Nesi • Chapter 3: Wavelets for Dealing with Super-Imposed Objects in Recognition of Music Notation, Susan E. George Section 2: Handwritten Music Recognition • Chapter 4: Optical Music Analysis for Printed Music Score and Handwritten Music Manuscript, Kia Ng • Chapter 5: Pen-Based Input for On-Line Handwritten Music Nota- tion, Susan E. George Section 3: Lyric Recognition • Chapter 6: Multilingual Lyric Modeling and Management, Pierfrancesco Bellini, Ivan Bruno, Paolo Nesi • Chapter 7: Lyric Recognition and Christian Music, Susan E. George Section 4: Music Description and its Applications • Chapter 8: Towards Constructing Emotional Landscapes with Music, Dave Billinge, Tom Addis • Chapter 9: Modeling Music Notation in the Internet Multimedia Age, Pierfrancesco Bellini, Paolo Nesi viii Section 5: Evaluation • Chapter 10: Evaluation in the Visual Perception of Music, Susan E. George Description of Each Chapter In Chapter 1, Dr. Ichiro Fujinaga describes the issues involved in the detection and removal of stafflines of musical scores. This removal process is an important step for many optical music recognition systems and facilitates the segmentation and recognition of musical symbols. The process is complicated by the fact that most music symbols are placed on top of stafflines and these lines are often neither straight nor parallel to each other. The challenge here is to remove as much of the stafflines as possible while preserving the shapes of the musical symbols, which are superimposed on stafflines. Various problematic examples are illustrated and a detailed explanation of an algorithm is presented. Image processing techniques used in the algorithm include: run- length coding, connected-component analysis, and projections. In Chapter 2, Professor Pierfrancesco Bellini, Mr. Ivan Bruno and Professor Paolo Nesi compare OMR with OCR and discuss the O3MR system. An overview of the main issues and a survey of the main related works are discussed. The O3MR system (Object Oriented Optical Music Recognition) system is also described. The used approach in such system is based on the adoption of projections for the extraction of basic symbols that constitute graphic elements of the music notation. Algorithms and a set of examples are also included to better focus concepts and adopted solutions. In Chapter 3, Dr. Susan E. George investigates a problem that arises in OMR when notes and other music notation symbols are super-imposed upon stavelines in the music image. A general purpose knowledge-free method of image filtering using two-dimensional wavelets is investigated to separate the super-imposed objects. The filtering provides a unified theory of staveline removal/symbol segmentation, and practically is a useful pre-processing method for OMR. In Chapter 4, Dr. Kia Ng examines a method of recognising printed music — both handwritten and typeset. The chapter presents a brief background of the field, discusses the main obstacles, and presents the processes involved for printed music scores processing; using a divide-and-conquer approach to sub- segment compound musical symbols (e.g., chords) and inter-connected groups (e.g., beamed quavers) into lower-level graphical primitives such as lines and ellipses before recognition and reconstruction. This is followed by discussions on the developments of a handwritten manuscripts prototype with a segmentation approach to separate handwritten musical primitives. Issues and ix approaches for recognition, reconstruction and revalidation using basic music syntax and high-level domain knowledge, and data representation are also presented. In Chapter 5, Dr. Susan E. George concentrates upon the recognition of handwritten music entered in a dynamic editing context with use of pen-based input. The chapter makes a survey of the current scope of on-line (or dynamic) handwritten input of music notation, presenting the outstanding problems in recognition. A solution using the multi-layer perception artificial neural network is presented, explaining experiments in music symbol recognition from a study involving notation writing from some 25 people using a pressure- sensitive digitiser for input. Results suggest that a voting system among networks trained to recognize individual symbols produces the best recognition rate. In Chapter 6, Professor Pierfrancesco Bellini, Mr. Ivan Bruno and Professor Paolo Nesi present an object-oriented language capable of modelling music notation and lyrics. This new model makes it possible to “plug” on the symbolic score different lyrics depending on the language. This is done by keeping separate the music notation model and the lyrics model. An object-oriented model of music notation and lyrics are presented with many examples. These models have been implemented in the music editor produced within the WEDELMUSIC IST project. A specific language has been developed to associate the lyrics with the score. The most important music notation formats are reviewed focusing on their representation of multilingual lyrics. In Chapter 7, Dr. Susan E. George presents a consideration of lyric recognition in OMR in the context of Christian music. Lyrics are obviously found in other music contexts, but they are of primary importance in Christian music — where the words are as integral as the notation. This chapter (i) identifies the inseparability of notation and word in Christian music, (ii) isolates the challenges of lyric recognition in OMR providing some examples of lyric recognition achieved by current OMR software and (iii) considers some solutions outlining page segmentation and character/word recognition approaches, particularly focusing upon the target of recognition, as a high level representation language, that integrates the music with lyrics. In Chapter 8, Dr. Dave Billinge and Professor Tom Addis investigate language to describe the emotional and perceptual content of music in linguistic terms. They aim for a new paradigm in human-computer interaction that they call tropic mediation and describe the origins of the research in a wish to provide a concert planner with an expert system. Some consideration is given to how music might have arisen within human culture and in particular why it presents unique problems of verbal description. An initial investigation into a discrete, stable lexicon of musical effect is summarised and the authors explain how and