Collaborative Annotation for Reliable Natural Language Processing: Technical and Sociological Aspects

207 Pages·2016·2.99 MB·English

by  Fort

Collaborative Annotation for Reliable
Natural Language Processing
FOCUS SERIES
Series Editor Patrick Paroubek
Collaborative Annotation
for Reliable Natural
Language Processing

Technical and Sociological Aspects

Karën Fort
First published 2016 in Great Britain and the United States by ISTE Ltd and John Wiley & Sons, Inc.
Apart from any fair dealing for the purposes of research or private study, or criticism or review, as
permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced,
stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers,
or in the case of reprographic reproduction in accordance with the terms and licenses issued by the
CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the
undermentioned address:
ISTE Ltd John Wiley & Sons, Inc.
27-37 St George’s Road 111 River Street
London SW19 4EU Hoboken, NJ 07030
UK USA
www.iste.co.uk www.wiley.com

© ISTE Ltd 2016
The rights of Karën Fort to be identified as the author of this work have been asserted by her in
accordance with the Copyright, Designs and Patents Act 1988.
Library of Congress Control Number: 2016936602

British Library Cataloguing-in-Publication Data
A CIP record for this book is available from the British Library
ISSN 2051-2481 (Print)
ISSN 2051-249X (Online)
ISBN 978-1-84821-904-5
Contents
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
List of Acronyms . . . . . . . . . . . . . . . . . . . . . . . . xi
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Chapter 1. Annotating Collaboratively . . . . . . . . . . 1
1.1.The annotation process (re)visited . . . . . . . . . 1
1.1.1.Building consensus . . . . . . . . . . . . . . . . 1
1.1.2.Existing methodologies . . . . . . . . . . . . . . 3
1.1.3.Preparatory work . . . . . . . . . . . . . . . . . 7
1.1.4.Pre-campaign . . . . . . . . . . . . . . . . . . . . 13
1.1.5.Annotation . . . . . . . . . . . . . . . . . . . . . 17
1.1.6.Finalization . . . . . . . . . . . . . . . . . . . . . 21
1.2.Annotation complexity . . . . . . . . . . . . . . . . 24
1.2.1.Example overview . . . . . . . . . . . . . . . . . 25
1.2.2.What to annotate? . . . . . . . . . . . . . . . . . 28
1.2.3.How to annotate? . . . . . . . . . . . . . . . . . 30
1.2.4.The weight of the context . . . . . . . . . . . . 36
1.2.5.Visualization . . . . . . . . . . . . . . . . . . . . 38
1.2.6.Elementary annotation tasks . . . . . . . . . . 40
1.3.Annotation tools . . . . . . . . . . . . . . . . . . . . 43
1.3.1.To be or not to be an annotation tool . . . . . 43
1.3.2.Much more than prototypes . . . . . . . . . . . 46
vi CollaborativeAnnotationforReliableNaturalLanguageProcessing
1.3.3.Addressing the new annotation challenges . 49
1.3.4.The impossible dream tool . . . . . . . . . . . . 54
1.4.Evaluating the annotation quality . . . . . . . . 55
1.4.1.What is annotation quality? . . . . . . . . . . . 55
1.4.2.Understanding the basics . . . . . . . . . . . . 56
1.4.3.Beyond kappas . . . . . . . . . . . . . . . . . . . 63
1.4.4.Giving meaning to the metrics . . . . . . . . . 67
1.5.Conclusion . . . . . . . . . . . . . . . . . . . . . . . 75
Chapter 2. Crowdsourcing Annotation . . . . . . . . . . 77
2.1.What is crowdsourcing and why should we be
interested in it? . . . . . . . . . . . . . . . . . . . . . . . 77
2.1.1.A moving target . . . . . . . . . . . . . . . . . . 77
2.1.2.A massive success . . . . . . . . . . . . . . . . . 80
2.2.Deconstructing the myths . . . . . . . . . . . . . . 81
2.2.1.Crowdsourcing is a recent phenomenon . . . 81
2.2.2.Crowdsourcing involves a crowd
(of non-experts) . . . . . . . . . . . . . . . . . . . . . . 83
2.2.3.“Crowdsourcing involves (a crowd of)
non-experts” . . . . . . . . . . . . . . . . . . . . . . . . 87
2.3.Playing with a purpose . . . . . . . . . . . . . . . . 93
2.3.1.Using the players’ innate capabilities and
world knowledge . . . . . . . . . . . . . . . . . . . . . 94
2.3.2.Using the players’ school knowledge . . . . . 96
2.3.3.Using the players’ learning capacities . . . . 97
2.4.Acknowledging crowdsourcing specifics. . . . . . 101
2.4.1.Motivating the participants . . . . . . . . . . . 101
2.4.2.Producing quality data . . . . . . . . . . . . . . 107
2.5.Ethical issues . . . . . . . . . . . . . . . . . . . . . . 109
2.5.1.Game ethics. . . . . . . . . . . . . . . . . . . . . 109
2.5.2.What’s wrong with Amazon Mechanical
Turk? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
2.5.3.A charter to rule them all . . . . . . . . . . . . 113
Contents vii
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
Preface
Thisbookpresentsauniqueopportunityformetoconstruct
what I hope to be a consistent image of collaborative manual
annotation for Natural Language Processing (NLP). I partly
rely on work that has already been published elsewhere, with
some of it only in French, most of it in reduced versions and
allofitavailableonmypersonalwebsite.1Wheneverpossible,
the original article should be cited in preference to this book.
Also, I refer to publications in French. I retained these
publications because there was no equivalent one in English,
hoping that at least some readers will be able to understand
them.
This work owes a lot to my interactions with Adeline
Nazarenko (LIPN/University of Paris 13) both during and
after my PhD thesis. In addition, it would not have been
conducted to its end without (a lot of) support and help from
Benoît Habert (ICAR/ENS of Lyon).
Finally,Iwouldliketothankallthefriendswhosupported
me in writing this book and proofread parts of it, as well as
the colleagues who kindly accepted that their figures be part
of it.
1Here:http://karenfort.org/Publications.php.
List of Acronyms
ACE Automatic Content Extraction
ACK Annotation Collection Toolkit
ACL Association for Computational Linguistics
AGTK Annotation Graph Toolkit
API Application Programming Interface
ATALA Association pour le Traitement Automatique des
LAngues (French Computational Linguistics
Society)
HIT Amazon Mechanical Turk Human Intelligence
Task
LDC Linguistic Data Consortium
NLP Natural Language Processing
POS Part-Of-Speech
Introduction
I.1. Natural Language Processing and manual annotation:
Dr Jekyll and Mr Hy|ide?
I.1.1. Where linguistics hides
Natural Language Processing (NLP) has witnessed two
major evolutions in the past 25 years: first, the extraordinary
success of machine learning, which is now, for better or for
worse (for an enlightening analysis of the phenomenon see
[CHU 11]), overwhelmingly dominant in the field, and
second, the multiplication of evaluation campaigns or shared
tasks. Both involve manually annotated corpora, for the
training and evaluation of the systems (see Figure I.1).
These corpora progressively became the hidden pillars of
our domain, providing food for our hungry machine learning
algorithms and reference for evaluation. Annotation is now
the place where linguistics hides in NLP.
However, manual annotation has largely been ignored for
quite a while, and it took some time even for annotation
guidelines to be recognized as essential [NÉD 06]. When the
performance of the systems began to stall, manual
annotation finally started to generate some interest in the
xiv CollaborativeAnnotationforReliableNaturalLanguageProcessing
community, as a potential leverage for improving the
obtained results [HOV 10, PUS 12].
This is all the more important, as it was proven that
systemstrainedonbadlyannotatedcorporaunderperform.In
particular, they tend to reproduce annotation errors when
these errors follow a regular pattern and do not correspond to
simple noise [REI 08]. Furthermore, the quality of manual
annotation is crucial when it is used to evaluate NLP
systems. For example, an inconsistently annotated reference
corpus would undoubtedly favor machine learning systems,
therefore prejudicing rule-based systems in evaluation
campaigns. Finally, the quality of linguistic analyses would
suffer from an annotated corpus that is unreliable.
(cid:1)(cid:2)(cid:2)(cid:3)(cid:4)(cid:1)(cid:4)(cid:5)(cid:3)(cid:2)
(cid:6)(cid:6)(cid:6)(cid:6)(cid:7)(cid:2)(cid:8)(cid:5)(cid:2)(cid:7)
(cid:8)(cid:3)(cid:9)(cid:10)
FigureI.1.Manuallyannotatedcorporaandmachinelearningprocess
Although some efforts have been made lately to address
some of the issues presented by manual annotation, there is
still little research done on the subject. This book aims at
providing some (hopefully useful) insights into the subject. It
is partly based on a PhD thesis [FOR 12a] and on some
published articles, most of them written in French.