Arabic text to Arabic sign language example-based translation system

181 Pages·2016·26.6 MB·English

University of Southampton Research Repository
ePrints Soton
Copyright © and Moral Rights for this thesis are retained by the author and/or other
copyright owners. A copy can be downloaded for personal non-commercial
research or study, without prior permission or charge. This thesis cannot be
reproduced or quoted extensively from without first obtaining permission in writing
from the copyright holder/s. The content must not be changed in any way or sold
commercially in any format or medium without the formal permission of the
copyright holders.

When referring to this work, full bibliographic details including the author, title,
awarding institution and date of the thesis must be given e.g.
AUTHOR (year of submission) "Full thesis title", University of Southampton, name
of the University School or Department, PhD Thesis, pagination
http://eprints.soton.ac.uk
University of Southampton
Faculty of Engineering, Sciences and Mathematics
School of Electronics and Computer Science
Arabic Text to Arabic Sign Language Example-Based
Translation System
by
Abdulaziz Almohimeed
——————————————————————————————-
A dissertation submitted in fulfilment of the requirements for the
award of Doctor of Philosophy
November 13, 2012
Dr. Mike Wald Professor Robert I. Damper
Supervisor Supervisor
Dr. Adam Pru¨gel-Bennett Professor Stephen Cox
Internal Examiner External Examiner
Declaration of Authorship
I, Abdulaziz Almohimeed, declare that the dissertation entitled “Arabic Text to
Arabic Sign Language Example-Based Translation System” and the work pre-
sented in the dissertation are both my own, and have been generated by me as
the result of my own original research. I confirm that:
this work was done wholly or mainly while in candidature for a research degree
•
at this University;
where any part of this dissertation has previously been submitted for a degree
•
or any other qualification at this University or any other institution, this has been
clearly stated;
where I have consulted the published work of others, this is always clearly
•
attributed;
where I have quoted from the work of others, the source is always given. With
•
the exception of such quotations, this dissertation is entirely my own work;
I have acknowledged all main sources of help;
•
where the dissertation is based on work done by myself jointly with others,
•
I have made clear exactly what was done by others and what I have contributed
myself.
Signature :
Date : November 13, 2012
ii
Acknowledgements
During my research, I was privileged to work with two great super-
visors, Professor Robert Damper and Dr.Mike Wald, who helped
me from the first day and were always there to patiently provide
guidance and support. I express my sincere gratitude to Profes-
sor Damper for giving his time and attention to shape my draft
and final thesis. His extensive knowledge and writing experience
improved my thesis. I could not have published my work on well-
known conferences, such as the EMNLP conference, without his help
and guidance. I also sincerely thank Dr.Wald for his guidance and
encouragement. His knowledge of deafness and assistive technologies
wasinvaluable. Iamgratefulforhiscommentsandsuggestionsduring
the research and writing process.
I could not have completed my research without the help of these
ArSL native signers and experts: Ahmed Alzaharani, Kalwfah Al-
shehri, Abdulhadi Alharbi, and Ali Alholafi. In addition, I thank
Mohamad Luhidan for analysing the morphological details in Cor-
pus 1.
I also thank my parents, Ibrahim and Ruqaia, for supporting and
encouraging me during my study. I am sincerely grateful to my father
for funding the building of the very first two ArSL corpora. I thank
my loving family-my wife Noha for travelling with me far from home,
and taking care of our daughter Ruqaia and our son Saad.
Finally, I acknowledge and thank the government of the Saudi Arabia
for funding my studies and my living expenses. When I return home,
I plan to repay the nation by applying the knowledge that I have
gained overseas.
Abstract
Thisdissertationpresentsthefirstcorpus-basedsystemfortranslation
from Arabic text into Arabic Sign Language (ArSL) for the deaf and
hearing impaired, for whom it can facilitate access to conventional
media and allow communication with hearing people. In addition to
the familiar technical problems of text-to-text machine translation,
building a system for sign language translation requires overcoming
some additional challenges. First, the lack of a standard writing
system requires the building of a parallel text-to-sign language corpus
from scratch, as well as computational tools to prepare this parallel
corpus. Further, the corpus must facilitate output in visual form,
which is clearly far more difficult than producing textual output. The
time and effort involved in building such a parallel corpus of text and
visual signs from scratch mean that we will inevitably be working
with quite small corpora. We have constructed two parallel Arabic
text-to-ArSL corpora for our system. The first was built from school-
level language instruction material and contains 203 signed sentences
and 710 signs. The second was constructed from a children’s story
and contains 813 signed sentences and 2,478 signs. Working with
corpora of limited size means that coverage is a huge issue. A new
technique was derived to exploit Arabic morphological information
to increase coverage and hence, translation accuracy. Further, we
employ two different example-based translation methods and combine
them to produce more accurate translation output. We have chosen
to use concatenated sign video clips as output rather than a signing
avatar, both for simplicity and because this allows us to distinguish
moreeasilybetweentranslationerrorsandsignsynthesiserrors. Using
leave-one-outcross-validationonourfirstcorpus,thesystemproduced
translated sign sentence outputs with an average word error rate of
36.2% and an average position-independent error rate of 26.9%. The
corresponding figures for our second corpus were an average word
error rate of 44.0% and 28.1%. The most frequent source of errors is
missing signs in the corpus; this could be addressed in the future by
collecting more corpus material. Finally, it is not possible to compare
the performance of our system with any other competing Arabic text-
to-ArSL machine translation system since no other such systems exist
at present.
Contents
1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Research Challenges and Goals . . . . . . . . . . . . . . . . . . . 4
1.3 Outline of the Dissertation . . . . . . . . . . . . . . . . . . . . . . 8
1.4 Contributions and Publications List . . . . . . . . . . . . . . . . . 9
2 Linguistic Background and MT 13
2.1 Arabic Language . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.1.1 Orthography . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.1.2 Phonology, Morphology, and Structure . . . . . . . . . . . 14
2.2 Arabic Sign Language . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.1 Orthography and Sources . . . . . . . . . . . . . . . . . . 18
2.2.2 Phonology . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.2.3 Grammar and Structure . . . . . . . . . . . . . . . . . . . 21
2.2.4 Vocabulary . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.2.5 Common Misconceptions . . . . . . . . . . . . . . . . . . . 25
2.2.5.1 Sign Language is Not a Universal Language . . . 25
2.2.5.2 ArSL is Not Based on the Arabic Language . . . 26
2.3 Overview of MT . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.3.1 MT Architectures . . . . . . . . . . . . . . . . . . . . . . . 26
2.3.2 Current Paradigms . . . . . . . . . . . . . . . . . . . . . . 28
2.3.3 EBMT Methods . . . . . . . . . . . . . . . . . . . . . . . . 29
2.3.4 Problems of Arabic Machine Translation . . . . . . . . . . 31
vi
CONTENTS
3 Sign Language Processing 32
3.1 Arabic Morphological Analysers . . . . . . . . . . . . . . . . . . . 32
3.1.1 Buckwalter’s Arabic Morphological Analyser . . . . . . . . 34
3.1.2 Xerox Arabic Morphological Analyser . . . . . . . . . . . . 36
3.1.3 Al-Khalil Arabic Morphological Analyser . . . . . . . . . . 37
3.2 Related SL Translation Systems . . . . . . . . . . . . . . . . . . . 37
3.2.1 Rule–Based Systems . . . . . . . . . . . . . . . . . . . . . 38
3.2.2 Corpus-Based Systems . . . . . . . . . . . . . . . . . . . . 43
3.2.3 Proposed ArSL systems . . . . . . . . . . . . . . . . . . . 46
3.3 Notation Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.3.1 Sutton SignWriting . . . . . . . . . . . . . . . . . . . . . . 49
3.3.2 Hamburg Notation . . . . . . . . . . . . . . . . . . . . . . 49
3.3.3 Gloss Notation . . . . . . . . . . . . . . . . . . . . . . . . 50
3.4 Related Corpora . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.4.1 General Purpose Corpora . . . . . . . . . . . . . . . . . . 51
3.4.1.1 European Cultural Heritage On-line SL Corpora . 51
3.4.1.2 Irish SL Corpus . . . . . . . . . . . . . . . . . . . 52
3.4.1.3 British SL Corpus Project . . . . . . . . . . . . . 53
3.4.1.4 Netherlands SL Corpus Project . . . . . . . . . . 54
3.4.2 MT Corpora . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.4.2.1 Weather Cast DGS Corpus . . . . . . . . . . . . 55
3.4.2.2 AmericanSignLanguageLinguisticResearchProject
(ASLLRP) . . . . . . . . . . . . . . . . . . . . . 56
3.4.2.3 Air Travel Information System (ATIS) Corpus . . 56
3.4.2.4 Czech SL Corpus . . . . . . . . . . . . . . . . . . 56
3.5 Sign Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.5.1 Concatenated Video-based Sign Synthesis . . . . . . . . . 59
3.5.2 Avatar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.5.2.1 Integrated Avatar . . . . . . . . . . . . . . . . . 62
3.5.2.2 Exclusive Avatar . . . . . . . . . . . . . . . . . . 67
3.6 Translation Evaluation Techniques . . . . . . . . . . . . . . . . . 69
vii
CONTENTS
4 Building ArSL Corpora 70
4.1 Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.1.1 Annotating Tool . . . . . . . . . . . . . . . . . . . . . . . 71
4.1.2 Notation System . . . . . . . . . . . . . . . . . . . . . . . 72
4.1.3 Compiling Tool . . . . . . . . . . . . . . . . . . . . . . . . 73
4.1.4 Size and Domain . . . . . . . . . . . . . . . . . . . . . . . 73
4.1.5 Spatial Reference Points . . . . . . . . . . . . . . . . . . . 75
4.2 Composition of the Team of Signers . . . . . . . . . . . . . . . . . 76
4.3 Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
4.4 Overview of Corpus Construction Process . . . . . . . . . . . . . 78
4.4.1 Video Recording . . . . . . . . . . . . . . . . . . . . . . . 79
4.4.2 Transcript and Annotation . . . . . . . . . . . . . . . . . . 79
4.4.3 Compiling the Parallel Corpus . . . . . . . . . . . . . . . . 82
4.4.4 Extracting Features . . . . . . . . . . . . . . . . . . . . . . 83
4.4.5 Sign Dictionary . . . . . . . . . . . . . . . . . . . . . . . . 84
4.4.6 Word-Sign Alignment . . . . . . . . . . . . . . . . . . . . . 84
4.5 Dealing with Size Limitation . . . . . . . . . . . . . . . . . . . . . 86
4.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
5 Implementing the ArSL System 92
5.1 Arabic Processing Unite (APU) . . . . . . . . . . . . . . . . . . . 92
5.1.1 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
5.1.2 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
5.2 Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
5.3 Three-Phase EBMT . . . . . . . . . . . . . . . . . . . . . . . . . 100
5.3.1 System Design . . . . . . . . . . . . . . . . . . . . . . . . 101
5.3.2 Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
5.3.3 Alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
5.3.4 Recombination . . . . . . . . . . . . . . . . . . . . . . . . 106
5.4 Classical EBMT . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
5.5 Similarity-Based EBMT . . . . . . . . . . . . . . . . . . . . . . . 108
5.5.1 System Design . . . . . . . . . . . . . . . . . . . . . . . . 108
5.5.2 Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
5.5.3 Alignment and Recombination . . . . . . . . . . . . . . . . 110
5.6 Phrase-Based EBMT . . . . . . . . . . . . . . . . . . . . . . . . . 111
viii
CONTENTS
5.6.1 System Design . . . . . . . . . . . . . . . . . . . . . . . . 111
5.6.2 Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.6.3 Alignment and Recombination . . . . . . . . . . . . . . . . 111
5.7 Combining Similarity- and Phrase-Based EBMT . . . . . . . . . . 112
5.7.1 Concatenated Video-based Sign Synthesis . . . . . . . . . 113
5.7.2 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
5.7.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
6 Evaluation 119
6.1 Human Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 119
6.2 Automatic Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 122
6.3 A New Automatic Evaluation Metric for SL Translation . . . . . . 124
6.3.1 Defining the Problem . . . . . . . . . . . . . . . . . . . . . 125
6.3.2 Sign Language Error Rate (SiER) technique . . . . . . . . 125
6.3.3 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
6.3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
6.4 Cross Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
6.5 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . 132
7 Conclusions and Future Work 139
ix