Final Summarization System LING 573 Deliverable #04

88 Pages·2015·1.92 MB·English

.
Final Summarization System
LING 573 Deliverable #04
.
Eric Garnick (
Outline
.
System Overview
Content Selection
Information Ordering
Content Realization
Issues and Deadends
ROUGE scores
In�uences
Future Directions
. . . . . . . . . . . . . . . . . . . .
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
egarnick, jtm37, olgaw LING 573 D4
System Overview
.
Python 3, nltk
Gigaword corpus
3 high-level, independent components:
content selection, ordering, realization
Barzilay, et. al. 2002
�nal ROUGE F scores for evaltest data:
R1 R2 R3 R4
0.31068 0.09708 0.03683 0.01701
. . . . . . . . . . . . . . . . . . . .
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
egarnick, jtm37, olgaw LING 573 D4
Content Selection
.
log likelihood ratio (Gigaword corpus)
tokenization, data cleanup
topic signatures (cluster title, article headlines)
tf-idf cosine similarity scores
. . . . . . . . . . . . . . . . . . . .
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
egarnick, jtm37, olgaw LING 573 D4
Information Ordering
.
added three different methods to re-order summaries, all part of a
discrete step:
chronological (Barzilay, 2002; publication date and relative
position in document)
majority (Barzilay, 2002; heuristic topological search, relative
sentence positions)
traveling sales person (Conroy, 2006; consider all possible
pairs, minimize distances)
. . . . . . . . . . . . . . . . . . . .
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
egarnick, jtm37, olgaw LING 573 D4
Information Ordering Evaluation
.
Earlier we evaluated reorderings by random spot check, gauging an
overall \feel", which ranked majority highest.
Now we have two different review methods:
manual, semi-blinded evaluation, 3 levels of comprehensibility
(Barzilay, 2002)
automatic, correlation to gold-standard summaries
. . . . . . . . . . . . . . . . . . . .
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
egarnick, jtm37, olgaw LING 573 D4
Information Ordering Evaluation: Manual
.
System Comprehensible Somewhat Comp. Incomp.
Original 8 9 4
Majority 11 7 3
Chronological 6 11 4
Similarity 6 10 5
Majority the least bad?
. . . . . . . . . . . . . . . . . . . .
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
egarnick, jtm37, olgaw LING 573 D4
Information Ordering Evaluation: Automatic
.
Re-ordering processes applied to the gold-standards, and then
calculated Spearman’s rank correlation.
Dataset Algorithm rho p-value
evaltest chrono 0.0604 0.3190
evaltest majority -0.0384 0.5270
evaltest pass 0.8825 4.4456e-91
evaltest tsp 0.4102 1.5206e-12
devtest chrono 0.03787 0.5265
devtest majority 0.0379 0.5259
devtest pass 0.9816 1.3553e-203
devtest tsp 0.2655 6.1884e-06
TSP the least bad?
. . . . . . . . . . . . . . . . . . . .
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
egarnick, jtm37, olgaw LING 573 D4
Content Realization
.
Sentence �ltering:
pruning extraneous words (email addresses, phone numbers,
paraentheticals, &c.) found by regexes
remove attributions
remove adverbials
remove ALL CAPS items (but with an exception list)
exclude sentences without any verbs (as determined by SpaCy
POS tagging)
. . . . . . . . . . . . . . . . . . . .
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
egarnick, jtm37, olgaw LING 573 D4
Issues and Deadends
.
Items that were tried and abandoned, or were deemed unhelpful:
co-reference resolution
sentence position weighting
stemming / lemmatization
augmented chronological re-ordering
various parameter tunings
. . . . . . . . . . . . . . . . . . . .
.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
egarnick, jtm37, olgaw LING 573 D4