Exploiting Syntactic Structure for Language Modeling Ciprian Chelba and Frederick Jelinek Center for Language and Speech Processing The Johns Hopkins University, Barton Hall 320 3400 N. Charles St., Baltimore, MD-21218, USA {chelba,jelinek}@jhu.edu 0 0 Abstract 0 ended_VP’ 2 The paper presents a language model that devel- with_PP n ops syntactic structure and uses it to extract mean- loss_NP a ingful information from the word history, thus en- J abling the use of long distance dependencies. The 5 of_PP model assigns probability to every joint sequence 2 contract_NP loss_NP cents_NP of words–binary-parse-structure with headword annotation and operates in a left-to-right manner — ] the_DT contract_NN ended_VBDwith_IN a_DT loss_NN of_IN 7_CD cents_NNS after L therefore usable for automatic speech recognition. C Themodel,itsprobabilisticparameterization,anda . set of experiments meant to evaluate its predictive Figure 1: Partial parse s powerare presented; animprovementoverstandard c [ trigram modeling is achieved. 2 The Basic Idea and Terminology 2 1 Introduction Considerpredictingthe wordafterinthesentence: v 2 Themaingoalofthepresentworkistodevelopalan- the contract ended with a loss of 7 cents 2 guage model that uses syntactic structure to model after trading as low as 89 cents. 0 long-distance dependencies. During the summer96 A 3-gram approach would predict after from 1 DoD Workshop a similar attempt was made by the (7, cents) whereas it is intuitively clear that the 1 dependency modeling group. The model we present strongestpredictorwouldbeendedwhichis outside 8 is closely related to the one investigated in (Chelba the reach of even 7-grams. Our assumption is that 9 / et al., 1997), however different in a few important what enables humans to make a good prediction of s aspects: after is the syntactic structure in the past. The c : • our model operates in a left-to-right manner, al- linguistically correct partial parse of the word his- v lowing the decoding of word lattices, as opposed to tory when predicting after is shown in Figure 1. Xi theonereferredtopreviously,whereonlywholesen- The word ended is called the headword of the con- tencescouldbe processed,thusreducingitsapplica- stituent (ended (with (...)))andendedisanex- r a bilityton-bestlistre-scoring;thesyntacticstructure posed headword when predicting after — topmost is developed as a model component; headwordinthelargestconstituentthatcontainsit. • our model is a factored version of the one The syntactic structure in the past filters out irrel- in(Chelbaetal.,1997),thusenablingthecalculation evant words and points to the important ones, thus ofthejointprobabilityofwordsandparsestructure; enabling the use of long distance information when this wasnotpossible inthe previouscase due to the predicting the next word. huge computational complexity of the model. Our model will attempt to build the syntactic Ourmodeldevelopssyntacticstructureincremen- structure incrementally while traversing the sen- tallywhiletraversingthesentencefromlefttoright. tenceleft-to-right. Themodelwillassignaprobabil- This is the main difference between our approach ityP(W,T)toeverysentenceW witheverypossible andotherapproachestostatisticalnaturallanguage POStag assignment, binary branching parse, non- parsing. Our parsing strategy is similar to the in- terminal label and headword annotation for every cremental syntax ones proposed relatively recently constituent of T. in the linguistic community (Philips, 1996). The Let W be a sentence of length n words to which probabilistic model, its parameterization and a few we have prepended ~~and appended~~ so experimentsthataremeanttoevaluateitspotential that w0 = ~~and wn+1 =~~. Let Wk be the for speech recognition are presented. word k-prefix w0...wk of the sentence and WkTk h_{-m} = (, SB) h_{-1} h_0 = (h_0.word, h_0.tag) hh__{{--22}} hh__{{--11}} hh__00 T_{-m} .................. T_{-2} T_{-1} T_0 (~~, SB) ....... (w_p, t_p) (w_{p+1}, t_{p+1}) ........ (w_k, t_k) w_{k+1}....~~ <> Figure 2: A word-parse k-prefix Figure 4: Before an adjoin operation (, TOP) h’_{-1} = h_{-2} h’_0 = (h_{-1}.word, NTlabel) (, TOP’) h_{-1} T’_0 h_0 T’_{-m+1}<- ............... T’_{-1}<-T_{-2} T_{-1} T_0 (~~, SB) (w_1, t_1) ..................... (w_n, t_n) (~~, SE) Figure 5: Result of adjoin-left under NTlabel Figure 3: Complete parse headword and non-terminal label assignments for the word-parse k-prefix. To stress this point, a word-parse k-prefix contains — for a given parse the w1...wk word sequence can be generated. The followingalgorithmformalizestheabovedescription — only those binary subtrees whose span is com- of the sequential generation of a sentence with a pletely included in the word k-prefix, excluding complete parse. w =. Single words along with their POStag 0 can be regarded as root-only trees. Figure 2 shows Transition t; // a PARSER transition a word-parse k-prefix; h_0 .. h_{-m} are the ex- predict (, SB); posed heads, each head being a pair(headword,non- do{ terminal label), or (word, POStag) in the case of a //WORD-PREDICTOR and TAGGER root-only tree. A complete parse — Figure 3 — is predict (next_word, POStag); any binary parse of the //PARSER (w1,t1)...(wn,tn) (, SE) sequence with the do{ restriction that (, TOP’) is the only allowed if(h_{-1}.word != ~~){ head. Note that ((w1,t1)...(wn,tn)) needn’t be a if(h_0.word ==~~ ) constituent, but for the parses where it is, there is t = (adjoin-right, TOP’); no restrictiononwhich of its wordsis the headword else{ or what is the non-terminal label that accompanies if(h_0.tag == NTlabel) the headword. t = [(adjoin-{left,right}, NTlabel), The model will operate by means of three mod- null]; ules: else • WORD-PREDICTOR predicts the next word t = [(unary, NTlabel), wk+1 given the word-parse k-prefix and then passes (adjoin-{left,right}, NTlabel), control to the TAGGER; null]; • TAGGER predicts the POStag of the next word } tk+1 given the word-parse k-prefix and the newly } predicted word and then passes control to the else{ PARSER; if(h_0.tag == NTlabel) • PARSER grows the already existing binary t = null; branching structure by repeatedly generating the else transitions: t = [(unary, NTlabel), null]; (unary, NTlabel), (adjoin-left, NTlabel) or } (adjoin-right, NTlabel) until it passes control }while(t != null) //done PARSER to the PREDICTOR by taking a null transition. }while(!(h_0.word== && h_{-1}.word==)) NTlabel is the non-terminal label assigned to the t = (adjoin-right, TOP); //adjoin _SB; DONE; newly built constituent and {left,right}specifies where the new headword is inherited from. The unary transition is allowed only when the The operations performed by the PARSER are most recent exposed head is a leaf of the tree — illustrated in Figures 4-6 and they ensure that all a regular word along with its POStag — hence it possible binary branching parses with all possible canbe takenatmostonce ata givenpositionin the halts with probability one. Consequently, certain h’_{-1}=h_{-2} h’_0 = (h_0.word, NTlabel) PARSER and WORD-PREDICTOR probabilities must be given specific values: h_{-1} h_0 • P(null/WkTk) = 1, if h_{-1}.word = ~~and h_{0}6=(~~, TOP’)—thatis,beforepredicting T’_{-m+1}<- ~~............... T’_{-1}<-T_{-2} T_{-1} T_0~~ — ensures that (~~, SB) is adjoined in the last step of the parsing process; • P((adjoin-right, TOP)/WkTk)=1, if Figure 6: Result of adjoin-right under NTlabel h_0 = (~~, TOP’) and h_{-1}.word = ~~and input word string. The second subtree in Figure 2 P((adjoin-right, TOP’)/WkTk)=1, if provides an example of a unary transition followed h_0 = (~~, TOP’) and h_{-1}.word6= by a null transition. ensurethattheparsegeneratedbyourmodeliscon- It is easy to see that any given word sequence sistent with the definition of a complete parse; with a possible parse and headword annotation is • P((unary, NTlabel)/WkTk) = 0, if h_0.tag 6= generated by a unique sequence of model actions. POStag ensures correct treatment of unary produc- This will prove very useful in initializing our model tions; parameters from a treebank — see section 3.5. • ∃ǫ>0,∀Wk−1Tk−1,P(wk=/Wk−1Tk−1)≥ǫ ensures that the model halts with probability one. 3 Probabilistic Model The word-predictor model (2) predicts the next The probabilityP(W,T) ofa wordsequence W and word based on the preceding 2 exposed heads, thus a complete parse T can be broken into: making the following equivalence classification: P(W,T)= P(wk/Wk−1Tk−1)=P(wk/h0,h−1) Qnk=+11[ P(wk/Wk−1Tk−1)·P(tk/Wk−1Tk−1,wk)· Afterexperimentingwithseveralequivalenceclas- Nk sifications of the word-parse prefix for the tagger k k k YP(pi/Wk−1Tk−1,wk,tk,p1...pi−1)](1) model, the conditioning part of model (3) was re- i=1 duced to using the word to be tagged and the tags where: of the two most recent exposed heads: • Wk−1Tk−1 is the word-parse (k−1)-prefix •wk isthewordpredictedbyWORD-PREDICTOR P(tk/wk,Wk−1Tk−1)=P(tk/wk,h0.tag,h−1.tag) • tk is the tag assigned to wk by the TAGGER Model(4)assignsprobabilitytodifferentparsesof • Nk −1 is the number of operations the PARSER the word k-prefix by chaining the elementary oper- executes before passing control to the WORD- ations described above. The workings of the parser PREDICTOR (the Nk-th operation at position k is modulearesimilartothoseofSpatter(Jelineketal., t•hpeknduelnlottersanthsietiio-nth);PNAkRiSsEaRfuonpcetrioantioonfcTarriedout 1994). The equivalence classification of the WkTk i word-parseweusedfortheparsermodel(4)wasthe at position k in the word string; same as the one used in (Collins, 1996): pk ∈{(unary, NTlabel), 1 (adjoin-left, NTlabel), k k P(pi/WkTk)=P(pi/h0,h−1) (adjoin-right, NTlabel), null}, pki ∈{ (adjoin-left, NTlabel), It is worth noting that if the binary branching (adjoin-right, NTlabel)},1 token; 3.5 Initial Parameters • POS tag vocabulary: 40, closed; • non-terminal tag vocabulary: 52, closed; Each model component — WORD-PREDICTOR, • parser operation vocabulary: 107, closed; TAGGER, PARSER — is trained initially from a The training data was split into “development” set set of parsed sentences, after each parse tree (W,T) — 929,564wds (sections 00-20) — and “check set” undergoes: — 73,760wds (sections 21-22); the test set size was • headwordpercolation and binarization — see sec- 82,430wds (sections 23-24). The “check” set has tion 4; been used for estimating the interpolation weights • decomposition into its derivation(W,T). and tuning the search parameters; the “develop- Then, separately for each m model component, we: • gather joint counts C(m)(y(m),x(m)) from the ment” set has been used for gathering/estimating counts; the test set has been used strictly for evalu- derivations that make up the “development data” ating model performance. using φ(W,T)=1; Table1showstheresultsofthere-estimationtech- • estimate the deleted interpolation coefficients on niquepresentedinsection3.4. Weachievedareduc- joint counts gathered from “check data” using the tionintest-dataperplexitybringinganimprovement EM algorithm. over a deleted interpolation trigram model whose These are the initial parameters used with the re- perplexitywas167.14onthesametraining-testdata; estimation procedure described in the previous sec- the reduction is statistically significantaccordingto tion. a sign test. 4 Headword Percolation and Binarization iteration DEV set TEST set number L2R-PPL L2R-PPL In order to get initial statistics for our model com- E0 24.70 167.47 ponents we needed to binarize the UPenn Tree- E1 22.34 160.76 bank (Marcus et al., 1995) parse trees and perco- E2 21.69 158.97 late headwords. The procedure we used was to first E3 21.26 158.28 percolate headwords using a context-free (CF) rule- 3-gram 21.20 167.14 based approach and then binarize the parses by using a rule-based approach again. Table 1: Parameter re-estimation results The headword of a phrase is the word that best represents the phrase, all the other words in the phrase being modifiers of the headword. Statisti- Simplelinearinterpolationbetweenourmodeland cally speaking, we were satisfied with the output the trigram model: of an enhanced version of the procedure described in (Collins, 1996) — also known under the name Q(wk+1/Wk)= “Magerman & Black Headword PercolationRules”. λ·P(wk+1/wk−1,wk)+(1−λ)·P(wk+1/Wk) Once the position of the headword within a con- yielded a further improvement in PPL, as shown in stituent — equivalent with a CF production of the Table 2. The interpolation weight was estimated on type Z → Y1...Yn , where Z,Y1,...Yn are non- check data to be λ=0.36. terminal labels or POStags (only for Yi) — is iden- tifiedtobe k,webinarizethe constituentasfollows: depending on the Z identity, a fixed rule is used Anoverallrelativereductionof11%overthetrigram to decide which of the two binarization schemes in model has been achieved. Figure 8 to apply. The intermediate nodes created 6 Conclusions and Future Directions by the above binarization schemes receive the non- terminal label Z′. The large difference between the perplexity of our model calculated on the “development” set — used iteration TEST set TEST set References number L2R-PPL 3-gram interpolated PPL W. Byrne, A. Gunawardana, and S. Khudanpur. E0 167.47 152.25 1998. Information geometry and EM variants. E3 158.28 148.90 Technical Report CLSP Research Note 17, De- 3-gram 167.14 167.14 partmentof ElecticalandComputer Engineering, The Johns Hopkins University, Baltimore, MD. Table 2: Interpolation with trigram results C.Chelba,D.Engle,F.Jelinek,V.Jimenez,S.Khu- danpur, L. Mangu, H. Printz, E. S. Ristad, R.Rosenfeld,A.Stolcke,andD.Wu. 1997. Struc- formodelparameterestimation—and“test”set— ture and performance of a dependency language unseendata—showsthattheinitialpointwechoose model. In Proceedings of Eurospeech, volume 5, for the parameter values has already captured a lot pages 2775–2778.Rhodes, Greece. of information from the training data. The same MichaelJohnCollins. 1996. Anewstatisticalparser problemisencounteredinstandardn-gramlanguage basedonbigramlexicaldependencies. InProceed- modeling;however,ourapproachhasmoreflexibility ings of the 34th Annual Meeting of the Associ- indealingwithitduetothepossibilityofreestimat- ation for Computational Linguistics, pages 184– ing the model parameters. 191. Santa Cruz, CA. We believe that the above experiments show the A.P.Dempster,N.M.Laird,andD.B.Rubin. 1977. potential of our approach for improved language Maximumlikelihoodfromincompletedataviathe models. Our future plans include: EM algorithm. In Journal of the Royal Statistical • experimentwithother parameterizationsthan the Society, volume 39 of B, pages 1–38. twomostrecentexposedheadsinthewordpredictor Frederick Jelinek and Robert Mercer. 1980. Inter- model and parser; polated estimation of markov source parameters • estimate a separate word predictor for left-to- fromsparsedata. InE.GelsemaandL.Kanal,ed- rightlanguage modeling. Note that the correspond- itors, Pattern Recognition in Practice, pages 381– ing model predictor was obtained via re-estimation 397. aimed at increasing the probability of the ”N-best” F.Jelinek, J.Lafferty,D. M.Magerman,R.Mercer, parses of the entire sentence; A. Ratnaparkhi, and S. Roukos. 1994. Decision • reduce vocabulary of parser operations; extreme tree parsing using a hidden derivational model. case: no non-terminal labels/POS tags, word only In ARPA, editor, Proceedings of the Human Lan- model; this will increase the speed of the parser guage Technology Workshop, pages 272–277. thus rendering it usable on larger amounts of train- M. Marcus, B. Santorini, and M. Marcinkiewicz. ing data and allowing the use of deeper stacks — 1995. Building a large annotated corpus of En- resulting in more “N-best” derivations per sentence glish: the Penn Treebank. Computational Lin- during re-estimation; guistics, 19(2):313–330. • relax — flatten — the initial statistics in the re- Colin Philips. 1996. Order and Structure. Ph.D. estimationofmodelparameters;thiswouldallowthe thesis, MIT. Distributed by MITWPL. model parameters to converge to a different point that might yield a lower word-level perplexity; • evaluate model performance on n-best sentences output by an automatic speech recognizer. 7 Acknowledgments This researchhas been funded by the NSF IRI-19618874grant (STIMULATE). The authors would like to thank to Sanjeev Khu- danpur for his insightful suggestions. Also to Harry Printz,EricRistad, AndreasStolcke,DekaiWu and all the other members of the dependency modeling group at the summer96 DoD Workshop for useful comments on the model, programming support andanextremelycreativeenvironment. Alsothanks to Eric Brill, Sanjeev Khudanpur, David Yarowsky, Radu Florian, Lidia Mangu and Jun Wu for useful input during the meetings of the people working on our STIMULATE grant.

Expoiting Syntactic Structure for Language Modeling

Similar Expoiting Syntactic Structure for Language Modeling