Table Of Content

Shortcut Sequence Tagging Huijia Wu1,3, JiajunZhang1,3, andChengqing Zong1,2,3 1NationalLaboratory ofPatternRecognition,InstituteofAutomation,CAS 2CAS Center forExcellenceinBrain Science and IntelligenceTechnology 3UniversityofChineseAcademyofSciences {huijia.wu,jjzhang,cqzong}@nlpr.ia.ac.cn 7 1 0 2 n a Abstract inant in sequence tagging problems due to J the superior performance (Wangetal.,2015; 3 DeepstackedRNNsareusuallyhardtotrain. Lampleetal.,2016). The horizontal hierarchy of Adding shortcut connections across different ] LSTMswithbidirectional processing canremember L layers is a common way to ease the training the long-range dependencies without affecting the C of stacked networks. However, extra short- short-term storage. Although the models have . cuts make the recurrent step more compli- s c cated. Tosimplythestackedarchitecture,we a deep horizontal hierarchy (the depth refers to [ propose a framework called shortcut block, sequence length), the vertical hierarchy is often 1 which is a marriage of the gating mecha- shallow, which may not be efficient at representing v nismandshortcuts,whilediscardingtheself- each token. Stacked LSTMs are deep in both 6 connected part in LSTM cell. We present dimensions, but become harder to train due to the 7 extensiveempiricalexperimentsshowingthat 5 feed-forward structure ofstackedlayers. thisdesignmakestrainingeasyandimproves 0 generalization. We propose various shortcut Shortcut connections (shortcuts, or skip con- 0 . blocktopologiesandcompositionstoexplore nections) enable unimpeded information flow 1 its effectiveness. Based on this architecture, by adding direct connections across differ- 0 7 we obtain a 6% relatively improvement over ent layers (Raikoetal.,2012; Graves,2013; 1 thestate-of-the-artonCCGbanksupertagging HermansandSchrauwen,2013). Recent works : dataset. We also get comparable results on v have shown the effectiveness of using short- POStaggingtask. i X cuts in deep stacked models (Heetal.,2015; r Srivastavaetal.,2015; Wuetal.,2016b). These a 1 Introduction works share a common way of adding shortcuts as increments totheoriginalnetwork. In natural language processing, sequence tagging mainly refers to the tasks of assigning discrete la- We focus on the refinement of shortcut stacked bels to each token in a sequence. Typical examples models to make the training easy. Particularly, for include Part-of-Speech (POS) tagging and Combi- stackedLSTMs,theshortcutsmakethecomputation natory Category Grammar (CCG) supertagging. A of LSTMblocks more complicated. Wereplace the regular feature of sequence tagging is that the in- self-connected parts in LSTM cells with the short- put tokens in a sequence cannot be assumed to be cutstosimplifytheupdates. Basedonthisconstruc- independent since the same token in different con- tion, we introduce the shortcut block, which can be texts can be assigned to different tags. Therefore, viewed as a marriage of the gating mechanism and theclassifiershouldhavememoriestorememberthe theshortcuts, whilediscard theself-connected units contextstomakeacorrectprediction. inLSTMs. Bidirectional LSTMs Our contribution is mainly in the exploration of (GravesandSchmidhuber, 2005) become dom- shortcutblocks. Weproposeaframeworkofstacked LSTMs within shortcuts, using deterministic or layerl+1,...,L. Inthissection, wefirstintroduce stochastic gates to control the shortcut connections. thetraditional stacked LSTMs. Based onthisarchi- We present extensive experiments on the Combina- tecture,weproposeashortcutblockstructure,which toryCategoryGrammar(CCG)supertagging taskto isthebasicelementofourstackedmodels. compare various shortcut block topologies, gating 3.1 TranditionalStackedLSTMs functions, and combinations of the blocks. We also evaluate our model on Part-of-Speech (POS) tag- StackedLSTMswithoutskipconnectionscanbede- gingtasktotestthegeneralizationperformance. Our finedas: modelobtainsthestate-of-the-artresultsonCCGsu- i sigm pertagging andcomparableresultsonPOStagging.  f  =  sigm Wl hlt−1 (5) 2 Recurrent Neural Networks for o sigm (cid:18) hl (cid:19)     t 1 Sequence Tagging  s   tanh  −     cl = f ⊙cl +i⊙sl Consider a recurrent neural network applied to t t−1 t (6) sequence tagging: Given a sequence x = hl = o⊙tanh(cl) t t (x ,...,x ), the RNN computes the hidden state 1 T Duringtheforwardpass,LSTMsneedtocalculatecl h = (h1,...,hT) and the output y = (y1,...,yT) t and hl, which isthe cell’s internal state and the cell byiteratingthefollowingequations: t outputsstate,respectively. cl iscomputedbyadding t h = f(x ,h ;θ ) (1) twoparts: oneisthecellincrementsl,controlledby t t t 1 h t − theinput gateil,theother isthe self-connected part yt = g(ht;θo) (2) t cl , controlled by the forget gate fl. The cell out- t 1 t wheret ∈ {1,...,T}represents thetime. x repre- pu−ts hl are computed by multiplying the activated t t sents the input at time t, h and h are the previ- cell state by the output gate ol, which learns when t 1 t t − ousandthecurrent hiddenstate,respectively. f and toaccess memorycellandwhentoblock it. “sigm” garethetransitionfunctionandtheoutputfunction, and“tanh”arethesigmoidandtanhactivationfunc- respectively. θ andθ arenetworkparameters. tion, respectively. Wl ∈ R4n 2n is the weight ma- h o × We use a negative log-likelihood cost to evaluate trixneedstobelearned. theperformance, whichcanbewrittenas: 3.2 ShortcutBlocks 1 N ThehiddenunitsinstackedLSTMshavetwoforms. C = − logy (3) N tn One is the hidden units in the same layer {hl,t ∈ nX=1 t 1,...,T}, which are connected through an LSTM. where tn ∈ N is the true target for sample n, and The other is the hidden units at the same time step y is the t-th output in the softmax layer given the {hl,l ∈ 1,...,L}, which are connected through a tn t inputsxn. feed-forward network. LSTM can keep the short- Stacked RNN is one type of deep RNNs, which term memory for a long time, thus the error sig- referstothehiddenlayersarestackedontopofeach nalscanbeeasilypassedthrough{1,...,T}. How- other, eachfeedinguptothelayerabove: ever,whenthenumberofstackedlayersislarge,the feed-forward network will suffer the gradient van- hlt = fl(hlt−1,hlt 1) (4) ishing/exploding problems, which make the gradi- − entshardtopassthrough {1,...,L}. wherehl isthet-thhiddenstateofthel-thlayer. t Shortcut connections can partly solve the above problem byadding adirect link between layers. An 3 Explorations ofShortcuts intuitive explanation is that such link can make the Shortcuts(skip connections) are cross-layer connec- error signal passing jumpthelayers, not justoneby tions, which means the output of layer l − 1 is not one. Thismaylead tofaster convergence and better only connected to the layer l, but also connected to generalization. To clarify notations, we introduce theshortcut block, which iscomposed of thediffer- Why Only Increments? Obviously, we can keep entlayersconnected throughshortcuts. the self-connected part with the increments in the Our shortcut block is mainly based on Wu et al. cell state. But the most important reason to this de- (2016b),whichintroduce gatedshortcuts connected sign is that it is much easier to compute. We do tocelloutputs. Themaindifferenceiswereplacethe not need extra space to preserve the cell state. This self-connected parts with shortcuts to compute the makesdeepstacked modelsmucheasiertotrain. internal state. LSTMblock (1997) composes mem- Discussion. The shortcut block can be seen as a orycellssharingthesameinputandoutputgate. He generalization of several multiscale RNN architec- et al. (2015) create a residual block which adds tures. It is up to the user to define the block topol- shortcut connections across different CNN layers. ogy. For example, when h l := hl 1, this is simi- Alltheseinspiredustobuildashortcutblockacross −t t− lartorecurrenthighwaynetworks(Zillyetal.,2016) differentLSTMlayers. Ourshortcutblockisdefined and highway LSTMs (Zhangetal.,2016). When asfollows: h l := hl 2, this becomes the traditional shortcuts −t t− i sigm forRNNs:  go  =  ssiiggmm Wl(cid:18) hhllt−1 (cid:19) (7) m = i⊙slt+g⊙hlt−2     t 1 (11)  s   tanh  − hl = o⊙tanh(m)+g⊙hl 2     t t− m = i⊙sl +g⊙h l t −t 3.3 GatesSesign (8) hlt = o⊙tanh(m)+g⊙h−t l Shortcut gates are used to make the skipped path deterministic (Srivastavaetal.,2015) or stochastic whereh listheoutputfromoneofthepreviouslay- −t (Huangetal.,2016). We explore many ways to ers1,...,l−1. g isthegatewhichisusedtoaccess compute the shortcut gates (denoted by gl). The theskippedoutputh l orblockit. t −t simplest case is to use gl as a linear operator. In t Comparison with LSTMs. LSTMs introduce a thiscase,glisaweightmatrix,andtheelement-wise t memory cell with a fixed self-connection to make productgl⊙h l inEq. (7)becomesamatrix-vector t −t the constant error flow. They compute the follow- multiplication: ingincrementtotheself-connected cellateachtime gl ⊙h l := Wlh l (12) step: t −t −t c = c +s (9) t t 1 t We can also get gl under a non-linear mapping, − t Hereweremovethemultiplicative gatestosimplify which is similar to the computation of gates in theexplanation. Theself-connected cellct cankeep LSTM: the recurrent information for a long time. s is the t gl = σ(Wlhl 1) (13) increment to the cell. While in our shortcut block, t t− we only consider the increment part. Since in se- Here we use the output of layer l−1 to control the quencetagging problem, theinputsequence andthe shortcuts,e.g. hl 2. Noticethatthisnon-linearmap- t− output sequence areexactly match. Specifically, the pingisnotunique, wejustshowthesimplestcase. input token x ,i ∈ {1,...,n} in a input sequence i Furthermore, inspired by the dropout withlengthnprovidesthemostrelevantinformation (Srivastavaetal.,2014) strategy, we can sam- topredictthecorresponding labely ,i ∈ {1,...,n} i plefromaBernoulli stochastic variabletogetgl. In t in a output sequence. We want to focus the infor- this case, a deterministic gate is transformed into a mation flow in the vertical direction through short- stochastic gate. cuts, rather than in the horizontal direction through the self-connected units. Therefore, we only con- gl ∼ Bernoulli(p) (14) t sider the increment to the cell and ignore the self- where gl is a vector of independent Bernoulli ran- connected part. Ourcellstatebecomes: t domvariableseachofwhichhasprobabilitypofbe- m = h−t l +st (10) ing 1. We can either fix p with a specific value or learnitwithanon-linearmapping. Forexample,we usingalocalwindowapproach, oracombination of canlearnpby: word and character-level representation. Following Wuetal. (2016b), weusealocal window approach p = σ(Hlhl 1) (15) t− together with a concatenation of word representations, character representations, and capitalization Attesttime,h l ismultiplied byp. −t representations. Discussion. The gates of LSTMs are essential Formally, we can represent the distributed word parts to avoid weight update conflicts, which are feature f using a concatenation of these embed- wt also invoked by the shortcuts. In experiments, we dings: find that using deterministic gates is better than the stochastic gates. We recommend using the logistic fwt = [Lw(wt);La(at);Lc(cw)] (16) gatestocomputegl. t wherew ,a represent thecurrent wordanditscap- t t 3.4 CompositionsofShortcutBlocks italization. cw := [c1,c2,...,cTw], where Tw is the lengthofthewordandc ,i ∈ {1,...,T }isthei-th In the previous subsections, we introduce the short- i w character for theparticular word. L (·) ∈ RVw n, cut blocks and the computation of gating functions. w | |× L (·) ∈ RVa m and L (·) ∈ RVc r are the look- To build deep stacked models we need to compose a | |× c | |× up tables for the words, capitalization and charac- these blocks together. In this section, we discuss ters,respectively. f ∈ Rn+m+r representsthedis- several kinds of compositions, as shown in Figure wt tributed feature of w . A context window of size d 1. Thelinkswith⊙representthegatedidentitycon- t surrounding thecurrentwordisusedasaninput: nections across layers. There are two types of connections in Figure 1: one is the direct connections x = [f ;...;f ] (17) t wt−⌊d/2⌋ wt+⌊d/2⌋ between adjacent layers, the other is the gated connections acrossdifferentlayers. where x ∈ R(n+m+r) d is a concatenation of the t × In Figure 1, Type 1 is to connect the input of the contextfeatures. Inthefollowingwediscussthew , t firsthidden layer h1 to all the following layers L = a andc indetail. t t w 2,3,4,5. Type 2 and Type 3 are composed of the Word Representations. Allwords inthe vocabu- shortcutblockswithspan1and2,respectively. Type laryshare acommonlook-up table, whichisinitial- 4andType5arethenestedshortcutblocks. Wewish ized with random initializations or pre-trained em- to find an optimal composition to pass information beddings. Each word in a sentence can be mapped indeepstackedmodels. to an embedding vector w . The whole sentence is t 4 Neural Architecture forsequence then represented by a matrix with columns vector Tagging [w1,w2,...,wT]. Following Wu et al. (2016a), we use a context window of size d surrounding with Sequence tagging can be formulated as P(t|w;θ), a word w to get its context information. Rather, t where w = [w ,...,w ] indicates the T words in 1 T we add logistic gates to each token in the con- a sentence, and t = [t ,...,t ] indicates the cor- 1 T text window. The word representation is computed responding T tags. In this section we introduce an as w = [r w ;...;r w ], t t d/2 t d/2 t+ d/2 t+ d/2 neuralarchitectureforP(·),whichincludesaninput where r :=−⌊ [r⌋ −⌊ ,.⌋..,r ⌊ ] ⌋∈ ⌊Rd ⌋is t t d/2 t+ d/2 layer, a stacked hidden layers and an output layer. −⌊ ⌋ ⌊ ⌋ a logistic gate to filter the unnecessary contexts, Since the stacked hidden layers have already been w ,...,w is the word embeddings in t d/2 t+ d/2 introduced in the previous section, we only intro- −⌊ ⌋ ⌊ ⌋ thelocalwindow. ducetheinputandtheoutputlayerhere. Capitalization Representations. We lowercase 4.1 NetworkInputs the words to decrease the size of word vocabulary Networkinputs arethe representation ofeach token to reduce sparsity, but we need an extra capitaliza- inasequence. Therearemanykindsoftokenrepre- tion embeddings to store the capitalization features, sentations, such as using a single word embedding, whichrepresentwhetherornotawordiscapitalized. l=1 l=1 l=1 l=1 l=1 l=2 l=2 l=2 l=2 l=2 l=3 l=3 l=3 l=3 l=3 l=4 l=4 l=4 l=4 l=4 l=5 l=5 l=5 l=5 l=5 Type1 Type2 Type3 Type4 Type5 −l l−2 Figure1: Compositionsofshortcutblocks.Wecallashortcutblockwithspan1whenht :=ht . Character Representations. We concate- of rules. Detailed explanations of CCG refer to nate character embeddings in a word to get the (Steedman,2000;SteedmanandBaldridge, 2011). character-level representation. Concretely, given Another similar task is POS tagging, in which a word w consisting of a sequence of characters the tags are part of speeches. Technically, the two [c ,c ,...,c ], where l is the length of the kinds of tags classify the words in different ways: 1 2 lw w word and L(·) is the look-up table for characters. CCG tags implicate the semantics of words, while We concatenate the leftmost most 5 character the POS tags represent the syntax of words. Al- embeddings L(c ),...,L(c ) with its rightmost 5 though these distinctions are important in linguis- 1 5 character embeddings L(c ),...,L(c ) to get tics, here we all treat them as the activations of lw 4 lw − c . When a word is less than five characters, we neurons, usingdistributed representations toencode w pad the remaining characters with the same special these tags. This high-level abstraction greatly im- symbol. proves the generalization, and heavily reduces the costofthemodelredesign. 4.2 NetworkOutputs For sequence tagging, we use a softmax activation 5.1.1 DatasetandPre-processing function g(·)intheoutputlayer: Our experiments are performed on CCG- −→ ←− Bank (Hockenmaier andSteedman,2007), y = g(Why[h ;h ]) (18) t t t which is a translation from Penn Treebank (Marcusetal.,1993) to CCG with a coverage where y is a probability distribution over all possi- t 99.4%. Wefollowthestandardsplits,usingsections ble tags. y (t) = exp(hk) isthe k-th dimension k Pk′exp(hk′) 02-21 for training, section 00 for development and of y , which corresponds to the k-th tag in the tag t section 23 for the test. We use a full category set set. Why isthehidden-to-output weight. containing 1285tags. Alldigitsaremappedintothe samedigit‘9’,andallwordsarelowercased. 5 Experiments 5.1 CombinatoryCategory Grammar 5.1.2 NetworkConfiguration Supertagging Initialization. There are two types of Combinatory Category Grammar (CCG) supertag- weights in our experiments: recurrent and ging is a sequence tagging problem in natural lan- non-recurrent weights. For non-recurrent guage processing. The task is to assign supertags weights, we initialize word embeddings with to each word in a sentence. In CCG the supertags the pre-trained 100-dimensional GolVe vectors stand for the lexical categories, which are com- (Pennington etal.,2014). Other weights are initial- posed of the basic categories such as N, NP and ized with the Gaussian distribution N(0, 1 ) √fan-in PP, and complex categories, which are the com- scaledbyafactorof0.1,wherefan-inisthenumber bination of the basic categories based on a set of units in the input layer. For recurrent weight matrices, following (Saxeetal.,2013) we initialize Model Dev Test with random orthogonal matrices through SVD ClarkandCurran(2007) 91.5 92.0 MLP(Lewisetal.(2014)) 91.3 91.6 to avoid unstable gradients. All bias terms are Bi-LSTM(Lewisetal.(2016)) 94.1 94.3 initialized withzerovectors. Elman-RNN(Xuetal.(2015)) 93.1 93.0 Hyperparameters. Our context window size is Bi-RNN(Xuetal.(2016)) 93.49 93.52 Bi-LSTM(Vaswanietal.(2016)) 94.24 94.5 setto3. Thedimensionofcharacterembedding and 9-stackedBi-LSTM(Wuetal.(2016b)) 94.55 94.69 capitalization embeddings are 5. Thesize of the in- 7-stackedshortcutblock(Ours) 94.74 94.95 put layer after concatenation is 465 ((word embed- 9-stacked: shortcutblock(Ours) 94.82 94.99 ding100+capembedding5+characterembedding 11-stacked: shortcutblock(Ours) 94.66 94.86 50) × window size 3). The number of cells of the 13-stacked: shortcutblock(Ours) 94.73 94.97 stacked bidirectional LSTM is also set to 465 for Table1:1-bestsupertaggingaccuracyonCCGbank orthogonal initialization. All stacked hidden layers havethesamenumberofcells. Theoutputlayerhas 1286neurons,whichequalstothenumberoftagsin different depths. Wepresent experiments trained on thetrainingsetwitha RAREsymbol. the training set and evaluated on the test set using the highest 1-best supertagging accuracy on the de- Training. We train the networks using the back- velopmentset. propagation algorithm,usingstochastic gradientde- Our 9-stacked model presents state-of-the-art re- scent (SGD) algorithm with an initial learning rate sults (94.99 on test set) comparing with other sys- 0.02. The learning rate is then scaled by 0.5 when tems. Notice that 9 is the number of stacked Bi- thefollowingcondition satisfied: LSTM layers. The total layer of the networks con- |ep −ec| tains 11 (9 + 1 input-to-hidden layer + 1 hidden- <= 0.005andlr >= 0.0005 e to-output layer) layers. We find the network with p stacked depth 7 or 9 achieves better performance where e is the error rate on the validation set on p than depth 11 or 13, but the difference is tiny. Our the previous epoch. e is the error rate on the cur- c stackedmodelsfollowEq. (11)andgatingfunctions rent epoch. The explanation of the rule is when the refertoEq. (13). growth of the performance become lower, we need to use a smaller learning rate to adjust the weights. 5.1.4 ExplorationofShortcuts We use on-line learning in our experiments, which Togetabetterunderstandingoftheshortcutarchi- meanstheparameterswillbeupdatedoneverytrain- tecture proposed in Eq. (7), we experiment with its ingsequences, oneatatime. variants to compare the performance. Our analysis Regularization. Dropout (Srivastavaetal.,2014) mainlyfocusesonthreeparts: thetopologyofshort- is the only regularizer in our model to avoid over- cutblocks, the gatingmechanism, and theircompo- fitting. Otherregularization methodssuchasweight sitions. The default number of the stacked layers is decay and batch normalization do not work in our 7. We also use the shared gates and Type 2’s archi- experiments. We add a binary dropout mask to the tecture as our default configurations, which are de- local context windows with a drop rate p of 0.25. scribed inEq. (11). Thecomparison issummarized Wealso apply dropout tothe output of the first hid- asfollows: den layer and the last hidden layer, with a 0.5 drop Shortcut Topologies. Table 2 shows the compar- rate. At test time, weights are scaled with a factor ison of shortcut topologies. Here we design two 1−p. kinds of models for comparison: one is both cl and t 5.1.3 ComparisonwithOtherSystems hl connected through shortcuts (Case 1), the other t Table 1 shows the comparison with other models is using m to replace cl (Case 2), as defined in the t for supertagging. The comparison does not include shortcutblock. Wefindtheskipconnections toboth any externally labeled data or POS tags. We eval- the internal states and the cell outputs with multi- uate the models composed of shortcut blocks with plicative gating achieves the highest accuracy (case Case Variant Dev Test hltupdated(Wuetal.,2016b) withgate: hlt =h˜lt+g⊙hlt−2 94.51 94.67 nogate: clt =c˜lt+hlt−2,hlt =h˜lt+hlt−2 93.84 93.84 withgate: clt =c˜lt+g⊙hlt−2,hlt =˜hlt+g⊙hlt−2 94.72 95.08 bothcltandhltupdated(Case1) highwaygate: hcltlt==((11−−gg))⊙⊙c˜h˜ltlt++gg⊙⊙hhlt−lt−22 94.49 94.62 shortcutsforbothcl andhl: clt =c˜lt+gc⊙clt−2 94.72 94.98 t t hlt =h˜lt+gh⊙hlt−2 nogateinhlt:hlt =o⊙tanh(m)+hlt−2 94.15 94.29 nogateinm:m=i⊙slt+hlt−2 94.77 94.97 shortcutblock(Case2) sharegate:hlt =o⊙tanh(m)+o⊙hlt−2 94.68 94.83 noshortcutininternal: hlt =o⊙tanh(it⊙st)+g⊙hlt−2 93.83 94.01 noshortcutincelloutput: hl =o⊙tanh(m) 93.58 93.82 t Table 2: Comparsionof shortcuttopologies. We use ˜hl to representthe originalcelloutputof LSTM block, which t equalso⊙tanh(clt),similartoc˜lt :=i⊙slt+f ⊙ct−1. 1,95.08%)onthetestset. Butcase2cangetabetter Type Dev Test validationaccuracy(94.77%). Weprefertousecase 1 94.22 94.38 2sinceitgeneralizes wellandmucheasiertotrain. 2 94.79 94.94 3 94.53 94.80 5.1.5 ComparisonofGatingFunctions 4 94.55 94.70 5 94.76 94.95 Weexperiment withseveral gatingfunctions proposed in Section 3.3. Detailed discussions are de- Table 4: Comparsion of shortcut block combinations. scribedbelow. Densecompositions(Type2and5)performsbetterthan sparseones. Identity Mapping. We use the tanh function to the previous outputs to break the identity link. The result is 94.81% (Table 3), which is poorer than the 5.1.7 ComparisonofHyper-parameters identityfunction. Wecaninferthattheidentityfunc- tionismoresuitablethanotherscaledfunctionssuch assigmoidortanhtotransmitinformation. AsdescribedinSection4.1,weuseacomplexin- put encoding for our model. Concretely, we use a Exclusive Gating. We find deterministic gates context window approach, together with character- performs better than stochastic gates. Further, non- level information to get a better representation for linear mapping gl = σ(Wlhl 1) achieves the best t t− the raw input. We give comparisons for the system test accuracy (Table 3, 94.79%), while other types with/without thisapproaches whilekeeping thehid- such as linear or stochastic gates are not generalize denandtheoutputpartsunchanged. well. Table5showstheeffectsofthehyper-parameters 5.1.6 ComparisonofShortcutBlock onthetask. Wefindthatthemodeldoesnotperform Compositions well(94.06%)withoutusinglocalcontextwindows. We experiment with several kinds of composi- Although LSTMs can memorize recent inputs for a tions of shortcut blocks, as shown in Table 4. We long time, it is still necessary to use a convolution- find that shortcut block with span 1 (Type 2 and 5) like operator to convolve the input tokens to get a performbetterthanotherspans(Type1,3and4). In better representation. Character-level information experiments, we use Type 2 as our default configu- also plays an important role for this task (13% rela- ration since it is much easier to compute than Type tively improvement), but the performance would be 5. heavily damagedifusingcharacters only. Case Variant Dev Test scaledmapping replacehtl−2withtanh(htl−2) 94.60 94.81 linearmapping gtl⊙h−tl=wl⊙h−tl 92.07 92.15 gll=σ(Wlhtl−1) 94.79 94.91 non-linearmapping gl=σ(Ulhl ) 94.21 94.56 l t 1 gll=σ(Vlhtl−−2) 94.60 94.78 gl∼Bernoulli(p),p=0.5 91.12 91.47 stochasticsampling t gtl∼Bernoulli(p),p=σ(Hlhtl−1) 93.90 94.06 Table3:Comparsionofgatingfunctions.Thenon-linearmappinggl =σ(Wlhl−1)isthepreferredchoice. l t Case Variant Dev Test Model Test k =0 93.96 94.06 Søgaard(2011) 97.5 windowsize k =5 94.27 94.81 Lingetal. (2015) 97.36 k =7 94.52 94.71 Wangetal. (2015) 97.78 characteronly 92.17 93.0 Vaswanietal. (2016) 97.4 l =0 93.59 93.71 Wuetal. (2016b) 97.48 character-level w l =3 94.21 94.41 7-stackedmixed+non-lineargate 97.48 w l =7 94.43 94.75 9-stackedmixed+non-lineargate 97.53 w 13-stackedmixed+non-lineargate 97.51 Table5: Comparsionofhyper-parameters Table6:AccuracyforPOStaggingonWSJ 5.2 Part-of-SpeechTagging Part-of-speech tagging is another sequence tagging useofskipconnections instacked RNNs. However, task, which is to assign POS tags to each word in a the researchers have paid less attention to the anal- sentence. Itisvery similar tothe supertagging task. ysis of various kinds of skip connections, which is Therefore,thesetwotaskscanbesolvedinaunified ourfocusinthispaper. architecture. ForPOStagging, weusethesamenet- Recently, deep stacked networks have been work configurations as supertagging, except for the widelyusedforapplications. Srivastavaetal. (2015) word vocabulary size and the tag set size. We con- and He et al. (2015) mainly focus on feed-forward duct experiments on the Wall Street Journal of the neural network, using well-designed skip connec- Penn Treebank dataset, adopting the standard splits tions across different layers to make the informa- (sections 0-18 for the train, sections 19-21 for vali- tion pass more easily. The Grid LSTM proposed dationandsections 22-24fortesting). by Kalchbrenner et al. (2015) extends the one di- Although thePOStagging result presented inTa- mensional LSTMs to many dimensional LSTMs, ble 6 is slightly below the state-of-the-art, we nei- which provides a more general framework to con- ther doany hyper-parameter tunings nor change the structdeepLSTMs. network architectures, just use the one getting the Yao et al. (2015) and Zhang et al. (2016) pro- best test accuracy on the supertagging task. This pose highway LSTMs by introducing gated direct proves the generalization of the model and avoids connections between internal states in adjacent lay- heavyworkofmodelre-designing. ers. Zilly et al. (2016) introduce recurrent highway networks(RHNs)whichuseasinglerecurrent layer 6 Related Work to make RNN deep in a vertical direction. These Skip connections have been widely used for train- works do not use skip connections, and the hier- ing deep neural networks. For recurrent neural net- archical structure is reflected in the LSTM internal works, Schmidhuber (1992); El Hihi and Bengio states or cell outputs. Wu et al. (2016b) propose a (1995)introducedeepRNNsbystackinghiddenlay- similar architecture for the shortcuts in stacked Bi- ersontopofeachother. Raikoetal. (2012);Graves LSTMs. The difference is we propose generalized (2013);HermansandSchrauwen(2013)proposethe shortcut block architectures as basic units for constructing deep stacked models. We also discuss the Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian compositions oftheseblocks. Sun. 2015. Deepresiduallearningforimagerecogni- tion. arXivpreprintarXiv:1512.03385. There are also some works using stochastic gates to transmit the information. Zoneout (2016) pro- Michiel Hermans and Benjamin Schrauwen. 2013. Training and analysing deep recurrent neural net- vides a stochastic link between the previous hid- works. InAdvancesinNeuralInformationProcessing den states and the current states, forcing the current Systems,pages190–198. statestomaintaintheirpreviousvaluesduringthere- Sepp Hochreiter and Ju¨rgen Schmidhuber. 1997. Lstm currentstep. Chungetal. (2016)proposesastochas- can solve hard long time lag problems. Advances tic boundary state to update the internal states and inneuralinformationprocessingsystems,pages473– cell outputs. These stochastic connections are con- 479. nected between adjacent layers, whileour construc- Julia Hockenmaier and Mark Steedman. 2007. Ccg- tionsoftheshortcutsaremostlycross-layered. Also, bank: a corpus of CCG derivations and dependency the updating mechanisms of LSTM blocks are dif- structuresextractedfromthe penntreebank. Compu- ferent. tationalLinguistics,33(3):355–396. GaoHuang,YuSun,ZhuangLiu,DanielSedra,andKil- 7 Conclusions ianWeinberger. 2016. Deepnetworkswithstochastic depth. arXivpreprintarXiv:1603.09382. Inthispaper, wepropose theshortcut blockasaba- Nal Kalchbrenner, Ivo Danihelka, and Alex Graves. sic architecture for constructing deep stacked mod- 2015. Grid long short-term memory. arXiv preprint els. We compare several gating functions and find arXiv:1507.01526. that the non-linear deterministic gate performs the David Krueger,Tegan Maharaj, JańosKrama´r, Moham- mad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, best. We also find the dense compositions perform Anirudh Goyal, Yoshua Bengio, Hugo Larochelle, better than the sparse ones. These explorations can AaronCourville, etal. 2016. Zoneout: Regularizing help us to train deep stacked Bi-LSTMs success- rnnsbyrandomlypreservinghiddenactivations. arXiv fully. Based on this shortcuts structure, we achieve preprintarXiv:1606.01305. thestate-of-the-artresultsonCCGsupertaggingand Guillaume Lample, MiguelBallesteros, Sandeep Subra- comparable results on POS tagging. Our explo- manian, Kazuya Kawakami, and Chris Dyer. 2016. rations could easily be applied to other sequence Neural architectures for named entity recognition. processing problems, which can be modeled with arXivpreprintarXiv:1603.01360. RNNarchitectures. MikeLewisandMarkSteedman. 2014. ImprovedCCG parsingwith semi-supervisedsupertagging. Transac- tionsoftheAssociationforComputationalLinguistics, References 2:327–338. Mike Lewis, Kenton Lee, and Luke Zettlemoyer. 2016. Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Lstmccgparsing. InProceedingsofthe15thAnnual 2016. Hierarchical multiscale recurrent neural net- ConferenceoftheNorthAmericanChapteroftheAs- works. arXivpreprintarXiv:1609.01704. sociationforComputationalLinguistics. Stephen Clark and James R Curran. 2007. Wide- Wang Ling, Tiago Lu´ıs, Lu´ıs Marujo, Ramoń Fernan- coverageefficientstatisticalparsingwithccgandlog- dez Astudillo, Silvio Amir, Chris Dyer, Alan W linearmodels. ComputationalLinguistics,33(4):493– Black, and Isabel Trancoso. 2015. Finding func- 552. tion in form: Compositional character models for Salah El Hihi and Yoshua Bengio. 1995. Hierarchical open vocabulary word representation. arXiv preprint recurrentneuralnetworksforlong-termdependencies. arXiv:1508.02096. InNIPS,volume400,page409.Citeseer. MitchellPMarcus,MaryAnnMarcinkiewicz,andBeat- Alex Graves and Ju¨rgen Schmidhuber. 2005. Frame- rice Santorini. 1993. Buildinga largeannotatedcor- wise phoneme classification with bidirectional lstm pusofenglish:Thepenntreebank. Computationallin- and other neural network architectures. Neural Net- guistics,19(2):313–330. works,18(5):602–610. Jeffrey Pennington, Richard Socher, and Christopher D AlexGraves. 2013. Generatingsequenceswithrecurrent Manning. 2014. Glove: Globalvectorsforwordrep- neuralnetworks. arXivpreprintarXiv:1308.0850. resentation. InEMNLP,volume14,pages1532–43. Tapani Raiko, Harri Valpola, and Yann LeCun. 2012. KaishengYao,TrevorCohn,KaterinaVylomova,Kevin Deeplearningmadeeasierbylineartransformationsin Duh,andChrisDyer. 2015. Depth-gatedlstm. InPre- perceptrons. InAISTATS,volume22,pages924–932. sented at Jelinek Summer Workshop on August, vol- Andrew M Saxe, James L McClelland, and Surya Gan- ume14,page1. guli. 2013. Exact solutions to the nonlinear dynam- YuZhang,GuoguoChen,DongYu,KaishengYaco,San- ics of learning in deep linear neuralnetworks. arXiv jeev Khudanpur, and James Glass. 2016. Highway preprintarXiv:1312.6120. longshort-termmemoryrnnsfordistantspeechrecog- nition. In 2016 IEEE International Conference on Ju¨rgenSchmidhuber. 1992. Learningcomplex,extended Acoustics, Speech and Signal Processing (ICASSP), sequencesusingtheprincipleofhistorycompression. pages5755–5759.IEEE. NeuralComputation,4(2):234–242. Julian Georg Zilly, Rupesh Kumar Srivastava, Jan AndersSøgaard. 2011. Semisupervisedcondensednear- Koutn´ık, and Ju¨rgen Schmidhuber. 2016. Recurrent est neighbor for part-of-speech tagging. In Proceed- highwaynetworks. arXivpreprintarXiv:1607.03474. ingsofthe49thAnnualMeetingoftheAssociationfor Computational Linguistics: Human Language Tech- nologies: shortpapers-Volume2,pages48–52.Asso- ciationforComputationalLinguistics. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research,15(1):1929–1958. Rupesh Kumar Srivastava, Klaus Greff, and Ju¨rgen Schmidhuber. 2015. Highway networks. arXiv preprintarXiv:1505.00387. Mark Steedman and Jason Baldridge. 2011. Combina- tory categorialgrammar. Non-TransformationalSyn- tax: FormalandExplicitModelsof Grammar.Wiley- Blackwell. Mark Steedman. 2000. The syntactic process, vol- ume24. MITPress. Ashish Vaswani, Yonatan Bisk, Kenji Sagae, and Ryan Musa. 2016. Supertaggingwithlstms. InProceedings oftheHumanLanguageTechnologyConferenceofthe NAACL. PeiluWang,YaoQian,FrankKSoong,LeiHe, andHai Zhao. 2015. Part-of-speech tagging with bidirectional long short-term memory recurrent neural network. arXivpreprintarXiv:1510.06168. Huijia Wu, Jiajun Zhang, and ChengqingZong. 2016a. A dynamic window neuralnetwork for ccg supertagging. arXivpreprintarXiv:1610.02749. Huijia Wu, Jiajun Zhang, and ChengqingZong. 2016b. An empirical exploration of skip connections for se- quentialtagging. arXivpreprintarXiv:1610.03167. Wenduan Xu, Michael Auli, and Stephen Clark. 2015. CCG supertagging with a recurrent neural network. Volume2: ShortPapers,page250. Wenduan Xu, Michael Auli, and Stephen Clark. 2016. Expected f-measure training for shift-reduce parsing with recurrent neural networks. In Proceedings of NAACL-HLT,pages210–220.