Table Of Content

Combining Language and Vision with a Multimodal Skip-gram Model AngelikiLazaridou NghiaThePham MarcoBaroni CenterforMind/BrainSciences UniversityofTrento {angeliki.lazaridou|thenghia.pham|marco.baroni}@unitn.it 5 1 0 2 r Abstract tion has led to the development of multimodal dis- a M tributionalsemanticmodels(MDSMs)(Brunietal., WeextendtheSKIP-GRAMmodelofMikolov 2014; Feng and Lapata, 2010; Silberer and Lapata, 2 etal.(2013a)bytakingvisualinformationinto 2014), that enrich linguistic vectors with perceptual 1 account. Like SKIP-GRAM, our multimodal information, most often in the form of visual fea- ] models(MMSKIP-GRAM)buildvector-based turesautomaticallyinducedfromimagecollections. L word representations by learning to predict C linguistic contexts in text corpora. However, MDSMs outperform state-of-the-art text-based . for a restricted set of words, the models are approaches, not only in tasks that directly require s c also exposed to visual representations of the access to visual knowledge (Bruni et al., 2012), but [ objects they denote (extracted from natural also on general semantic benchmarks (Bruni et al., 3 images),andmustpredictlinguisticandvisual 2014;SilbererandLapata,2014). However,current v features jointly. The MMSKIP-GRAM mod- MDSMs still have a number of drawbacks. First, 8 els achieve good performance on a variety of 9 semantic benchmarks. Moreover, since they they are generally constructed by first separately 5 propagatevisualinformationtoallwords,we building linguistic and visual representations of the 2 0 use them to improve image labeling and re- same concepts, and then merging them. This is ob- . trieval in the zero-shot setup, where the test viouslyverydifferentfromhowhumanslearnabout 1 conceptsareneverseenduringmodeltraining. 0 concepts, by hearing words in a situated perceptual 5 Finally,theMMSKIP-GRAMmodelsdiscover context. Second,MDSMsassumethatbothlinguis- 1 intriguingvisualpropertiesofabstractwords, ticandvisualinformationisavailableforallwords, : pavingthewaytorealisticimplementationsof v with no generalization of knowledge across modal- i embodiedtheoriesofmeaning. X ities. Third, because of this latter assumption of r fulllinguisticandvisualcoverage,currentMDSMs, a 1 Introduction paradoxically,cannotbeappliedtocomputervision Distributional semantic models (DSMs) derive tasks such as image labeling or retrieval, since they vector-based representations of meaning from pat- do not generalize to images or words beyond their ternsofwordco-occurrenceincorpora. DSMshave trainingset. been very effectively applied to a variety of seman- We introduce the multimodal skip-gram models, tictasks(Clark,2015;Mikolovetal.,2013b;Turney two new MDSMs that address all the issues above. and Pantel, 2010). However, compared to human Themodelsbuildupontheveryeffectiveskip-gram semantic knowledge, these purely textual models, approach of Mikolov et al. (2013a), that constructs just like traditional symbolic AI systems (Harnad, vectorrepresentationsbylearning,incrementally,to 1990;Searle,1984),areseverelyimpoverished,suf- predictthelinguisticcontextsinwhichtargetwords feringoflackofgroundinginextra-linguisticmodal- occur in a corpus. In our extension, for a subset ities(GlenbergandRobertson,2000). Thisobserva- of the target words, relevant visual evidence from natural images is presented together with the cor- from early-acquired concrete words to a larger vo- puscontexts(justlikehumanshearwordsaccompa- cabulary. However, they use subject-generated fea- nied by concurrent perceptual stimuli). The model tures as surrogate for realistic perceptual informa- must learn to predict these visual representations tion, and only test the model in small-scale simula- jointly with the linguistic features. The joint objec- tions of word learning. Hill and Korhonen (2014), tive encourages the propagation of visual informa- whose evaluation focuses on how perceptual infor- tion to representations of words for which no direct mation affects different word classes more or less visualevidencewasavailableintraining. Theresult- effectively, similarly to Howell et al., integrate per- ingmultimodally-enhancedvectorsachieveremark- ceptualinformationintheformofsubject-generated ably good performance both on traditional seman- featuresandtextfromimageannotationsintoaskip- tic benchmarks, and in their new application to the gram model. They inject perceptual information “zero-shot” image labeling and retrieval scenario. by merging words expressing perceptual features Very interestingly, indirect visual evidence also af- with corpus contexts, which amounts to linguistic- fectstherepresentationofabstractwords,pavingthe context re-weighting, thus making it impossible to waytoground-breakingcognitivestudiesandnovel separate linguistic and perceptual aspects of the in- applicationsincomputervision. duced representation, and to extend the model with non-linguisticfeatures. Weuseinsteadauthenticim- 2 RelatedWork ageanalysisasproxytoperceptualinformation,and we design a robust way to incorporate it, easily ex- There is by now a large literature on multimodal tendible to other signals, such as feature norm or distributional semantic models. We focus here on brainsignalvectors(Fysheetal.,2014). a few representative systems. Bruni et al. (2014) The recent work on so-called zero-shot learning propose a straightforward approach to MDSM in- to address the annotation bottleneck in image la- duction,wheretext-andimage-basedvectorsforthe beling (Frome et al., 2013; Lazaridou et al., 2014; samewordsareconstructedindependently,andthen Socher et al., 2013) looks at image- and text-based “mixed” by applying the Singular Value Decompo- vectorsfromadifferentperspective. Insteadofcom- sition to their concatenation. An empirically supe- bining visual and linguistic information in a com- rior model has been proposed by Silberer and La- mon space, it aims at learning a mapping from pata (2014), who use more advanced visual repre- image-totext-basedvectors. Themapping,induced sentations relying on images annotated with high- from annotated data, is then used to project images level “visual attributes”, and a multimodal fusion of objects that were not seen during training onto strategy based on stacked autoencoders. Kiela and linguisticspace,inordertoretrievethenearestword Bottou (2014) adopt instead a simple concatena- vectors as labels. Multimodal word vectors should tion strategy, but obtain empirical improvements by be better-suited than purely text-based vectors for usingstate-of-the-artconvolutionalneuralnetworks thetask,astheirsimilaritystructureshouldbecloser to extract visual features, and the skip-gram model tothatofimages. However,traditionalMDSMscan- for text. These and related systems take a two- notbeusedinthissetting,becausetheydonotcover stage approach to derive multimodal spaces (uni- wordsforwhichnomanuallyannotatedtrainingim- modal induction followed by fusion), and they are ages are available, thus defeating the generalizing only tested on concepts for which both textual and purpose of zero-shot learning. We will show be- visual labeled training data are available (the pio- low that our multimodal vectors, that are not ham- neering model of Feng and Lapata (2010) did learn pered by this restriction, do indeed bring a signifi- from text and images jointly using Topic Models, cant improvement over purely text-based linguistic but was shown to be empirically weak by Bruni et representationsinthezero-shotsetup. al.(2014)). Multimodal language-vision spaces have also Howelletal.(2005)proposeanincrementalmul- been developed with the goal of improving cap- timodal model based on simple recurrent networks tion generation/retrieval and caption-based image (Elman, 1990), focusing on grounding propagation retrieval (Karpathy et al., 2014; Kiros et al., 2014; Mao et al., 2014; Socher et al., 2014). These methods rely on necessarily limited collections of cap- tioned images as sources of multimodal evidence, = + whereasweautomaticallyenrichaverylargecorpus the cute little sat on the mat CAT with images to induce general-purpose multimodal word representations, that could be used as input maximize context prediction maximize similarity embeddingsinsystemsspecificallytunedtocaption processing. Thus,ourworkiscomplementarytothis lineofresearch. map to visual space cat 3 MultimodalSkip-gramArchitecture Figure 1: “Cartoon” of MMSKIP-GRAM-B. Lin- 3.1 Skip-gramModel guistic context vectors are actually associated to We start by reviewing the standard SKIP-GRAM classes of words in a tree, not single words. SKIP- model of Mikolov et al. (2013a), in the version GRAM is obtained by ignoring the visual objective, we use. Given a text corpus, SKIP-GRAM aims MMSKIP-GRAM-A byfixingMu→v totheidentity at inducing word representations that are good at matrix. predicting the context words surrounding a target word. Mathematically, it maximizes the objective visual representation of the concepts they denote function: (just like in a conversation, where a linguistic   T 1 (cid:88) (cid:88) utterance will often be produced in a visual scene T  logp(wt+j|wt) (1) including some of the word referents). The visual t=1 −c≤j≤c,j(cid:54)=0 representation is also encoded in a vector (we where w1,w2,...,wT are words in the training describe in Section 4 below how we construct corpus and c is the size of the window around it). We thus make the skip-gram “multimodal” by target wt, determining the set of context words to adding a second, visual term to the original linguis- be predicted by the induced representation of wt. ticobjective,thatis,weextendEquation1asfollow: Following Mikolov et al., we implement a subsam- T 1 (cid:88) pling option randomly discarding context words as (L (w )+L (w )) (3) ling t vision t T aninversefunctionoftheirfrequency,controlledby t=1 hyperparameter t. The probability p(wt+j|wt), the where Lling(wt) is the text-based skip-gram ob- core part of the objective in Equation 1, is given by jective (cid:80) logp(w |w ), whereas the −c≤j≤c,j(cid:54)=0 t+j t softmax: L (w )termforceswordrepresentationstotake vision t eu(cid:48)wt+jTuwt visualinformationintoaccount. Notethatifaword p(wt+j|wt) = (cid:80)Ww(cid:48)=1eu(cid:48)w(cid:48)Tuwt (2) wsytstiesmnaotitcaalslysotchiaetecdasteo, ev.igs.u,alfoirnfdoertmeramtiionne,rsasanids where u and u(cid:48) are the context and target vector non-imageable nouns, but also more generally for w w representations of word w respectively, and W is any word for which no visual data are available, the size of the vocabulary. Due to the normaliza- L (w )issetto0. vision t tion term, Equation 2 requires O(|W|) time com- Wenowproposetwovariantsofthevisualobjec- plexity. A considerable speedup to O(log|W|), is tive,resultingintwodistinguishedmulti-modalver- achieved by using the hierarchical version of Equa- sionsoftheskip-grammodel. tion2 (MorinandBengio,2005),adoptedhere. 3.3 Multi-modalSkip-gramModelA 3.2 Injectingvisualknowledge One way to force word embeddings to take visual We now assume that word learning takes place in a representations into account is to try to directly situated context, in which, for a subset of the target increase the similarity (expressed, for example, words, the corpus contexts are accompanied by a by the cosine) between linguistic and visual representations, thus aligning the dimensions of the OurtextcorpusisaWikipedia2009dumpcompris- linguistic vector with those of the visual one (recall ingapproximately800Mtokens.1 Totrainthemulti- that we are inducing the first, while the second is modal models, we add visual information for 5,100 fixed),andmakingthelinguisticrepresentationofa words that have an entry in ImageNet (Deng et al., concept “move” closer to its visual representation. 2009), occur at least 500 times in the corpus and We maximize similarity through a max-margin have concreteness score ≥ 0.5 according to Turney framework commonly used in models connecting et al. (2011). On average, about 5% tokens in the language and vision (Weston et al., 2010; Frome et textcorpusareassociatedtoavisualrepresentation. al., 2013). More precisely, we formulate the visual Toconstructthevisualrepresentationofaword,we objectiveL (w )as: sample 100 pictures from its ImageNet entry, and vision t − (cid:88) max(0,γ−cos(uwt,vwt)+cos(uwt,vw(cid:48))) (4) extracta4096-dimensionalvectorfromeachpicture w(cid:48)∼Pn(w) using the Caffe toolkit (Jia et al., 2014), together where the minus sign turns a loss into a cost, γ is withthepre-trainedconvolutionalneuralnetworkof themargin,uwt isthetargetmultimodally-enhanced Krizhevsky et al. (2012). The vector corresponds word representation we aim to learn, vwt is the cor- to activation in the top (FC7) layer of the network. responding visual vector (fixed in advance) and vw(cid:48) Finally, we average the vectors of the 100 pictures ranges over visual representations of words (fea- associated to each word, deriving 5,100 aggregated tured in our image dictionary) randomly sampled visualrepresentations. fromdistributionP (w ). Theserandomvisualrep- n t resentations act as “negative” samples, encouraging Hyperparameters For both SKIP-GRAM and the u to be more similar to its own visual representa- MMSKIP-GRAM models, we fix hidden layer size wt tionthantothatofotherwords. Thesamplingdistri- to300. TofacilitatecomparisonbetweenMMSKIP- butioniscurrentlysettouniform,andthenumberof GRAM-AandMMSKIP-GRAM-B,andsincethefor- negativesamplescontrolledbyhyperparameterk. mer requires equal linguistic and visual dimension- ality, we keep the first 300 dimensions of the visual 3.4 Multi-modalSkip-gramModelB vectors. For the linguistic objective, we use hierar- The visual objective in MMSKIP-GRAM-A has the chicalsoftmaxwithaHuffmanfrequency-baseden- drawback of assuming a direct comparison of lin- coding tree, setting frequency subsampling option guisticandvisualrepresentations,constrainingthem t=0.001 and window size c=5, without tuning. to be of equal size. MMSKIP-GRAM-B lifts this The following hyperparameters were tuned on the constraintbyincludinganextralayermediatingbe- text9 corpus:2 MMSKIP-GRAM-A: k=20, γ=0.5; tweenlinguisticandvisualrepresentations(seeFig- MMSKIP-GRAM-B:k=5,γ=0.5,λ=0.0001. ure1forasketchof MMSKIP-GRAM-B).Learning 5 Experiments this layer is equivalent to estimating a cross-modal mapping matrix from linguistic onto visual repre- 5.1 Approximatinghumanjudgments sentations, jointly induced with linguistic word embeddings. Theextensionisstraightforwardlyimple- Benchmarks AwidelyadoptedwaytotestDSMs mented by substituting, into Equation 4, the word and their multimodal extensions is to measure how representation u with z = Mu→vu , where well model-generated scores approximate human wt wt wt Mu→v is the cross-modal mapping matrix to be in- similarity judgments about pairs of words. We put duced. To avoid overfitting, we also add an L2 reg- together various benchmarks covering diverse as- ularization term for Mu→v to the overall objective pects of meaning, to gain insights on the effect of (Equation3),withitsrelativeimportancecontrolled perceptualinformationondifferentsimilarityfacets. byhyperparamerλ. Specifically, we test on general relatedness (MEN, Bruni et al. (2014), 3K pairs), e.g., pickles are re- 4 ExperimentalSetup lated to hamburgers, semantic (≈ taxonomic) simi- Theparametersofallmodelsareestimatedbyback- 1http://wacky.sslmit.unibo.it propagationoferrorviastochasticgradientdescent. 2http://mattmahoney.net/dc/textdata.html larity(Simlex-999,Hilletal.(2014),1Kpairs;Sem- onwordswithoutdirectvisualrepresentations. Sim, Silberer and Lapata (2014), 7.5K pairs), e.g., pickles are similar to onions, as well as visual sim- Results The state-of-the-art visual CNN FEA- ilarity (VisSim, Silberer and Lapata (2014), same TURES aloneperformremarkablywell,outperform- pairsasSemSimwithdifferenthumanratings),e.g., ing the purely textual model (SKIP-GRAM) in two pickleslooklikezucchinis. tasks, and achieving the best absolute performance on the visual-coverage subset of Simlex-999. Re- Alternative Multimodal Models We compare garding multimodal fusion (that is, focusing on our models against several recent alternatives. We the visual-coverage subsets), both MMSKIP-GRAM test the vectors made available by Kiela and Bottou models perform very well, at the top or just below (2014). Similarly to us, they derive textual features it on all tasks, with comparable results for the two with the skip-gram model (from a portion of the variants. Their performance is also good on the WikipediaandtheBritishNationalCorpus)anduse full data sets, where they consistently outperform visual representations extracted from the ESP data- SKIP-GRAM and SVD (that is much more strongly set (von Ahn and Dabbish, 2004) through a convo- affected by lack of complete visual information). lutional neural network (Oquab et al., 2014). They They’re just a few points below the state-of-the-art concatenatetextualandvisualfeaturesafternormal- MEN correlation (0.8), achieved by Baroni et al. izingtounitlengthandcenteringtozeromean. We (2014) with a corpus 3 larger than ours and exten- alsotestthevectorsthatperformedbestintheevalu- sivetuning. MMSKIP-GRAM-B isclosetothestate ationofBrunietal.(2014),basedontextualfeatures of the art for Simlex-999, reported by the resource extracted from a 3B-token corpus and SIFT-based creators to be at 0.41 (Hill et al., 2014). Most im- Bag-of-Visual-Wordsvisualfeatures(SivicandZis- pressively, MMSKIP-GRAM-A reaches the perfor- serman, 2003) extracted from the ESP collection. manceleveloftheSilbererandLapata(2014)model Bruniandcolleaguesfuseaweightedconcatenation on their SemSim and VisSim data sets, despite the ofthetwocomponentsthroughSVD.Wefurtherre- fact that the latter has full visual-data coverage and implement both methods with our own textual and uses attribute-based image representations, requir- visual embeddings as CONCATENATION and SVD ing supervised learning of attribute classifiers, that (withtargetdimensionality300,pickedwithouttun- achieve performance in the semantic tasks compa- ing). Finally, we present for comparison the results rable or higher than that of our CNN features (see onSemSimandVisSimreportedbySilbererandLa- Table 3 in Silberer and Lapata (2014)). Finally, if pata (2014), obtained with a stacked-autoencoders themultimodalmodels(unsurprisingly)bringabout architecture run on textual features extracted from a large performance gain over the purely linguistic WikipediawiththeStrudelalgorithm(Baronietal., modelonvisualsimilarity, theimprovementiscon- 2010)andattribute-basedvisualfeatures(Farhadiet sistently large also for the other benchmarks, con- al.,2009)extractedfromImageNet. firming that multimodality leads to better semantic All benchmarks contain a fair amount of words models in general, that can help in capturing differ- forwhichwedidnotusedirectvisualevidence. We ent types of similarity (general relatedness, strictly are interested in assessing the models both in terms taxonomic,perceptual). ofhowtheyfuselinguisticandvisualevidencewhen While we defer to further work a better un- they are both available, and for their robustness in derstanding of the relation between multimodal lack of full visual coverage. We thus evaluate them grounding and different similarity relations, Table intwosettings. Thevisual-coveragecolumnsofTa- 2 provides qualitative insights on how injecting ble1(thoseontheright)reportresultsonthesubsets visual information changes the structure of se- forwhichallcomparedmodelshaveaccesstodirect mantic space. The top SKIP-GRAM neighbours of visualinformationforbothwords. Wefurtherreport donuts are places where you might encounter them, results on the full sets (“100%” columns of Table whereasthemultimodalmodelsrelatethemtoother 1) for models that can propagate visual information take-away food, ranking visually-similar pizzas at and that, consequently, can meaningfully be tested the top. The owl example shows how multimodal MEN Simlex-999 SemSim VisSim Model 100% 42% 100% 29% 100% 85% 100% 85% KIELAANDBOTTOU - 0.74 - 0.33 - 0.60 - 0.50 BRUNIETAL. - 0.77 - 0.44 - 0.69 - 0.56 SILBERERANDLAPATA - - - - 0.70 - 0.64 - CNNFEATURES - 0.62 - 0.54 - 0.55 - 0.56 SKIP-GRAM 0.70 0.68 0.33 0.29 0.62 0.62 0.48 0.48 CONCATENATION - 0.74 - 0.46 - 0.68 - 0.60 SVD 0.61 0.74 0.28 0.46 0.65 0.68 0.58 0.60 MMSKIP-GRAM-A 0.75 0.74 0.37 0.50 0.72 0.72 0.63 0.63 MMSKIP-GRAM-B 0.74 0.76 0.40 0.53 0.66 0.68 0.60 0.60 Table1: Spearmancorrelationbetweenmodel-generatedsimilaritiesandhumanjudgments. Rightcolumns reportcorrelationonvisual-coveragesubsets(percentageoforiginalbenchmarkcoveredbysubsetsonfirst row of respective columns). First block reports results for out-of-the-box models; second block for visual andtextualrepresentationsalone;thirdblockforourimplementationofmultimodalmodels. Target SKIP-GRAM MMSKIP-GRAM-A MMSKIP-GRAM-B donut fridge,diner,candy pizza,sushi,sandwich pizza,sushi,sandwich owl pheasant,woodpecker,squirrel eagle,woodpecker,falcon eagle,falcon,hawk mural sculpture,painting,portrait painting,portrait,sculpture painting,portrait,sculpture tobacco coffee,cigarette,corn cigarette,cigar,corn cigarette,cigar,smoking depth size,bottom,meter sea,underwater,level sea,size,underwater chaos anarchy,despair,demon demon,anarchy,destruction demon,anarchy,shadow Table2: Orderedtop3neighboursofexamplewordsinpurelytextualandmultimodalspaces. Onlydonut andowlweretrainedwithdirectvisualinformation. models pick taxonomically closer neighbours of over the more abstract measurement sense picked concrete objects, since often closely related things by the MMSKIP-GRAM neighbours. For chaos, also look similar (Bruni et al., 2014). In particular, theyrankademon,thatis,aconcreteagentofchaos both multimodal models get rid of squirrels and at the top, and replace the more abstract notion of offer other birds of prey as nearest neighbours. despair with equally gloomy but more imageable No direct visual evidence was used to induce the shadows and destruction (more on abstract words embeddingsoftheremainingwordsinthetable,that below). are thus influenced by vision only by propagation. The subtler but systematic changes we observe in 5.2 Zero-shotimagelabelingandretrieval such cases suggest that this indirect propagation The multimodal representations induced by our is not only non-damaging with respect to purely models should be better suited than purely text- linguistic representations, but actually beneficial. basedvectorstolabelorretrieveimages. Inparticu- For the concrete mural concept, both multimodal lar,giventhatthequantitativeandqualitativeresults models rank paintings and portraits above less collected so far suggest that the models propagate closely related sculptures (they are not a form of visual information across words, we apply them to painting). For tobacco, both models rank cigarettes imagelabelingandretrievalinthechallengingzero- and cigar over coffee, and MMSKIP-GRAM-B shotsetup(seeSection2above).3 avoids the arguably less common “crop” sense cued by corn. The last two examples show how the 3We will refer here, for conciseness’ sake, to image label- multimodal models turn up the embodiment level ing/retrieval,but,asourvisualvectorsareaggregatedrepresen- in their representation of abstract words. For depth, tationsofimages,thetaskswe’remodelingconsist,morepre- their neighbours suggest a concrete marine setup cisely,inlabelingasetofpicturesdenotingthesameobjectand retrievingthecorrespondingsetgiventhenameoftheobject. Setup We take out as test set 25% of the 5.1K P@1 P@2 P@10 P@20 P@50 words we have visual vectors for. The multimodal SKIP-GRAM 1.5 2.6 14.2 23.5 36.1 MMSKIP-GRAM-A 2.1 3.7 16.7 24.6 37.6 models are re-trained without visual vectors for MMSKIP-GRAM-B 2.2 5.1 20.2 28.5 43.5 these words, using the same hyperparameters as above. For both tasks, the search for the correct Table3: Percentageprecision@kresultsinthezero- word label/image is conducted on the whole set of shotimagelabelingtask. 5.1Kword/visualvectors. In the image labeling task, given a visual vector P@1 P@2 P@10 P@20 P@50 representing an image, we map it onto word space, SKIP-GRAM 1.9 3.3 11.5 18.5 30.4 and label the image with the word corresponding MMSKIP-GRAM-A 1.9 3.2 13.9 20.2 33.6 to the nearest vector. To perform the vision-to- MMSKIP-GRAM-B 1.9 3.8 13.2 22.5 38.3 languagemapping,wetrainaRidgeregressionby5- Table4: Percentageprecision@kresultsinthezero- foldcross-validationonthetestset(forSKIP-GRAM shotimageretrievaltask. only,wealsoaddtheremaining75%ofword-image vectorpairsusedinestimatingthemultimodalmod- elstotheRidgetrainingdata).4 embeddings we are inducing, while general enough In the image retrieval task, given a linguis- to achieve good performance in the semantic tasks tic/multimodal vector, we map it onto visual space, discussed above, encode sufficient visual informa- andretrievethenearestimage. ForSKIP-GRAM,we tion for direct application to image analysis tasks. use Ridge regression with the same training regime Thisisespeciallyremarkablebecausethewordvec- as for the labeling task. For the multimodal mod- tors we are testing were not matched with visual els, since maximizing similarity to visual represen- representations at model training time, and are thus tationsisalreadypartoftheirtrainingobjective,we multimodal only by propagation. The best perfor- donotfitanextramappingfunction. ForMMSKIP- manceisachievedby MMSKIP-GRAM-B,confirm- GRAM-A, we directly look for nearest neighbours ing our claim that its Mu→v matrix acts as a multi- of the learned embeddings in visual space. For modalmappingfunction. MMSKIP-GRAM-B, we use the Mu→v mapping functioninducedwhilelearningwordembeddings. 5.3 Abstractwords Results In image labeling (Table 3) SKIP-GRAM We have already seen, through the depth and chaos is outperformed by both multimodal models, con- examples of Table 2, that the indirect influence of firming that these models produce vectors that are visualinformationhasinterestingeffectsontherep- directly applicable to vision tasks thanks to visual resentation of abstract terms. The latter have re- propagation. The most interesting results however ceivedlittleattentioninmultimodalsemantics,with are achieved in image retrieval (Table 4), which Hill and Korhonen (2014) concluding that abstract is essentially the task the multimodal models have nouns, in particular, do not benefit from propagated been implicitly optimized for, so that they could be perceptual information, and their representation is appliedtoitwithoutanyspecifictraining. Thestrat- even harmed when such information is forced on egy of directly querying for the nearest visual vec- them (see Figure 4 of their paper). Still, embod- tors of the MMSKIP-GRAM-A word embeddings iedtheoriesofcognitionhaveprovidedconsiderable worksremarkablywell,outperformingonthehigher evidence that abstract concepts are also grounded ranks SKIP-GRAM, which requires an ad-hoc map- in the senses (Barsalou, 2008; Lakoff and John- ping function. This suggests that the multimodal son,1999). Sincethewordrepresentationsproduced by MMSKIP-GRAM-A, including those pertaining 4WeuseonefoldtotuneRidgeλ,threetoestimatethemap- to abstract concepts, can be directly used to search pingmatrixandtestinthelastfold. Toenforcestrictzero-shot for near images in visual space, we decided to ver- conditions, we exclude from the test fold labels occurring in ify,experimentally,ifthesenearimages(ofconcrete the LSVRC2012 set that was employed to train the CNN of Krizhevskyetal.(2012),thatweusetoextractvisualfeatures. things) are relevant not only for concrete words, as expected, but also for abstract ones, as predicted by global |words| unseen |words| all 48% 198 30% 127 embodiedviewsofmeaning. concrete 73% 99 53% 30 More precisely, we focused on the set of 200 abstract 23% 99 23% 97 wordsthatweresampledacrosstheUSFnormscon- creteness spectrum by Kiela et al. (2014) (2 words Table 5: Subjects’ preference for nearest visual had to be excluded for technical reasons). This neighbourofwordsinKielaetal.(2014)vs.random set includes not only concrete (meat) and abstract pictures. Figure of merit is percentage proportion (thought) nouns, but also adjectives (boring), verbs of significant results in favor of nearest neighbour (teach), and even grammatical terms (how). Some acrosswords. Resultsarereportedforthewholeset, words in the set have relatively high concreteness aswellasforwordsabove(concrete)andbelow(ab- ratings, but are not particularly imageable, e.g.: stract) the concreteness rating median. The unseen hot, smell, pain, sweet. For each word in the set, columnreportsresultswhenwordsexposedtodirect we extracted the nearest neighbour picture of its visual evidence during training are discarded. The MMSKIP-GRAM-A representation, and matched it wordscolumnsreportsetcardinality. with a random picture. The pictures were selected from a set of 5,100, all labeled with distinct words freedom theory wrong (the picture set includes, for each of the words associated to visual information as described in Sec- tion 4, the nearest picture to its aggregated visual representation). Since it is much more common for god together place concrete than abstract words to be directly repre- sented by an image in the picture set, when search- ing for the nearest neighbour we excluded the pic- turelabeledwiththewordofinterest,ifpresent(e.g., Figure 2: Examples of nearest visual neighbours of we excluded the picture labeled tree when picking some abstract words: on the left, cases where sub- the nearest neighbour of the word tree). We ran a jects preferred the neighbour to the random foil; on CrowdFlower5 survey in which we presented each theright,caseswheretheydidnot. test word with the two associated images (random- izing presentation order of nearest and random picture), and asked subjects which of the two pictures significant preference for the model-predicted near- they found more closely related to the word. We estpictureforaboutonefourthoftheabstractterms. collected minimally 20 judgments per word. Sub- Whether a word was exposed to direct visual evi- jectsshowedlargeagreement(medianproportionof denceduringtrainingisofcoursemakingabigdif- majoritychoiceat90%),confirmingthattheyunder- ference, and this factor interacts with concreteness, stoodthetaskandbehavedconsistently. as only two abstract words were matched with im- We quantify performance in terms of proportion ages during training.6 When we limit evaluation to ofwordsforwhichthenumberofvotesforthenear- word representations that were not exposed to pic- est neighbour picture is significantly above chance tures during training, the difference between con- according to a two-tailed binomial test. We set sig- crete and abstract terms, while still large, becomes nificanceatp<0.05afteradjustingallp-valueswith lessdramaticthanifallwordsareconsidered. theHolmcorrectionforrunning198statisticaltests. Figure 2 shows four cases in which subjects ex- The results in Table 5 indicate that, in about half pressed a strong preference for the nearest visual the cases, the nearest picture to a word MMSKIP- neighbour of a word. Freedom, god and theory are GRAM-A representation is meaningfully related to strikinglyinagreementwiththeview, fromembod- theword. Asexpected,thisismoreoftenthecasefor iedtheories,thatabstractwordsaregroundedinrel- concretethanabstractwords. Still,wealsoobservea 6Inbothcases,theimagesactuallydepictconcretesensesof 5http://www.crowdflower.com thewords:amemoryboardformemoryandastopsignforstop. evant concrete scenes and situations. The together Model ρ exampleillustrateshowvisualdatamightgroundab- WORDFREQUENCY 0.22 KIELAETAL. -0.65 stractnotionsinsurprisingways. Forallthesecases, SKIP-GRAM 0.05 we can borrow what Howell et al. (2005) say about MMSKIP-GRAM-B 0.04 visualpropagationtoabstractwords(p.260): MMSKIP-GRAM-A -0.75 MMSKIP-GRAM-B* -0.71 Intuitively, thisissomethingliketryingtoexplain an abstract concept like love to a child by using (a) (b) concrete examples of scenes or situations that are associatedwithlove. Theabstractconceptisnever Figure 3: (a) Distribution of MMSKIP-GRAM-A fullygroundedinexternalreality,butitdoesinherit vectoractivationformeat (blue)andhope(red). (b) somemeaningfromthemoreconcreteconceptsto Spearmanρbetweenconcretenessandvariousmea- whichitisrelated. suresontheKielaetal.(2014)set. Ofcourse,notallexamplesaregood: thelastcol- umn of Figure 2 shows cases with no obvious rela- MMSKIP-GRAM-A representations and those gen- tionbetweenwordsandvisualneighbours(subjects eratedbymappingMMSKIP-GRAM-Bvectorsonto preferredtherandomimagesbyalargemargin). visual space (MMSKIP-GRAM-B*) achieve very Themultimodalvectorsweinducealsodisplayan high correlation (but, interestingly, not MMSKIP- interestingintrinsicpropertyrelatedtothehypothe- GRAM-B). This is further evidence that multimodal sis that grounded representations of abstract words learning is grounding the representations of both are more complex than for concrete ones, since ab- concreteandabstractwordsinmeaningfulways. stractconceptsrelatetovariedandcompositesitua- tions(BarsalouandWiemer-Hastings,2005). Anat- 6 Conclusion ural corollary of this idea is that visually-grounded representationsofabstractconceptsshouldbemore Weintroducedtwomultimodalextensionsof SKIP- diverse: Ifyouthinkofdogs,verysimilarimagesof GRAM. MMSKIP-GRAM-A is trained by directly optimizing the similarity of words with their visual specificdogswillcometomind. Youcanalsoimag- representations, thus forcing maximum interaction inetheabstractnotionoffreedom, butthenatureof the related imagery will be much more varied. Re- betweenthetwomodalities. MMSKIP-GRAM-Bin- cludes an extra mediating layer, acting as a cross- cently,Kielaetal.(2014)haveproposedtomeasure modalmappingcomponent. Theabilityofthemod- abstractness by exploiting this very same intuition. elstointegrateandpropagatevisualinformationre- However,theyrelyonmanualannotationofpictures sultedinwordrepresentationsthatperformedwellin via Google Images and define an ad-hoc measure bothsemanticandvisiontasks,andcouldbeusedas of image dispersion. We conjecture that the repre- inputinsystemsbenefitingfrompriorvisualknowl- sentations naturally induced by our models display edge(e.g.,captiongeneration). Ourresultswithab- a similar property. In particular, the entropy of our stract words suggest the models might also help in multimodalvectors,beinganexpressionofhowvar- tasks such as metaphor detection, or even retriev- ied the information they encode is, should correlate ing/generating pictures of abstract concepts. Their withthedegreeofabstractnessofthecorresponding incrementalnaturemakesthemwell-suitedforcog- words. As Figure 3(a) shows, there is indeed a dif- nitivesimulationsofgroundedlanguageacquisition, ferenceinentropybetweenthemostconcrete(meat) anavenueofresearchweplantoexplorefurther. andmostabstract(hope)wordsintheKielaetal.set. To test the hypothesis quantitatively, we mea- Acknowledgments sure the correlation of entropy and concreteness on the 200 words in the Kiela et al. (2014) set.7 We thank Adam Liska, Tomas Mikolov, the re- Figure 3(b) shows that the entropies of both the viewersandtheNIPS2014LearningSemanticsau- dience. We were supported by ERC 2011 Start- 7Since the vector dimensions range over the real number ing Independent Research Grant n. 283554 (COM- line, we calculate entropy on vectors that are unit-normed af- teraddingasmallconstantinsuringallvaluesarepositive. POSES). References [Fysheetal.2014] AlonaFyshe,ParthaPTalukdar,Brian Murphy, and Tom M Mitchell. 2014. Interpretable [Baronietal.2010] Marco Baroni, Eduard Barbu, Brian semanticvectorsfromajointmodelofbrain-andtext- Murphy, and Massimo Poesio. 2010. Strudel: A basedmeaning. InInProceedingsofACL,pages489– distributionalsemanticmodelbasedonpropertiesand 499. types. CognitiveScience,34(2):222–254. [GlenbergandRobertson2000] Arthur Glenberg and [Baronietal.2014] Marco Baroni, Georgiana Dinu, and David Robertson. 2000. Symbol grounding and Germań Kruszewski. 2014. Don’t count, pre- meaning: A comparison of high-dimensional and dict! asystematiccomparisonofcontext-countingvs. embodied theories of meaning. Journal of Memory context-predicting semantic vectors. In Proceedings andLanguage,3(43):379–401. ofACL,pages238–247,Baltimore,MD. [Harnad1990] StevanHarnad. 1990. Thesymbolground- [BarsalouandWiemer-Hastings2005] Lawrence Barsa- ingproblem. PhysicaD:NonlinearPhenomena,42(1- lou and Katja Wiemer-Hastings. 2005. Situating 3):335–346. abstractconcepts. InD.PecherandR.Zwaan,editors, [HillandKorhonen2014] FelixHillandAnnaKorhonen. Grounding Cognition: The Role of Perception and 2014. Learning abstract concept embeddings from Action in Memory, Language, and Thought, pages multi-modal data: Since you probably can’t see what 129–163. Cambridge University Press, Cambridge, I mean. In Proceedings of EMNLP, pages 255–265, UK. Doha,Qatar. [Barsalou2008] Lawrence Barsalou. 2008. Grounded [Hilletal.2014] Felix Hill, Roi Reichart, and Anna Ko- cognition. AnnualReviewofPsychology,59:617–645. rhonen. 2014. SimLex-999: Evaluating se- [Brunietal.2012] EliaBruni,GemmaBoleda,MarcoBa- mantic models with (genuine) similarity estimation. roni, and Nam Khanh Tran. 2012. Distributional se- http://arxiv.org/abs/arXiv:1408.3456. manticsinTechnicolor. InProceedingsofACL,pages [Howelletal.2005] Steve Howell, Damian Jankowicz, 136–145,JejuIsland,Korea. and Suzanna Becker. 2005. A model of grounded [Brunietal.2014] Elia Bruni, Nam Khanh Tran, and language acquisition: Sensorimotor features improve Marco Baroni. 2014. Multimodal distributional se- lexicalandgrammaticallearning. JournalofMemory mantics. Journal of Artificial Intelligence Research, andLanguage,53:258–276. 49:1–47. [Jiaetal.2014] Yangqing Jia, Evan Shelhamer, Jeff Don- [Clark2015] Stephen Clark. 2015. Vector space mod- ahue,SergeyKarayev,JonathanLong,RossGirshick, els of lexical meaning. In Shalom Lappin and SergioGuadarrama,andTrevorDarrell. 2014. Caffe: Chris Fox, editors, Handbook of Contemporary Se- Convolutionalarchitectureforfastfeatureembedding. mantics, 2nd ed. Blackwell, Malden, MA. In arXivpreprintarXiv:1408.5093. press; http://www.cl.cam.ac.uk/˜sc609/ [Karpathyetal.2014] Andrej Karpathy, Armand Joulin, pubs/sem_handbook.pdf. andLiFei-Fei. 2014. Deepfragmentembeddingsfor [Dengetal.2009] Jia Deng, Wei Dong, Richard Socher, bidirectionalimagesentencemapping. InProceedings Lia-Ji Li, and Li Fei-Fei. 2009. Imagenet: A large- ofNIPS,pages1097–1105,Montreal,Canada. scale hierarchical image database. In Proceedings of [KielaandBottou2014] Douwe Kiela and León Bottou. CVPR,pages248–255,MiamiBeach,FL. 2014. Learning image embeddings using convolu- [Elman1990] JeffreyLElman. 1990. Findingstructurein tional neural networks for improved multi-modal se- time. Cognitivescience,14(2):179–211. mantics. In Proceedings of EMNLP, pages 36–45, [Farhadietal.2009] Ali Farhadi, Ian Endres, Derek Doha,Qatar. Hoiem,andDavidForsyth. 2009. Describingobjects [Kielaetal.2014] Douwe Kiela, Felix Hill, Anna Korho- by their attributes. In Proceedings of CVPR, pages nen, and Stephen Clark. 2014. Improving multi- 1778–1785,MiamiBeach,FL. modal representations using image dispersion: Why [FengandLapata2010] Yansong Feng and Mirella Lap- lessissometimesmore. InProceedingsofACL,pages ata. 2010. Visual information in semantic represen- 835–841,Baltimore,MD. tation. In Proceedings of HLT-NAACL, pages 91–99, [Kirosetal.2014] RyanKiros,RuslanSalakhutdinov,and LosAngeles,CA. Richard Zemel. 2014. Unifying visual-semantic em- [Fromeetal.2013] Andrea Frome, Greg Corrado, Jon beddings with multimodal neural language models. Shlens, Samy Bengio, Jeff Dean, Marc’Aurelio Ran- In Proceedings of the NIPS Deep Learning and Rep- zato, and Tomas Mikolov. 2013. DeViSE: A deep resentation Learning Workshop, Montreal, Canada. visual-semanticembeddingmodel. InProceedingsof Published online: http://www.dlworkshop. NIPS,pages2121–2129,LakeTahoe,NV. org/accepted-papers.