Loving Common Lisp, or the Savvy Programmer’s Secret Weapon

233 Pages·2020·4.914 MB·English

by  Mark Watson

Loving Common Lisp, or the Savvy
Programmer’s Secret Weapon
Mark Watson
Thisbookisforsaleathttp://leanpub.com/lovinglisp
Thisversionwaspublishedon2020-08-03
ThisisaLeanpubbook.LeanpubempowersauthorsandpublisherswiththeLeanPublishing
process.LeanPublishingistheactofpublishinganin-progressebookusinglightweighttoolsand
manyiterationstogetreaderfeedback,pivotuntilyouhavetherightbookandbuildtractiononce
youdo.
ThisworkislicensedunderaCreativeCommonsAttribution-NonCommercial-NoDerivatives4.0
InternationalLicense
Contents
CoverMaterial,Copyright,andLicense . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
NotesontheSixthEditionPublishedJune2020 . . . . . . . . . . . . . . . . . . . . . . . . . . 2
NotesontheFifthEditionPublishedSeptember2019 . . . . . . . . . . . . . . . . . . . . . . . 2
WhyUseCommonLisp? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
ARequestfromtheAuthor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
OlderBookEditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
SettingUpYourCommonLispDevelopmentSystemandQuicklisp . . . . . . . . . . . . . . 5
ListofQuicklispProjectsandSmall ExamplesinthisBook . . . . . . . . . . . . . . . . . . . 6
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
WhyDidIWritethisBook? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
FreeSoftwareToolsforCommonLispProgramming . . . . . . . . . . . . . . . . . . . . . . . 9
HowisLispDifferentfromLanguageslikeJavaandC++? . . . . . . . . . . . . . . . . . . . . 9
AdvantagesofWorkinginaLispEnvironment . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
CommonLispBasics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
GettingStartedwithSBCL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
MakingthereplNicerusingrlwrap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
TheBasicsofLispProgramming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Symbols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
OperationsonLists. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
UsingArraysandVectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
UsingStrings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
UsingHashTables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
UsingEvaltoEvaluateLispForms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
UsingaTextEditortoEditLispSourceFiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
RecoveringfromErrors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
GarbageCollection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
LoadingyourWorkingEnvironmentQuickly. . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
FunctionalProgrammingConcepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Quicklisp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
CONTENTS
UsingQuicklisptoFindPackages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
UsingQuicklisptoConfigureEmacsandSlime . . . . . . . . . . . . . . . . . . . . . . . . . . 41
DefiningLispFunctions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
UsingLambdaForms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
UsingRecursion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Closures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
UsingtheFunctioneval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
DefiningCommonLispMacros . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
ExampleMacro . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
UsingtheSplicingOperator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Usingmacroexpand-1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
UsingCommonLispLoopMacros . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
dolist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
dotimes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
do . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
UsingtheloopSpecialFormtoIterateOverVectorsorArrays . . . . . . . . . . . . . . . . . 55
CommonLispPackageSystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
InputandOutput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
TheLispreadandread-lineFunctions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
LispPrintingFunctions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
PlottingData . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
ImplementingtheLibrary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
PackagingasaQuicklispProject . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
CommonLispObjectSystem-CLOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
ExampleofUsingaCLOSClass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
ImplementationoftheHTMLstreamClass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
UsingDefstructorCLOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
HeuristicallyGuidedSearch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
NetworkProgramming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
AnintroductiontoDrakma. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
AnintroductiontoHunchentoot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
CompleteRESTClientServerExampleUsingJSONforDataSerialization . . . . . . . . . . 84
NetworkProgrammingWrapUp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
UsingtheMicrosoftBingSearchAPIs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
GettinganAccessKeyforMicrosoftBingSearchAPIs . . . . . . . . . . . . . . . . . . . . . . 88
ExampleSearchScript . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
CONTENTS
Wrap-up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
AccessingRelationalDatabases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
DatabaseWrapUp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
UsingMongoDB,SolrNoSQLDataStores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
MongoDB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
ACommonLispSolrClient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
NoSQLWrapup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
NaturalLanguageProcessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
LoadingandRunningtheNLPLibrary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
PartofSpeechTagging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
CategorizingText . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
DetectingPeople’sNamesandPlaceNames . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
SummarizingText . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
TextMining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
InformationGathering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
DBPediaLookupService . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
WebSpiders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
UsingApacheNutch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
WrapUp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
UsingTheCLMachine-LearningLibrary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
UsingtheCLMLDataLoadingandAccessAPIs . . . . . . . . . . . . . . . . . . . . . . . . . . 135
K-MeansClusteringofCancerDataSet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
SVMClassificationofCancerDataSet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
CLMLWrapUp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
BackpropagationNeuralNetworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
HopfieldNeuralNetworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
UsingPythonDeepLearningModelsInCommonLispWithaWebServicesInterface . . 162
SettingupthePythonWebServicesUsedinthisChapter . . . . . . . . . . . . . . . . . . . . 162
InstallingthespaCYNLPServices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
InstallingtheCoreferenceNLPServices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
CommonLispClientforthespaCyNLPWebServices . . . . . . . . . . . . . . . . . . . . . . 164
CommonLispClientfortheCoreferenceNLPWebServices . . . . . . . . . . . . . . . . . . 166
TroubleShootingPossibleProblems-SkipifthisExampleWorksonYourSystem . . . . . 167
PythonInteropWrap-up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
UsingthePY4CLLibrarytoEmbedPythoninCommonLisp . . . . . . . . . . . . . . . . . . . 169
ProjectStructure,BuildingthePythonWrapper,andRunninganExample . . . . . . . . . . 169
CONTENTS
Implementationofspacy-py4cl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
TroubleShootingPossibleProblems-SkipifthisExampleWorksonYourSystem . . . . . 172
Wrap-upforUsingPy4CL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
AutomaticallyGeneratingDataforKnowledgeGraphs . . . . . . . . . . . . . . . . . . . . . . 174
ImplementationNotes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
GeneratingRDFData . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
GeneratingDatafortheNeo4jGraphDatabase . . . . . . . . . . . . . . . . . . . . . . . . . . 179
ImplementingtheTopLevelApplicationAPIs . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
ImplementingTheWebInterface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
CreatingaStandaloneApplicationUsingSBCL . . . . . . . . . . . . . . . . . . . . . . . . . . 187
KGCreatorWrapUp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
KnowledgeGraphNavigator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
ExampleOutput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
ProjectConfigurationandRunningtheApplication . . . . . . . . . . . . . . . . . . . . . . . . 195
ReviewofNLPUtilitiesUsedinApplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
DevelopingLow-LevelSPARQLUtilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
ImplementingtheCachingLayer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
UtilitiestoColorizeSPARQLandGeneratedOutput . . . . . . . . . . . . . . . . . . . . . . . 202
TextUtilitiesforQueriesandResults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
UsingLispWorksCAPIUIToolkit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
WritingUtilitiesfortheUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
WritingtheUI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
Wrap-up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
BookWrapup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
Cover Material, Copyright, and
License
Copyright2011-2020MarkWatson.Allrightsreserved.ThisbookmaybesharedusingtheCreative
Commons“shareandsharealike,nomodifications,nocommercialreuse”license.
ThiseBookwillbeupdatedoccasionallysopleaseperiodicallychecktheleanpub.comwebpagefor
thisbook¹forupdates.
Thisisthesixtheditionreleasedsummerof2020.
Pleasevisittheauthor’swebsite².
Ifyoufoundacopyofthisbookonthewebandfinditofvaluethenpleaseconsiderbuyingacopy
atleanpub.com/lovinglisp³tosupporttheauthorandfundworkforfutureupdates.
¹https://leanpub.com/lovinglisp
²http://markwatson.com
³https://leanpub.com/lovinglisp
Preface
Notes on the Sixth Edition Published June 2020
Two examples optionally use the CAPI user interface toolkit provided with LispWorks Common
Lisp⁴ and work with the free personal edition. The first CAPI application is Knowledge Graph
Navigator⁵ and the second CAPI example is Knowledge Graph Creator⁶. Both of these examples
builduputilitiesforworkingwithKnowledgeGraphsandtheSemanticWeb.
I expand the Plot Library chapter to generate either PNG graphics files or if you are using the free
personal edition of LispWorks you can also direct plotting output to a new window in interactive
programs.
Iaddedanewchapteronusingthepy4cllibrarytoembedPythonlibrariesandapplicationcodeinto
aCommonLispsystem.IprovidenewexamplesforembeddingspaCyandTensorFlowapplications
in Common Lisp applications. In earlier editions, I used a web services interface to wrap Python
codeusingspaCyandTensorFlow.Iamleavingthatchapterintact,renamingitfrom“UsingPython
DeepLearningModelsInCommonLisp”to“UsingPythonDeepLearningModelsInCommonLisp
With a Web Services Interface.” The new chapter for this edition is “Using the PY4CL Library to
EmbedPythoninCommonLisp.”
Notes on the Fifth Edition Published September 2019
Thereweretwochaptersadded:
• A complete application for processing text to generate data for Knowledge Graphs (targeting
theopensourceNeo4JgraphdatabaseandalsosupportRDFsemanticweb/linkeddata)
• A library for accessing the state of the art spaCy natural language processing (NLP) library
andalsoastateoftheartdeeplearningmodel.ThesemodelsareimplementedinthinPython
wrappers that use Python libraries like spaCy, PyTorch, and TensorFlow. These examples
replaceasimplehybridJavaandCommonLispexampleinpreviouseditions.
IhaveaddedtextandexplanationsasappropriatethroughoutthebookandIremovedtheCouchDB
examples.
Ihavemadelargechangestohowthecodeforthisbookispackaged.Ihavereorganizedtheexample
codeonGitHubbyprovidingtheexamplesasmultipleQuicklisplibrariesorapplications.Inowdo
⁴https://lispworks.com
⁵http://knowledgegraphnavigator.com
⁶http://kgcreator.com
Preface 3
this with all of my Common Lisp code and it makes it easier to write smaller libraries that can be
composed into larger applications. In my own workflow, I also like to use Makefile targets to build
standalone applications that can be run on other computers without installing Lisp development
environments. Please follow the directions at the end of the Preface for configuring Quicklisp for
easybuildsanduseoftheexamplesoftwareforthisbook.
Why Use Common Lisp?
WhyCommonLisp?Isn’tCommonLispanoldlanguage?DomanypeoplestilluseCommonLisp?
I believe that using Lisp languages like Common Lisp, Clojure, Racket, and Scheme are all secret
weapons useful in agile software development. An interactive development process and live
production updates feel like a breath of fresh air if you have development on heavy weight like
JavaEnterpriseEdition(JEE).
Yes, Common Lisp is an old language but with age comes stability and extremely good compiler
technology. There is also a little inconsistency between different Common Lisp systems in such
thingsashandlingthreadsbutwithalittleupfrontknowledgeyoucanchoosewhichCommonLisp
systemswillsupportyourrequirements.
A Request from the Author
I spent time writing this book to help you, dear reader. I release this book under the Creative
Commons “share and share alike, no modifications, no commercial reuse” license and set the
minimum purchase price to $5.00 in order to reach the most readers. Under this license you can
share a PDF version of this book with your friends and coworkers and I encourage you to do so. If
you found this book on the web (or it was given to you) and if it provides value to you then please
consider doing one of the following to support my future writing efforts and also to support future
updatestothisbook:
• Purchase a copy of this book leanpub.com/lovinglisp/⁷ or any other of my leanpub books at
https://leanpub.com/u/markwatson⁸
• Hiremeasaconsultant⁹
I enjoy writing and your support helps me write new editions and updates for my books and to
developnewbookprojects.Thankyou!
⁷https://leanpub.com/lovinglisp/
⁸https://leanpub.com/u/markwatson
⁹https://markwatson.com/
Preface 4
Older Book Editions
ThefourtheditionofthisbookwasreleasedinMay2017andthemajorchangeswere:
• Added an example application KGCreator that processes text data to automatically generate
data for Knowledge Graphs. This example application supports the Neo4J graph database as
wellassemanticweb/linkeddatasystems.Themajorchangeswere:
• Addedabackpropagationneuralnetworkexample
• Added a deep learning example using the Java based Armed Bear Common Lisp with the
popularDeepLearning4jlibrary
• Addedaheuristicsearchexample
• Addedtwomachinelearningexamples(K-MeansclusteringandSVMclassification)usingthe
CLMLlibrary
• Afeweditstotheprevioustext
ThethirdeditionwasreleasedinOctober2014.Themajorchangesmadeinthe2014editionare:
• IreworkedthechapterCommonLispBasics.
• IaddedmaterialtothechapteronusingQuickLisp.
The second edition was released in 2013 and was derived from the version that I distributed on my
websiteandImovedproductionofthebooktoleanpub.com¹⁰.
Acknowledgments
IwouldliketothankJansAasman¹¹forcontributingastechnicaleditorforthefourtheditionofthis
book. Jans is CEO of Franz.com¹² which sells Allegro Common Lisp¹³ as well as tools for semantic
webandlinkeddataapplications.
I would like to thank the following people who made suggestions for improving previous editions
ofthisbook:
SamSteingold,AndrewPhilpot,KennyTilton,MathewVilleneuve,EliDraluk,ErikWinkels,Adam
Shimali,andPaoloAmoroso.
I would like to also thank several people who pointed out typo errors in this book and for specific
suggestions:MartinLightheart,Tong-KiatTan,RainerJoswig,GeroldRupprecht,andDavidCortesi.
I would like to thanks the following Reddit /r/lisp readers who pointed out mistakes in the fifth
¹⁰https://leanpub.com/u/markwatson
¹¹https://en.wikipedia.org/wiki/Jans_Aasman
¹²http://franz.com/
¹³http://franz.com/products/allegro-common-lisp/