Kestrel: An Interface from Optimization Modeling Systems to - ampl

14 Pages·2008·1.22 MB·English

INFORMSJournalonComputing
informs
Vol.20,No.4,Fall2008,pp.525–538 ®
issn1091-9856(cid:1)eissn1526-5528(cid:1)08(cid:1)2004(cid:1)0525 doi10.1287/ijoc.1080.0264
©2008INFORMS
Kestrel: An Interface from Optimization
Modeling Systems to the NEOS Server
s).g/. Elizabeth D. Dolan
(r
ro
hos. IndustrialEngineeringandManagementSciencesDepartment,NorthwesternUniversity,
utm Evanston,Illinois60208;andMathematicsandComputerScienceDivision,
aor ArgonneNationalLaboratory,Argonne,Illinois60439
enf
ths.i Robert Fourer
oal IndustrialEngineeringandManagementSciencesDepartment,NorthwesternUniversity,
tn
r Evanston,Illinois60208,[e-mail]
yu
urtesp://jo Powernext,750J0e2aPnar-iPs,iFerranrece,[e-mail]
cohtt Todd S. Munson, Jason Sarich
aat
asble MathAermgoantincse,aInlldinCoiosm6p0u43te9r{[e-mail],lA.grogvo,[e-mail]}tory,
a
pyail
ov T
ca he NEOS server provides access to a variety of optimization resources via the Internet. The new Kestrel
sis interface to the server enables local modeling environments to request NEOS optimization services and
thies, retrievetheresultsforlocalvisualizationandanalysissothatusershavethesameconvenientaccesstoremote
dci NEOS solvers as to those installed locally. Kestrel agents have been implemented for the AMPL and GAMS
butepoli mlatoedrerleintrgieevnavliorofnremseunlttss,;mthaekseinaggpenotsssihbaleveabreuednimdeensitganryedfosromthoaftpsuarbapllreolbplermocsescsainngb.equeuedforexecutionand
rin
distsio Keywords: optimization;mathematicalprogramming;Internetservices;modelinglanguages
dmis History: Accepted by John Chinneck, former Area Editor for Modeling: Methods and Analysis; received
aner September 2002; revised December 2004, December 2006, August 2007; accepted October 2007. Published
p onlineinArticlesinAdvanceJuly7,2008.
e
clnd
tia
r
as
sht 1. Introduction with our solutions to those problems as embodied
thirig TheNEOSserver(Czyzyketal.1997,1998;Groppand in the design of the NEOS application programming
tong Moré 1997) provides Internet access via http://neos. interfaceandtheKestrelagentsformodelingsystems.
htudi mcs.anl.govtoover50solversforarangeofoptimiza- Theremainderofthepaperillustrateshowwehave
rigncl tion problem types. This paper describes the design designed the Kestrel interface to make it useful from
pyn,i and creation of Kestrel, a callable interface to the the modeler’s point of view. Section 4 shows how
scomatio NpeErOmSittsienrgvearcctehsasttehxrtoeungdhs athberoseardvevra’rsievteyrsoaftimlitoydbely- Kenetstorpetlimagizeanttisonenmaobdleeltihnegluasnegrusaogfest—woAMquPiLte(Fdoiuffreerr-
d
olor ing languages and systems. et al. 1990, 2003) and GAMS (Bisschop and Meeraus
hnf The Kestrel interface is called from a locally run- 1982, Brooke et al. 1992)—to invoke NEOS solvers in
i
MSal ning environment and returns results to that envi- virtually the same way as locally installed solvers.
n
Ro ronment. Kestrel thus gives a modeling system the Section5thenpresentstwomoresophisticatedexam-
FOditi same access to remote NEOS solvers as to solvers ples: one that uses Kestrel/AMPL as part of a larger
Nd
A installed locally. As a result, analysts can consider iterative scheme to solve and visualize solutions to
I
a wider variety of solvers, problems may be solved a partial differential equation, and one that incor-
more effectively, and new solver technologies may be porates the Kestrel interface to make a GAMS-to-
disseminated more rapidly. AMPL preprocessing service available via the NEOS
Section 2 reviews the aspects and limitations of the server. By using a paradigm similar to that described
NEOS server that have led to our design of Kestrel. by Ferris and Munson (2000), an extension to the
Section 3 then presents Kestrel from a technical point AMPL and GAMS Kestrel agents enables subprob-
ofview.Itconsidersthecommunicationandinterface lems to be queued for execution and later retrieval
problems inherent in calling the NEOS server from of results; §6 describes this and related features and
a program somewhere else on the Internet, together their applications.
525
Dolanetal.: Kestrel: AnInterfacefromOptimizationModelingSystemstotheNEOSServer
526 INFORMSJournalonComputing20(4),pp.525–538,©2008INFORMS
2. The NEOS Server files are then transmitted to the workstation, and the
The NEOS server is the most ambitious realization remote solver is initiated there.
to date of the optimization server concept (Fourer Compressed input is automatically detected and
and Goux 2001). Operated by the Optimization Tech- expanded as part of the unpacking procedure.
nology Center of Argonne National Laboratory and More complex preprocessing, such as automatic
Northwestern University, it represents a collaborative differentiation or modeling language translation, is
effort involving over 40 designers, developers, and performed as part of the remote solver activities after
administrators around the world. The project solic- the problem has been sent to a workstation.
s).g/. its solvers for all kinds of local and global optimiza- Duringtheremotesolver’sexecution,alltextinthe
(r
uthorms.o tnioonnliannedar,rsetloactehdasptirco,balnedmsse,minidcleufidniintegplrinoegarar,mins,tewgiethr, sthtaendNaErOdSousteprvuetra,nadndertrhoernsttroeatmhes iussperasisnedthbeacfokrmto
aor linear, nonlinear, complementarity, and more general of a listing returned through the original submission
thes.inf nondifferentiable constraints. Some of the solvers are interface. Some modelers are content to look at this
oal free or are available through open-source licenses; listing, while others either write their own processing
yturn others are proprietary to varying degrees, but their programs or cut and paste from the listing to other
so software. However, a listing of this sort is problem-
urtep://j othwronuegrshhtahveeseargvreere.d to permit their use at no charge atical when the results are to be manipulated by a
cohtt Since its inception in 1996, the NEOS server has program rather than directly by a human modeler, as
aat been progressively enhanced and now regularly han- when the optimization is part of some larger compu-
asble dles over 40,000 submissions per month from a vari- tational scheme or is embedded in a broader model-
a ing context, and a listing may be insufficiently secure
copyavail e(Dtyoloafnbeutsailn.e2s0s0,2ead,ubc)a.Atiosnidael,faronmd sthcieenKteifistcreelnindteearvfaocres when the results involve proprietary information.
Effective use of an optimization modeling lan-
thises,is tmoabile, dWeesbc-rbibaesded, pfroorbmlesm, asnadrethseenNtEtoOtShgersaeprhviecralviuaseer- greutaugrne,edininpaafrotricmulaapr,prroepqruiairteestoththaetatshseociraetseudltmsobde-
dci interface(theNEOSGUI).Althoughtheserver’sloca-
butepoli tionisfixed,eachofthesolverscanrunonanywork- ealcicnegsseannvdiromnamneipnut.laTtehethehurmesaunltsmthordoeulegrh cthane etnhvein-
rin station connected to the Internet. Hence, available
sto ronment’s interface, which may provide specialized
disi computing resources have been able to grow along
s facilities for browsing data and results, for dealing
dmi with the number of solvers and users (Dolan 2001).
aner The large number of optimization problem types withmultiplefiles,andforinteractingwithdatabases
p andspreadsheets.Theserequirementshavemotivated
e and solvers precludes standardization on any one
clnd our interest in an alternative interface that enables a
tia input format. The NEOS server instead allows opti-
ars mization solvers to receive whatever text or binary local modeling environment to send an optimization
sht problemtotheserverandretrievetheresultsinmuch
thirig inputtheyareequippedtoaccept,whetherexplicitor the same way that the environment would interact
ttoding scyompebwoliitchotrhiisnvtahreieftoy,rmtheosfeCrveorrmFoarintrtaaninrsoaudtianteasb.aTsoe wiTthheanloecxatlsseocltvioern. describes the technical demands,
hu
gcl that records the characteristics and needs of each design, and implementation of a callable interface
yriin solver. A solver’s database entry includes the input to NEOS, and the specifics of accessing this inter-
opon, format it accepts, the remote workstations on which facefromalgebraicmodelingsystemsviaKestrel.The
scmati it may be run, user documentation for the various reader who is mainly interested in the application of
d interfaces,andotherinformationusedtoconstructthe
olor these ideas may want to skip to §§4 to 6, where we
hnf NEOS server’s Web-based form interface. If a solver describe several uses of Kestrel in the context of the
MSali canberunwithdifferentinputformats,itmusthavea algebraic modeling languages AMPL and GAMS.
n database entry for each. With over 50 distinct solvers,
Ro
NFOdditi theUnnudmerbethriosfadraratanbgaesmeeenntt,rieeascihs asbuobumti1s2s0io.n to the 3. Building the Kestrel Modeling
A
I
NEOSserverisasingletextinputfilecomprisingone Language Agents
or more parts delimited by XML tags. (See Skonnard Inthesimplestterms,acallableinterfacetotheNEOS
and Gudgin 2001 for a concise introduction to XML.) server requires some mechanism for making function
Fore-mailsubmissions,theuserinsertstagsexplicitly, calls over the Internet. A local program known as an
while the Web interface and NEOS GUI insert appro- agent can then translate a user request into a remote
priate tags automatically. Consulting the database function call that actually runs on the computer host-
entry for the requested solver, the server unpacks the ing the NEOS server.
input’s parts into separate files and chooses a remote In designing the callable interface to NEOS, we
workstationonwhichthesubmissionwillberun.The sought to assure that this arrangement would be
Dolanetal.: Kestrel: AnInterfacefromOptimizationModelingSystemstotheNEOSServer
INFORMSJournalonComputing20(4),pp.525–538,©2008INFORMS 527
sufficiently flexible to handle the broad variety of server—the same kind of server that handles Web
modeling environments and solvers. To that end, we page requests. The http server parses the XML docu-
adopted the following design goals: ment,dispatchestheargumentstotherequestedfunc-
• Agents should be implementable on a choice of tion,formsanotherXMLdocumentencapsulatingthe
platforms and in a choice of languages. results of the function, and returns the latter docu-
• Agentsshouldbeabletoissuebothblockingand ment as the response to the POST request. Although
nonblocking calls on the NEOS server. XML is designed for text, binary data can also be
• Agents should be able to send both text and transmitted via XML-RPC by encoding it as base64
s).g/. binary data. text that is inserted into the XML document.
r(or • A broad variety of applications should be imple- This mechanism requires a well-defined API to
uthoms. mentable with reasonable effort. define agent calls that match all of the server func-
aor • The calling mechanism should be compatible tions. The NEOS API—described at http://neos.
enf with the rest of the NEOS server infrastructure. mcs.anl.gov/neos/NEOS-API.html—contains over 25
othals.i • Theservershouldbeabletohandlemanysimul- calls accessible to C, C++, Java, Python, Perl, PHP,
trn taneous agent calls. Ruby,andanyotherlanguageforwhichanXML-RPC
yu
so Fortunately, standard “middleware” is readily library is available.
urtep://j available to help meet these goals. Competing tech- As an example, the following Python call submits
cohtt nologiesincludeMicrosoft’s.NETtechnology(http:// an XML-delimited optimization request (or “job”) to
aat www.microsoft.com/net); DCE, the Open Software the NEOS server in the form of a character string:
asble Foundation’s Distributed Computing Environment (cid:1)jobNumber(cid:2)password(cid:3)
a (http://www.opengroup.org/dce); the Nexus library,
opyvail a component of the Globus Alliance (http://www. =neos(cid:4)submitJob(cid:1)xmlstring(cid:2)user="(cid:2)
ca globus.org); the Object Management Group’s CORBA
s
thises,i (XhMttpL:-/R/PwCw(wht.tcpo:r/b/aw.owrgw);.xmanlrdpc.Ucosmer)L.and Software’s interface="(cid:2)id=0(cid:3)
dci Ifthesubmissionfails,suchaswhentheNEOSqueue
butepoli libTrahrey,fiarsntdimthpelesmeceonntadtiownasofbKaseesdtreolnusCeOdRthBeAN. Bexouths fortherequestedsolverisfull,thereturnedjobNumber
rin is 0 and the password is an error message. Otherwise,
sto versionsreliedonaseparate“Kestrelserver”running
disi the jobNumber and password are used in subsequent
s in cooperation with the NEOS server and required a
dmi calls that manage the job, such as the following to
aner shareddiskdriveforefficiency.Whenversion5ofthe retrieve real-time output from a NEOS solver while a
p NEOS server was implemented in 2005, however, a
e job is running:
clnd callable interface through XML-RPC was made a cen-
tia
r tral component of its design, and a separate Kestrel-
as (cid:1)msg(cid:2)offset(cid:3)
sht only server was no longer necessary.
thirig Thus,webeginthissectionwithadescriptionofthe =neos(cid:4)getIntermediateResults(cid:1)jobNumber(cid:2)
tong applicationprogramminginterface(the“NEOSAPI”) password(cid:2)offset(cid:3)
tdi thatNEOS5providestoallofitsusersthroughXML-
hu
gcl RPC. We next introduce the concept of an agent that Thebase64-encodedmsgcontainstheresultsofthejob
rin
pyn,i calls the NEOS API. The remainder of this section from character number offset up to the last received
oo then focuses on the internal design of Kestrel agents data. If the job is still running, then this function
scmati for optimization modeling systems. Subsequent sec- hangsuntilthenextpacketofoutputissentorthejob
d
olor tions argue that these agents can provide a variety of finishes.Anonblockingcallisalsoprovidedandthere
hnf valuable services to users of modeling systems. areanalogousfunctionsforretrievingfinaloutput.An
i
MSal elementaryPythonscriptforacompleteNEOSserver
n
Ro 3.1. Fundamentals of the NEOS API agent using these functions can be as simple as the
FOditi AllsubmissionstotheNEOSserverareperformedby program shown in Figure 1.
NAd creating an XML document file to define the input, API calls are also available to retrieve information
I
submittingthisfiletotheserver,andreceivingaresult about the state of the NEOS server, to obtain the list
file from the server. of solvers, and to kill a submission currently being
Communication with the NEOS server is accom- executed. Calls of another kind register and maintain
plished by use of the XML-RPC (XML remote proce- solvers on the server.
durecall)standard(Winer2003).Thisstandardallows
an agent process to invoke a specific member func- 3.2. Agents
tion of a remote object by creating an XML docu- Although XML document creation and NEOS server
mentencapsulatingthefunctionnameandarguments communicationcanbecarriedoutexplicitlybyauser,
and making a POST request to a knowledgeable http most of the time these activities are conducted by an
Dolanetal.: Kestrel: AnInterfacefromOptimizationModelingSystemstotheNEOSServer
528 INFORMSJournalonComputing20(4),pp.525–538,©2008INFORMS
#!/usr/bin/env python
import sys AMPL session GAMS session
import xmlrpclib
from config import variables
if len(sys.argv) < 2 or len(sys.argv) > 3: AMPL Kestrel GAMS Kestrel
agent agent
sys.stderr.write("Usage: NeosClient <xmlfilename
| help | queue>")
sys.exit(1)
E-mail client Web browser
neos = xmlrpclib.ServerProxy("http://neos.mcs.anl.
s).g/. gov:3332")
(r
ro
hos. msg = neos.ping( )
utm if not msg.find("OK"):
aor sys.stderr.write("Could not contact NEOS server.") E-mail agent NEOS GUI XML-RPC client Web agent
enf sys.exit(1)
tothnals.i ifssyyss..satrdgovu[t1.]wr=i=te"(h"eHlepl"p:not yet available(cid:4)(cid:4)(cid:4)") XML-RPC
r
yu
so elif sys.argv[1] == "queue":
courtehttp://j mssygs.=stndeooust..pwrriintteQ(umesuge)( ) NEOS server
else:
aat xmlfile = open(sys.argv[1], "r")
asble xmlstring = ""
a buffer = 1 Solver workstation Solver workstation
pyail while buffer:
ov
ca buffer = xmlfile.read( )
sis xmlstring += buffer Solver workstation
thies, xmlfile.close( )
dci (jobNumber, password) = neos.submitJob(xmlstring) Figure2 StructureoftheVariousNEOSInterfaces,Showingthe
butepoli sys.stdout.write("jobNumber = %d " % jobNumber) CUosinnngecthtieonXsMBLe-RtwPeCenAPthIeNEOSServerandPossibleAgents
rin offset = 0
distsio status = neos.getJobStatus(jobNumber, password)
s while status == "Waiting" or status == "Running": 3.3. Kestrel Agents for Modeling Systems
dmi
aner (mspga,ssowfofrsde,t)of=fsneeto)s.getIntermediateResults(jobNumber, Themechanismforcommunicationbetweenamodel-
p ingsystemandNEOSisalsoimplementedbycreating
ticleand ssytsa.tsutsdo=utn.eworsi.tgee(tmJsogb.Sdtaattau)s(jobNumber, password) an agent. In this case, the agent is responsible for
r gathering a problem instance and related information
as msg = neos.getFinalResults(jobNumber, password).data
thisright sys.stdout.write(msg) fmroemnttfhroemmtohdiseliinnfgorsmysatteimon,,gseunbemraittitninggatnheXXMMLLddocouc--
tong Figure1 AnElementaryPythonScriptforaCompleteNEOS umenttotheNEOSserver,gettinglistingsandresults
tdi ServerAgent fromtheserver,andconvertingtheresultstoaformat
hu
gcl that the modeling system can use.
rin
yi intermediaryprogram,calledanagent,thatmakesthe Our implementations of such agents, initially for
pn,
oo appropriate NEOS API calls. the AMPL and GAMS systems, are written in the
cati As an example, consider what happens when a Python programming language. Python has several
sm
oldor request is submitted to NEOS via the Web form features that make it ideal for this purpose:
hnf interface. Clicking the “Submit to NEOS” button ini- • It is freely available and compatible with most
i
MSal tiates an agent (a Python CGI script) that makes a computing platforms.
n
Ro NEOS API call to get a list of the XML tags used by • It is distributed with an easy-to-use XML-RPC
FOditi the selected solver, reads the data that the user has library.
NAd entered through text boxes, check boxes, and radio • Third-party tools are available for converting
I
buttons of the Web form, and combines this infor- Python programs to executable binaries for Microsoft
mation to create an XML document. The agent then Windows.
makes further NEOS API calls to submit the XML The Python code for our Kestrel agents for
document and to check the server intermittently for AMPL and GAMS can be downloaded from the
output and job status. When the job is finished, the NEOS Web pages (http://neos.mcs.anl.gov/neos/
agent directs the user’s Web browser to the results downloads.html), but a few key code fragments
page. Figure 2 shows how different NEOS agents are presented here to give an idea of the issues
facilitate communications between local systems and involved in getting communication via Kestrel to
the NEOS server. work. Section 4 complements this discussion by
Dolanetal.: Kestrel: AnInterfacefromOptimizationModelingSystemstotheNEOSServer
INFORMSJournalonComputing20(4),pp.525–538,©2008INFORMS 529
describing how the Kestrel agents appear from the 120,000
user’s standpoint. AMPL or GAMS
Although the AMPL and GAMS agents are sep- 100,000 Other
arate programs, their code dealing with NEOS ns 80,000
o
communications is similar. The following fragments missi 60,000
are taken from the AMPL agent, but their GAMS b
equivalents are much the same. Su 40,000
The AMPL Kestrel agent is invoked when AMPL’s 20,000
s).g/. solvecommandisgiven.Theagentencodesinforma- 0
uthor(ms.or trieolanteadboeuntvitrhoenpmreonbtlevmariianbstlaesncien,tosoalnveXrMopLtitoenxst,daoncd- Jan-05Apr-05Jul-05Oct-05Jan-06Apr-06Jul-06Oct-06Jan-07Apr-07Jul-07Oct-07Jan-08Apr-08
aor ument stored as a string in the Python variable xml.
enf Then, the agent creates an object self.neos that will Figure3 Predominance of AMPL and GAMS in Submissions to the
tothnals.i bwehiucshedis lfiosrtencoinmgmautnaicsapteincigfiewditUhRtLh.e NEOS server, lost whenNEitOSseSerrvveers merely as a NEOS input format.
r
yu The NEOS server proceeds in such a case as follows:
so Communication with the server can now begin.
courtehttp://j Fniirnsgt,atnhde aagcceenpttimngusctonchneeccktiotnhsatbtyheussienrgvetrheisArPuIn’s- satsa•ttihoTenhrteehqasueterhsvtaeesrd’assmoinlvopedur.etliinsgplaasnsgeudagtoe parorceemssootreaws worekl-l
aat ping call:
• The modeling language processor interprets the
asble self(cid:4)neos(cid:4)ping(cid:1) (cid:3) input, converts it to a format that the solver recog-
a
pyail Next, the agent uses the submitJob call to send the nizes, and invokes the solver.
coav XMLdocumentstoredinthestringvariablexmltothe • When the solver run is complete, the processor
sis NEOS server: retrieves results and generates result displays speci-
thies, (cid:1)jobNumber(cid:2)password(cid:3)=self(cid:4)neos(cid:4)submitJob(cid:1)xml(cid:3) fied by additional modeling language commands.
dci For security reasons, the modeling language pro-
butepoli Whentheserverreceivesthiscall,itperformsthetasks cessorrunsinamodethatsuppressesfilewritingand
rin necessaryforinitiatingajobandreturnsajobnumber other local features. Thus, only the solver’s output
sto
disi andpasswordifthesubmissionissuccessful,orazero andtheprocessor’sresulttablesarereturnedthrough
s
dmi andanerrormessageifanexceptionoccurred. the NEOS server. Any interactivity, database connec-
aner Finally, several calls are used to wait until the tivity, or other flexibility that the modeling system
p
ticleand NreEsuOltSs:submission is finished, and to retrieve the ceannviproronvmidenetw. henlocallyinstalledislostintheserver
r
as Thus,oneoftheKestrelinterface’sprincipaldesign
thisright whi=l=e"sWataittuinsg=":= "Running" or status raenqduiAreMmPeLntpsrowcaesssteos tpoercmaliltrelomcaoltleyNruEnOnSinsgolvGeArsMinS
tong status = kestrel.neos.getJobStatus a way that would preserve all of the familiarity and
htudi (jobNumber, password) power of the GAMS and AMPL environments. This
gcl time.sleep(5)
rin section first describes the use of the Kestrel/AMPL
yi results = self.neos.getFinalResults
pn, and Kestrel/GAMS agents for such a purpose, and
oo (jobNumber, password)
cati if isinstance(results, xmlrpclib.Binary): then considers their advantages over other client-
sm server arrangements for optimization.
oldor results = results.data
hnf 4.1. Using the Kestrel/AMPL Agent
i Because the results for a Kestrel submission are
MSnal returned as a base64-encoded string, the data data AMPL specifies the solver name and options through
Ro a mechanism modeled on Unix and Windows envi-
NFOAdditi mvaelumeberreporfesthenetreedtubrny vthaeluestrisinugs.eTdhtiosgveatlutheeisbinthaerny reoxnchmaenngtevoafraiallboletshearnidnfoursmesattieomnpwoirtahryasfiollevserf.oTrhtuhse,
I writtentoafilefromwhichtheAMPLmodelingenvi-
from the AMPL modeling system’s standpoint, the
ronment extracts the solution.
invocationofasolverproceedsinthreemainsteps:
• Write a representation of the current problem
4. Using the Kestrel Modeling
instance to a file, and set appropriate environment
Language Agents
variables.
The great majority of current submissions to the • Locate a specified local solver, invoke it as an
NEOSserverarewrittenintheAMPLorGAMSmod- independent program with appropriate arguments,
elinglanguage(Figure3).Yetaspreviouslyremarked, and wait for it to write a solution file.
some of the advantages of a modeling language are • Read the solution file and resume processing.
Dolanetal.: Kestrel: AnInterfacefromOptimizationModelingSystemstotheNEOSServer
530 INFORMSJournalonComputing20(4),pp.525–538,©2008INFORMS
The Kestrel interface for AMPL substitutes the outlev=1
Kestrel/AMPL agent program for the local solver in ···
the second step. The agent obtains a solution to the LOQO 6.06: optimal solution
problem instance via the NEOS server, rather than (11 QP iterations, 11 evaluations)
from some locally installed solver. But to the AMPL primal objective 191999.9989204328
system,theagentisjustanotherprogramthataccepts dual objective 191999.996903256
problem instances and returns solutions. Thus, no ampl:
special modification of AMPL is necessary to make it
When AMPL is being used interactively, text contin-
(s).rg/. compatible with the Kestrel/AMPL agent. ues to appear a few lines at a time as the solver
hors.o As an example, the AMPL commands for solv- runs,withthefrequencydependingonhowthesolver
utm ing the steel.mod sample problem (http://www. managesitsbuffers.Afterthesolverruniscompleted,
aor ampl.com/BOOK/EXAMPLES2/steel.mod) by using
enf all solution values are returned to the AMPL system
ths.i a locally installed copy of the LOQO solver (Vander- and can be accessed through subsequent commands.
tonal bei and Shanno 1999) are as follows: The display command can be used to browse the
r
yu ampl: model steel.mod; solution, for example:
so
courtehttp://j aammppll:: doapttaiosnteseoll.vdeart;loqo; aMmapkle:[d∗i]sp:l=ay Make;
ampl: option loqo_options ’mindeg sigfig=8
aat bands 6000
asble ampl: sooluvtel;ev=1’; coils 1400
a ;
copyavail Thecorrespondingcommandstosolvethesameprob- The state of the AMPL system is in fact exactly the
lem by using Kestrel/AMPL to run LOQO through
sis sameatthispoint(exceptforthesettingofthesolver
thies, the NEOS server are much the same: option) as it would be if the solver had been run
dci ampl: model steel.mod; locally.
butepoli ampl: data steel.dat; Echoing the algorithm progress log interactively, as
rin ampl: option solver kestrel; the solver generates it, is a substantial enhancement
sto
disi ampl: option kestrel_options ’solver=loqo’; madepossiblebythedecisiontouseXML-RPCasthe
s
dmi ampl: option loqo_options ’mindeg sigfig=8 communication protocol for NEOS 5. Earlier versions
aner outlev=1’; couldonlyreturnthesolver’soutputallatonce,after
p
clend ampl: solve; the solver run was complete. To view current output,
tia the job number and password could be entered into
sarhts TsohethsaotltvheerAopMtiPoLnsisolcvheancgomedmfaronmd nloowqoitnovokeksetsrtehle, a Web form (Figure 4) so that the results could be
thirig localKestrelagentprogram.Also,acommandsetting viewed through a browser, although they could still
tong kestrel_options is added, to tell the Kestrel interface not be returned to the local AMPL environment. This
htudi which remote NEOS solver is being requested. In all facility remains available in NEOS 5 but is mainly of
gcl other respects, the AMPL session to this point is the value for submissions through noninteractive inter-
opyrion,in samOnecaestbheefoprer.oblem instance has been successfully facTehsissuecnhhaasncee-mmaeinl.t extends to situations in which
cati the local Kestrel or AMPL process is terminated and
sm conveyed to the NEOS server, the Kestrel agent dis-
d then restarted while the solver continues to run. Our
olor plays the assigned job number and password:
hnf designofKestrel/AMPLforthissituationisdiscussed
MSali Job 831764 sent to newton.mcs.anl.gov in §6 in the context of the Kestrel modeling language
Ron password: ZQPNBVCc agents’ submission and retrieval features.
FOditi Then, it starts to echo any text that the solver sub-
NAd sequently writes to the standard output and error 4.2. Using the Kestrel/GAMS Agent
I
streams: The GAMS modeling system communicates with
solvers entirely via temporary files, so its commu-
---------- Begin Solver Output -----------
nication with the Kestrel interface is also file based.
Executing /home/neosotc/neos-5-solvers/
Prior to sending the GAMS problem file to the NEOS
kestrel-ampl/kestrelSolver.py
server, the Kestrel/GAMS agent removes all refer-
kestrel_options:solver=loqo
ences to the license and other system components, so
loqo_options:mindeg sigfig=8 outlev=1 that proper licenses and information for the remote
LOQO 6.06: mindeg solver can be inserted later. This practice helps pre-
sigfig=8 serve the integrity of the client’s system by not
Dolanetal.: Kestrel: AnInterfacefromOptimizationModelingSystemstotheNEOSServer
INFORMSJournalonComputing20(4),pp.525–538,©2008INFORMS 531
s).g/.
(r
ro
hos.
utm
aor
enf
ths.i
oal
tn
r
yu
so
urtep://j
cohtt
aat
asble
a
pyail
ov
ca
s
si
thies,
dci
butepoli
rin
sto
disi
s
dmi
aner
p
e
clnd
tia
r
as
thisright Figure4 TChuerreNnEtOOSutSpeurtvoerr’FsinWaelbJoFborRmesfuolrtsR,eaqsuAepsptirnogprRiaetseultsfromaSolver,InResponsetoWhichtheServerGeneratesaPageContainingthe
og
tn
htudi sending license information over the Internet unnec- option lp = kestrel;
rigncl essarily. The Kestrel/GAMS agent preprocessing also solve transport using lp minimizing z;
yi
opon, converts all absolute path names to relative ones. The added statement transport.optfile = 1 speci-
cati ToruntheGAMSLIBtrnsportmodel(http://www. fies that an options file, named kestrel.opt, is pro-
sm gams.com/modlib/libhtml/trnsport.htm), as an ex-
d vided. This options file conveys the remote solver
olor ample, the command gams trnsport is issued. The
hnf name as well as any algorithmic directives for
i GAMS system then looks for additional commands
MSnal within the file trnsport.gms. To solve the problem by the remote solver. Thus, for our MINOS example,
Ro kestrel.opt includes the line
NFOdditi ucosminmgaanldoscaalrecoapsyfoolflotwhes:MINOS solver, the relevant kestrel_solver minos
A
I
model transport /all/; and optionally any MINOS directives on additional
option lp = minos; lines.
solve transport using lp minimizing z; After the problem instance has been submitted, the
Kestrel/GAMS agent displays a confirmation with a
To solve the same problem using MINOS remotely
job number and password, and the URL of a page
throughtheNEOSserver,wesimplychangethelinear
where current or final results can be viewed. After
programming solver to kestrel:
the remote solver finishes its work, the solution is
model transport /all/; returned to GAMS, which may further process and
transport.optfile = 1; reportitjustasiftheproblemhadbeensolvedlocally.
Dolanetal.: Kestrel: AnInterfacefromOptimizationModelingSystemstotheNEOSServer
532 INFORMSJournalonComputing20(4),pp.525–538,©2008INFORMS
4.3. Advantages of the Kestrel Modeling make it difficult, if not impossible, to use the classic
Language Agents NEOS submission mechanisms, but the Kestrel inter-
Some work and possibly expense is involved in face works well in these cases. As an example, this
preparing to use the Kestrel interface. The prospec- section describes an integration of Kestrel and other
tive user who does not already have local access to software to compute a so-called mountain pass solu-
AMPL or GAMS must acquire and install a copy— tion to a semilinear partial differential equation.
nospecialversionisneeded—eitherthroughpurchase The availability of the Kestrel interface also opens
or by downloading a freely available version limited up possibilities for NEOS services that involve more
s).g/. to 300 variables and constraints (and for GAMS also thanoneresource.Inparticular,anoptimizationprob-
r(or 2,000nonzeros,1,000nonlinearnonzeros,and50inte- lem may be preprocessed in some manner prior to
uthoms. ger variables). submitting it to the final solver for solution. The lat-
aor Also Python, available without charge for all pop- ter part of this chapter presents an example in which
otheals.inf upylatrhopnl.aotrfogr/mdos,wmnluosatd)b,eanindstaallePdyth(soene shctrtipp:t//iwmwplwe-. Ksoelsvterre”l opnretphreocNesEsOinSgseisrvuesr.ed by a “GAMS/AMPL
trn menting the AMPL or GAMS Kestrel agent must
syou be downloaded (from http://neos.mcs.anl.gov/neos/ 5.1. Integrating with Other Software
urtep://j kestrel.html). The Python scripts are only about 300– Many applications need to interact with third-party
cohtt 350 lines. We also offer the option of compiled Win- softwaretogenerateproblemdataandtovalidateand
aat dows versions of the Python scripts, in the form of visualize results obtained from an optimization prob-
asble application and auxiliary files occupying four to five lem. This software cannot be provided to the NEOS
a megabytes in total. server and is awkward to integrate with the inputs
opyvail Once this initial investment is made, the local user and outputs of the classical NEOS interfaces. Kestrel
ca canprovideanidealmechanismforsuchapplications.
gainsaccesstoalltheNEOSsolversthathavemodel-
s
dthiscies,i irnetgailnaningguatgheeinbteenrfeafictess—of18thateculorcraelntfweartiutirnegs—owf htihlee couAlsdasnerevxeamtoplein,twegeractoensNidEeOrSawcaisthe wMhAeTreLAKBestfroerl
butepoli AcoMmPmLanodrsGoAfMASMPmLodreemlinaginsayvstaeimlab.leT,hfeorinetxearamctpivlee, fifenrdenintigalneoqnutraitviioanl.sWoluetcioonnssitdoearssepmeciilfiinceaallryptahretiaLladnief--
strion to permit browsing through the solution and experi- Emden-Fowler equation
disi
s mentation with changes to the model or data. Multi-
dmi −(cid:5)u=up
aner ple files can also be read or written, optionally with
p connections to databases, spreadsheets, or the user’s
e with the Dirichlet boundary conditions u(cid:1)x(cid:3)=0 for
ticland own programs. all x∈(cid:9)(cid:10), where u is the function that we are trying
ars In the early stages of developing an application, to find, (cid:10) is the domain, (cid:9)(cid:10) is the boundary of the
sht the Kestrel agents also offer the advantage of allow-
thirig ing syntax and logic errors in the model and data domain, and p>0 is a constant exponent. Instead of
pyrightton,including tbvKoeaelsibdtdereeptlceracaoutgbegeldenhmttb,yliwonchsAateaMlrlneycPa.esLCs’oserterhltaoaeccrihnaNlthiEnpeOfrNeeSasEsoiiOlnbvtSieelirstfpeiaerhcsvaeesscrearv.neicOaeantilhvslyoee stioolnvainlgfutnhcFist(cid:1)ioupn(cid:3)ro=b(cid:1)le(cid:10)m(cid:2)(cid:8)d(cid:13)iure(cid:1)xct(cid:3)l(cid:8)y2,−weup(cid:1)cx+o(cid:3)np1+s1tr(cid:3)udcxt the varia-
coatio numerous AMPL and GAMS submissions that result with the Dirichlet boundary conditions. All of this
sm in nothing more than syntax error reports.
oldor Finally,theKestrelinterfacecanofferaconsiderable function’snontrivialcriticalpoints(whereu∗(cid:9)=0)cor-
hnf degree of security because it sends NEOS only prob- respondtosolutionstotheoriginalpartialdifferential
i
MSal lem instances in low-level (usually binary) formats, equation.
n
Ro andreceivesresultsonlyinsimilarformats.Theother Subject to reasonable assumptions, the Mountain
FOditi NEOS modes require models and data to be sent in Pass Theorem (Moré and Munson 2004) states that
INAd text forms that people can read—indeed, these forms between any two local minimizers of some function
f(cid:16) (cid:1)n →(cid:1), there exists a continuous path connect-
have been designed to promote readability—and are
ing these minimizers whose maximum value is min-
limited to producing results in readable listings (as
imal over all such paths. The critical point where the
discussed in §2).
maximum value is achieved along this optimal path
is termed the mountain pass. A mountain pass is not
5. Representative Further Uses
a locally optimal point of f but is rather a saddle
Optimization applications often take input from pointthatgivestheminimalchangeinfunctionvalue
numerous sources and require specific solver out- needed to make a transition from one minimizer to
put for further postprocessing. Such requirements another. In our example, each mountain pass of F is a
Dolanetal.: Kestrel: AnInterfacefromOptimizationModelingSystemstotheNEOSServer
INFORMSJournalonComputing20(4),pp.525–538,©2008INFORMS 533
critical point of F, so that solving the original partial
differential equations corresponds to finding moun- 9
tain passes of F. 8
7
We compute mountain passes by use of the elas-
6
tic string algorithm (Moré and Munson 2004). First,
5
we approximate the variational function F by dis- 4
cretizing the domain (cid:10) into a finite number of trian- 3
gular elements and applying the trapezoidal rule to 2
s).g/. approximate the integral on each of these elements. 1
r(or The approximate variational function, f(cid:1)u(cid:3), is then –1.00
authoorms. tahpeprsouxmimmaatetioinnteogvrearl oavllertrtihanegeulelamreenltesm. Uenstisngoff(cid:1)tuh(cid:3)e, –0.5 0 0 –0.5 –1.0
enf the elastic string algorithm solves the following non- 0.5 0.5
othals.i linearoptimizationproblemtofindanapproximation 1.0 1.0
trn to a mountain pass: Figure5 Nontrivial Solution to the Lane-Emden-Fowler Equation
syou withp=3
courtehttp://j Mu1i(cid:2)n(cid:4)(cid:4)(cid:4)i(cid:2)muK(cid:2)izwe w thiKsessittruealtitohnu.sOpnrolyviSdteespt(h4e) iinsteprefrafcoermoefdchroemiceotefolyr
subject to w≥f(cid:1)u (cid:3)(cid:2) k=1(cid:2)(cid:4)(cid:4)(cid:4)(cid:2)K(cid:2)
aat k on the NEOS server by use of the Kestrel/AMPL
asble (cid:8)uk−uk+1(cid:8)2≤h2k(cid:2) k=0(cid:2)(cid:4)(cid:4)(cid:4)(cid:2)K(cid:2) agentaspreviouslydescribed.Therestofthearrange-
opyvaila u0=ua(cid:2) mgreanmtsu.sAest MthAeTbLeAgiBnnainndgoothfeSrtelopca(l5l)y, eaxeficnuatlinAgMprPoL-
ca
sis uK+1=ub(cid:4) command writes a local file in AMPL’s standard
dthicies, Here,K isthenumberofbreakpointsinthepiecewise- fporromblaetmfoirnsretapnrceesse,ntwathioicnhoifsnroenaldinebayr ompetiamnsizaotfiona
butepoli lsiinoenaarldapispcrroetxiizmataitoinonotfothue,(cid:8)p·a(cid:8)this, uthkeisEuacfilindietaen-dnimoremn-, MAATcLoAmBplmeteexAfiMlePaLsmdeosdcerilbceodntbayinGinagys(e1v9e9r7a)l.sample
rin
distssio bhektwiseeansguivcecnesspivaerapmoeintetsr,acnodnsutraainndinug tahreetwdiostmanicne- davoamilaaibnlse,afrnodmthhetMtpA:/T/LwAwBws.cmricpst.sanfol.rgtohve/a∼ntamlyusniss,oanr/e
dmi a b
aner imizers of F separated by a mountain range. models/mountain.zip. A nontrivial solution to the
p For this particular application, we fix the number Lane-Emden-Fowler equation with p=3 on an annu-
e
clnd of breakpoints to a relatively small number and solve lus domain generated by this code is depicted in
artisa the optimization problem to obtain an approximate Figure 5.
sht mountain pass. Then, we use an efficient Newton
thirig method to compute an accurate mountain pass. The 5.2. Preprocessor Services on the NEOS Server
og key steps are as follows: Preprocessing techniques are applied to an optimiza-
tn
tdi (1) Construct the finite-dimensional approximation tion problem instance to make it more amenable
hu
gcl to the variational problem by decomposing the given to solution, such as by reducing the number of
rin
yi domain into elements, using a meshing package such variables and constraints (Andersen and Andersen
pn,
oo as TRIANGLE (Shewchuk 1996). 1995, Brearley et al. 1975, Ferris and Munson 2001,
cati (2) Calculate the starting and ending points u Savelsbergh1994)ortranslatingittoadifferentform.
sm a
oldor and ub. Thus,ratherthansubmitaproblemdirectlytoasolver,
hnf (3) ChooseparametersK andhfortheelasticstring a user may wish to submit it to some kind of pre-
i
MSal algorithm. processor that is available through the NEOS server.
n
Ro (4) Solve the nonlinear optimization problem. After simplifying or converting the problem in some
FOditi (5) Compute an accurate solution in MATLAB way, the preprocessor can then use a Kestrel agent to
NAd (Chapman 2005, Palm 2001) and analyze and visual- submitthemodifiedproblemtoaNEOSsolver,with-
I
ize the results. out further intervention by the user. Results from the
Only Steps (2)–(4) could be carried out remotely by solver are returned via the Kestrel agent to the pre-
the standard NEOS scheme of submitting modeling processor, which can optionally undo some effects of
language declarations and commands. The decompo- the preprocessing before returning results to the user.
sition in Step (1) requires specialized local software Thepreprocessorandthesolverneednotresideonthe
and Step (5) requires the execution of modeling lan- samemachineorhaveanythingelseincommonother
guage commands to produce a local file of descrip- thantheiravailabilitythroughtheNEOSserver.
tions that a MATLAB session can use to perform This arrangement is currently used to provide a
function, gradient, and Hessian evaluations. way of submitting GAMS models to Kestrel-enabled
Dolanetal.: Kestrel: AnInterfacefromOptimizationModelingSystemstotheNEOSServer
534 INFORMSJournalonComputing20(4),pp.525–538,©2008INFORMS
solvers requiring AMPL input. The preprocessor is 6. Submission and Retrieval Features
the GAMS/CONVERT tool (Bussieck et al. 2003), of the Kestrel Modeling
which takes as input a GAMS problem instance, and
Language Agents
produces as output a scalar model—having no index
We have assumed in previous examples that the local
sets—in any of several languages and formats.
processes—AMPL or GAMS, and the corresponding
UsingtheGAMS/CONVERTtool,theNEOSserver
Kestrel agent—remain active until results from the
can offer a “GAMS/AMPL solver” to which the user
remote solver have been returned. This arrangement
submits a GAMS model and an AMPL solver choice.
s).g/. The server routes this submission to a workstation works well for a user whose number of submissions
r(or where GAMS and AMPL are installed along with the and solution time per submission are moderate. In
hos. GAMS/CONVERTtoolandtheKestrelagent.Thefol- othercircumstances,however,theKestrelfacilitiescan
autorm lowing steps are performed on that workstation: be made considerably more useful and fault toler-
enf (1) The GAMS software package converts the ant by allowing job submission and retrieval to be
ths.i GAMS model and data to a GAMS problem instance. requested separately. Should a failure occur on the
tonal (2) TheGAMS/CONVERTtoolconvertstheGAMS local machine or in the NEOS server while a problem
r
syou problem instance to an equivalent problem instance is being solved on a remote workstation, the user can
urtep://j in scalar AMPL model format. request results again at a later time after the failure
cohtt (3) TheAMPLsoftwarepackageconvertsthescalar hasbeencorrected.Theusermayalsodeliberatelyter-
aat AMPL model to an AMPL problem instance. minatetheKestrelagentandperformothermodeling
asble ins(t4a)nTcehetoKtehsetreNlEaOgeSntsesruvbemr,iatslotnhgewAiMthPtLheprAobMlePmL teaxspkesctbeedfotroetarekqeuaeslotinnggttihmee.reTshueltussoefr’as isnotlevreimthtaatskiss
a
pyail solver choice. may even include the submission of other problems
ov
ca Atthispoint,theNEOSserverisinafamiliarstate, through the Kestrel interface; by automating this pos-
s
si essentially the same as when a modeler has made a sibility, we have been able to go beyond our original
thies, Kestrel submission from a locally running AMPL or designgoalsbyprovidingarudimentaryformofpar-
butedpolici GprAocMeeSdssetsosiosenndasthdeesAcrMibPeLd pinro§b4le.mThinusst,atnhcee stoervtheer allIenl pthrioscesesscitniogn., we first describe two simple Kestrel
strion specified AMPL solver, on a (usually different) work- extensions: one for retrieving results from submis-
disi station that is set up to run that solver. sions that have become disconnected, and the other
s
dmi Once the specified solver finishes its work, all for requesting cancellation of a submission. We then
aner stepsarereversed.Thesecondworkstationreturnsan discuss the enhanced utilities for managing multiple
p
e AMPLsolutionfiletotheNEOSserver,whichreturns
clnd submissions through Kestrel.
tia it to the first workstation, where GAMS/CONVERT
r
sahts translates it to the form GAMS expects. GAMS then 6.1. Retrieving Disconnected Submissions
thirig cboamckptloetethseitusswerovrkiabthyesNenEdOinSgsethrveerre.quested output Inevitably, some Kestrel agent processes terminate
og prematurely. If the termination of a Kestrel agent
ttdin The initial submission may be made through any occurs after the problem-submission call has com-
hu of the NEOS interfaces. In particular, when the initial
gcl pleted,however,thenfurtheroperationsontheNEOS
yriin submission is itself made through the Kestrel/GAMS server side are not affected. In particular, the NEOS
opon, interface, then the single line like server still queues the problem and solves it on a
cati kestrel_solver minos remote workstation.
sm
d Because the Web pages of intermediate results
olor in the GAMS kestrel.opt file described in §4 need
hnf only be replaced by the corresponding two lines are generated by the NEOS server independently of
i
MSal how the submission was made, termination of the
Ron kestrel_solver gams_ampl Kestrelagentdoesnotpreventsuchresultsfrombeing
FOditi option solver loqo; viewedthroughaWebinterfaceliketheoneshownin
NAd where LOQO is an example of a solver that sup- Figure4.Followingcompletionoftheremotesolver’s
I
ports AMPL but not GAMS input. Subsequent lines run,thecompleteresultlistingcanbeviewedthrough
may contain other AMPL option statements. When thesameWebinterfaceforaslongastheNEOSserver
thesubmissiontotheGAMS/AMPLsolverismadein keeps it on file—currently a few days.
this way, the problem instance passes twice through Termination of Kestrel/AMPL or Kestrel/GAMS
the NEOS server, once each in GAMS and AMPL also has no effect on the processes that eventually
form. Step (1) above can be skipped, however, be- return the results from the NEOS server. Thus, the
cause translation of the GAMS model and data to a local modeling system that requested the results can
probleminstancehasalreadybeendonelocallyonthe still obtain them, provided that it can restart the
GAMS user’s computer. Kestrelagentandcaninstructittoretrieveaprevious