MCS, a new approach to data treatment in astronomical projects

0.11 MB·English

by  G. Calderone

Proceedingsofthe363.WE-HeraeusSeminaron:“NeutronStars and Pulsars”(Postersand contributedtalks)
Physikzentrum BadHonnef, Germany,May.14−19,2006, eds.W.Becker,H.H.Huang, MPEReport291,pp.226-229
MCS, a new approach to data treatment in astronomical
projects
G. Calderone1 and L. Nicastro2
1
Istituto Nazionale diAstrofisica, IASF,Via U.La Malfa 153, 90146 Palermo, Italy
2
Istituto Nazionale diAstrofisica, IASF,Via P. Gobetti 101, 40129 Bologna, Italy
7
0
0
2
n Abstract. Today’s astronomicalprojects need computa- data. In this comparison customizing the MCS server is
a tionalsystemscapabletostoreandanalyzelargeamounts like writing a web page.
J of scientific data, to effectively share data with other re- MCS is the evolution of a software named SDAMS

4 searchInstitutes andtoeasilyimplementinformationser- (SPOrt Data Archiving and Management System)

vices to present data for different purposes (scientific, Nicastro & Calderone 2002,thatwasimplementedtosup-
1
maintenance, outreach, etc.). Due to the wide scenario of port the SPOrt experiment (Cortiglioni et al. 2004), an
v
2 astronomical projects there isn’t yet a standardized ap- Italian Space Agency (ASI) funded project which has
0 proach to implement the software needed to support all been“frozen”becauseoftheproblemswiththeColumbus
1 the requirements of a project. The new approachwe pro- module on the International Space Station. At that time
1
pose here is the use of a unified model where all data are the approach used to build SDAMS seemed to be easily
0
stored into the same data base becoming available in dif- portabletootherexperimentssoitsfeaturesandusability
7
0 ferent forms, to different users with different privileges. has been generalized until it became the actual MCS. In
/ fact it is now used to manage the data collected by the
h
optical camera ROSS (Tosti et al. 2004) mounted on the
p
- robotic telescope REM (Covino et al. 2004) at La Silla,
o Chile (see Nicastro & Calderone, this volume).
r 1. Introduction
t MCShasbeendevelopedonthe GNU/Linuxplatform
s
a Informationservicescanbeseparatedintwoclasses:those and is released under the GPL license. It can be freely
: in which the information produced are addressed to hu- downloadedfromthesiteross.iasfbo.inaf.it/mcs/.Thissite
v
i mans, and those in which they are meant for the use by containsallnews,updates,documentationanddownload-
X other software applications. In the former case there is able software packages. The documentation is present as
r a quite standardized way to develop such an information a user manual (in pdf format) which focuses on the main
a
service, essentially based upon a web server, a database aspectsofanMCS-basedapplication,describesthe server
server,ascriptinglanguageandHTMLpages.Inthelatter environment, commands, and the interfaces for various
case instead there is no such standardization, and here’s languages,andasatechnicalreferencedocumentationfor
why MCS (My Customizable Server) was implemented. all MCS’s classes (in html format, automatically gener-
ated using Doxygen). The site is still under development,
MCS is a set of software tools aimed at easily imple-
so check for updates.
mentinformationservices,thatisanapplicationthatpro-
vides a service over the network. At the core of a MCS-
based system there is a TCP server which listen for user 2. The MCS architecture
connections, and once a user is connected it will send all
The MCS core is basically a set of highlevelC++ classes
the requested information. However the transmitted data
aimedatprovidingallthefunctionalitiesofanapplication
aren’t in free format (like in a web page), but are packed
server, that is an application that will listen for user’s
inawelldefinedfashion(usingtheMCSprotocol)sothat
requestcoming from the network,eventually will accessa
on the other side a software can understand what is be-
databaseand/orexecuteanexternalprogram,thenitwill
ing sent. The MCS high level classes will hide all code
send the answer to the user using the MCS protocol.The
implementations related to multi-threading, networking,
main features of application servers built with MCS are:
database access, etc., and require no low-level knowledge
of these issues by the users. MCS can also be customized – easy to configure;
through the derivation of some classes. So MCS and its – authentication and grant support;
protocol are for software applications what a web server – secure connections (through SSL);
and HTTP are for the WWW: a simple way to access – database access (MySQL);
G. Calderone & L. Nicastro: MCS, a new approach to data treatment in astronomical projects 227
Table 1. MCS and Unix “shell” comparison.
TCP/IP socket
Unix shell MCS server
Database server MCS server Client
C, C++, Fortran, IDL, PHP stdin and stdout bidirectional TCP socket
system account MySQL account
Shell script internal commands base commands
IDSLQL programs, shell scripts external programs (EXEC command)
Perl home directory work directory
Fig.1. Main components of an MCS based system.
only supported database server is MySQL1. In the future
other servers may become accessible through MCS.
– base commandsset,supporttocreatenew customized
Application server:the applicationserveris the coreof
commands;
the information system. It implements the client/server
– accessibility (as clients) from other languages such as
model:a clientopens a TCPsockettowardsthe hostrun-
C, Fortran, IDL, PHP, etc.;
ningMCSandsendsarequest,thentheserver“computes”
– logging facility, etc.
an answer, eventually querying the database and/or exe-
cuting some external programs, and sends it back to the
All these features are already available, without perform-
client.ThebehavioroftheMCSservercanbecustomized
ing any customization. So to implement a simple service
deriving some classes.
with the above features you’ll only need to install MCS
External programs:externalprogramsaresoftwareap-
and configure it through a simple configuration file. The
plications written in any language, which interact with
only code needed is as follows:
the applicationserverviacommandline andthestandard
//Mandatory includes for every MCS-based appl. output. Support to these programs was added to easily
#include <mcs.hh> integrate already existing applications within MCS.
using namespace mcs; Clients: clients are programs which access the MCS ser-
//Main program vice over the network. Such programs can be written in
int main(int argc, char *argv[]) { anylanguageandrunonanyplatform,providedthatthey
//Start the server... implement the MCS protocol. Interfaces that implement
Env* env = mcsStart("simplest"); the MCS protocol are provided by the MCS library for
//...and wait for its end the following languages on the Linux platform: C++, C,
mcsWait(env); Fortran, IDL, PHP. Support for other languages (such as
} Java,PythonandPerl)andtheWindowsplatformwillbe
available soon.
For more complex services you can customize the server Note that sucha systemwould not be a closedmodel,
behavior in several ways: in the sense that existing databasesand/orsoftwaretools
maybeintegratedintotheMCS-basedsystems.That’sbe-
– addingexternalprograms,eitherrealexternalapplica-
cause MCS doesn’t make any assumption about type and
tions or batch lists of MCS commands;
quantityofdatayouneedtodealwith.Itusesarelational
– addingSQLprograms,tobeexecutedonthedatabase
databasesystemtostoremanyofitsdata.Databaseshave
server;
become today the most useful, scalable and flexible way
– adding customized commands, deriving the
to store and give access to any sort of data.
UserThreadclass;
Fromauser’spointofview MCSis verysimilarto the
– modifying the behavior of the server side thread, de-
usageofaclassicUnixshell,thatisacommandlineinter-
riving the LocalThreadclass;
facewithapromptonwhichuserscanexecutecommands
intheirownenvironmentandwaitfortheoutputbeforea
In a typical application you should implement a
new command is issued. It is therefore possible to make a
database with all the tables needed to store all the rel-
comparisonbetweenthe “components”of a shell,and the
evant data related to the project, and eventually prepare
ones from an MCS connection (see Tab. 1).
the required external programs. Figure 1 shows the typi-
The output of a command won’t be ASCII text like
cal architecture of an MCS-based system:
in a shell. Data will be instead formatted using the MCS
Database server: the database server is used to handle
protocol and can be sent and received in binary form in
clientsauthentication,tostoreallapplicationspecificdata
both directions. Figure 2 shows the typical sequence of
and anything else necessaryto the applicationitself. This
events during an MCS session.
server isn’t accessible directly from the clients, but it is
visible only to the application server. At the moment the 1 www.mysql.com
228 G. Calderone & L. Nicastro: MCS, a new approach to data treatment in astronomical projects
classcanbe usedto implementsomeserverside taskslike
CLIENT SERVER
data quick look or reduction, database maintenance, etc.
START
WAIT 3. A real life example
CONNECTION
OPEN A
SOCKET In a typical scientific experiment we have an instrument
THREAD producing data, a main storage system, a set of software
CREATION
tools to perform analysis,and people with different needs
who wish to access the data. In this section we’ll analyze
SEND the components of an informative system based on MCS,
WELCOME MSG
applied to such an experiment. A common use of MCS in
RECEIVING
WELCOME MSG suchareallifeprojectwouldbeasfollows:anMCSserver
WAIT is running on the computer attached to the scientific in-
COMMAND
strumentandcollectsdataonlocaldisks.Overthesedata
SEND
COMMAND the MCS server will eventually perform some automatic
"BYE" ?V THREAD analysisfordataquick-look,thenwillstoretheresultsand
DESTRUCTION
F the raw data in the database. The quick-look tool can be
implemented in C++, or it can be a previously written
PROCESSING software; in the latter case you only need to implement a
REQUEST
simple interfacebetweenthis toolandMCS. Atechnician
canthenconnectvia a simple telnetterminalto the MCS
SEND serverandcheck if everything goesfine. This canbe done
ANSWER
usingmnemoniccommandcodes,verysimilartothecom-
RECEIVE
ANSWER mands issued in an ordinary Unix shell. These command
codes can of course be customized. A researcher can use
the “Client” class to develop a client tool to connect to
T OTHER
COMMANDS? the MCS server and perform a remote control of the in-
strument or retrievescientific data in a variety of formats
F
TIME (typically formatsareraw,ASCII, FITSorVOTable).As
already mentioned, if the researcher doesn’t want to use
END
C++ to implement the client tool, he can use one of the
Fig.2. Flow diagram of a typical MCS session. available interfaces to MCS from other programming lan-
guages such as C, IDL, Fortran or PHP. One of the main
feature of these interfaces is that the function names are
MCS classes can also be used outside an application thesameinalllanguages.Finallytherecanbeawebserver
server, for example it is possible to write a program in that connects to MCS through a PHP interface and pro-
C++ which uses the DBConn and Query classes to videsbookkeepingandoutreachinformationaswebpages
access a database, or one of the VOT Parser Tree or toexternalusers.Eachafore-mentionedusercanofcourse
VOT Parser Stream classes to read VOTable file. The havedifferentprivileges,andsobeingabletoviewandre-
same programcanalsobe written in one ofthe languages trieve different kinds of data.
forwhichwehaveaninterface(atthemomentC,Fortran,
IDL, PHP; other will be developed in the near future). 4. MCS and the Virtual Observatory
MCS doesn’t pretend to be an all-comprehensive, multi-
2.1. Deriving MCS classes wavelength repository for astronomical data like the Vir-
tual Observatory is, nor a general purpose data analysis
DerivingaC++classmeanscreatinganewclassthathas
software. In this sense MCS is (of course) not a Virtual
all the behaviour characteristics of an existing one (the
Observatory competitor. Instead it was implemented to
parent class), plus some more specific behaviour added.
easily setup real time information services, while reusing
In the MCS case the server behaviour can be customized
much of the already existing software tools and database
through the derivation of the UserThread class (more
tables. However in the near future MCS will implement
specifically onlythe “virtual”methods shouldbe derived,
the PLASTIC2 interface to connect and use the Virtual
asdescribedinthedocumentation).Thiswayitispossible
Observatory facilities.
tocreatecustomcommandsavailableasiftheywere“base
MCScanalsobeseenasastartingpointtoimplement
commands” (see Tab. 1). Another class that can be de-
orsimply makepracticewitha distributedcomputingen-
rivedforcustomizationisLocalThread.Itrunsinaserver
sidethread,independently fromotherclientthreads.This 2 plastic.sourceforge.net
G. Calderone & L. Nicastro: MCS, a new approach to data treatment in astronomical projects 229
vironment like in the “GRID” based projects. A network 7. MCS is already used by...
ofcomputers,eachrunninganMCSserver,canconnectto
MCS is already used for managing the data produced by
alltheothersasclienttoperformcomputationordatabase
the ROSS camera. The SPOrt experiment, whenever it
queries.Ofcourse,alsointhiscaseMCSand“GRID”are
will be resumed, will have the data fully managed by an
not competitors because they are supposedto be usedfor
MCS based system. The Astrometry group of the Turin
differentpurposes:infactimplementingaserviceandset-
AstronomicalObservatoryismakinguseoftheMCSclient
ting up an MCS server is much more simple than setting
library through the IDL interface to cross match HTM
up a node or putting a service on the “GRID”. On the
indexedcatalogues.Otherastronomicalexperimentsfrom
otherhandthe“GRID”projectsoffermanymorefeatures
the gamma-raysto the radio band have shown interest to
than MCS does.
use MCS for their data management. One of them is the
Italian gamma-rays satellite AGILE.
5. The MCS companion tools
We have developed a number of software tools that can 8. Conclusions
beusedwithMCStoimproveitsfunctionalities(ofcourse
Several astronomical projects are still supported by soft-
these software can also be used as standalone):
waredevelopedunderanoldconception:handlingalldata
– MyRO (My Record Oriented privilege system): this fileswith“ad-hoc”developedscripts,runningstandalone
software provides a natural extension to the MySQL applications which can read only proprietary ASCII or
privilegesystem,offeringthepossibilitytospecifypriv- binary files, relying on users responsibility to follow the
ileges on a record level. It supports users and groups format and naming specifications for files, absence of any
likethoseofaUnixsystem,givingthepossibilitytoset mechanism to automatically update a web site, etc. How-
a read and/or write permission to each record. Needs evertodayinformationtechnologyoffersalotofnewtools
MySQL version 5. which can give several benefits to astronomers: database
– DBEngine for FITS and VOTable: a DBEngine systems to efficiently store andretrieve huge quantities of
is a software library linked into the database server data,librariestoeasilydistributedataacrossthenetwork,
(MySQL in this case) which let users read and write widely usedstandardformats for informationinterchange
table in a format different from the MySQL propri- usingfreelyavailablelibraries,newhighlevellanguagesto
etary. Our software let you read and write FITS and quickly implement sophisticated data handling and pro-
VOTablefilesasiftheyweredatabasetables(thissoft- cessing, and so on. All of these tools are being effectively
ware is still under development). usedbytheVirtualObservatoryandGRIDprojects.They
– Database interface to HEALPix and HTM: will become the standard “de-facto” in the near future,
this software provides some of the functions of the so software developers involved in astronomical projects
HEALPixandHTMlibraryasMySQLfunctions,tobe should start to get used to these new technologies. MCS
useddirectlyinSQLqueries.Functionsallowing(fast) aims at being the simplestand fastesttool to let develop-
circular and rectangular selections of entries in HTM ers build new software to support scientific projectsusing
indexed database tables will also be available soon. these new technologies.
Acknowledgements. The SPOrt experiment was supported by
6. The future
ASI.The REM Observatory is supported by INAF.
MCSisincontinuousdevelopment.Itwillsoonimplement
interfaces for other languages such as Python, Perl, Java,
References
Tcl/Tk. Another feature that will be implemented soon
is the integration with MyRO to handle a more complex Cortiglioni S., Bernardi B., Carretti E., et al., 2004, New As-
privilege system. tron. 9, 297
User contributed libraries for the various supported Covino S., Stefanon M., Sciuto G., et al., 2004, SPIE Sym-
posium, Proc 5492, 1613, G. Hasinger & M. J. L. Turner
languages are being built. This will allow an even eas-
eds.
ier access to the MCS functionalities to non expert pro-
Nicastro L., Calderone G., 2002, AIP Conf. Proc. 609, 279, S.
grammers. Moreover, commonly used, independently de-
Cecchini, S. Cortiglioni, R. Sault & C. Sbarra eds.
veloped externalpackages will also be included and made
Tosti G., Bagaglia M., Campeggi C., et al., 2004, SPIE Sym-
accessibletroughMCS.Theyinclude the aforementioned
posium, Proc 5492, 689, G. Hasinger & M. J. L. Turner
HEALPix and HTM libraries for sky pixelizationscheme, eds.
the Naval Observatory Vector Astrometry Subroutines
(NOVAS – aa.usno.navy.mil/software/novas)used for as-
trometriccalculationsandtransformations,theWorldCo-
ordinate System (WCS – fits.gsfc.nasa.gov/fits wcs.html)
library and tools.