Data storage, processing and visualisation for the ATCA

0.54 MB·English

by  T. Murphy

Data storage, processing and visualisation for the ATCA
T. MurphyA,C,D, P. LambB, C. OwenC, and M. MarquardingC
ASchool of Physics, University of Sydney,NSW,2006, Australia
BCSIRO ICT Centre, Canberra, ACT, 2601, Australia
CAustralia Telescope National Facility, Epping, NSW,1710, Australia
DEmail: [email protected]
Abstract: We present three Virtual Observatory tools developed at the ATNF for the storage, pro-
cessingandvisualisationofATCAdata. ThesearetheAustraliaTelescopeOnlineArchive,aprototype
datareductionpipeline,andtheRemoteVisualisation System. Thesetoolsweredevelopedinthecon-
6 text of the Virtual Observatory and were intended to be both useful for astronomers and technology
0
demonstrators. Wediscuss thedesign and implementation of these tools, as well as issues that should
0
be considered when developing similar systems for futuretelescopes.
2
Keywords: astronomical data bases: miscellaneous — methods: data analysis
n
a
J
1 Introduction 2 The ATCA
6
1
TheAustraliaTelescopeCompactArray(ATCA)isan

Theso-calleddataexplosioninastronomypromisesex- east-westearth-rotationsynthesisinterferometer,with
1
v citing new scientific developments, but brings with it six 22 m antennas on a 6 km baseline. It has been in
4 manytechnicalchallenges,incollecting,storing,trans- operation at Narrabri since 1990. The telescope can
5 porting, processing and visualising data. Virtual Ob- observe at 6 bands with wavelengths 20 cm, 12 cm, 6
3 servatories(VO)havedevelopedtomeetsomeofthese cm,3cm,1cmand3mm. Eachantennaobservestwo
1 technical challenges. Falling under the broad area of frequencies simultaneously. There are six bandwidths
0 e-science (which incorporates other scientific domains available on Frequency 1 (128, 64, 32, 16, 8 and 4
6 facingsimilar challenges, suchasgeneticsandparticle MHz) and two bandwidths on Frequency 2 (128 and
0 physics) theaim of VirtualObservatory research is to 64 MHz). The telescope produces ∼ 0.5 GB of raw
/ providethetools necessary for dealing with this data. dataperday,andthisislikelytoincreasesignificantly
h
p TheAustralianVirtualObservatory(Aus-VO)1was with future telescope upgrades.
- started in 2003 with the aim of both contributing to In the rest of this section we outline the exist-
o the international VO effort, and developing tools of ing systems for archiving, processing and visualising
r ATCA data.
use to Australian astronomers. Australia has many
t
s areas of expertise (for example radio astronomy) and
a
it makes sense to focus our efforts on providing mod- 2.1 Data archiving
:
v ern tools for working in these areas. In this context
i theATNFdecidedtodeveloparangeoftoolsforstor- SincethecommencementofoperationoftheATCAin
X ing,processingandvisualisingdatafromtheAustralia June1990,acompleterecordofalldataobservedfrom
r Telescope Compact Array. The aim was that these the telescope has been maintained offline at the tele-
a tools would be useful to astronomers now, and at the scopesite,mostlyrecordedonCD.Inconjunctionwith
sametimeletusexplorethetechnologythatwouldbe this,theATNFmaintainedarecordoftheprojectpro-
necessary fordevelopingsoftware forfuturetelescopes posals for observations onthetelescope –the Projects
such as theSquareKilometre Array (SKA). database—andashortformoftheobservationparam-
etersforeachdaysobserving–thePositions database.
This paper is based on a talk given at the ASA
After a proprietary period of 18 months, in which the
Annual Meeting in Sydney in July 2005. After giv-
observing team has sole access to the data obtained
ing some background about the ATCA and the Vir-
inanobservation, thedataismadepubliclyavailable.
tual Observatory, we discuss three tools developed at
AstronomerscansearchforobservationsontheATNF
the ATNF over the last few years. Firstly the Aus-
webpage, submit the details of the observation data
tralia Telescope Online Archive which contains all of
requiredviae-mailandhaveaCDcontainingthedata
thedatacollectedsofarbytheATCA;secondlyapro-
prepared for them at nominal cost.
totype data reduction pipeline for ATCA data; and
finally the Remote Visualisation System for viewing
2.2 Data processing
large datasets.
ATCA Data processing (reduction) is generally per-
formed with one of the standard radio data reduction
1http://www.aus-vo.org packages; most commonly Miriad (Sault et al. 1995),
1
2 Publications of theAstronomical Society of Australia
but also AIPS2 and AIPS++3. After loading, editing • Data Curation IG
and calibrating the data, the resulting product is an • Theory IG
intensity map referred to as the dirty image. At this
stage a deconvolution algorithm, usually a variant of whichfocusontherequirementsofparticularapplica-
clean(H¨ogbom1974),isrequiredtoproducethefinal tion domains.
image. TheaimoftheAustralianVirtualObservatory(Aus-
At each stage in the process there are a range of VO)istoprovidedistributed,uniforminterfacestothe
parameters that can beset tocontrol thetypeof pro- data archives of Australia’s major observatories and
cessing performed. Both general parameters (such as the archives of simulation data. Aus-VO is a collab-
calibration strategy, clean method and type of data oration between many Australian institutions, includ-
editing)aswellasfine-grainedparameters(suchascal- ingtheUniversitiesofMelbourne,Sydney,NewSouth
ibration solution interval, numberof clean iterations WalesandQueensland,MonashUniversity,Swinburne
and median filter size) need to be modified to obtain UniversityofTechnology,theAustralianNationalUni-
the best results. Hence data processing is typically a versityandMountStromloObservatory,theVictorian
highly interactiveprocess. Partnership for AdvancedComputing, theATNFand
There is an existing system operating at the tele- theAAO.
scopeCAONISdesignedforon-the-flyimagingofATCA There are a range of VO projects underway in
data. However the design of this system makes it dif- Australia, including the development of data archives
ficult to port to current Linuxsystems. and software for HIPASS (Meyer et al. 2004), RAVE
(Siebert 2004), 2QZ (Croom et al. 2004) and SUMSS
(Bock et al. 1999). The initial focus of most of these
2.3 Data visualisation
projectshasbeentomakedatafromAustralianprojects
widely available within the international community,
The final images from the ATCA are usually visu-
in a VO compliant format. In addition there are sev-
alisedusingtoolssuchasMiriadorkvis(Gooch1996).
eral projects investigating novel methods for astro-
These are well established tools that cover many of
nomical data mining and data analysis, for example
thevisualisation requirementsofATCAobservers. As
Rohdeet al.(2005)applymachinelearningtechniques
mentioned in the previous section, the CAONIS system
to catalogue crossmatching. The Melbourne Univer-
which runs at Narrabri also allows basic visualisation
sitygrouphasalsobeensettingupinfrastructuresuch
of images.
asaregistryforAustralianwebservicesanddataarchives.
3 The Virtual Observatory
4 The Australia Telescope
TheumbrellaorganisationforVirtualObservatorywork Online Archive
istheInternationalVirtualObservatoryAlliance(IVOA).
TheIVOAwasformed inJune2002 withamission to In June 2003, a joint project between the ATNF and
the CSIRO ICT Centre was commenced to make the
facilitatetheinternationalcoordinationand
ATNF archive data available online as the Australia
collaborationnecessaryforthedevelopment
Telescope Online Archive (ATOA). This was planned
and deployment of the tools, systems and
as anew dataresource for astronomers, as well as the
organizational structures necessary to en-
foundationforthedevelopmentofonlinedataprocess-
able the international utilization of astro-
ing systems to make the raw data more accessible to
nomical archives as an integrated and in-
non-expert users (see Section 5). The construction of
teroperating virtual observatory.
the ATOA required the copying of the offline archive
TheIVOAisacollaborationbetweenover15mem- (atthetime,∼2700CDs,totalling∼1.7TB)fromthe
bercountriesincludingAustralia. Thefocussofarhas telescopesitetoCanberrawheretheonlinearchivewas
been developing the standards required for interoper- to be developed, creating a meta-database describing
abilitybetweensoftwaredevelopedanddataproduced the data, and making a web front-end to search and
inallareasofastronomy. Anothersignificantaimisto download thedata.
develop the infrastructure required (networks and or- Thedatabaseconsistsoftwoparts. Thefirstisthe
ganisations)forthelargescalestorageanddistribution raw data from the telescope (RPFITS files) which is
of astronomical data. stored as normal files on the host system. In addition
The IVOA working groups address a range of is- thereisarelationaldatabasewhichstoresallthemeta-
suessuchasgridandwebservices,datamodellingand data (discussed in Section 4.1). The ‘vital statistics’
standards for the data access. There are also four in- of the ATOA are shown in Table 1. The current rate
terest groups of growth of the archive is ∼ 0.5 Gb/day. However
• Applications IG this is likely to increase significantly in the future as
new instruments come online. To maintain an grow-
• Astronomy Grid IG ingarchive(ratherthanastaticone)itisnecessaryto
2Astronomical Image Processing System (AIPS), ensure the RPFITS files are stored in a readily acces-
sible way (currently on a RAID system) that is easily
http://www.cv.nrao.edu/aips.
3Astronomical Image Processing System (AIPS++), distributed over a number of drives. Also, that the
http://aips2.nrao.edu. database itself is easy to update in a robust manner.
www.publish.csiro.au/journals/pasa 3
The ATOA was made publicly available in December 4.2 A data model for the ATCA
2004andcanbeaccessedfromhttp://atoa.atnf.csiro.au.
A data model is a comprehensive scheme describing
how data is to be represented, for manipulation by
humansor computerprograms. Data models are crit-
Table 1: ATOA Statistics ical for planning how data will be organised within a
Projects 2261 databaseastheydescribealltherelationshipsbetween
Files 57147 thedifferent entities.
Sources 128111 AsectionofourdatamodelfortheATCAisshown
Metadata size ∼4 Gb inFigure1. WenowbrieflyexplaintheUML(Unified
RPFITS data size ∼2 Tb ModellingLanguage)notationusedinthedatamodel.
Each boxcontainsan entity(e.g. Scan) that hasbeen
Growth rate ∼0.5 Gb/day
identified in the metadata. Each entity has attributes
(e.g. restFreq), each of which are of a specified data
type (e.g. float). Associated entities are connected
to each other with lines, which also specify the cardi-
nality of the relationship. For example
4.1 Metadata
Scan SPW
1..* 0..*
Metadata is simply data which describes other data,
for example the project code or the name of the pri- should bereadas“A scan has 0 or more spectral
mary calibrator. The meta-database for the ATOA windows. A spectral windows has 1 or more
consists of three main parts; thecontentsof theorigi- scans.”
nalATNFonlineProjectsdatabase,metadatadescrib- The development of a data model that covers the
ingtheobservation thatisextracteddirectly from the wholeofastronomyisanongoingprojectwithinthein-
rawdatafilesproducedbythetelescopessoftware,and ternational VO community. We havecontributed this
metadatainferredfromalloftheavailabledatasources data model to the IVOA Data Model WG as an ex-
to assist in the automation of reducing the telescopes ample of a data model for radio astronomy. For more
raw data to images. The types of metadata used in information on this topic, see the IVOA Data Mod-
theATOA are summarised in Table 2. elling website4.
The ATOA archive database structure is created
directlyfromthedefinitionsintheATOAdatamodel.
Table 2: Metadata in the ATOA Partsofthedatamodelcontaininformationforspecific
databaseimplementationssothatalloftheimplementation-
Metadata source Examples specific parts of the database creation are handled in
Projects database proposal; observer name this process. The data model in Figure 1 corresponds
country; institution tothepartofthedatamodelthatdescribesthemeta-
Positions database∗ source names & positions data contained directly in the archive RPFITS files.
Thedatamodelfor theinferreddataisavailablefrom
observing band; receivers
theATOA web pages5.
RPFITS files scans; polarisations
array configuration
Inferred calibrator names & roles 4.3 Implementation
calibrator–targetmatches
The ATOA web interface was implemented as a Java
∗ The positions database is included in our data (ver. 1.4.2)6applicationandishostedusingtheApache
model, and some of the metadata is used to Tomcat(ver. 5.0.28)7webcontainer. Relationaldatabase
reconstruct the observation metadata. However, servicesareprovidedbyanOracle9iinstancerunning
it is not actually loaded into the ATOA. on thesame machine. Aweb based interface was cho-
sensoastomaximiseinteroperabilityandprovideeasy
access to users. For example, RPFITS files may sim-
plybedownloadedbyconstructingasuitableURLfor
the ATOA file server. This allows files on the server
Most of the metadata available in the ATNF Po-
tobedownloadedbyaWebbrowser,bycommand-line
sitions database is also available from the metadata
programsthatallowuserstofetchthedatareferredto
in the raw data files, and is finer-grained, since the
by a URL, or by application programs using libraries
Positionsdataisadailysummary,whilethefilemeta-
that allow a URL to be opened in a similar way to a
data is available for each telescope pointing. The in-
file on a local file system.
ferred metadata in the ATOA is ‘value added’ infor-
mation that is automatically determined from the ex- 4http://www.ivoa.net/twiki/bin/view/IVOA/IvoaDataModel
isting metadata, for example the calibration role of 5http://www.atnf.csiro.au/computing/web/atoa/implementation.html
each source (primary calibrator, secondary calibrator, 6http://java.sun.com
target, etc). This is discussed furtherin Section 5.2. 7http://tomcat.apache.org
4 Publications of theAstronomical Society of Australia
The user interface centres around two main web pipeline which can process single pointing continuum
pages: the query page which allows users to specify data from the ATOA and is available for testing at
criteria for selecting RPFITS files from the archive, http://atoa.atnf.csiro.au/test.
and aresults page which providesthemeans for users
toinspectthemetadataofmatchingfilesanddownload
5.1 Metadata for automatic
particular files if desired. The results page initially
processing
presentstheuserwithabroad,globalviewofthequery
resultsintabularformlistingdetailssuchasfilename,
The metadata in the Project and Position databases,
filesize,principalinvestigatorandarrayconfiguration.
whileprovidinginformationaboutwhichastronomical
Theusermayalsointeractively‘drill-down’foramore
sources have been recorded in an observing session,
detailed view of any file in the list. RPFITS files can
does not (in general) provide any information about
be downloaded individually or in batches.
therolethattheobserverintendedthesourcetoplayin
AsmentionedinSection2.1ATCAdatahasapro-
theobservation(eg. primarycalibrator,targetsource).
prietary period of 18 months, in which it is only ac-
This would be relatively easy to record in a new sys-
cessible to members of the project team. Authorised
tem, but as we are dealing with existing data we had
accessissupportedfordatawithintheproprietaryac-
to infer the roles of sources.
cessperiod. Ifauserwishestoaccessproprietarydata
Another problem for automatic processing is the
theymustfirstgothroughamanuallyverifiedauthen-
grouping of data into valid ‘observations’. An expert
ticationprocessafterwhichapasswordisissuedtothe
would typically choose an appropriate subset of files
Principal Investigator for that project. In the future
from the archive to image. However, a non-radio as-
we plan to replace this authentication and authorisa-
tronomermaychooseansubsetthatcontainsfilesthat
tion method with a streamlined system that links the
newATNFproposalsystem,OPAL8,anditsauthenti- shouldbeimagedseparately,orfilesthatcontaindata
that should be ignored entirely. Although it is impos-
cationdatabasetotheATOA.Userswillthenbegiven
sible to deal with all cases, our aim was to have the
accesstoproprietarydatabyusingtheirOPALcreden-
pipelinegrouptogethertheselecteddatainsuchaway
tials basedon theprojectstheyareassociated with in
that an image could bemadein at least 80% of cases.
theOPAL system
A wide range of observation types can be recognised
TheATOAwebserveranddatabasearehostedon
andcharacterisedusingthemeta-databasebutarenot
aDellPowerEdge750 runningDebianLinux3.0. The
yet processed by the prototype pipeline (e.g. millime-
hosthasa2.8GHzPentium4processor, 2GBofRAM
tre and spectral line observations).
andisattachedtoa3terabyteAppleXserveRAIDfor
In the following section we discuss how we assign
archive storage.
the source roles within an observation, and the algo-
rithmweusedtomatchtargetsourceswiththeappro-
5 A data processing pipeline priate calibrators.
framework
5.2 Determining source roles
The data products in radio astronomy are often less While matching target sources with their calibrators
accessible to the non-expert than those in other do- wouldbestraightforwardforanastronomeritisachal-
mainssuchasopticalastronomy. Itrequiresareason- lenge for an automatic system. In a typical (simple)
ably highlevel ofdomain expertisetoprocess theraw observing session the primary calibrator is recorded
data and produce an image. Obviously for carrying for a short period at the start or end of the observing
out detailed scientific analysis it would be necessary session;andalternatingpointingsaremadetothesec-
to develop this expertise, or collaborate with a radio ondary,and thesource of interest, or target. However
astronomer. Howeverinaneraofmultiwavelengthas- there is a great variety of different ways that the ob-
tronomy, astronomers expect to download and com- servercanchoosetostructuretheirobservations. Ifan
pare data from a variety of telescopes, at a variety of observationcontainsmorethanonetarget,thetargets
wavelengths. mayshare,orhavedistinct,secondarycalibrators, de-
Withthisinmindwehavedevelopedanautomatic pending on their separation in thesky. There may be
pipeline for people who want to quickly inspect the several secondary calibrators for each target, and the
data in the ATOA, to see if it was suitable for fur- same source may be used for primary and secondary
ther processing. One of the aims of this project was calibration. Inaddition,someobserversusesecondary
to test the viability of ‘driving’ the pipeline using the calibratorsthatarenotinthelistofrecommendedcal-
metadata discussed in Section 4.1. In other words ibrators, and that list has itself changed overtime.
the pipeline should make decisions about what kind Inordertoclassifythesourcesinanobservingses-
of processing to do — both on a general (e.g. contin- sion the following metadata is used
uum/spectral line) and specific level (e.g. number of
clean iterations). • The locations and names of sources extracted
Inthissectionwediscussthedevelopmentofextra from the raw telescope data
metadatarequiredtodrivingthepipeline,inparticular • Thetimesanddurationsofthesourcepointings
thecalibrationprocess. Wethenoutlineourprototype
• The names and locations of the four primary
8http://opal.atnf.csiro.au/ calibrators commonly used at the ATCA
www.publish.csiro.au/journals/pasa 5
• A recent ATCA catalogue of recommended sec- minutes. We used the OpenPBS Batch Queuing Sys-
ondary calibrators tem (ver. 2.3)13 for queue management, but unfortu-
• Names of sources extracted from project titles nately it has no mechanism for reporting job comple-
tion to another program. After processing for a web
• Apre-assembledlistofpossiblecalibratorsources service completes, the batch job doing the processing
Once the source roles have been determined, the sends a completion message to the program invoked
proximityin theskyand theproximity in observation by the web service that controls the execution of the
timeofthetargetsandtheirsecondarycalibrators are processing for theservice. However, at this point, the
used to match targets with their respective calibra- batch processing system has not yet transferred the
job’s output data back to the pipeline server. The
tor(s). Foreachtargetpointing,aweightiscalculated
for each secondary calibration pointing made within control program then polls the PBS batch queue at
five second intervals to ensure that the batch job has
two hours of observation of thetarget pointing:
completed.
wt,s=ΣPΣSe−(3a/a2max)2e−(3∆t/∆2tmax)2 filesTfrhoemratwheddataatafrpormoctehsesinAgT,OthAe,laolgltfihleesi,natenrdmtehdeiartee-
sulting images are stored temporarily on the pipeline
∆tt,s<∆tmax server. The first web service call made by a pipeline
where S is the set of candidate secondary calibrators, client reserves a private location for storage, and re-
aistheangularseparationbetweenthetargetandthe quests a lifetime for the storage. The pipeline server
secondary calibrator, amax is the maximum desirable has a configurable maximum lifetime, and the stored
separation between the target and the secondary cal- data will be deleted after this time expires. Only
ibrator (and is a function of the observing frequency clients who have the name of the storage area (a ran-
band). ∆t is the separation of the time midpoints of domly generated string) can access it. There is no
the target and calibrator pointings and ∆tmax is the quota on the storage use of any individual temporary
maximum desirable time separation (two hours). The storagearea. However,aquotamaybeimposedonthe
summation is over all pointings at a target (ΣP) and total amount of storage available to all active storage
all secondary calibrators within two hours of a target areas.
pointing(ΣS). Thewt,sareusedtoselectsuitablesec-
ondary calibrators for the respective targets from the
calibrators whosewt,s weightsdominateforaparticu- ATOA ATOA
lar target. metadata raw data
This procedure constructs the metadata required
forcontinuumimagingatcentimetrewavelengths. The
ATOA query server ATOA file server
algorithm works well in general, but there are some
problematic cases, for example where the target is a
source from thesecondary calibrator catalogue.
5.3 Implementation
Pipeline server
The underlying processing of the ATCA data is car-
riedoutusingtheGlishscriptinglanguageinAIPS++. Session state
The ATOA imaging Web Services interface was con-
structed using the Apache Axis tools (ver. 1.1)9, and
interfacestotheprocessingscriptsthroughaPerl(ver.
5.4.8)10 script that deals with the control of the exe- Remote Visualisation
Pipeline client
cution of the Glish scripts. System
The pipeline client is written using Python (ver.
2.3)11,andtheSOAPpywebservicestools(ver. 0.11.3)12.
There were some minor, but difficult to find, prob- Figure 2: System architecture. This schematic
lemsininteroperation betweentheSOAPpytoolsand shows the relationship between the three tools we
ApacheAxis;thedata structuresused in theweb ser- have developed.
vices calls are possibly more complicated than had
been previously used between the two web services
implementations. Documentation in both was not as
TheATOAandpipelinewebservicesreturnaURL
informative or complete as it might havebeen.
forthegeneratedimagestotheend-user’ssystem. This
Thepipelinewebservicescanbeconfiguredtorun
allows the URL to be passed on to the Remote Visu-
directly on the server host, or be directed to run on
alisation System(seeSection6)imageviewingsystem
othermachinesthroughabatchqueuingsystem,since
so thattheimage can beviewed onlinewhile it isstill
some stages in the pipeline can run for several CPU
residentonthepipelineserver. Figure2showtheover-
9http://ws.apache.org/axis all system architecture, in particular how the ATOA,
10http://www.perl.com pipeline and RVSinteract.
11http://python.org
12http://pywebsvcs.sourceforge.net 13http://www.openpbs.org
6 Publications of theAstronomical Society of Australia
6 The Remote Visualisation
large datasets. For example a 1.5 Gb data cube from
System theGalacticAll-SkySurvey(GASS)(McClure-Griffiths et al.
2005) takes about one minute to load. Compare this
with downloading the full cube from say the U.S. to
TheRemoteVisualisationSystem(RVS)wasdesigned Australiawhichcouldtake∼1−2hours. Formorein-
to enable visualisation of and interaction with large
formationanddirectaccesstoRVS,seehttp://www.atnf.csiro.au/vo/r
astronomical images in the context of the VO.As op-
posed to other VO image displays, such as CDS Al-
adin (Ferniqueet al. 2004), RVS does not require the 7 Discussion
usertodownload thedatatotheclient machine. Fur-
thermoreitprovidesrenderingofimagecubes,suchas TheATOAhasbeenpublicsinceDecember2004. We
spectral-linecubescreatedfromATCAdata. TheRVS
hope that it will encourage the reuse of ATCA data
serveracceptsFITSimages-whichcanbecompressed
forprojectsotherthanthoseitwasoriginallyintended
- through local file:// URLsand remote http or ftp
for. The framework used for the ATOA could easily
access. Thedatashouldbeco-located withoratleast beextendedtoincludedatafromothertelescopesand
be available to the server on high bandwidth connec-
can be updatedas additional metadata is required.
tion,whileitplacesnosuchrequirementsontheclient.
The most significant improvement of the ATOA
Only minimal data transfer to the client is necessary over existing online archives (such as NRAO15 and
and this is independent of the size of the source data MAST16) is the data delivery mechanism. Most ex-
set. The server-side architecture is distributed to en-
isting archives do not support on demand delivery of
able workload sharing and extensibility. RVS makes
data over the web, instead requiring the user to sub-
use of several software components: CORBA to make
mitaformrequestingfilesthatthenhavetobetrans-
it distributed, AIPS++ as the image rendering com-
feredtoapubliclyaccessibleftpsiteortoothermedia
ponent and Java for the web services and client. The
(such as CD) for physical delivery. In the ATOA, the
architecture of theRVS system is shown in Figure 3.
batch downloading of multiple files is handled by a
streaming TAR or ZIP archiving algorithm that per-
formsdynamicarchivingasfilesarestreamedoverthe
web, requiring no additional disk space on the server
for theseoperations.
IndevelopingtheATCAdatamodelandconsider-
ing the type of metadata required for automatic pro-
cessing we identified several new metadata types that
would be useful to store in the RPFITS files. As a
result thefollowing fieldshavebeenadded totheRP-
FITS files and will be available in all future ATCA
data:
• four calibrator codes
C (standard phasecalibrator)
F (primary flux calibrator)
B (bandpass calibrator)
Figure 3: RVS system architecture.
P (pointingcalibrator)
• Pointing offsets
RVSisexposedthroughawebserviceinterfaceus- • Weatherdata: added rain gauge and phase rms
ing the standard Web Service Description Language and difference
(WSDLver. 1.1)14. Thiscan easily beintegratedinto • Attenuatorsettings at start of scan
custom applications. Several client applications make
use of the web service interface; the RVSViewer - a • Subreflectorposition
traditionalimageviewer,athumbnailservice-provid- • Correlator configuration
ing preview images and a session viewer. The session
• Scan type
viewer connects to an existing RVSViewer via a key.
Multiple instances can be runat thesame time, mak- • Coordinate type
ing it apossible to useit as aconferencing tool where • Linemode
people can observe and interact with the data. The
ATOA pipeline re-uses the existing RVSViewer client • CACAL counter
by passing it thefile location of theoutput image. Thesewillhelpbothautomaticprocessingsystemsand
RVSisnotspecifictotheATOAorprototypepipeline astronomersassessthedataqualityintheobservations
andthereareplanstouseitforallATCAarchives. It theyareinterestedin. Afulle-logbook systemwill be
hasbeensuccessfully testedonimagesanddatacubes used in the future as currently the logs are all stored
from various surveys and has good performance on
15http://archive.nrao.edu/archive/e2earchive.jsp
14http://www.w3.org/TR/wsdl 16http://archive.stsci.edu/
www.publish.csiro.au/journals/pasa 7
on paperat thetelescope and henceare not easily ac- H¨ogbom, J. A. 1974, A&AS,15, 417
cessible to ATOAusers.
We have developed a prototype pipeline for pro- Meyer, M. J., et al. 2004, MNRAS,350, 1195
cessing of raw data for single-pointing continuum im-
McClure-Griffiths, N. M., et al. 2006, ApJ, in press,
ages. This is attached to the ATOA to provide an
astro-ph/0510304
improvedservicefor usersoftheATOA.Atthisstage
the image quality is suitable for previewing the data Rohde, D.J., et al. 2005, MNRAS,360, 69
in archive to see if it is of interest. Further manual
processingwould thenberequiredtoobtainimagesof Sault, R. J., Teuben, P. J., & Wright, M. C. H. 1995,
scientific quality. in ASP Conf. Ser. 77: Astronomical Data Analysis
A significant challenge in developing the ATOA Software and Systems IV,433
andtheprototypepipelinewereintegratingpre-existing
software with modern software tools. For example, Siebert, A. 2004, in SF2A-2004: Semaine de
the Glish scripting language has no web service li- l’AstrophysiqueFrancaise, 567–+
braries and so an extra layer had to be developed be-
tween the data processing level and the web services.
If re-implementing from scratch, a language such as
Python would be a better alternative for developing
thepipeline.
In developing these tools we have started to ex-
plore the techniques necessary for astronomical soft-
ware development in the VO era. This is essential for
future telescopes and surveys that Australia will pro-
duce. Making access to existing Australian data as
easy as possible will maximise its use in the interna-
tional community.
Acknowledgments
The authors would like to acknowledge the software
development done on the RVS project, primarily by
Anil Chandra and also by Praveena Tokachichu. The
ATNF side of the prototype pipeline and ATOA de-
velopment was managed by Neil Killeen and Jessica
Chapman. Vince McIntyre contributed extensively to
allthreeprojects, inparticularinsettingupthehard-
ware required.
A number of ATNF staff put in significant effort
to get the ATOA set up, in particular Robin Wark,
Bob Sault and Mark Wieringa. Warwick Wilson and
MarkWieringaimplementedthechangestoaddextra
metadata to the RPFITS files.
FromtheICTCentre,RobertPowermadetheini-
tial data model designs, the ATOA data loader soft-
ware and ATOA query front end. Geoff Squire and
Bella Robinson made significant contributions to the
prototypepipeline.
References
Bock,D.C.-J.,Large,M.I.,&Sadler,E.M.1999,AJ,
117, 1578
Clark, B. G. 1980, A&A,89, 377
Croom, S. M., et al. 2004, MNRAS,349, 1397
Fernique, P., et al. 2004, in Toward an International
Virtual Observatory,271–+
Gooch, R.1996, in ASPConf. Ser.101: Astronomical
Data Analysis Software and Systems V,80
8 Publications of theAstronomical Society of Australia
Figure 1: A portion of the data model for the ATCA. For an explanation of the no-
tation see Section 4.2. The complete data model is available from the ATOA website:
http://www.atnf.csiro.au/computing/web/atoa/implementation.html.