First On-Site Data Analysis System for Subaru/Suprime-Cam

19 Pages·2011·3.19 MB·English

PASJ:Publ.Astron.Soc.Japan63,S585–S603,2011March25
(cid:2)c 2011.AstronomicalSocietyofJapan.
(cid:3)
First On-Site Data Analysis System for Subaru/Suprime-Cam
HisanoriFURUSAWA,1 YukiOKURA,1SogoMINEO,2TadafumiTAKATA,1FumiakiNAKATA,3ManobuTANAKA,4
NobuhikoKATAYAMA,4RyosukeITOH,4NaokiYASUDA,5,1SatoshiMIYAZAKI,1YutakaKOMIYAMA,1
YousukeUTSUMI,1,6TomohisaUCHIDA,4 andHiroakiAIHARA2,5
1NationalAstronomicalObservatoryofJapan,2-21-1Osawa,Mitaka,Tokyo181-8858
[email protected]
2DepartmentofPhysics,SchoolofScience,TheUniversityofTokyo,7-3-1Hongo,Bunkyo-ku,Tokyo133-0033
3SubaruTelescope,NationalAstronomicalObservatoryofJapan,650NorthA’ohokuPlace,Hilo,HI96720,USA
4InstituteofParticleandNuclearStudies,HighEnergyAcceleratorResearchOrganization(KEK),1-1OhoTsukuba,Ibaraki305-0801 D
o
5InstituteforthePhysicsandMathematicsoftheUniverse(IPMU),TheUniversityofTokyo, w
n
5-1-5Kashiwanoha,Kashiwa,277-8583 loa
6DepartmentofAstronomicalScience,TheGraduateUniversityforAdvancedStudies(SOKENDAI), de
d
2-21-1Osawa,Mitaka,Tokyo181-8588 fro
m
(Received2010September30;accepted2010October17) http
s
Abstract ://a
c
a
We developed an automated on-site quick analysis system for mosaic CCD data of Suprime-Cam, which is d
e
a wide-field camera mounted at the prime focus of the Subaru Telescope, Mauna Kea, Hawaii. The first version m
ic
of the data-analysissystem was constructed, and started to operate in generalobservations. This system is a new .o
u
function of observing support at the Subaru Telescope to provide the Subaru user community with an automated p
.c
on-sitedataevaluation,aimingatimprovementsofobservers’productivity,especiallyinlargeimagingsurveys.The om
newsystem assiststhedataevaluationtasksin observationsbythecontinuousmonitoringofthecharacteristicsof /p
a
every data frame during observations. The evaluation results and data frames processed by this system are also sj/a
usefulforreducingthedata-processingtimeinafullanalysisafteranobservation. Theprimaryanalysisfunctions rtic
implementedinthedata-analysissystemarecomposedofautomatedrealtimeanalysisfordataevaluationandon- le
-a
demand analysis, which is executedupon request, includingmosaicing analysisand flat makinganalysis. In data b
s
evaluation,whichiscontrolledbytheorganizingsoftware,thedatabasekeepstrackoftheanalysishistories,aswell tra
c
astheevaluatedvaluesofdataframes,includingseeingandskybackgroundlevels;italsohelpsintheselectionof t/6
3
framesformosaicingandflatmakinganalysis.Weexaminedthesystemperformanceandconfirmedanimprovement /s
p
inthedata-processingtimebyafactorof9withtheaidofdistributedparalleldataprocessingandon-memorydata 2
/S
processing,whichmakestheautomateddataevaluationeffective. 5
8
Key words: Astronomical data bases: miscellaneous — Methods: data analysis — Techniques: 5/1
5
imageprocessing 7
0
6
9
9
b
y
1. Introduction Suprime-Cam data using the established data-analysis g
u
pipelines, observers at the Subaru Telescope have suffered es
The Subaru Prime-focusCamera (Suprime-Cam; Miyazaki from the lack of an established quality check mechanism t on
et al. 2002) is a wide-field camera at the Subaru Telescope, during observations for data obtained with the instrument. 29
MaunaKea, Hawaii, whichhasbeenoperatedfora decadein Data evaluation is a necessary task in observations in order M
a
general (open-use) observations. The field of view (FOV) of to make sure the data taken with the instrument have the rc
h
Suprime-Camisa34(cid:4)27squarearcmin,whichiscoveredby required qualities, such as seeing and sky transparency, and 2
0
ten 2k (cid:4) 4k CCD’s; the corresponding data-production rate alsotodynamicallyoptimizeanobservationplanbasedonthe 19
is 0.18Gbytes per exposure. In the long and stable operation data qualities during the observing night. The development
of Suprime-Cam, we have collected extensive experience in of the data evaluation mechanism for assisting observers has
the data handling of the wide-field imaging data sets, most beena subjectto be resolvedin operatingthe observatoryfor
of which were well-implemented in data-analysis pipelines alongperiod.
dedicatedfor the Suprime-Camdata (Yagiet al. 2002; Ouchi In the Suprime-Cam observation, most of the data evalua-
2003). Those pipelines have been used in many sciences, tionhasbeenperformedbytheobserversbythemanualquick
includingSubarusurveyprojects(e.g.,Kashikawaetal.2004; look of the data, while they are performing their observa-
Furusawaetal.2008). tions. Manual data evaluation is often a burdensometask for
Despite the successful scientific outcomes from the observers,sincetheyareresponsibleforexecutingtheobserva-
tionbycheckingthedataandmakingadecisionconcerningthe
(cid:3) BasedondatacollectedattheSubaruTelescope,whichisoperatedbythe nextexposures. Theconflictofmultipletasksattheobserving
sitesometimescausesasignificantobservingoverheadtime.If
NationalAstronomicalObservatoryofJapan.
S586 H.Furusawaetal. [Vol.63,
dataevaluationforthedataframesisautomaticallyperformed, available in the system are explained in detail in section 4.
including checking seeing, sky background levels of data, In sections 5 and 6, the analysis framework and the orga-
count levels of standard stars, and photometric conditions as nizingsoftware,whichbuildupandorchestratetheentireanal-
well as bias levels and read noise levels, the observers can ysisfunctionsin thedata-analysissystem, arepresented. The
concentrateon executingand planningobservations,and thus performanceofthesystemisdiscussedinsection7.Wediscuss
theproductivityinobservationswillbesignificantlyimproved. thefutureprospectsofthesysteminsection8,andasummary
Continuous monitoring of such data characteristics during ofthispaperispresentedinsection9.
observationsbyautomatedon-sitedataevaluationwithimme-
diate data reduction will help in the detection of even minor 2. OverviewoftheOn-SiteDataAnalysisSystem
changes,or possibleproblemsin the data, which arelikely to
beoverlookedbyonlythemanualquicklook. Theresultsfor The on-site data-analysis system is designed to assist in
every data frame by on-site data evaluation are also useful to nightly general observations at the Subaru Telescope, by D
o
knowwhichdataframesfortargetskyfieldsandstandardstars providingtoolstocheckandevaluatethecharacteristicsofthe w
n
shouldbeusedforthefinaldataproductsforsciencework.The Suprime-Camdataimmediatelyafterdataacquisition. lo
a
d
data-evaluationresultssignificantlyreducethedata-processing Withthison-sitedata-analysissystem,allofthedataframes e
d
timefordataframeselectioninthefinaldataanalysisafterthe areautomaticallyreducedafterthedataisobtained,andstatis- fro
observingprogramisfinished. Themonitoringof thecharac- ticalinformationconcerningthedataisextracted.Theanalysis m
h
teristics of everydata frame on site is particularly helpfulfor resultsareshowntoobserverssoonafterthedataevaluationis ttp
largeimagingsurveys,whichproducehugedatasets. donethroughauserinterface(awebbrowser),typicallyduring s://a
Thus, the automated data evaluation mechanism at the the next exposure. Those data-evaluation results are used to c
a
observing site is very important to increase the productivity assist observers performingtheir observation. We define this de
m
of observations and the data analysis, and is desired in the immediate‘automated’dataprocessingforthedataevaluation ic
observationsupport. of every data frame as ‘realtime analysis’, which is the key .o
u
p
In some observing facilities around the world, systems for functionofthissystem. .c
o
assisting the data-evaluation tasks are already implemented After data evaluation for each data frame is completed, m
at observing sites. The Canada–France–Hawaii Telescope otheranalysisfunctionscanbeexecuteduponrequestsbythe /pa
s
(CFHT) at Mauna Kea, Hawaii has established tools for observers, which are defined as ‘on-demand analysis’. On- j/a
observingassistanceanddataanalysis—Skyprobe(Cuillandre demandanalysisconductsfurtheranalysisonadataset,which rtic
le
et al. 2002; Steinbring et al. 2009) for monitoring the sky are once passed through realtime data evaluation, in order -a
b
transparency, and Elixir (Magnier & Cuillandre 2004) anal- mainlytochecktheachievementsoftheobservation. Theon- s
ysis systems for performing quick on-site reduction and demandanalysis includes mosaicing analysis to stack a spec- tra
c
flux/astrometric calibrations for imaging data frames at the ified rangeofdataframes, andtoestimate theachieveddepth t/6
3
observatory. The Very Large Telescopeoperated by ESO has ofthestackeddata,aswellasperformingflatmakinganalysis /s
p
2
amechanismfordata-qualityevaluationinqueue-modeobser- in order to create accurate flat frames by combining multiple /S
vations(Hanuschiketal.2002;Hanuschik2007). dataframes. 58
5
Thedevelopmentoftheautomateddataevaluationsystemas Combiningthetwotypesofanalysisfunctionscomplemen- /1
5
observingsupportattheSubaruTelescopehasbecomeamore tarilyworksforeffectiveobservingsupport.Completedescrip- 7
0
6
importantchallenge.Inthenextseveralyears,unprecedentedly tions of all the analysis functions available in the system are 9
9
wide-field imaging surveys are planned in world-wide facili- presentedinsection4. b
y
ties,includingtheSubaruTelescope(e.g.,Miyazakietal.2006; To realize this new system, we employ (1) mechanisms to g
u
Sweeney 2006; Morgan et al. 2008). In such large surveys, improve data-processing speed: minimal disk inputs/outputs es
the data-evaluation system should play an essential role for (I/O’s) in data processingand paralleldata processing, which t on
assistinglong-periodobservationsandderivingreliablescience make the data-evaluation time as short as possible so as to 29
outputsfromhugedatasetsinareasonabletimescale. Hence, catch up with the repeated exposures, and (2) a database for M
a
weaimtodevelopanautomatedon-sitedataevaluationsystem the managementof data-analysishistories to supportefficient rc
h
attheSubaruTelescope. data analysis. With the aid of the database in this system, 2
0
1
In this paper, we present the first automated on-site quick the analysis histories and results obtained in the data evalu- 9
data-analysis and evaluation system for Suprime-Cam at the ation are efficiently recorded, and necessary information can
Subaru Telescope. The system is designed to assist in data- be easily searched and used in analysis functions. Both of
evaluation tasks in observations that have been handled by the abovemechanisms are introducedfor the first time in the
observers and supporting staff, as a new observatory’s func- Subaru observing support tools at the Subaru Telescope, and
tion,includingloggingthedatacharacteristics,andquickdata areimportantachievementsofthiswork.
analysis for checking the achievements of observations. The
firstversionofthesystemstartedoperationforgeneralobser- 3. MachinesandDataFlow
vationsin2010March.
This paper is constructed in the following manner: in Thedata-analysissystemconsistsof10PC’sthatrunon64-
section2,anoverviewofthenewon-sitedata-analysissystem bit Linux (CentOS version 5), all of which are located in the
is described, and in section 3 machines and data flow in the data-archiveroomatthebasefacilityoftheSubaruTelescope.
data-analysissystem are shown. All of the analysisfunctions All of the machines belong to the same network segment,
No.SP2] Suprime-CamOn-SiteDataAnalysisSystem S587
D
o
w
n
lo
a
d
e
d
fro
m
h
ttp
s
://a
c
a
d
e
m
ic
.o
u
p
.c
o
m
/p
a
Fig.1. Machinesanddataflowinthedataanalysissystem. Themachinesinthedata-analysis systemareshownin10separatedroundedrectangles sj/a
(fboerlroewalttihmeeswanitaclhyisnisg,hwuhberueni2t)C,aClDlo’sfawrheipchroacreessceodnnoenceteadchtoanthaelyssaismseenrveetwrsoirmkusletganmeeonuts.lyT.hOen5eamnaolysasiicsinsegrvaenrasly(ssihsoswenrvaetrtihsealsesfitg-nheadndfeodrtshideem)oarseaiucsinegd rticle
analysis. Alloftheprocesseddataarestoredinthefileserverfora2-monthlimitedperiod. Theremaining3servers(Control,Database,andWeb)are -a
usedfortheanalysisorganizingsoftware.Theobserverscanobtaintheanalysisresultsthroughawebserverbyusingawebbrowser. bs
tra
c
and are plugged in to a gigabit-ethernet switching hub unit, protocol and communicating with the database server (see t/6
3
connecting to the gateway of the network. Figure 1 shows section6fordetails). /s
p
2
themachinesusedinthissystem,andthenetworkconnections /S
5
amongthemachines. 4. AnalysisFunctions 8
5
Fiveanalysisservermachinesareassignedforthereal-time /1
5
analysis. Those machines have 4 or 8 CPU cores with 2 to In this section, we describe functions implemented in the 70
6
4GBmemoryoneachnode. FITSfilesoftheseparatedCCD data-analysissystemthatareavailabletogeneralobservers.As 9
9
data obtained with Suprime-Cam are immediately sent to the mentionedinsection2,theanalysisfunctionsareclassifiedinto b
y
end-user machine of the observation control system (OCS), the two categoriesbased on how they are executed: ‘realtime g
u
e
and then sent to analysis servers of the on-site data analysis analysis’forautomateddataevaluation,and‘on-demandanal- s
systembyFTP.DataframesoftwoCCD’sareprocessedwith ysis’, which performs additional analysis by using extracted t on
analysisapplicationsoftwareoneachanalysisserver. Another informationbytherealtimeanalysis. 29
separated analysis server machine is dedicated for mosaicing Theentireanalysisfunctionsavailabletoallobserversinthis M
a
analysis. This machine mounts working directories of all 5 on-sitedataanalysissystemarealsosummarizedinfigure2. rc
h
analysisserversforrealtimeanalysisbyNetworkFileSystem 2
4.1. ApplicationSoftware 01
(NFS),inordertoeasilycollectreduceddatatobeinputforthe 9
mosaicinganalysis. Theapplicationsoftwarepakages,whichconstructthedata-
Onefileserverisusedtoretainbackupdataprocessedonthe processingenginesintheanalysisfunctionsoftheon-sitedata-
previous observing nights. In the current operation, the data analysissystem,arebuiltupbycombiningdedicatedsoftware
takeninrecenttwomonthswithSuprime-Camareretainedin packagesdevelopedbyourselvesandseveralopen-sourcesoft-
thismachine. warepackages.
Theotherthreemachinesareresponsiblefororganizingdata The in-house components of the application software are
analysisinthissystem,consistingofthemanagementofweb- written in C, C++, and Python languages together with shell
based user interface (Web), control of the analysis task flow scripts. For open-source packages, we use the CFITSIO
(Control), and the management of the database (Database). (Pence 1999) library and the PyFITS module (v2.1.1; devel-
Theorganizingsoftware onthosemachinescontrolsthedata- oped by STScI) for FITS file manipulations in our in-house
analysis task flow in the on-site data-analysis system by software. WCSTools (v3.7.8; Mink 2002) is used for FITS
connectingtotheanalysisserversovertheSecureShell(SSH) header handling, and DS9 (Joye & Mandel 2003) is used
S588 H.Furusawaetal. [Vol.63,
D
o
w
n
lo
a
d
e
d
fro
m
h
ttp
s
://a
c
a
d
e
m
ic
.o
u
p
.c
o
m
/p
a
s
j/a
Fig.2. Analysisfunctionsinthedata-analysissystem. Thefunctionsavailableinthissystemaresummarizedinthisfiguretogetherwiththerelation rtic
betweenthesystemcomponentsoftheanalysisorganizingsoftware(RCM),includingthededicateddatabase(boxesattheupperpart),andtheanalysis le-a
servers(boxesatthelowerpart). b
s
tra
c
for browsing FITS images. Three TERAPIX software pack- data-analysis stages performed for every data frame, each of t/6
3
ages1 [SExtractor (Bertin & Arnouts 1996), SCAMP, and which is a separated application software written in C, C++, /s
p
SWarp]whichwereoriginallydesignedfordatareductionfor or Python. The analysis stages include (1) overscan region 2/S
MegaCamatCFHT,arecombinedwiththein-housesoftware subtraction, (2) flat fielding, (3) trimming useless pixels at 5
8
5
toconstructthewholeanalysispipelinesusedinthesystem. the edge, (4) measurements of the statistical values of the /1
5
frames(seeing,sky-backgroundlevel,overscannoise,number 7
4.2. RealtimeAnalysis 0
of objects), (5) astrometric calibration, and (6) determination 69
9
Realtimeanalysisisakeyfunctionoftheon-sitedataanal- ofthephotometriczeropointforstandardstarframes. b
y
ysis system. This function performs the automated on-site Resultant images and extracted values by the above- g
u
evaluation of Suprime-Cam data, by extracting the following mentioned realtime analysis stages are collected by the orga- e
s
information,whichcharacterizesthequalityofalldataframes: nizingsoftware,registeredinthedatabase,andshownonweb- t o
n
(1)forscientificobjectandstandardstars—seeing,skyback- based summary windows. Figures 4 and 5 show examplesof 2
9
ground levels, and overscan noise, (2) for flat frames — sky summary windows for showing results of the realtime anal- M
a
background levels and overscan noise. The data frames are ysis. Processed imaging data can be seen in JPEG thumbnail rc
h
tagged in the database based on the derived quality infor- images,andcanalsobebrowsedwithaDS9,whichisopened 2
0
mation to assist in the observation and latter data analysis. fromtheremoteanalysisserversontotheusers’displaybythe 19
Also,acoarseastrometriccalibrationisperformedforalldata X11forwardingprotocol.
frames,andthephotometriczeropointisestimatedforthestan- The configuration files for controlling analysis algorithms
dard star frames. This analysis functionusually runsmost of (e.g., a SExtractor configuration file for object detection
thetimeduringanobservation. and measurements) are specified by the observers when the
The realtime-analysis mode becomes ready to work at realtime-analysismodeisstarted. Thesystemdefaultconfigu-
the beginning of an observation by using the user interface rationispreparedbytheobservatorystaff.
(figure3).Bypollingnewfilesinthepre-defineddirectoriesof 4.2.1. Algorithmsoftherealtimeanalysisstages
eachanalysisserver,therealtimeanalysisstartsdataprocessing 1. Overscan region subtraction (Overscansub): A data
whennewdataframesarriveinthedirectories. frame of Suprime-Cam is read out through 4 independent
Realtime analysis consists of a sequence of quick readout channels, and each of the effective pixels has associ-
ated overscanregions, which are attached at the ends of both
1 hhttp://terapix.iap.fr/i; currently moved to hhttp://www.
serial and parallel readouts in each channel, i.e., there are x
astromatic.neti.
No.SP2] Suprime-CamOn-SiteDataAnalysisSystem S589
D
o
w
n
lo
a
d
e
d
fro
m
h
ttp
s
://a
c
a
d
e
m
ic
.o
u
p
.c
o
m
/p
a
s
j/a
rtic
le
-a
b
s
tra
c
t/6
3
/s
p
2
/S
5
8
5
/1
5
7
0
6
9
9
b
y
Fig.3. Windowtostartuptherealtimeanalysismode. Observerscanspecifytheanalysisparametersandflatframesforeachfiltertobeusedinthe g
realtimeanalysisprocesses. ue
s
t o
n
andyoverscanregionsforeachchannel(figure6).Thispartin 3. Trimming (Agpmask): Suprime-Cam data frames have 2
9
therealtimeanalysissoftwaremeasuresthemedianofthepixel a shadowed area on the top of the FOV, which is due to M
countsin each line of the serial overscanregionsof the input vignettingbytheauto-guider(AG)probe. Thisshadowedarea arc
dataframes. First,itsubtractsthosemedianvalueslinebyline cannotbeflatfieldedcorrectly,andthefluxcalibrationisdiffi- h 2
fromtheeffectivepixels.Then,theoverscanregionsandblank cult to perform. The analysis software trims the area that is 01
9
pixels, which are added in the data readout, are trimmed out, determinedby the positionof the AG probeat the time when
and thus the output data frames do not have any gaps within thedataistaken.
the2k(cid:4)4k-pixelarea. Thisoperationisdoneinallchannels 4. Statistical value measurement (Stat): In this stage,
separately. Atthisstage, themeanandthestandarddeviation statisticalvaluesrepresentingthedataqualitiesarederivedby
of the pixel counts in the overscan regions are also derived, severalseparatedtasks. First,SExtractorv2.5.0detectsobjects
andaremergedtotheresultsofthebelow-describedstatistical forthelaterastrometriccalibrationanalysis,andmeasuresthe
valuemeasurement. size (FWHM in pixels) and the sky-background level of the
2.Flatfielding(Ffield):Thisreductionstagedividesthepixel data frame at the same time. Then, the software derives the
values in the input data frames by those of the flat frames, average seeing size of the data frames. The seeing measure-
which are defined by the observers, or preset by the observa- mentisdescribedinaseparatedparagraphbelow. Acomplete
torystaff. Thedetailsofchoiceoftheflatframesaredescribed listofthederivedqualitiesinthisstageis: (1)sky-background
insub-subsection4.2.2. level (ADU), (2) RMS of sky background(ADU), (3) seeing
S590 H.Furusawaetal. [Vol.63,
D
o
w
n
lo
a
d
e
d
fro
m
h
ttp
s
://a
c
a
d
e
m
ic
Fig.4. Mainsummarywindowfortherealtime(andre-analysis)forSuprime-Camdataofeachexposure.Statisticalvaluesandevaluatedflagsarelisted .o
u
inthesectionsofeachexposure,aswellasbuttonstoopenanotherwindowshowingdetailedinformationforeachexposureorwritingcomments.When p
theUpdatebuttonisclicked,thelatestinformationisloadedfromthedatabase,andthesummarywindowisupdated. .co
m
/p
a
s
j/a
rtic
le
-a
b
s
tra
c
t/6
3
/s
p
2
/S
5
8
5
/1
5
7
0
6
9
9
b
y
g
u
e
s
t o
n
2
9
M
a
rc
h
2
0
Fig.5. Screenshotoftheshow-detailwindow,whichshowsdetailedresultsofrealtime(orre-analysis)foreachCCDdatainaSuprime-Camexposure. 1
9
FlagsfortheanalysisresultstaggedinthedatabaseforeachCCDtogetherwiththestatisticalvaluesandthumbnailimagesareshown.
size(arcsec),(4)numberofobjectsdetectedwitha2(cid:2) signifi- procedure. Note that those object detection processes are
cancelevel,(5)thethreshold(ADU)usedinobjectdetection. independently done from the above-mentioned SExtractor
Seeingmeasurement(Seeing): sessionfortheastrometriccalibrationanddeterminationofthe
The seeing measurement of this stage is carefully treated, sky-backgroundlevel.
since cosmic-ray events or spurious sources occasionally The first object detection is performed with SExtractor, in
contaminate a group of the most compact objects, which are which a 2(cid:2) detection threshold is adopted to allow seeing
thought to represent the point spread function (PSF) or the measurements,eveninrelativelypoorweatherconditions.The
seeing size of the data frame. In order to pick up only detectedobjectsare sortedoutaccordingtotheir totalmagni-
real point sources, the seeing measurement software in this tude(MAG AUTO),forwhichthephotometriczeropointisset
system follows two iterative stages of the object detection to zero. Then, in order to estimate a good magnitude range
No.SP2] Suprime-CamOn-SiteDataAnalysisSystem S591
frame,andisusedforaseeingassumptionforthesecondrunof
objectdetection.
The second SExtractor object detection is run by
using the tentative seeing in the configuration param-
eter SEEING FWHM. With a reasonable assumption of
SEEING FWHM,SExtractorperformsabetterjobfordiscrim-
inating point-source objects (star–galaxy separation) based
on the stellarity index CLASS STAR. To have more reliable
clean point source samples, the analysis software relies on
theCLASS STARindex,andpicksuponlythoseobjectsthat
satisfytheCLASS STARindexgreaterthan0.9,andtheirsizes
sit within the tentative seeing ˙00.03. In figure 7, the final D
o
point sources picked up from all objects are also shown with w
n
filledcircles. lo
a
d
5. Astrometric calibration (Astmt): SCAMP (TERAPIX ed
software;v1.4.6-MP)isusedforthemainprocessinthistask. fro
m
Inthecurrentimplementation,individualCCD’sareseparately h
processedwithSCAMP,andgiventheresultantWCSkeyword ttp
s
values. Theobjectsin eachdataframe, whicharedetectedin ://a
the above-mentioned ‘Stat’ analysis stage, are matched with ca
d
thereferencesourcestodeterminetheabsolutecoordinatesof e
m
CCD pixels by SCAMP. In this step, no relationship in the ic
.o
geometricalarrangementbetweenCCD’sinashotisassumed. u
p
TheUSNO-B1.0catalog(Monetetal.2003)isusedforrefer- .c
o
ence sources, which are located in the local disk of the anal- m
/p
ysis system. The FITS keywords of the input data frame are a
s
updated with the resultant WCS values. The coefficients of j/a
thesecond-orderpolynomialsusedtodescribedistortedworld rtic
le
coordinates, which are expressed by PV keywords, are also -a
b
derived and written into the output FITS header in this soft- stra
warepart. c
t/6
6. Photometric zeropoint determination (Photm): For the 3
/s
data frametaken fora standardstar field, the Photm software p2
executesanestimationofthephotometriczeropointinunitsof /S
5
(magADU(cid:5)1s(cid:5)1),bycomparingthefluxesofdetectedobjects 85
/1
Fig.6. Data format of a Suprime-Cam CCD image. CCD data has intheframeandcatalogmagnitudes. 5
7
4readoutsegments,eachofwhichisoutputfromtherelevantreadout Calibration sources for determining the photometric zero- 06
channel. In the figure, the rectangular regions shown in white in 9
pointarederivedfromthestandardstarfieldsbyLandolt(1983, 9
each channel are the effective scientific pixel areas, and the neigh- b
2009). The celestial coordinates and magnitudes of the cali- y
boringshadedregionsareassociatedoverscanpixelsforserialreadout g
(attached to one side of each effective regions) and parallel readout brationsourcesinavailablewavebandsarestoredinatableof ue
s
(attached tothebottom). Thetwoboxesinthechannel1arearepre- the PostgreSQL (v8.1.11) database. The comparison is done t o
saennatlythsiesp(idxoetlteadreslinuesefdofroCrnCoDrm’sawlizitahtoiountovfigcnoeutntitnlgevbeylstihneflAatG-mparkoibneg, by assuming WCS information that is already determined in n 29
anddashedlineforthosewiththevignetting). the above astrometric calibration stage. The resultant photo- M
metriczeropointisshownontheweb-basedsummarywindows a
rc
(figure 8). Currently, this analysis stage is only available for h 2
forderivingpoint-sourcesamples,acumulativenumbercount data frames in the wavebands covered by Landolt’s photo- 01
9
of the objects as a function of the magnitude is calculated. metric standard stars, and no color term or airmass factor is
Our examinationshows that approximatelya 15% fractionof consideredinthezeropointestimation.
the total objects from the brightest magnitude (15 percentile 4.2.2. Flatframes
brightest objects) are likely to be classified as point sources Realtimeanalysisrequiresflatframesregisteredintheanal-
withoutanyseriouscontaminationwithgalaxies,cosmicrays, ysis system to perform flat fielding of the input data frame.
and bad pixels (figure 7). The software picks up 50 of Theflatframesarepreparedandregisteredbytheobservatory
the most compact objects that satisfy the criterion in size of staffpriortotheobservingnight.Thosepre-definedflatframes
FWHM = 00.03–20.04, out of the 15 percentile objects. This areset tothe ‘observatory-recommendedflats’ inthe analysis
step also removes saturated objects at the bright end magni- system, and are automatically used for data frames taken in
tudesfrom the sample. The mode FWHM of the thus-picked correspondingfilters.
objects is determined. This mode FWHM is set to ‘tenta- When the observers starts realtime analysis, they are
tive seeing’, which is close to the average seeing size of the requestedtospecifyflatframestobeusedinrealtimeanalysis
S592 H.Furusawaetal. [Vol.63,
D
o
w
n
lo
a
d
e
d
fro
m
h
ttp
s
://a
c
a
d
e
m
ic
.o
u
p
.c
o
m
/p
a
s
j/a
rtic
le
-a
b
s
tra
c
t/6
3
/s
p
2
/S
5
8
5
/1
5
7
0
6
9
9
b
y
g
u
e
s
Fig.7. Selectionofstellarobjectsamplesfortheseeingmeasurement.Thelowerpanelshowsthesizes(FWHM’s)ofdetectedobjectsinatypicalCCD t o
frameasafunctionofarbitrarymagnitude(smalldots). Theupperpanelshowsthecumulativenumbercountofobjectsforthesamedataframethat n
2
isnormalizedbythetotalnumberofobjects. Itisfoundthatthemagnitudewhichincludesa15%fractionofdetectedobjectsfromthebrightendis 9
agoodthresholdingmagnitude(shownwiththeverticaldashedlineinthefigure)topickupreasonablycleansampleofstellarobjects.Thefinalsample M
fortheseeingmeasurementisalsoshownwithfilledcircles(magenta). Notethatdotslocatedatthebrightestmagnitudes.(cid:5)15aresaturatedobjects, arc
andthoseatthefaintestmagnitudes>(cid:5)11.5arehighlycontaminatedbybadpixelsandcosmicrays,whichareremovedfromthesampleintheseeing h 2
measurements.Alsoseethetext. 0
1
9
from the database. Although the observers may leave the Suprime-Cam,againthedefaultflatframesareused.
flat frames as the observatory-recommended flats, they can
4.3. On-DemandAnalysis
also select another set of flat frames for a particular filter.
The observers can select as many flat frames for different On-demandanalysisisagroupofanalysisfunctionsthatare
filters as needed. Those flat frames explicitly selected by the invoked when the observers intentionally request to process
observersaretreatedas‘user-selectedflats’,whichoverridethe data which have already gone through the realtime analysis,
observatory-recommendedflatframes. and their results have been registered in entries of the anal-
When neither the user-selected nor the observatory- ysisdatabase.
recommended flat frames are set, the ‘default flat’ frames 4.3.1. Re-analysis
(currently,flatforRc-band)areused.Iftheinputdataistaken Thisfunctionisusedtore-processaparticularrangeofdata
withabrand-newfilterthatisbeingusedforthefirsttimewith frames that have already been processed by realtime analysis
No.SP2] Suprime-CamOn-SiteDataAnalysisSystem S593
D
o
w
n
lo
a
d
e
d
fro
m
h
ttp
s
://a
c
a
d
e
m
ic
.o
u
p
.c
Fig.8. Exampleresultofadeterminationofthephotometriczeropointusingphotometricreferencestars.Thehorizontalsolidline,whichisthebest-fit o
m
linetoallthedatapoints,showstheresultingphotometriczeropointassumingthattheSuprime-Cambandsystemisexactlythesameasthestandard /p
bandsystemusedformagnitudesofreferencestars. as
j/a
after it has finished. The re-analysis is used mainly for the 4.3.2. Flat-makinganalysis rticle
purpose of conducting analysis with different configuration Flat-making analysis is a function for creating flat frames -a
b
parameters, or by using differentflat frames other than those from data frames that are specified by the observers based stra
used in realtime analysis. Typical-use cases for this function on the statistical values of the data extracted in the real- c
t/6
include:(1)tuningofthedetectionthresholdsforobjectdetec- time or re-analysis, such as the sky-background levels and 3
/s
tion data taken under relatively poor weather conditions, or the number of objects. The analysis system collects the data p
2
dataforcrowdedfieldsetc, (2)changingflatframeswhenthe frames into a working directory, which are selected by the /S
5
observerswould like to use other flat frames created by their observers’query. 85
ownnight-skydata. Theobserverscanspecifythedataframes The algorithm used to create the flat frames is as follows: /15
7
to be re-processed and set the analysis parameters, including first,thesoftwarenormalizesthepixelcountsoftheinputdata 0
6
9
flatframesthroughtheuserinterface.Currently,there-analysis frames by dividing the counts in each pixel by the median 9
b
function can be executed only when the realtime analysis is representative of the entire frame area. Here, we use only y
g
disactivated,duetothelimitedsystemresources. the pixels within the far-left single readout channel to deter- u
e
Once the observers perform re-analysis for a certain set of minethemediancount. Sinceweshoulduseexactlythesame st o
data frames, those data frames have two or more different regionfor all data frames for the same CCD, the shaded area n
2
analysis sets (analysis branches), depending on the number bytheAGprobevignettingneedstobetakencareof.Thepixel 9
M
of re-analysis trials, each of which is usually associated with regionsusedforthemediandeterminationare:(x,y)=(1,31) a
different analysis parameters or flat frames. Since the anal- (cid:5) (512,3400)for data of the five CCD’s locatedat the upper rch
ysis system needs to identify which analysis branch is the FOV, and (x,y) = (1,31) (cid:5) (512,4060) for the other five 20
1
successful one to be used in later analysis, the observers are CCD’s at the lower FOV (figure 6). Second, the normalized 9
requested to specify one of those analysis branches (real- frames are collected by each CCD, and combined into a flat
time and re-analysis) as the ‘successful’ to be used in the frame while taking the median value in each pixel, in which
followinganalysis. a 3(cid:2) clipping of outlying values is performed. Finally, the
Theanalysissystemprovidestheobserverswithauserinter- resultantflatframesareregisteredintheanalysisdatabaseand
face to select successful analysis branches. The observers shownonthesummarywindow(figure9),whichcanbeused
can set a result tag (successful or unsuccessful)of each anal- byobservers’requests.
ysis branch based on the analysis result through the inter- 4.3.3. Mosaicinganalysis
face.Theinformationfortheanalysisbranchesinthedatabase, The observers can request the analysis system to generate
whicharesetto‘unsuccessful’,areignoredinanyqueriesfor a mosaic-stacked image from a particular set of data frames
searching those data frames in the following Flat-making or registered in the database by the realtime analysis or re-
Mosaicinganalysis. analysis. The data frames can be searched based on the
S594 H.Furusawaetal. [Vol.63,
D
o
w
n
lo
a
Fig.9. Screenshotoftheresultingflatframescreatedbyaflat-makinganalysisoftheanalysissystem. d
e
d
fro
m
h
statistical values again in the same manner as in the Flat- 4.4. StatusMonitor ttp
making analysis. The selection parameters available in the Theobserverscanmonitorthevariationagainsttimeofthe s://a
user interface include the statistical values — seeing, sky- c
statistical values derived from the realtime analysis through a
backgroundlevel, andnumberof objects, aswellasthe basic de
plotsonawebbrowserattheobservingroom.Figure13shows m
poabrjeacmtentaemrsef,ofirlitderenntaimfyein,gactqhueistaitrigoentdoabtjee,catnsd—fraemxpeo/CsuCreDtiImDe’s, ascreenshotofthestatusmonitor. Themonitorpageisauto- ic.o
maticallyupdatedeveryoneminuteandstatisticalvaluesofthe up
(figure 10). After choosing data frames from the output list latestdataareaddedtotheplots. .co
returnedbytheframeselection(figure11),themosaicanalysis m
processisexecutedbyclickingtheExecMosaicbutton. The plots shown on the monitor include basic information /pa
of the latest frame, the variation in the seeing size, the ellip- s
The analysis in this part is conducted as follows: first, j/a
SExtractor detects control stars in each data frame. Next, ticity of point-likesources, bias level, noise level in overscan rtic
pixels, and the sky-backgroundlevel. In every morningafter le
the control star catalogs are input to the SCAMP to deter- theobservingnight,themonitorwebpageisresetandtheold -ab
mine the geometrical alignment and flux scaling by internal s
cross-identificationamongobjectsthatarecommonlydetected data is automatically archived, which can be reviewed on the tra
on multiple data frames. The cross-identification between monitorwebpagebyselectingthedateofobservation. ct/6
3
the detected objects and external catalog sources is also 4.5. MiscellaneousFunctions /s
p
2
upseerfothrmeeUdSfNorOt-hBe1a.0bscoalutateloagstargoamine.tricTchaelibrersautilotann,twihneforermwae- Other various functions available in the analysis system /S58
are listed. By the functions to download processed data and 5
tion, including WCS keywords and flux scaling factors, is /1
asummarylogtable,includingstatisticalvaluesofeachexpo- 5
recorded in ascii files (.head files) for each of the input data 7
sure, the observers can conduct their full analysis smoothly 06
frames. Finally, the SWarp (TERAPIX software; v2.17.6) 9
attheirinstitute. 9
performsgeometricaltransformationwithpixelresamplingof b
y
each frame using the head files, and stacks all of the trans- (cid:6) A function to download processed data onto users’ g
u
e
formedframesintoonelargemosaic-stackedimage.Theresul- machines s
tantstackedimageisshownonthesummarywindowtogether (cid:6) A function to obtain a summary log table containing t on
with the other stacked images already created in the same data-evaluationresults 29
observingprogram(figure 12). The algorithmand configura- (cid:6) Afunctiontoperformaquicklook-upofprocesseddata M
a
tionfilesforthisanalysisfunctionisoptimizedforimagesfor withaDS9browser rc
h
typical blank sky fields, which are taken under good weather (cid:6) A function to attach users’ comments to registered 2
0
1
conditions, without dark clouds or crowded/extended objects images in the database, which are also reflected in the 9
intheFOV. summarylog
The mosaicing analysis function also includes several
follow-upanalysestoseestatisticalpropertiesoftheresultant 5. AnalysisFramework
stacked image. It (1)creates an objectcatalog by SExtractor,
(2) derives the number count of objects, while assuming the Theanalysisframeworkissoftwareforconstructingadata-
photometriczeropointinagoodskycondition,whichislisted analysispipeline,byconnectingtheseparatedapplicationsoft-
on the Suprime-Cam instrument web page, and (3) measures ware programs(written in C/C++) in a row, and sequentially
the noise statistics over the mosaic-stacked image, and executingthemonthememory.
roughlyestimates the achieved depth, again assuming a good We employ the analysis framework software Roobasf (Lee
skycondition. et al. 2010; Mineo et al. 2010) to construct the anal-
ysis pipeline used in the realtime and re-analysis functions.