Reducing APT Data
Notes taken from here
Very quick guide
To reduce a run's worth of data
* run aptproc.pl in each night's directory until sky flats are processed
* select good sky flats and combine to form master flat
* reducce one good night's data using new master flat
* cread master image and catalogue for each new field from good night's data
* reduce remaining nights with aptproc.pl
* for each field, generate lightcurves with res2lc.pl
Details
I have started every line where user input is requred with '**'. Stuff in
between is just additional information.
** copy data from CD/DVD to /data2/jessiec/apt on Wink
A separate directory is used for each night's data, and each directory is
named after the date at the start of the night.
eg. /data2/jessiec/apt/030303/
/data2/jessiec/apt/030304/
/data2/jessiec/apt/030305/
etc...
** > source /home/jessiec/mike/apt_setup
This will define various useful aliases for programs, scripts, etc... You
may find it useful to put this line into your .cshrc file, so that these
definitions are automatically loaded every time you open a new shell.
** open a new xterm and cd to a raw data directory, eg 030324/ (this is so that
several directories can be processed at once - unfortunately the pipeline
doesn't seem to run as a background job).
** > aptproc
This will will run for an hour or two (from raw images to photometry
results) and do the following:
- Unzip files, if there are any .gz or .bz2 files in the directory.
- Set permission to 664 for all .fits files
- Update image headers, adding the keywords TRIMSEC and BIASSEC required
for later processing, also changing FILTERS keyword to FILTER
- Sort the images into the subdirectories sky/ and star/ based on the
IMAGETYP header keyword. Also write a list of images for each field in a file
star/l
- If there are any sky flats, process them (subtract bias, trim overscan
region and correct for non-linearity). Form a nightly master sky flat by
combining the processed sky flats (creating sky/msky.fits).
- At this point, unless a master sky flat has been specified with the '-f'
option (see below), the process will stop and ask for a flat to be specified,
offering the msky.fits from the current night as default (though it may not
exist). However, in general this is not what we want to use. Instead, once we
have reached this point for all the nights within a run (or even a couple of
consecutive runs), we want to combine all the processed twilight images into
a single master flat, and use that same flat to flatfield all the images from
that run (or that season).
A wonderfully friendly Perl script called mflat.pl will make this much
easier. Once you have enough processed twilights for a season (say beginning
with nights 0501??), go to /data1/mgh/master/ and do this:
** > mflat.pl 0501skyI.fits /data2/jessiec/apt/0501*/sky
where 'I' is the filter used. Make sure you only include flats that were
taken with that filter. This will generate a master flat (0501skyI.fits), and
a text file (0501skyI.l) listing the individual processed twilight flats that
were combined to make the master.
** Inspect the new master flat (eg. in DS9) to check for any 'strange'
features, such as stars or images of the APT entrance pupil due to
scattered light (it happens!), or in fact any small-scale
features. Actually, scattered light problems seem to be unavoidable,
but make sure they are not present in the processed nightly flats (see
below). Smooth gradients across the field are not a problem as they
are handled by later processing. There are also a few 'spots', an
obvious vertical line and some vignetting in the corners that will
always be there (see other master flats in /data1/mgh/master for
examples).
** You should also check the individual, nightly combined flats (msky.fits),
after flatfielding them with the newly created master flat. To help you do
this, mflat.pl automatically creates a tmp/ subdirectory in the location where
you ran it (usually /data1/mgh/master/), links all the msky.fits files in
there (with names like sky050106.fits), then processes them with the new
master flat.
The resulting psky050106.fits (etc...) file should be 'flat', with
the exception of shallow, smooth gradients. In particular, the spots,
vertical line and vignetting should be absent. Stars should also be absent!
Some flatfield features seem to be quite difficult to actually fully remove,
but the amplitude of any residual features in these images should be less
than 0.1% of the overall flux level. An easy way to check is to go into IRAF
and use commands like
> imarith 'a/median(a)' nsky06 psky050106
... to divide each image by its median value. If any of these images
look bad, the corresponding night's sky/ subdirectory should be
renamed to something like 'badsky' (or the pim*.fits files deleted)
and then mflat.pl re-run without them.
** go back to each aptproc process and specify the master flat to use. Again,
make sure you reduce images with the correct master flat (right filter and
from twilights taken within a few months of the data you're reducing.
Most of our images are now in I band, but there may be some other data too.
aptproc then goes on to...
- Decide which fields need to be processed. By default, all fields will be
processed, but fields can be selected or excluded (see below for command line
options).
- Trim, bias subtract, linearise and flatfield star images.
- Re-run AptAstrom on every processed image, updating the headers with the
correct world coordinate system information (this is needed as trimming the
overscan region off the images changes the image coordinates, upon which the
old WCS transformation was based). Processed images with smag<-1
are deleted, as they don't tend to give useful photometry. The
corresponding raw images are not removed. The smag limit can be adjusted by
changing the values in the %smaglim hash in aptproc.pl.
- (The original version fo the pipeline used to run and IRAF script to add
ST, UT, JD and AIRMASS to image headers. The AptAstrom program now takes care
of this, so I have turned this off. However it could be turned back on for
older data if need be.)
- Move processed images to separate directories for each field, renaming
them from .fits to .fit (Mike's photometry programs use .fit for images and
.fits for objects catalogues in FITS table format). Any images that were not
copied over because the AptAstrom fit failed or the 'smag' was less than -1
(i.e. cloudy and probably useless) are listed in a file called 'bad'.
- Generate object catalogue for each image. For each .fit image, this will
create a _cat.fits table and also a _cat.ell ascii listing (which can be used
to check what objects are in the catalogue using the regions feature in ds9).
- If a master catalogue for the current field has not been chosen yet (by
default this is in the directory /data1/mgh/master), select the lowest airmass
image from the present night and copy it, along with its image catalogue to the
master directory. (This is only used when we first want to quickly reduce data
on a new field. In practice we then generate a master image "manually", by
stacking a dozen or two good images from a single night and then generate the
master catalogue from this stacked image.)
- Use the individual object catalogues to find six-parameter linear
coordinate transformations from each image to the master image. These are
stored in a .trans file for each image.
- Transform the master catalogue objects into the frame of each image and
measure these stars. The photometry apertures used in each image are written to
a _list.ell file for easy plotting. The results of the photometry itself are
output in binary fits tables with names ending _list.fits
- For compatibility with earlier versions of the pipeline, the data is
then read out of the _list.fits files and converted to a single binary
(though readable as ascii) file called 'resultsn'. This file contains 5760-byte
records. The first of these lists the JD (-2450000) from each image (the first
four numbers are dummies). Each subsequent record contains data for a star form
the master list. The first four numbers are the x & y coordinate, the median
mag of the star, and something else i'm not sure about. The numbers after that
are magnitude-airmass pairs corresponding to the JDs in the first record. The
values in this file are instrumental magnitudes (2.5*log(flux in ADU)).
- Perform a frame-to-frame magnitude zero-point correction. This is done by
the program 'cllc', which allows the zero-point to vary as a linear function of
x, y, xy and y**2 (x and y being the pixel coordinates). The final output is a
'resultsn.xyy' file, in the same format as the original resultsn file.
This program also uses Tycho stars (in a master.ty file linked in from
the /data1/mgh/master directory) to calibrate the magnitude zero-point. In I
band, this involves a guess at each Tycho star's I magnitude based on its B and
V mags (using tables for main-sequence stars from AQ5/Cox2000).
The output magnitudes are still instrumental, but adjusted so that a
fixed zero-point of 22.5 will turn them into real mags (this is done by
res2lc, below).
- After the last iteration, cllc checks the magnitude rms for bright stars
in each image, and if it is ever greater than 15mmag, outputs the sequence
number of that image to the file fort.33. aptproc.pl renames this to
resultsn.ex, and re-runs cllc, which now excludes the noisy images from the
calculations (and from resultsn.xyy).
- Write a detailed log file, including the output from each step, to a
'LOG' file. (Previous LOG files are saved as LOG001, LOG002, etc...)
aptproc has now finished running, but the fun's not over yet!. From here
on, data is treated one field at a time. Individual nights of data can be
quickly inspected using:
** > lightcurves resultsn.xyy
This program will first calculate the overall magnitude offset of each
image from the median, as well as the RMS error on the magnitues of bright
stars in the image. These values are printed in columns 2 and 3
respectively. The offsets should be within a few millimag of 0, since they have
already been corrected for, and the RMS should be around 3-5mmag, but up to
9-10mmag is still ok.
After pressing Enter a few times, it will display a plot of RMS vs. mag,
along with a theoretical limit (Poisson noise - for this to match the data,
the values of the sky flux and magnitude zero-point need to be recorded in a
file named 'lcpar' - see an example in /data2/jessiec/apt). Stars brighter
than about I=7.7 will have higher RMS values since they are saturated (in
60sec images).
It will then display the image offsets and RMS values for each image (the
same information it printed to the terminal), followed by star lightcurves in
order of decreasing magnitude, starting with the saturated ones. Images for
which the RMS was greater than 9mmag are have their corresponding datapoints
highlighted by red circles in all plots. The only way to avoid looking through
all lightcurves is to do a Ctrl-C.
If there are still images you think should be removed from the lightcurves,
add their sequence numbers (i.e. 1 for the first image, etc..) to the
'resultsn.ex' file. You can also specify negative numbers to count from the
end (eg. -1 is the last image). To help you with this, the lightcurves program
outputs such a list, containing all the images for which RMS>9mmag, to a file
called 'fort.33'. Remember, however, that the sequence numbers in resultsn.ex
should be relative to the complete set if images in that directory, while the
image numbers output by lightcurves are only the ones it found in resultsn.xyy.
So if there was already a resultsn.ex there, you'll have to do a bit of mental
arithmetic...
If all of the data in this directory is useless, rename the directory from
'fld
Once all the data for a particular field has been reduced to 'resultsn.xyy'
files, we're ready to create some lightcurve files!
** > cd /data2/jessiec/apt/
** > mkdir
** > res2lc.pl Jan06_1/ 14.0 */fldJan06_1/resultsn.xyy
This will read all the 'resultsn.xyy' files and write a lightcurve file
(Jan06_1/nnnnn.lc) for each star brighter than 14.0 (set this to -1 to include
all stars) and sufficiently far from the CCD edges and corners (parameters for
this are hard-coded into res2lc). 'nnnnn' is the star's id, which is its
sequence number in the master catalogue for the field. The catalogue is sorted
in order of increasing y coordinate (approximately). A log is written to
Jan06_1/res2lc.log so that we can easily check later on where those
lightcurves came from.
In practice we generate several sets of lightcurves for each field, adding
more nights of data as they are taken, so within Jan06_1/ there will be
subdirectories like 'upto060126_14', 'upto060201_13', etc... (the number at the
end is the magnitude limit).
And you're ready to do some eyeballing... or whatever you wish!
Command line options for aptproc
+all do everything (this is the default)
-all do nothing, except tasks individually turned on by subsequent
'+' switches
+pre do preprocessing (update headers + sort images to sky/ & star/)
-pre skip preprocessing
+sky process and combine sky images to form master sky flat
-sky don't process sky flats
+star process star images and move to field dir
-star don't process star images
+cat generate object catalogues for all images in each field dir
-cat don't generate object catalogues
+phot do photometry on images in each field dir using master list
(and generate resultsn and resultsn.xyy files)
-phot don't do photometry
+recal Repeat calibration after excluding bad images (done by default
with +all or +phot)
-recal Don't recalibrate
-lin skip linearity correction when processing images
-f flat.fits use flat.fits to flatfield star images (instead of master flat
from the current dataset)
-d f1,f2,... process only the fields in the list (comma-separated)
-s f1,f2,... skip fields in the list (comma-separated)
-m master If master is a *_cat.fits file, use it as the master catalogue
for photometry. If it is a directory, look for master catalogue
files in that directory, with the name
-h field Here: only do catalogue generation and photometry on images in
the current directory, which are of the field specified. If
aptproc is run from a directory ending in /fld
-r rcore Set the photometry aperture radius in pixels. The default is 3.
(The software measures stars with apertures of radius rcore,
and also factors of 1/sqrt(2), sqrt(2), 2, and 2sqrt(2) times
rcore. These mags are all stored in the _list.fits files)
Quick guide to files and directories used in the reduction process
* In each night's directory (/data2/jessiec/apt/YYMMDD/), you'll find:
aptYYMMDD.log observer's log (brief comment on night, sometimes absent)
obsYYMMDD.log output from observing script
obs
LOG log file from most recent run of aptproc.pl
LOG??? previous log files
sky/ containing raw (im*), processed (pim*) and combined (msky.fits)
twilight images
star/ containing raw star images (and lists of image names belonging
to each field observed)
fldF1/ reduced data for field F1
* the each field directory (eg. fldF1/) will have:
LOG* log files from aptproc run just on this field/night
bad list of non-photometric images that were not processed
fort.* debugging output from fortran programs (not really important)
images list of processed images
master_cat.fits copy of master catalogue for field
master.ty copy of list of Tycho stars in field
pim*.fit processed image
pim*.trans coefficients of coordinate transformation to master frame
pim*_cat.ell for plotting catalogue objects (DS9 region file)
pim*_cat.fits fits catalogue of objects found in this image
pim*_list.ell for plotting photometry apertures used to measure image
pim*_list.fits bits table output from photometry program (contains mags
measured with several aperture sizes, and more...)
resultsn base-aperture-size (3pix) photometry compiled in single file
resultsn.ex sequence numbers of datapoints to exclude from lightcurves
resultsn.xyy results file after zero-point correction
tttt temporary file (not important)
(some older data may also have:
results_0.ps plot of rms vs. mag and residual extinction coeffs vs. mag
results_1.ps as above, after residual extinction has been corrected for
)
* Also used in the reduction are the following files in the /data1/mgh/master/
directory:
YYMMskyI.fits master sky img in I filter (combined from twilights obtained
on several nights within the run, listed in YYMMskyI.l)
F1.fit image of field F1 from which master catalogue was obtained
F1.l list of images stacked to create F1.fit
F1.ty list of Tycho stars matched to stars in the master catalogue
(columns: star id, Tycho id, x, y, V mag, B-V colour,
match distance in pixels)
F1_cat.ell DS9 region file for master catalogue
F1_cat.fits master catalogue for field F1
F1_cat.rd same list as _cat.ell, but with RA and Dec instead of x & y
F1_mat.ell DS9 region file of matched objects (Tycho)
Programs and scripts used in the data reduction process
Note: most of these are aliased (in /home/jessiec/mike/apt_setup) so that you
can run them from any directory.
* scripts & programs in /home/jessiec/mike/apt/
aptproc.pl The main pipeline script
translate_keywords.pl Change FILTERS keyword in FITS header to FILTER
sortdata.pl Sort images into sky/ and star/ subdirs
combflat.pl Combine twilight flats, wrapper for fitsio_combine_flat
combstar.pl Combine star images, wrapper for fitsio_combine_star
tycho.pl Find Tycho stars in the field and match them up with
Mike's object catalogue
lot_merge.pl Calculate coordinate transformations to master frame
dolistr Shell script wrapper used to run imcore_list on all
processed images in the current directory (this used
to be done by lot_list.pl). Takes rcore as an input.
cllc Perform relative photometric calibration on a results
file to remove atmospheric extinction variations
lightcurves Display RMS vs. mag and individual lightcurves from
results file
res2lc Generate individual .lc files from results file
res2lc.pl Wrapper for res2lc, saves log of output
* Mike's Perl scripts in /home/jessiec/mike/processing/
hedit.pl Edit FITS headers
imcore.pl Generate object catalogues for images (wrapper for
imcore_conf)
lot_classify.pl Classify detected objects
* Mike's Fortran programs in /home/jessiec/mike/soft/fitsio/
fitsio_combine Stack images (newer version, but not currently used)
fitsio_combine_flat Combine flatfield images, rejecting those with less
than 10000 counts on average (based on an earlier
version by Mike)
fitsio_combine_star Combine star images, rejecting those with more than
5000 counts on average (based on an earlier version)
fitsio_ccdproc Process (flatfield, debias, etc...) an image
imcore_conf Generate object catalogue for and image
imcore_list Do photometry on an image, using an input object list
* IRAF scripts in /home/mgh/iraf/scripts/new/
AptAstrom Find Tycho stars in image, do quick astrometry & photometry on
them to set WCS info in header & estimate sky transparency.
Also adds ephemeris info such as JD, UT, ST, AIRMASS, etc...
(../setheader used to be for seting JD, UT, ST, in headers , but no longer
needed now that AptAtrom does this)