Reducing Data from the 40-inch Telescope
Before you do anything, make sure you have saved a back-up of the raw data.
This will ensure that if you screw something up, all is not lost.
Step 1: Trimming and Subtracting Bias, flatfield correction from Images
1. cd into directory of one night's data and make sure that the directory has biases, v-flats, and i-flats. If not, copy flats and biases in from the nearest night which has them.
2. Make a new directory using the name of the night plus "reduced" (in the 060505 directory, make a directory called "060505reduced"). The reduced data will go into this directory.
3. cd into this directory
4. Open iraf
5. Run "apt". This loads the automated APT script.
6. cd into reduced directory within iraf (eg, "cd /home/duane/May06_40/060505/060505reduced"). This tells iraf where the reduced data will be sent.
Typing "pwd" in the directory will give you the "address" of that directory.
7. Open ds9
8. Open up a bias image (be sure to set "scale" to "zscale"). Zoom into the bottom left-hand corner. Write down the x and y coordinates to trim off the dark column on the left hand side. Biases only need the bottom left-hand coords. Write biassec coords in this fashion: [x,y:*]
9. Now open a flat image and do the same. However, you will also need to write down the x and y coords of the upper-right hand corner, being sure to cut off dodgy columns and the overscan region.
Write down the coords in this fashion: [x1:x2,y1:y2]
(Where x1 and y1 are the bottom left-hand coords and x2, y2 are the upper right-hand coords)
10. Run "freduce". This is the script for automated reduction of 40-inch data.
11. It will ask "Input directory, where the raw images are? (/home/duane/May06_40/raw/060508/):" Insert the address of the directory where the raw data is stored. If the above address in parantheses is correct, just press 'enter'.
12. It will then ask "TRIMSEC for all the images? ( [140:2097,40:4130]):"
Enter the coords as asked above in step 9 (if coords are correct, press enter).
It will then ask "BIASSEC for all the images; (y should be *)? ([2:138,*]):"
Enter the coords as asked above in step 8 (if coords are correct, press enter).
13. It will finally ask "Start the reduction from scratch? (yes):" (yes) means the default answer is yes, so hit enter. Now, the script should run. It will trim the overscan region and dodgy columns, produce a master bias and master flat in each colour, then subtract the bias and divide the flat from the images in each colour). This will typically take a few minutes. If the script stops and gives an error, it is usually because one of the images is bad (could be a bias, flat, or image). If this is the case, go back to the Linux shell and make a new directory called :junk:. Move the bad image to this directory and repeat steps 10-13. When the script is complete, it will give a list of data for the v and i images that looks like this:
IMAGE NPIX MEAN STDDEV MIN MAX
pflat1289v.fits 8010178 0.8565 0.057640 0.15000 89.44
pflat1291v.fits 8010178 1.0290 0.043160 0.14460 71.26
pflat1292v.fits 8010178 0.8500 0.024820 0.11420 38.15
pflat1293v.fits 8010178 0.9316 0.017160 0.12650 31.11
pflat1294v.fits 8010178 1.0610 0.006825 0.49360 7.095
pflat1295v.fits 8010178 1.3160 0.011910 0.12680 12.32
pflat1296v.fits 8010178 1.0630 0.007798 0.03137 3.546
pflat1297v.fits 8010178 0.8893 0.009709 0.02906 11.36
14. Check the STDDEV column. The flats that have low STDDEV are good (~0.005). If some of the flats give a higher STDDEV, especially anything over 1.0, then go back into the raw directory and move those particular flats into the junk directory. Remove all of the files in the reduced directory and re-reduce the data (steps 10-13). There should be at least 5 flats/biases in order to make a decent stacked Bias or Flat image. Check the Bias.fits, FlatV.fits and FlatI.fits images and make sure they are good (ie, they are trimmed, have NO STARS, etc).
15. Once all of the data for each of the nights is reduced, move them all into a separate directory: eg, "May06reduced".
Step 2: Reducing the Images to Extract the Photometry
REMINDER: When running IRAF in a directory, be sure to cd into that directory in the IRAF window (run pwd to get the address)... therefore IRAF knows which directory to work from!
The below scripts describe all of the steps. The actual process if finally automated, thanks to the Python programming skills of Tim Leslie.
cd into /data2/duane/src
python run.py /directory/name/with/reduced/data/
This will sort the data and put it into separate directories called UNSW-TR-##_V (or I) and do all of the processing steps up to Step 30.
You must manually choose the images to stack and create a master.fits file. once you make (and check visually) the master.fits file, the rest can be completed by running:
python run1.py /directory/name/with/reduced/data/separated/
This will do everything you need. Now just check the master.fits file to identify the target star, then flc into an lcs directory to see the light curves!
Sorting the fields and colours
Check and see what fields are in each night of reduced data.16. From within reduced night's directory, open IRAF, and cd into that particular directory within iraf (eg. cd /home/duane/May06reduced/060505reduced)
17. Within IRAF, run "epar hedit"
18. Set "edit", "add", and "delete" to "no"
19. Set "input" to "pim*i.fits" (or pim*v.fits or whatever filter it is)
20. Set the field to "OBJECT"
21. Set "value" to "."
22. run ":q" to get out
23. Run "hedit pim*i.fits OBJECT ." (or "hedit pim*v.fits OBJECT ." to list v-images)
24. Make a note of which images are in which fields, make directories for the different fields and different colours (I or V) then move all the the data into the appropriate directories.
For example, if the fields in a particular night are TR-41 and TR-2, then move the images(bias and flat images do not need to be moved - they have already been subtracted from the images) to the following: I-TR-41, I-TR-2, V-TR-41, V-TR-2... OR TR-41 with subdirectories "I-images" and "V-images"... whatever works for you.
Setting up Images for Aperture Photometry
25. cd into one of the subdirectories and run:imcore.pl "pim*.fits" noconf 10 6 1 4 >> LOG
lot_classify.pl pim*cat.fits >> LOG
A note on the numbers: 10 is the number of pixels that need to be connected together above the detection threshold to represent a detection; 6 is the number of sigma above the background noise the pixels need to be to be detected; 1 is a flag indicating the field may be crowded; 4 is the initial radius of the photometry aperture for the purposes of detection. You can play with these numbers to reflect different circumstances (e.g. for the APT we use 4 4 1 3).
REMEMBER: change i or v depending on which filter the images were taken in
Making a Master Image
26. Choose image with the lowest airmass. To do this, you can run what.pl, and the last column of the output is the airmass. Then link that file to the name master_cat.fits:what.pl "pim*i.fits"
ln -s pim----i_cat.fits master_cati.fits
(where pim----i_cat.fits is the name of the image with the lowest airmass)
28. Run the following:
lot_merge.pl master_cati.fits pim*i_cat.fits >> LOG
29. Now you can generate a stacked master image. Remove the master_cat.fits link (rm -rf master_cat.fits)
30. Run /home/mgh/pipeline/apt/transinfo.pl 550 1050 > output
31. "less output" and check for a series of images with similar shifts
32. Make a list (in a text file) of those images (emacs list - paste names of files, save)
Example (all images below have a shift of 0.59 to 0.72):
pim0160i
pim0162i
pim0163i
pim0164i
pim0165i
33. Open iraf:
run "epar imcombine"
change the "input" to "@list" (where list is the name of the text file from step 32)
change the "output" to "master.fits"
run ":g" (this will create the master.fits file)
STOP and visually check the generated image master.fits before continuing. If it is not a nice sharp image with low noise in the background, repeat steps 32 and 33 which a different list of images.
34. Run:
imcore.pl master.fits noconf 10 6 1 4 >> LOG
lot_classify.pl master_cat.fits >> LOG
35. Remove all the pim*cat* and pim*list* (if there are any) and pim*trans files, so now you should be left with just pim-----.fits files, and the master files (rm -rf pim*cat* pim*list* pim*trans*)
36. Run:
imcore.pl "pim*i.fits" noconf 10 6 1 4 >> LOG
lot_classify.pl pim*cat.fits >> LOG
lot_merge.pl master_cat.fits pim*cat.fits >> LOG
38. To set up the images to extract lightcurves, run:
dolist.pl 9 pim*i.fits >> LOG
If you get a warning about a variable only being used once, ignore it. I (Jessie) will try and fix this in the near future. dolist.pl should take a few minutes to run. If it takes a very short period of time, check the LOG file to make sure it has executed successfully. It should create pim*_list.fits and pim*_list.ell files. Check that the directory contains these files. 9 is the radius of the photometry aperture that we will use to generate the lightcurves. This number can be changed.
Possible Problems
1. No RA and DEC in the header - if there are other images of the same object in the same directory, can they be read in from there? Or will we need to set up a master list of RAs and DECs of all the objects we observed? This needs to be done before the setjd step.2. The image is blank/didn't read out correctly/all values are the same (non-saturated) value. Possibly just needs a standard deviation test on a small region of the image? These images cause the catalogue generating software to crash eventually.
3. The image is misaligned somehow, to the extent that the catalogue matching software crashes complaining of incomprehensible list input, so something is getting written incorrectly earlier due to this problem.
An example of this kind of image can be found at:
129.94.163.48:/data/jessiec/followup/test/try2/im0113i.fits
There may be more night-specific problems that I didn't come across with the test data. If the imcore/lot_classify/lot_merge/dolist steps fail at some stage, use the logs to work out what image the step has crashed on, and visually inspect that image and its header to see if you can work out what the problem might be. Try comparing that image with one that the program didn't crash on to see if that helps.
Step 3: Getting Lightcurves
39. Once you have a directory with pim*_list.fits files, run:
mef2res nord >> LOG
The 'nord' flag means you are not giving it a file with the RA and DEC of the apertures, which is the case for APT data reduction.
mef2res should create a series of files called "resultsn", "resultsn2", etc etc (each of these files is the photometry as measured in a different sized aperture, based on the number entered in Step 38). If these files have not been generated, check the LOG file to see what has happened and try to fix it.
40. Once you have the resultsn files, we can create lightcurves. We are not running cllc on the 40inch data as we do for the APT data for various reasons, so we can skip straight to the lightcurve step. Make a directory to put the lightcurves into (eg. mkdir lcs), and then run the following command:
res2lc.pl lcs/ 14. -40 resultsn
lcs/ is the directory that you have made to put the lightcurves in, 14. is the magnitude down to which you want to generate lightcurves (include the decimal point for correct variable typing), -40 is a flag to switch the CCD dimensions (hardwired into the code) to the values for the 40-inch telescope, and resultsn is the name of the results file you are extracting the lightcurve info from (you could try res2lc.pl for a couple of the different resultsn files and see what different lightcurves you get).
41. Move into the directory with the lightcurves, and check that they have been generated. You can check the quality of the lightcurves by running:
flc 1
This will generate a new window with the lightcurve plotted, of the first star. Back in the window with the prompt, keep pressing enter to go through the stars, and Ctrl-C to get out when you are happy that you understand what's going on with the lightcurves.
42. At this stage you will need to work out which star in the master.fits image is the target star. In the directory /data/jessiec/candidates/ you will find subdirectories for each of the candidates, including finder charts and region files to be loaded in ds9. The image trxx.jpg (where xx is the candidate number) has a close-up of the DSS image with the target star highlighted. If this is not enough to locate the star, get the coordinates of the star out of the trxx.reg file and use ds9 to download a larger DSS image from the DSS server. REMEMBER: A DSS image must be rotated by 90 degrees and inverted in Y in order to match the 40inch images.
43. To work out the ID number of the star, move back to the directory with the master.fits and master_cat.fits files, and run:
makereg.pl master_cat.ell
This generates a file called ds9.reg. Open the master.fits image in ds9, and load the region file ds9.reg. This will label all the stars with their ID numbers. Once you have identified the ID number of the target star, move back to the lightcurves directory. You can now check out the lightcurve of the target and see if we caught a transit.