Transfer of Polar Camera project
================================
Tasks Duration Start
========================================= ============ ========
- Enumeration of items to be returned 1 hour 31, 3 pm
to Calgary
- Summary of data processing and analysis 4 hours 01, 8 am
operations, and relevant software
- Preparation of briefing documents on >1 day 02, 8 am
all aspects of the system, its care and
feeding, and the scientific work
completed, begun, and intended
- Housecleaning workstation: archival and 2 hours 03, 8 am
removal of personal files
- Completion of field-of-view assessment 1-2 hours ??
for R.A. Doe
- Negotiation of transfer of workstation 4 hours ??
and software to the University of
Calgary (if desired) (CSA; UofSK)
- Completion of 1997-98 data processing 1-2 weeks ??
and generation of quick-look plots
- Modification of processing and analysis 1 week ??
software to run in Unix environment
------------------------------------------------------------------------
Camera: records/files related to design
" " construction
" " test
" " calibration
" " deployment
" " operations
system documentation: electronics
control software !!!
assembly
log books
miscellaneous hardware and cables
software and manuals
Data: raw data tapes
processed data tapes
processed data CD-ROMs
quick-look plot books
Science: reprints of existing papers
files on particular events
studies in progress:
- polar diffuse aurora
- single-station auroral stereoscopy
- passbands and OH (6,2) P- and Q-branch throughputs of OH
channels
- OH waves, Dec. 22-25, 1994
- N2+ M throughput of 785 nm channel - needed for quantitative
use
P120 workstation: ???This will need some negotiating!???
- Comtex computer (including internal DAT drive)
- CTX monitor
- Canon printer and ink
- APC Line-R 1250 power conditioner and Back-UPS Office
- Software, including IDL license and documentation
------------------------------------------------------------------------
Operations:
===========
Listing tape contents:
----------------------
Run C:\NOVABK\Novaback.pif | Utility Functions | Summarize a Backup
Tape. Store the results to C:\DATA\TAPES\EUnnmm.SUM, where nn = tape
number, mm = backup set number (1,2,...). If the last backup set spans
two tapes, it's better to abort at the end of tape than to try to run
onto the new tape.
Processing UT-day periods of data:
----------------------------------
Any IDL routines mentioned are found in C:\IDL\STEELE unless otherwise
indicated. Processing is done using PROCESS.PRO, and the steps involved
are the following:
- Load the required data from the data tape, using NOVABACK's RST
function called from the command line using IDL's SPAWN facility, with
arguments to restrict the restored data to the appropriate time
period; this is accomplished using C:\IDL\STEELE\NOVASTOR.PRO.
- Uncompress any data files that were compressed within the camera
control software before being written to disk; create an image list
file giving the start time in UT seconds and the full path name for
each data file from the given date, with simultaneous files from
Cameras 0 and 1 shown together on a line of the file; weed out any
damaged or incomplete data files by checking header and file size;
UCALL.PRO -> KIEXPAND.PRO for each data file, which checks for a
file name indicating compression, and
SPAWNs a DOS executable to uncompress it;
-> WRILIST.PRO, which assembles arrays of file names and
start times for all images files from the
specified date and one or both cameras,
sorts the lists by time, and writes the
lists to separate files by camera; unless
defeated, the two lists are then merged
into a single list with simultaneous files
listed on the same line, and then written
to C:\DATA\ILIST\yymmdd_2.ILI, after which
the individual lists are deleted;
-> WEEDOUT.PRO, which reads the image list file using
RDILIST.PRO, and reads each file in the
list, deletes any file that generates an
I/O error from the image list, and then
writes the corrected image list back to the
original file name.
- Reduce any calibration data included in the data files for the day,
using REDUCALN.PRO, which is well-commented (IMHO).
- Process the raw data files into images in geophysical units, Rayleighs
(R) or R/nm for background images, with up-to-date field calibration
information, using NEWQL.PRO; the routine also gives the option to
generate zenith brightness data files (from which a quick-look plot
can be made), and hourly thumbnail images from all channels as well
as dark frames, as a second quick-look data product; this routine was
originally an IDL program to be run interactively to display
calibrated images on the screen, but was converted to a subroutine
for unattended processing; it is over 1100 lines of extensively-
commented straight-line code (SORRY!); I just noticed that there is
no header defining the input and putput parameters - this must be
rectified immediately!; the user has control over all kinds of
parameters but the invocation of NEWQL.PRO in PROCESS.PRO gives the
default values and should not be tampered with lightly; the sequence
in the routine is as follows:
- get needed input either from command line or by prompting user;
- read image lists generated by WRILIST.PRO and WEEDOUT.PRO;
- define image masks to ignore pixels outside the fields of view;
- drop the calibration data files to simplify late processing;
- divide the data into 'cycles' by the time intervals between images
(a 'cycle' consists of all images acquired during one execution of
a data acquisition script by the Polar Camera control software -
typically 4-5 60-s images per camera - so a cycle lasts about 4.5
to 5.5 minutes);
- load the uniformity correction ("flat-field") images;
- load the field calibration results and filter characteristics;
- open the ASSOCiated files used to hold the calibrated images as they
are produced;
- identify the near-zenith pixels for later use;
- set up for inline background subtraction (NOTE: this option is NOT
used, by default, and should NOT be considered reliable; for
instance, it assumes the presence of THREE background channels, not
the current TWO, and will fail for any data after March 1994; this
operation has been implemented separately and is described later in
this document);
- load the default (mean) dark frames for use until actual frames are
read from the current day's data;
- for each 'cycle', do the following:
- surrender control of the PC to the OS for 1 second;
- open and define parameters of temporary storage for all images
in 'cycle';
- for each image pair in 'cycle', do the following:
- for each image in the pair, do the following:
- if the image exists, read it, and decode and validate the
header;
- extract the exposure start time, length, and type (i.e.,
auroral, airglow, calibration, dark, ...) from the header;
- subtract the appropriate dark frame, if the image was acquired
with the shutter open, or store the new dark frame for later
use, if not;
- rebin the image to 256x256 if necessary - images after March
1995 are all 128x128 to save disk and tape space;
- apply the uniformity (flat-field) correction appropriate to
the camera and filter number;
- define and apply the calibration factor, converting background
images to Rayleighs per nanometer;
- temporarily store the resulting calibrated image, after an
attempt at spike removal, until all images from the cycle have
been processed;
- save the zenith intensity to the appropriate quick-look data
file;
- store a thumbnail image for quick-look purposes (if the image
is the first of the current hour through its filter);
- ---------- end of loops over images, pairs ----------
- close the temporary storage, now containing all processed images
from cycle - this creates a large file on the hard drive;
- reopen the temporary storage;
- for each image pair in the current 'cycle', do the following:
- for each image in the pair; do the following:
- load the image from temporary storage;
- scale the image, up to the 99.5th percentile, by a power of 2
encoded in the header, to enable storage in 8 bits, unless
the image is a dark frame or a background image (and should
not need scaling);
- store the scaled image in the right ASSOCiated file;
- ---------- end of loops over images, pairs, cycles ----------
- delete the temporary storage file;
- close the quick-look files holding zenith brightnesses;
- close the ASSOCiated files holding calibrated images; they are named
D:\IMAGES\SAVE\yymmddcf.RAY, where yy, mm, dd, c, f are year, month,
day, camera, and filter;
- close the thumbnail image quick-look file and plot it if it is
complete;
- plot the zenith quick-look data to a PostScript file for later
printing using PLOT_QL.PRO, which is fairly self-explanatory;
- create a dedicated log file directory for the current day and store
there the various log files that accompany the data files, after
gzipping them to save room (STORLOG.PRO, again fairly self-
explanatory);
- store the final versions of the previous day's log files, which may
not have existed when the previous day's data were processed (for
historical reasons too complex to go into here, and which I don't
remember anyway!)
- plot the temperature log file using PLOTTLOG.PRO (again fairly self-
explanatory, although it calls RDTLOG.PRO to decode the temperature
logs, and CT0.PRO and CT1.PRO to assemble the data required to make
the plot);
- clean up (i.e., remove) the data files from the current day, as we
have no further use for them;
- clean up the residual data files from the previous day, which were
loaded because our time interval for loading ran from 10 min before
0 UT to 10 min after 24 UT;
NOTES:
1. This process can be automated to run for several days by means of an
IDL program such as C:\IDL\PROCESS\980212PM.PRO, which invokes
PROCESS.PRO for several days in succession and speeds up processing by
running overnight. Note that data from days which span two tapes must be
loaded manually by running C:\IDL\STEELE\NOVASTOR.PRO for each of the
backup sets.
2. The PS files created above (thumbnail images, zenith brightnesses,
and temperatures) are not queued for printing by the code as it now
stands, as this option was not available under Win95; the files must be
manually queued.
Doing background subtraction for specific periods of data (up to 1 day):
------------------------------------------------------------------------
Any IDL routines mentioned are found in C:\IDL\STEELE unless otherwise
indicated. The process requires that two .RAY files be present in
D:\IMAGES\SAVE: the channel from which background is to be subtracted,
and the appropriate background channel. The process involves two steps:
generating estimated background images for short sub-intervals of time
throughout the period over which background subtraction is to be done,
and then subtractng the appropriate background estimate from each image
over the period. The results are stored in D:\IMAGES\SAVE\yymmddcf.BGS,
where yy, mm, dd, c, and f are as above, and BGS stands for BackGround
Subtraction.
The first step is implemented using C:\IDL\STEELE\SAVBKGRD.PRO, is
called once to cover a defined time period using defined time
subintervals. The method is to assemble all of the images from a
specified background channel during each subinterval, and to define a
background image such that each pixel's value is the greatest lower
bound of the values at that pixel position in the images taken during
the subinterval. This minimizes the effects of transient changes like
intermittent cloud or moving stars. The resulting image is smoothed with
the LEEFILT routine supplied with IDL, and saved to an ASSOCiated
variable (i.e., a disk file) as a floating point image. The file
contains one such image for each subinterval spanning the period, except
for subintervals that don't contain any images. The file name is of the
form C:\IMAGES\BKGRD\yymmddcf.BKG.
The second step is implemented using TRAY2BGS.PRO. This routine must be
called once for each subinterval (defined above) in the desired time
period, and it subtracts the appropriate background image from all
images acquired in the selected channel during that subinterval. It
creates the file D:\IMAGES\SAVE\yymmddcf.BGS.(I never got around to
modifying it so the functionality was like that of SAVBKGRD.PRO -
sorry. I usually define two arrays, containing the start hour and minute
of each subinterval respectively, before running TRAY2BGS.PRO, and then
run it in a loop over the elements of these arrays, using them as hour
and minute arguments to TRAY2BGS.PRO.)
Viewing single-channel image sequences as a "video":
----------------------------------------------------
VIDEO.PRO is an IDL program, invoked using the .RUN command. It prompts
the user for date, camera/filter, and time information, and user
preferences regarding the display. It offers the option to transform
images to geographic or geomagnetic coordinates before display, but
these transformations are very slow and are better done using other
means (see below) where the results can be saved for later reference.
The basic procedure is very simple: after locating the image file in
D:\IMAGES\SAVE (*.RAY or *.BGS; if both exist a choice is offered), and
setting up the display window, each image is read in turn from the file,
reconstituted by undoing the scaling applied in NEWQL.PRO, and displayed
to the window with annotation as appropriate. Unless interrupted with an
'x' keystroke or paused with a 'p' keystroke (press any key to restart),
the display will continue until the end of the image file is reached.
The delay between images can be set, but the display proceeds in time
order, i.e., it cannot presently be made to run backward. (It would be a
simple adjustment.)
Determining image orientation w.r.t. stars:
-------------------------------------------
STARID.PRO is a procedure that searches for 'stars' (small, relatively
bright spots) in three consecutive raw background images proposed by the
user, and generates a list of the 'stars' found. This list is supplied
to STPLOTFIX.PRO, a procedure which allows the user iteratively to
identify individual 'stars' with actual features in the sky. This is
done by generating a synthetic star map based on the Yale Bright Star
Catalog, the date, time, location, and approximate orientation of the
cameras. The procedure identifies all stars that lie within a certain
pixel distance from the 'star', and displays them in sequence for the
user's inspection. When the user is satisfied which star matches the
'star', a 'y' keystroke tells the procedure to record the relevant data
and go on to the next 'star'. This process continues until all 'stars'
have been identified with stars or called 'planets' (i.e., ignored). The
recorded data are then edited to put them into the correct format for
input to Mathematica, and written to a file named like
C:\DATA\STARPOS\G0601060.M. (Here, G indicates airglow, 06 indicates day
06 of the month, 01 indicates camera 0, filter 1, 060 indicates 60-s
exposure, and the M suffix indicates a Mathematica program.
Mathematica is invoked with input from the file created by stplotfix:
mathematica < g0601060.m
and the output is g0601060.res. The last five lines of this file show
the best-fit values for the following parameters:
c0: column number of the pixel containing the zenith (0..255);
r0: row number of the pixel containing the zenith (0..255);
r: 'stretch' parameter in the transformation from radial pixel
distance to zenith angle (see the header of CR2AZ.PRO);
k: 'warp' parameter in the transformation from radial pixel
distance to zenith angle;
ca: azimuth (deg E of geographic N) of column c0 of the CCD
detector;
These parameters are extracted and manually stored in a file named
C:\DATA\MEANPARM\MNPyyddd.EU; where yy is the year and ddd is the day
number of the day (1..365) of the observations used to determine the
orientation. The file contains ten parameters on two lines; those for
Camera 0 are on the first line, and those from Camera 1 on the second.
File of this type are read by procedure RDMNPARM.PRO, which reads the
parameters for the specified camera on the specified date from the
appropriate file.
Plotting single-channel image sequences:
----------------------------------------
SHOWRAY.PRO is an IDL program to display sequences of calibrated images
from a single channel of the Polar Camera. The user is prompted for
date, cam/filt, and start time, as well as whether to create a
PostScript file or a screen display, whether to save the zenith
brightnesses (useful if displaying a .BGS file), whether to apply a van
Rhijn correction (no), the desired color table, the desired full scale
brightness, and whether to use a logarithmic or linear color scale. The
user can select a common fullscale for all images (any value greater
than 1), or a common maximum percentile to display (any value less than
1). Once these are defined, the program reads, scales, displays, and
annotates images in sequence. The user controls the display rate by
pressing the space bar once per image. The screen holds 12 images and
each new images overwrites the previous one in that position. Display
continues until the user presses 'q' to start a new display sequence, or
'x' to exit the program. Pressing 's' decrements the screen image
position pointer by 1, so the next image displayed overwrites the one
most recently displayed.
If the Post Script option is selected, the user can chose whether to
save the images as B/W (default 4-bit color depth) or color (default
8-bit depth). The filename will be of the form C:\PS\yymmddxn.{PS|CPS},
depending on whether B/W or Color display was chosen. yy, mm, dd have
their usual meanings, while x = 5*Camera + Filter (0..9) and n is larger
by 1 than the largest number among existing files sharing the same first
6 characters in their file names. For instance, if there exist files
C:\PS\98040220.PS and C:\PS\98040221.CPS, the next file created of
Camera 0, filter 2 images from April 2, 1998 will be C:\PS\98040222.PS.
This departure from the standard yymmddcf nomenclature was necessary to
allow multiple plot files from the same camera, filter, and day to
exist.
Plotting all-channel image sequences:
-------------------------------------
SHOWSEQ.PRO is an IDL program that behaves essentially the same way as
SHOWRAY.PRO, but takes data from all channels of the Polar Camera
instead of just one. All of the required .RAY files must exist at
D:\IMAGES\SAVE. The images are scaled individually to their 99.9th
percentiles. The PostScript option exists here, too, with filenames
C:\PS\yymmddAn.{PS|CPS}.
Mapping images to geographic or geomagnetic coordinates:
--------------------------------------------------------
POCAFLIC.PRO is yet another IDL program. As with those above, the user
is prompted for all needed inputs, which in this case include year,
month day, camera, and filter, start hour, minute, second, end hour,
minute, and second, assumed emission height to which to map the images,
the coordinates to map to, and others specific to the selected
coordinate system. If geographic coordinates are selected, the user can
choose to center the mapped image on Eureka or the nearby geomagnetic
pole, leave North at the top of the image or rotate it to put
geomagnetic noon at the top, and show coastlines and/or coordinate
graticules. If geomagnetic coordinates are selected, the user can
superimpose an MLAT/MLT clock dial. Full scale can be fixed or
autoscaled, but fixed is easier to interpret. The output is sent to a
file, implemented as an ASSOCiated variable, that is named
C:\DATA\PGM\yymmddcf.{GEO|PGM}, and is accompanied by a text file named
C:\DATA\PGM\yymmddcf.{GEI|PGI} where the trailing 'I' signifies
'information'. The information file contains a header indicating the
source data and the operations performed, followed by one line for each
image transformed, giving the image and transformation parameters. The
information file is important in the later use of the transformed image
file, and should not be tampered with. The contents of the image file
can be displayed using procedure SHOWFLIC.PRO, described below, or used
to generate a polar patch montage with MONTAGE.PRO, MINIVIDEO.PRO, and
MKMTG.PRO.
The transformation is accomplished by one of two routines: MAP_GEO.PRO
or MAP_PGM.PRO. Their inputs are broadly similar but differ in detail.
The MAP_GEO routine offers the opportunity to overplot a satellite
trajectory, but this has not yet been implemented in the PC environment.
(However, the FORTRAN source code exists, and there is a FORTRAN
compiler on the Polar Camera workstation; any takers?)
Displaying a sequence of mapped images:
---------------------------------------
SHOWFLIC.PRO is a procedure that allows the user high flexibility in
viewing a sequence of transformed images. The user supplies the
{GEO|PGM} file name, and any of several optional parameters, to the
procedure, and can view the images one at a time, or free-running,
either forward or backward in time. The images can be shown with a color
bar, and as positives or negatives. The user can skip to a particular
image, or skip forward or backward through the images 5 or 10 at a time.
All these options are available through single keypresses, and the
keypress menu can be displayed by pressing '?'. Selected images can be
flagged for later display as a PostScript file (although this segment of
the code may produce unexpected results because of unfinished
modifications).
Estimating Topocentric Positions of Sun and Moon:
-------------------------------------------------
EPHEM.EXE is a DOS executable to calculate solar and lunar topocentric
positions, given date, time, and 3D coordinates. The code also checks
the solar and lunar elevations against thresholds defined by the user,
and creates a 0-byte file called RUNKI if both bodies are below the
threshold elevations. The existence of this file is used by KI as a
signal that data can be acquired. The executable was compiled from a C
program that ought to be Unix-compatible.
========================================================================
Briefing Documents on All Aspects of the System, Its Care and Feeding,
and the Scientific Work Completed, Begun, and Intended
======================================================================
- System Installation
=====================
This is described in detail in documents prepared most recently for
Albert Maione prior to his trip to Eureka in October 1997.
- Assembly Instructions for Polar Camera ground tripod (2 pgs.)
- Installation Instructions for Polar Camera (3 pgs.)
- Installing and Testing the PoCa Host Watchdog Timer (1 pg.)
- System Start-Up
=================
This is described in detail in documents prepared most recently for
Albert Maione prior to his trip to Eureka in October 1997.
- Field Calibration
===================
This is described in detail in documents prepared most recently for
Albert Maione prior to his trip to Eureka in October 1997.
- Polar Camera Calibration Instructions (1 pg.)
- Polar Camera - Changing the Batteries n the Calibration Source (2
pgs.)
- Normal Field Operation at Eureka
==================================
This is described in detail in documents prepared most recently for
Albert Maione prior to his trip to Eureka in October 1997.
- Requested Attentions to Polar Camera - Winter 1997/98
- Polar Camera Tape Change Schedule, Eureka, Winter 1997/98
- TAPECHNG.PRO - allows the user to generate a skeleton tape change
schedule based on start date, end date, minimum solar depression angle
for data taking, estimated log-file overhead per day, tape size, and
length (min.) of one 'cycle'. The procedure models the amount of data
taken per day and determines when tapes fill up and must be changed.
- Syntax of System Control Language (or "How to Talk to KI")
- Laboratory Calibration
- Troubleshooting
- Scientific Studies:
- Polar Patches: December 8, 1993 (Steele and Cogger, 1996 Radio
Science; Prikryl et al., submitted to Annales Geophysicae)
- Polar Arcs: October 23, 1993 (Steele et al., 1995 DASP Winter
Workshop)
- Polar Diffuse Aurora: Dec. 94 - Jan. 95 (Steele et al., 1995 IUGG)
- Single-point Auroral Stereoscopy: October 23, 1993 (Steele and
Oznovich, 1995 Fall AGU)
- OH Waves: December 22-25, 1994 (Steele and Cogger, unfinished)
- Polar arc 630 nm emission tomography: Feb. 17, 1996 (Doe et al.,
1997 GRL; Doe et al., 1997 IAGA)
- CME Response: January 10-11, 1997 (Steele et al., 1998 GRL; McEwen
et al, submitted to JASTP)
- Collaborations: Leroy Cogger (UofC) regarding atmospheric optical
phenomena generally: mesopause and F-region
airglow, polar aurora (and Interball)
Don McEwen (UofSK) regarding Eureka optical data
Abas Sivjee (ERAU) regarding OH spectroscopy and
airglow at Eureka
John MacDougall (UWO) regarding CADI data
Paul Prikryl (CRC) regarding patch origins
Rick Doe (SRII) regarding polar arcs
Cesar Valladares (Boston College) - intended -
regarding polar diffuse aurora
Ed Weber (PL) - lapsed - regarding DMSP particle
data to accompany events observed by Polar
Camera
Marc Hairston (UTDallas) - lapsed - regarding DMSP
SSIES ion drift data and inferred E fields to
accompany patch drifts inferred optically from
Polar Camera images