CUPID General Info

CUPID_V2.0

CUPID General Info     "cupid" Command     All Pipelines     All Modules     


The CUPID package allows users to run the Spitzer IRS Pipelines.

==============================================================================
RUNNING CUPID
==============================================================================

The "cupid" command will run IRS pipelines in specified processing directories.

For details about running the "cupid" command,
      run the "cupid" command with no parameters, or see:

"cupid" Command

In the nominal case first the BCD pipeline produces bcd images, then the COADD pipeline produces coadd images and then the BKSUB pipeline produces background subtracted images. After this, there are three pipelines, BCDX, COADDX, and BKSUBX which do spectral extractions on the images. For a list of all pipelines and the steps in each, see:

All Pipelines

For a list of all modules and the parameters for each, see:

All Modules

You could run the 7 standard IRS pipelines on an AOR by doing the following: 1. Use Leopard to download the desired AOR. Only the "raw" data is needed. The "cal" data is also needed if you have the "_NOCAL" version of CUPID. (An AOR often contains 4 "processing directories", ch0, ch1, ch2, and ch3.) 2. Run cupid with a command similar to the following, but pointed at your downloaded AOR: cupid /data/r1234567/ch* ============================================================================== Additional HELP topics: ============================================================================== The following topics are discussed below: PROCESSING DIRECTORIES (contents of pipeline processing directories) STATUS (meaning of the status of a step or pipeline) LOG FILES AND STATUS FILES (module calls and status information) OPTIONS (how to specify customizations when running pipelines) VERBOSITY (how to get more or less information during processing) FETCHING CAL OR CDF FILES (how CUPID decides which CAL and CDF files to use) ALTERNATE CAL OR CDF ARCHIVES (how to use your own CAL or CDF archive) CUPID DIRECTORY (contents of the CUPID directory) WRAPPER SCRIPTS (how processing is specified in wrapper scripts) TROUBLESHOOTING (some things to try if things do not seem to be working) ============================================================================== PROCESSING DIRECTORIES ============================================================================== All pipelines are run in "processing directories". There can be four processing directories for each AOR, one per channel (ch0/ ch1/ ch2/ ch3/). The CUPID processing directories are very similar to what is in a full Leopard download. After running pipelines, each processing directory will contain the following subdirectories: raw/ contains original "raw" DCE files (from Leopard). bcd/ contains data files created by the single DCE pipelines. (BCD and BCDX) It also contains a .log file for each pipeline run. See "BCD Pipeline Description" document for details. pbcd/ contains data files created by the multi-DCE pipelines. (COADD, COADDX, BKSUB, BKSUBX) It also contains a .log file for each pipeline run. cal/ contains all calibration files used. cdf/ contains all "control data files" (namelists) used. =============================================================================== STATUS =============================================================================== Every step run returns a status. If a step returns a status >=64 then processing is halted immediately. The meaning of status levels is: 0 = completely nominal 1 = processing skipped because file is wrong type (not a problem) 2 - 31 = something happened out of the ordinary, but no problem 32 - 63 = something happened that may be a problem, processing continues 64 - 255 = something happened that is a problem, processing terminated =============================================================================== LOG FILES AND STATUS FILES =============================================================================== There is a log file for each pipeline run output subdirectory. The location of the log file is shown as the first line while running a pipeline. The log file contains pipeline run information and any error messages. In addition, there is a STATUS file in the output subdirectory which summarizes the status of the most recent run of each pipeline. ============================================================================== OPTIONS ============================================================================== Here are details of how the "options" work. Note that the "SET" option uses module names but other options use STEP names. Also note, these options apply to the BCD pipeline unless they are prefixed with "X/", in which case they apply to all of the extrqaction pipelines (BKSBX, COADDX and BCDX). SET (override parameter value): This is used to set the value for a parameter when particular module is called. Each override specifier has the form: SET.-=. For example, a new flat file could be specified with: SET.flatap-f=/urs/joe/data/newflat_1.fits Or the following to specify 100.0 as the estimated readnoise when creating the uncertainty file: SET.snest_irs-readnoise=100.0 START (start processing at a specified step): This will cause the first step(s) to be skipped over. The specified step where processing will have the input files just as if the preceding steps had been run. This means that the pipeline must have been run from the first step at least once. For example, to begin processing at the CUBESUB step, specify the following: START.CUBESUB (It will use inputs from a previously run DARKBASE step.) STOP (stop processing after a specified step): This will cause the the pipeline to stop processing after running the specified step. For example, to stop processing after the FLATAP step, specify the following: STOP.FLATAP SKIP (skip a step): This will cause the the pipeline to skip the specified step. The following step will take its inputs from the preceding step. (This is true even if a previous pipeline run has produced data files for the skipped step.) For example, to skip the STRAYCROSS step, specify the following: SKIP.STRAYCROSS (In this case the following FLATAP step will use inputs from the DROOPRES step.) DO.AFTER (move a step): This will cause the the pipeline to execute the first specified step after the second specified step. The first specified step must precede the second specified step in the nominal processing order. For example, to swap the order of CUBESUB and DARKBASE, specify the following: DO.DARKBASE.AFTER.CUBESUB The full command to produce "unstraycross" BCDs if the pipeline has already been run would be the following. Note that it starts with the FLATAP step to avoid needlessly re-running the first few steps. cupid BCD r123456/ch1 START.FLATAP SKIP.STRAYCROSS The "SKIP.STRAYCROSS" option here will cause FLATAP to get it's input from the DRROPRES output files rather that from the STRAYCROSS ones. =============================================================================== VERBOSITY =============================================================================== The default level of output should be sufficient in most cases, but there may be cases where more or less output is desired. There are environment variables to control the amount of information output. SCREEN OUTPUT (CUPID_SHOW environment variable): The default value of CUPID_SHOW is 2. It can be set to a lower value to reduce the clutter on the users terminal. As each pipeline runs, the default is to display the following: (1) the output log file location, (2) step-by-step status (if CUPID_SHOW >= 1), and (3) and final status. And then, after all pipelines have run, a summary is displayed. (if CUPID_SHOW >= 2) LOG FILES (CUPID_LOG environment variable) The default value of CUPID_LOG is 2. The optional contents are as follows: - the actual command line call used to process a step (if CUPID_LOG >= 1) - values of FITS words read from file (if CUPID_LOG >= 2) - information about fetching cal and cdf files (if CUPID_LOG >= 2) - values explicitly set environment variables (if CUPID_LOG >= 2) - values of defaulted environment variables (if CUPID_LOG >= 3) - timing summary per step (if CUPID_LOG >= 3) =============================================================================== FETCHING CAL OR CDF FILES =============================================================================== Whenever a module needs a CAL of CDF file, CUPID must decide which file to use. Here is the logic used: 1. If the file is already there in the local cal/ or cdf/ directory, it is used and no actual "fetch" is required. A file is considered "there" if: (a) it has the correct root file name, and (b) it has the correct channel number (b0_ b1_ b2_ b3_), and (c) if it is a dark, it must have the correct exposure time designator (r6_ r14_ etc.) The SCLK range in the file name is not considered. 2. And if the file is not in the local directory, if will be "fetched" (copied) from the CAL or CDF archive directory. The file fetched will be the one that is qualified as follows: (a) it has the correct root file name, and (b) it has the correct channel number (b0_ b1_ b2_ b3_), and (c) if it is a dark, it must have the correct exposure time designator (r6_ r14_ etc.) (d) it has a SCLK range that covers the data being processed. If more than one file qualifies, the one with the smallest SCLK range will be used. =============================================================================== ALTERNATE CAL OR CDF ARCHIVES =============================================================================== If you have a directory with an alternate CAL of CDF archive it can be used by setting the CUPID_CAL_ARCHIVE or CUPID_CDF_ARCHIVE environment variables. =============================================================================== CUPID DIRECTORY =============================================================================== The contents of the root CUPID/ directory is described in the README file. =============================================================================== WRAPPER SCRIPTS =============================================================================== The actual pipeline processing logic is in wrapper scripts in the CUPID wrappers/ subdirectory. These wrappers specify the pipeline "steps" and module calls for each pipeline. Parameters passed to modules are specified as follows: IN input data files - may not override - are prepended with prefix (e.g. SPITZER_12345678_S3_0003_0001_) - may use wrapper variable (e.g. ) as filename - may also set wrapper variable (e.g. ) to the filename OUT output data files - may not override - are prepended with prefix (e.g. SPITZER_12345678_S3_0003_0001_) - may set wrapper variable (e.g. ) to the filename UPD updated data files (data files which are input AND updated) - may not override - are prepended with prefix (e.g. SPITZER_12345678_S3_0003_0001_) CAL calibration files (input, in cal/ subdirectory) - may override - are in the /cal subdirectory if not overridden - fetched from CUPID cal archive if not there already CDF CDF (namelist) files (input, in cdf/ subdirectory) - may override - are in the /cdf subdirectory if not overridden - fetched from CUPID cdf archive if not there already V values (numeric or string) - may override - (no special handling) =============================================================================== TROUBLESHOOTING =============================================================================== 1. If processing starts but then crashes: At the start of each pipeline processing run, the name of the log file is printed out. It ends with ".log". Look inside this file for error messages which may explain what happened. 2. If processing doesn't even really start: Cupid may be having trouble determining either: (a) the type system you are running on, or (b) the location of a valid Perl executable. To find out, enter the following command, then try to run again: setenv CUPID_SHOW 3 When you try to run cupid again, you should get a line shown which contains: (a) "Using Perl:" followed by the path to where CUPID is running Perl (b) "Modules:" followed by one of: bin_sun, bin_mac, or bin_linux indicating what OS CUPID thinks you are using. To overide the Perl path, use the $CUPID_PERL environment variable to indicate a valid Perl executable. For example: setenv CUPID_PERL /local/perl-5.8.7_build/bin/perl To overide the OS type, use the $CUPID_BIN environment variable to indicate a valid OS, one of: bin_sun bin_mac bin_linux For example: setenv CUPID_BIN bin_linux