Knowledgebase: Spitzer: SHA

Spitzer SHA : Documentation, Known bugs and issues

All the Spitzer documentation can be found linked from here.
All the SHA known bugs are here.

Why does this IRAC observation appear to have no post-BCD (Level 2) data?

If the data was taken in IRAC's subarray mode, or for certain IERs (Instrument Engineering Requests), there will be no post-BCD data for the observation. Data in IRAC subarray mode are not processed through the online post-BCD pipeline. The post-BCD pipeline was not designed to work with the three-dimensional data cubes that are the subarray BCDs. When the software was developed it was deemed unlikely that subarray observations would be used for anything other than photometry of single bright objects and mosaics would not be a desired product. As a result, post-BCD products (e.g., mosaics and source lists) are not available from the pipeline. Please see the IRAC Instrument Handbook or IRAC Knowledgebase page for more information on dealing with subarray data.

I have a list of 200 Program IDs, and I want to download all the, say, associated Level 2 IRS High-Resolution spectra. How do I do this?

For such a long list of Program IDs (PIDs), you will want to write a script. For each PID, the script should:
  1. Search the SHA for all Level 2 products associated with the PID. For example,
    % wget "http://sha.ipac.caltech.edu/applications/Spitzer/SHA/servlet/DataService?PID=105&VERB=3&DATASET=ivo%3A%2F%2Firsa.ipac%2Fspitzer.level2" -O output.file
    For more details on constructing the URL, see http://irsa.ipac.caltech.edu/onlinehelp/heritage/api.html
  2. Choose only the rows with a "wavelength" column consistent with a High-Resolution observation (e.g. "IRS SH 9.9-19.6um").
  3. If there are any High-Resolution observations, use the column "externalname" to find the name of the associated Level 2 pbcd product (e.g. r12256512/ch1/pbcd/SPITZER_S1_12256512_0007_8_E7359549_tune.tbl) and the column "accessUrl" to learn the URLs that you need to visit to download that product (e.g. http://sha.ipac.caltech.edu/applications/Spitzer/SHA/servlet/ProductDownload?DATASET=level2&ID=7359549).
  4. Download any High-Resolution observations found:
    %wget "http://sha.ipac.caltech.edu/applications/Spitzer/SHA/servlet/ProductDownload?DATASET=level2&ID=7359549" -O SPITZER_S1_12256512_0007_8_E7359549_tune.tbl
Your script will have to loop over the above 4 steps to get through all of your 200 PIDs.

What is the filenaming scheme for all these files I downloaded from the archive?

Here are links to pages for each instrument with the filename conventions and contents spelled out:

What time keywords should I use from the header for the exact start of my AOR?

In a nutshell, DATE_OBS is the most accurate, absolute measurement for comparing with astronomical emphemerides. SCLK_OBS (in real sec) is good for relative timing comparisons within an AOR or even a campaign.

More words on this:

Both DATE_OBS and UTCS_OBS are computed from spacecraft telemetry clock counters (SCLK_OBS coarse and fine bit ticks) and calibrated using an atomic clock in Boulder to correct for on-board drifts.

What are the Post-BCD (pbcd) data products I downloaded from the archive? Can I use them for science?

The pbcd products available from the archive were originally intended as quick-look images of your data. Post-BCD products have been produced with generic namelists, and the parameters have not been optimized for your data set. Post-BCD products and Level 2 products are two names for the same thing.

More specifically, for the three Spitzer instruments, the IRAC pbcd data products (mosaics) are often of science quality, but there are no guarantees and users should go back and inspect the BCD data products to verify results as needed. The IRS pbcd data products are often of science quality, but users should always inspect the BCD products as well before analysis and publication, because the PBCD data products are made using simplified assumptions, and only basic reductions are performed. To achieve the ultimate spectral quality from their data, users are encouraged to start with the BCDs and use one or more of the SSC or contributed software packages (e.g., IRSCLEAN, SPICE, CUBISM, etc.) to reduce and analyze their data. The MIPS 24 micron PBCD data products are often of science quality, but there are no guarantees and users should go back and inspect the BCD data products to verify results as needed. Users will definitely need to be cautious with the 70 and 160 micron PBCD data products and going back to the BCD data products at these wavelengths is often required to eliminate artifacts and improve the quality of the final data products.

How do I visualize AORs for the time it was actually observed?

In the Spitzer Heritage Archive (SHA), search for an observation, then select the observation in the list and then select the 'AOR Footprint' tab on the right hand side of the page to visualize the AOR on the date it was observed.

What is the basic format for Spitzer processed data?

The data in the Spitzer Heritage Archive (SHA) data is electronically accessible as FITS files (no extensions). The end-product files include complete calibration information and intermediate files for individual processing 'steps.'

What are the primary data products that the Spitzer Science Center generated? What is a BCD? What is a post-BCD product?

In general, the eight Spitzer Science Center (SSC) data processing pipelines convert spacecraft engineering data into scientifically useful data. Raw data (engineering and science) were received at the ground tracking stations via telemetry and forwarded to the Spitzer Flight Operations Center at the Jet Propulsion Laboratory in Pasadena. The Flight Operations System at JPL received telemetry packets from the Spitzer spacecraft and repackaged them into FITS files containing sensor data, expressed in DN (data number) units. Instrument engineering and housekeeping data were also transferred to the SSC from the Flight Operations System. The archived raw data represent rationally organized, time-ordered data and include associated Observatory pointing data and calibration observations.

Basic Calibrated Data (BCD) are two-dimensional images in FITS format, and correspond to individual 'data collection events (DCEs)' within an observation. An image is flux and/or wavelength (if appropriate) calibrated, and surface brightness measurements are expressed in physical units. In addition, flat-fielding and cosmetic restoration (e.g., cosmic-ray removal) algorithms are applied to the BCD. Spatial 'world' coordinates are derived from Observatory pointing information only. The BCDs represent the most reliable product achievable through automated processing.

Extended pipeline products (also referred to as Post-BCD products, previously referred to as Browse Quality Data) are also available. Higher-level imaging products that include co-addition, mosaicing, and source extraction are available.

The description of SSC data products depends on the science instrument and observing mode. High-level descriptions of BCDs for each of the observing modes are available in the Spitzer Observer's Manual, and details appear in the corresponding Instrument Handbook (IRAC, IRS, MIPS).

The SSC provided routine calibrations for each of the observing modes. These data entered the Spitzer public archive immediately upon processing and verification (unless a proprietary observation embargoed the release of the data). The investigator was able to propose to undertake special calibrations. For Legacy Science projects, these special calibration data also entered the public archive immediately upon processing and verification.

How can I tell which pipeline processed my data? How can I tell if my data have been reprocessed under the most recent pipelines?

Check the header of one of your BCD files and look for the keyword 'CREATOR'. (e.g., imhead blah_bcd.fits | grep CREATOR). This is the software version that was used to process your data.

The pipeline history file for your instrument can be found in Appendix A of the Instrument Handbooks. If the value of your file's CREATOR is a number greater than or equal to the largest number on the pipeline history page, you've got the most recent version. If it does NOT match, you should re-download the data from the archive to get the latest version of the data.

How do I find out all of the, e.g., planetary nebulae that have been observed?

There are a few different ways you can search for objects such as PNe in our archive.

You can construct your own list of PNe. Unless you trust SIMBAD (or NED) to resolve the coordinates from the names, you should tabulate the RA and Dec and then you can search by position for a whole list of objects at once.

Another thing to do is have a look at the list of titles and abstracts for all accepted programs. At the top of this page, there is a complete list of titles (listed alphabetically by PI). By scanning the titles, you can find that the two GTO programs pid 77 and 68 look promising for PNe. You can also check ADS for access to the abstracts for Spitzer Programs. If the data are already public, the archive will let you download the data.

How do I search for Spitzer calibration star data observed in the same campaign as my target?

You can look up primary calibrators from the Reach et al. (2005) IRAC calibrator paper and then search for observations of these stars, constraining your search to the relevant time interval or campaign.

You can also go to the observing logs page and look for the calibration observations that were taken before your observation.

What does SHA/AOR/IER/BCD/PBCD/Level 1/Level 2 mean?

Here are some common definitions. The archive of Spitzer data is the Spitzer Heritage Archive (SHA). An individual Spitzer observation sequence is an AOR, or Astronomical Observation Request. In certain cases (often calibration or sometimes science observations), you may also see an IER, or Instrument Engineering Request. Either one involves many individual frames. The individual data frames that emerge, calibrated, from the Spitzer pipeline are Level 1, or Basic Calibrated Data, or BCDs. The products that come from combining these individual data frames (such as mosaics of individual pointings) are Level 2, or post-BCD, or PBCD data. Enhanced products come from combining AORs or doing post processing (such as synthetic photometry from spectra or source extraction from images). These can be contributed by the community, or generated by the Spitzer Science Center.

Do you have any tutorial videos for the SHA?

Yes. The IRSA YouTube channel has several tutorials that are relevant to the SHA -- see the Spitzer playlist as well as the set of "micro-tutorials" relevant for more than one archive. The videos are in HD; you may need to manually use the YouTube gear menu to force it to realize this.

What's the file format that I need to use for the SHA search-by-file upload?

See the the SHA help file.

How come the SHA is only showing me mosaic files in the Level 2/PBCD tab? Where are the other files I know were taken?

Yes, there are a lot more files than what are shown, such as the errors! We decided for clarity that only certain files of primary interest to users would appear for individual examination in the SHA interface, but of course the download includes many more files -- you don't even have to tell it to include ancillary products. The IRAC, IRS, and MIPS Instrument Handbooks list (and define) all of the files that are included in the downloads.  To first order, only the highest level products are shown in the Level 2/PBCD tab, and for that matter the Level 1/BCD tab. This is designed for you to inspect the data enough to see if you want to download it. If you package up the data for downloading, the most frequently used standard files (errors, coverage, masks, etc.) will come with the download; to ensure you have all the files, select "incuding ancillary" when packaging.

How do I download all of the data shown as a result of my SHA search at once?

Click on the checkbox at the top of the column of checkboxes, at the top left of the search results pane. Then click on "Prepare Download."

In the SHA, how come I can't make a 3-color image from Spitzer data I am exploring interactively?

The visualization tools currently only allow you to create 3-color images that you load from disk or from the web. If you want to create 3-color images using Spitzer data, download the data to disk and then load them into the SHA from disk as individual color planes.  Creating 3-color images within the SHA itself is on the wish list for future versions.

I did a search, left the SHA window up, and now that I've come back to this session, it's in a funny state and does not appear to be responding as it should. Why?

Sometimes, if there is a network hiccup (e.g., you pick up and move your laptop to a different wireless hub), the SHA can be left in a funny state. (In some extreme cases, you will need to not just restart the browser but clear your browser cache.) To avoid losing a search, if you've accomplished a complex search, log in to the SHA and save or tag the search. That way, you can recover it easily with a single click the next time you log in.

The SHA broke up my request into many zip files! Is there any way I can get it in larger pieces, or somehow streamline the data download?

When downloading large quantities of data (big programs, whole campaigns, etc.), the SHA will break up the downloads into "manageable" pieces, where "manageable" is defined as "not larger than common computers and software can handle." We understand your frustration if you computer is customized to handle much larger files than the average computer. If you don't want to click to download each piece, use the download script provided by the Background Monitor, available when the packaging is complete either in the Monitor itself or from the email you can arrange to have sent to you. The script can be configured to unzip the files too.

The Download Retrieval Script dialog gives you some options regarding which script you want to use. Generally speaking, the wget script is best for Linux and Unix users. The curl script is best for Mac users, because curl is part of the standard OS distribution; Mac users can also go retrieve and install wget and then use the wget scripts. For any of the scripts, you can also choose to include an option that unzips the zip files automatically.  The files stay on disk here for at least 72 hours, so you have a window of time to download them.

Save the script to a plain text file, and invoke the script.  You can copy and paste the script lines individually into your terminal window, or by  typing "csh [yourtextfile]" at the prompt.  The files will be automatically and sequentially downloaded to your disk, and if you've selected that option, unzipped as well. 

For Windows users, download and save the text file of URLs . Then follow the following steps to install the wget script and then download your data:

How long do the SHA-created zip files stick around on the disk at IRSA after they are prepared?

72 hours, sometimes longer if the SHA is not under heavy use.

What do I do with the zip files I get when I download data from the SHA?

To uncompress the files you have downloaded, type "unzip <file>.zip". To uncompress multiple files at once, type "unzip '*.zip'" (the single quotes are important), or "unzip \*.zip" -- you just have to escape out the wildcard.

Double-clicking on the zipfile should also uncompress the files.