Downloading Data with wget Scripts

Several archive services provide the user with scripts they can run on their local workstation to download large amounts of data. These scripts require that a utility called wget is installed on your workstation and used with a command-line interface (also known as a shell prompt). wget comes with some platforms, however, if it is not it can easily be installed.

These instructions explain how to install wget and use it with IRSA's scripts on Windows, Unix, and Mac OSX machines.

Examples for creating and using wget scripts can be found at the Gator Program Interface Guide.

About IRSA Download Scripts

IRSA's download scripts are sets of wget commands that can download one or more files, or an entire tree of files recursively, from our server to your workstation. The script structure allows the same file to be run as a Unix/Mac OSX sh script or a Windows batch file. As a result, there are occasional warning messages (particularly on Windows) that can be ignored. On Unix and Mac OSX, the only oddity is the file extension is .bat (a Windows requirement), though this has no effect on script execution.

Downloading, Installing, and Using wget

Unix and Mac OSX

1. Check your system to see if wget is already installed by typing which wget at a shell prompt within a terminal window or xterm. If it is not installed, you need to download and install wget in your personal path (ask your system administrator if you are not sure where this is). There are two ways to get wget:

   Start from scratch and get the source from: http://www.gnu.org/software/wget/wget.html

   Or, get a prebuilt binary.

   • For Mac OSX, consider installing Homebrew and running the following in your OS X terminal:
     
     $ brew update # optional
$ brew install wget

   • Since Unix download resources frequently, it's best to do a Google search on "download wget [your operating system]". If you have difficulty finding a provider, contact us and include your operating system and version, and we will try to to find a location for you.

2. Download and save the IRSA data file script associated with the data you want to download (for example, irsspect.bat). These scripts work entirely with relative file paths, so place it in the same directory where you want the downloaded data to reside. You may want to create a new directory specifically for that downloaded data.

3. Change the script file's permissions to make it executable (for example, chmod 755 irsspect.bat).

4. Once the data file script is in the desired directory and is executable, type the script file name at the shell prompt. The files should start downloading into the designated directory.

Windows

1. Although it is possible to build wget from scratch for Windows, it's easier to download a wget.exe file. Download the file from: https://eternallybored.org/misc/wget/.

2. When prompted to save the file, save it to your Windows executable path. This is usually a system directory like C:\WINNT or C:\Windows.

   Note: You will need Administrative access to your machine to save to these directories. If your Web browser automatically saves files to a pre-set directory that is not in your executable path, you must manually move the file to your C:\WINNT or C:\Windows directory.

3. Create a new directory on your Desktop or hard drive to store the files you will be downloading.

4. Download the data file script associated with the IRSA data you want to download, and save it to the directory you just created.

5. Check that the data file script has a .bat file extension. If the extension is .sh, change it to .bat.

6. Double-click on the .bat file, or run it manually at the command-line prompt.

7. The files should download into the same directory as the data file script.

   Note: Older Windows versions may display warning messages during the download process. These messages do not affect the data and can be ignored.

---

If you have trouble locating, installing, or using wget to retrieve archive data after using these instructions, contact our Help Desk.