or 
located at the top or bottom of your screenThis exercise assumes you have been introduced to the operating system Linux. If you do not have any Linux expertise, please let your instructor know right away.
Linux operates in two ways: 1) via window management much like Windows or on a Mac and 2) via command lines. We will need to be proficient in both ways. We’ll start with window management, but within a terminal (Shell-Konsole) or an xgterm, we will be using command lines.
Open up a terminal and within it, open up a second window, this time an xgterm, by typing
| xgterm & | ; the & means run the process in the background |
| ; the semicolon seen on this line and the one above means we’ve put in a comment in these directions | |
| ;it also means: Please read this information when and where it occurs! |
at the prompt. It should open up a new window that will also accept commands. You may note that these two windows have some subtle but important differences. We get into this later. You can kill the window by typing exit. For experimenting, type in just xgterm without the & and see what happens. This is a good way to tie up two terminal windows at once, not recommended.
The next task is to customize the way the mouse interacts with the active windows on your desktop. We need to change the default setting in order to work efficiently within IRAF. The result of this change will sometimes be frustrating, but it is absolutely necessary.
First, open up two active shell windows (terminals or xterms) on the desktop if they are not already open. For the programs we'll be using, we want the position of the mouse to determine which window is active, not any mouse clicks.
NOTE: there may be other ways to take care of these next steps depending on what window manager you are using and how many updates have occurred since these instructions were written. Figure it out if neither of the following ways work.
1. Click on or located at the top or bottom of your screen
2. Click on or located at the top or bottom of your screen
3. Click on the symbol at the upper left corner of a terminal.
Test this by hitting some carriage returns with the mouse over one window, then position it over the second one and hit a few more carriage returns. The active window should be determined by the mouse location, without your having to click to activate the window.
We will be producing many large files that do not need to be saved from year to year, and thus it is prudent as far as conserving disk space in the Department to set up directories that can then be deleted after the quarter is over. NOTE: THIS MEANS AT THE END OF SPRING QUARTER, YOU SHOULD NOT USE THIS DIRECTORY FOR DATA YOU WISH TO KEEP! Those data WILL disappear leaving you in tears and horribly depressed; you may even change your major to accounting.
Our course working directory is:
| /net/projects/Astro_480/spring-??/ | ;substitute your year, 09, 10, etc. |
where you will find that you all have personal directories set up. It is against the "computer use laws" to use your home directories for research and for courses like this one that will be creating numerous large files. All work must be saved in
/net/projects/Astro_480/spring-??/yourusername/
The images we will be using for learning IRAF in this course are located in subdirectories under:
/net/projects/Astro_480/exercises/
[larson@astrolab01 exercises]$ ls -l */*
-rw-rw-r-- 1 larson astro-480 2003648 Jun 9 2006 exercise1/bias_ave_050206.fits.gz
-rw-r--r-- 1 larson astro-480 4176516 Jun 2 2006 exercise1/photos.tar.gz
-rw-r--r-- 1 larson astro-480 2989021 Jun 2 2006 exercise1/test-f1.fits.gz
-rw-r--r-- 1 larson astro-480 2989726 Jun 2 2006 exercise1/test-f2.fits.gz
-rw-r--r-- 1 larson astro-480 1136390 Jun 2 2006 exercise2/m92.tar.gz
-r--r--r-- 1 larson astro-480 1459824 Jun 2 2006 exercise2/spicamfocus.fits.gz
-rwxrwxr-x 1 larson astro-480 4643621 Jun 2 2006 exercise6/GS1826.tar.gz
Exercises 3, 4 and 5 all use the same images contained in the compressed "tarball" m92.tar.gz; we just get more and more sophisticated in our data reductions.
In the terminal, type
xgterm &
to bring up an xgterm window if you don't already have one open. You should then see a white-background terminal window. Type pwd just to make sure you are in your COURSE directory (this is to keep us from filling up our home directory).
Before we can use the IRAF image analysis utility, we need to do some IRAF initialization tasks, mkiraf does this for you. Place the mouse cursor over the xgterm window to activate it and type
mkiraf
in order to set up the requisite IRAF initialization file. When prompted for the terminal type, you should answer with xgterm.
Running the mkiraf command produces a new file called login.cl. This file is extremely important as it contains setup and initialization parameters that IRAF uses when it starts up. It also creates a new directory called the "uparm" directory. This directory contains parameter files that will be modified as you use the various tasks in IRAF.
If for some reason things go wrong, you can repeat mkiraf, but if you do, it will ask if you want to reinitialize your uparm directory. You may or may not want to do this. But, if IRAF is misbehaving later on in the quarter, you may just want to start over and reinitialize the files in the uparm directory.
Caution: If you do remake IRAF, the process will overwrite any changes you personally made to your login.cl file.
Take a look at the "login.cl" file with the less utility:
less login.cl
Lines in login.cl that start with "#" are commented out, i.e. not read by IRAF when it starts up. The defaults that mkiraf creates are inappropriate for our system. We’ll need to do some minor editing on the login.cl file. [Type a ‘q’ to quit ‘less’ and move on to the next steps.]
The login.cl file in /net/projects/Astro_480/exercises/ has the entries that are appropriate for our system. Take a look at that file, compare with the one you just created, and make the necessary changes to your file. In particular be sure that you
set home ="/net/projects/Astro_480/spring-??/yourusername/"
set imdir ="home$images/"
You must include the quotation marks and the trailing slash symbol; don’t forget to substitute your year for the ??, otherwise, you won’t get far. And, yourusername should be your login user name. The "imdir" is the name of the directory where you could store your images. Many students prefer keeping their images within a subdirectory of the exercise they are working on.
[CHECK: do you actually have a subdirectory named images within your working directory? If not, and you want one, you should make the directory using mkdir images]
Then set up the default image type, over-write mode, and standard image display by setting :
set imtype ="fits"
set clobber = no
set stdimage = imt2048
Then update and quit emacs by saving via the icon or by typing <Ctrl>x <Ctrl>c.
Review the entire login.cl file to see what is there. Look down to where you see the declaration foreign. What’s going on there? Page down lower in the file to where particular IRAF packages that you specify can be loaded. This is especially handy if you work with ones that are not frequently used, or if you use a particular package all of the time.
It is often useful to reduce the size of a data file in order to reduce the storage space needed and to reduce the time it takes to transfer the file across the net. There are two basic classes of compression algorithms: lossy and lossless. Lossless compression retains all the information that exists in the original file, and basically packs it into fewer bits. Lossy compression, on the other hand, achieves a smaller data file but at the expense of information. In general, astronomers shy away from lossy compression algorithms for data files. It is usually a struggle to attain the desired signal-to-noise ratio, so why give it up?
When considering whether to just transfer a file as is, or to implement a sequence of compress-and-transfer-and-uncompress you need to consider the trade-off between reduced transfer speed vs. increased computational load on the two ends of the transfer pipe. In our experience there is not a unique optimum decision as it depends on the power of the two machines and the overall throughput of the network connection.
A commonly used lossless compression scheme under Linux is gzip (and the inverse operation, gunzip). Typical text files compress by about 50% when gzip’d. The syntax for gzip is pretty simple. To compress a file called foo.dat you would type
gzip foo.dat
And this would produce a file called foo.dat.gz
Compressed files generally have suffixes of .gz or .Z. Beware of files with such names, as trying to view them on the screen will likely lead to a variety of control characters being sent to the terminal, often sending it into some kind of trance.
You can uncompress a file with the gunzip command, so to undo what was done above you would type
gunzip foo.dat.gz (or just gunzip foo.dat)
Notice that the suffix .gz is implicit: you don’t really need to type it.
Imagine you’ve just finished an extended observing run and have generated literally hundreds of images, each residing in an individual data file. This is a common occurrence in high data rate astronomical observing systems. Rather than deal with the bookkeeping and labor of transferring each file, one at a time, to a remote system (or onto a CD) it’s much easier to deal with a single bundle that contains all the files of interest. The Linux utility "tar" accomplishes this.
The tar command will take a list of files and produce a single file that contains the files in question. This tar archive, a "tarball" can then be transferred around as easily as a single file. To bundle (copy) a group of files into a tar archive, use:
tar –cvf <tarfilename> <file list>
or, sometimes the file flag must come right before the 'tarfilename':
tar -cv <file list> -f <tarfilename>
For example, if we wanted to make a tar file called images.tar that contained all the files that end in .fits (a common extension for images) we would type:
tar –cvf images.tar *.fits
or
tar -cv *.fits -f images.tar
Depending on the version of Linux you are using (or on a Mac) the tar command does not necessarily conform to the syntax that is typical for Linux commands (command source destination). In order to unravel a tar archive and recover the constituent files, you would type
tar –xvf <tarfile>
Check out the man page on the tar command (type: man tar) to see the mind-numbing variety of options that the "tar" command has. It’s common to first make a tar file, and then compress the tar file in order have a convenient bundle of files that are a minimum size. It is common to see files that end in ".tar.gz".
The options listed above, however, are almost always the only ones you need. These 4 do the following:
-c: create
-x: extract
-v: verbose (print to the screen what is being (un)tarred)
-f: create a file, don't tar onto a tape (a bit of arcane linuxness that you may ignore)
Now that we know how to produce compact tar files of data, the next step is to learn how to transfer them between machines. There are a number of options, with a range of security properties. If you have access to an account on each end of the file transfer then the preferred method is to use the "scp" (for secure copy) utility. This will allow you to move a file without ever sending a password in the clear, but requires authentication.
To move a file called "mydata.tar.gz" from a directory on a hypothetical machine called hubble at the Apache Point Observatory (with network node hubble.apo.nmsu.edu) to my work directory for Astronomy 480 on the astrolab cluster, I would type all on one line:
scp larson@hubble.apo.nmsu.edu:~/mydata.tar.gz larson@astrolab18.astro.washington.edu:/net/projects/Astro_480/larson/.
The syntax is
scp user@node:path/sourcefile user@node:path/destfile
and sourcefile and destfile refer to the source and destination file names, respectively, and the paths are pathnames to those files. The "." at the end of the scp command means to keep the same file name.
You should try this out. Transfer a file or two between the machine you’re logged into, and another machine in the astrolab cluster. You’ll be prompted for a password on each file transfer. The protocol does support wildcards ( *.* ), etc., but only if you enclose the full file name in quotes e.g. user@node:'path/sourcefile' , but often you will still want to know the names of the file(s) you are transferring.
Other archaic information transfer protocols such as telnet and ftp (except anonymous ftp run by a careful system administrator) are now forbidden, as they are totally insecure.
The directory /net/projects/Astro_480/exercises/exercise1/ contains the images you will need for IRAF Exercise I. FOR PRACTICE, secure copy them over to a subdirectory called exercise1 in your course directory (make the directory if you don’t have it). Make sure you are in your exercise1 subdirectory:
| scp /net/projects/Astro_480/exercises/exercise1/*.gz . | ;there is a space after the .gz and before the period |
Since these are the only files now in this subdirectory, uncompress them by typing:
gunzip *
The tarball, photos.tar, contains images that you will need for Exercise 1. Extract these files by typing:
tar –xvf photos.tar
Since Linux is a multiuser operating system, some file protection protocols have been established. There are three categories of users: the entire world, your "group", and the individual user. The authority to read, write and (in the case of a program) execute a file can be granted or denied to each of these classes of users. When you do an "ls –l" listing of a directory, the information on the far left hand side of each line pertains to file permission characteristics. All files are owned by individual users, each of whom is a member of a group.
The first column shows whether the object is a directory or not. Directories are listed with a "d" in the first column. The next 9 columns show whether the world, group, and file owner have read, write and execute privileges for the file in question. A dash means they lack the relevant permission, whereas the letters r, w, and x in the relevant columns show that the respective action is allowed.
In moving data around and between Linux systems, file permission characteristics can be a major inconvenience. For example, you may wish for some foreign colleague to "scp" over your hot-off-the-telescope images, but the instrument data files might be generated with conservative permission properties, that deny non-group members the ability to read/copy the file. You can overcome most of these permission problems by changing the permission properties of the data files to allow open access. This is done with the "chmod" command, for example
chmod 777 *.fits
would allow all users all privileges on all the files ending in .fits that reside in the current directory. The chmod command takes two arguments. The first is a number that is used as a byte mask that establishes the permissions for the file that is named as the second argument. See the man pages on chmod and ls for more details.