The Shell

Introduction

• Most modern tools have a graphical user interface (GUI)
  • Because they're easier to use
• But command-line user interfaces (CLUIs) still have their place
  • Easier (faster) to build new CLUI tools
    • Building a GUI takes time
    • Building a good GUI takes a lot of time
  • Higher action-to-keystroke ratio
    • Once you're over the (steeper) learning curve
  • Easier to see and understand what the computer is doing on your behalf
    • Which is part of what this course is about
  • Most important: it's easier to combine CLUI tools than GUI tools
    • Small tools, combined in many ways, can be very powerful
• [Ray & Ray 2003] is a good introduction for newcomers
  • And despite its age, [Kernighan & Pike 1984] is still useful
• How to tell if you can skip this lecture
  • Do you know what a shell is?
  • Do you know the difference between an absolute path and a relative path?
  • Do you know what a process is?
  • Do you know what a pipe is?
  • Do you know what $PATH is?
  • Do you know what rwxr-xr-x means?
• Comment on this slide

The Shell vs. the Operating System

• The most important command-line tool is the command shell (often just called “the shell”)
  • Manages a user's interactions with the operating system by:
    • Reading commands from the keyboard
    • Figuring out what programs the user wants to run
    • Running those programs
    • Displaying their output on the screen
  • Looks (and works) like an interactive terminal circa 1980

    • Figure 4.1: A Shell in Action

• The shell is just one program among many
  • Many different ones have been written
  • sh was the first for Unix
    • Most others extend its capabilities in various ways
    • Which means that it's the lowest common denominator you can always rely on
  • We'll use bash (the Bourne again shell) in this course
    • Available just about everywhere
    • Even on Windows (thanks to Cygwin)
• In contrast, the operating system is not just another program
  • Automatically loaded when the computer boots up
  • The only program that can talk directly to the computer's hardware
    • I.e., read characters from the keyboard, or send drawing commands to the screen
  • Manages files and directories on the disk
  • Keeps track of who you are, and what you're allowed to do
  • You can run many instances of the shell on a computer at once, but it can only run one operating system at a time

  • Figure 4.2: Operating System and Shell

• Comment on this slide

The File System

• The file system is the set of files and directories the computer can access
  • “Everything that stays put when you turn the computer off and restart it”
• Data is stored in files
  • By convention, files have two part names, like notes.txt or home.html
  • Most operating systems allow you to associate a filename extension with an application
    • E.g., .txt is associated with an editor, and .html with a web browser
  • But this is all just convention: you can call files (almost) anything you want
• Files are stored in directories (often called folders)
  • Directories can contain other directories, too
  • Results in the familiar directory tree

  • Figure 4.3: A Directory Tree

  • Everything in a particular directory must have a unique name
    • Otherwise, how would you identify it?
  • But items in different directories can have the same name
• On Unix, the file system has a unique root directory called /
  • Every other directory is a child of it, or a child of a child, etc.
• On Windows, every drive has its own root directory
  • So C:\home\gvwilson\notes.txt is different from J:\home\gvwilson\notes.txt
  • When you're using Cygwin, you can also write C:\home\gvwilson as c:/home/gvwilson
  • Or as /cygdrive/c/home/gvwilson
    • Some Unix programs give ":" a special meaning, so Cygwin needed a way to write paths without it…
• A path is a description of how to find something in a file system
  • An absolute path describes a location from the root directory down
    • Equivalent to a street address
    • Always starts with "/"
    • E.g., /home/gvwilson is my home directory, and /courses/swc/lec/shell.swc is this file
  • A relative path describes how to find something from some other location
    • Equivalent to saying, “Four blocks north, and seven east”
    • E.g., from /courses/swc, the relative path to this file is lec/shell.swc
• Every program (including the shell) has a current working directory
  • “Where am I?”
  • Relative paths are deciphered relative to this location
  • It can change while a program is running
• Finally, two special names:
  • "." means “the current directory”
  • ".." means “the directory immediately above this one
    • Also called the parent directory
    • In /courses/swc/data, .. is /courses/swc
    • In /courses/swc/data/elements, .. is /courses/swc/data

    • Figure 4.4: Parent Directories

• Comment on this slide

A Few Simple Commands

• Easiest way to learn basic Unix commands is to see them in action
• First, I type pwd (short for "print working directory”) to find out where I am
  • Unfortunately, most Unix commands have equally cryptic names

  • pwd
    

    /home/gvwilson/swc
    

• I then type ls (for “listing”) to see what's in the current directory

  • ls
    

    LICENSE.txt     admin   data    graphics        lec     pdf     scraps  util
    Makefile        cgi-bin etc     img             mp3     publ    src     web
    

• What actually happens when I type ls is:
  • The operating system reads characters from the keyboard
  • Passes them to the shell (because it's the currently active window on my desktop)
  • The shell breaks the line of text it receives into words
  • Looks for a program with the same name as the first word (i.e., the command to run)
    • Describe in a moment how the shell knows where to look
  • Runs that program
  • Reads the program's output and sends it back to the operating system for display

  • Figure 4.5: Running a Program

• I can tell ls to produce more informative output by giving it some flags
  • By convention, flags start with "-", as in "-c" or "-l"
  • Show directories with trailing slash

    • ls -F
      

      LICENSE.txt  admin/    data/  exer/ lec/  publ/    soln/  util/
      Makefile     cgi-bin/  etc/   img/  pdf/  scraps/  src/   web/
      

  • Show all files and directories, including those whose names begin with .

    • ls -a
      

      .               ..              .svn            admin           data
      exer            img             lec             publ            soln
      src             util            tmpl            README.txt      license.txt
      todo.txt
      

    • By default, ls doesn't show anything whose name begins with .
    • Note the .svn directory: this is where Subversion keeps administrative information
    • Do not edit anything in this directory yourself, or Subversion will become very confused
• Comment on this slide

Creating Files and Directories

• Rather than messing with the course files, let's create a temporary directory and play around in there

  • mkdir temp
    

  • Note: no output
  • The -v (“verbose”) flag tells mkdir to print a confirmation message
• Now go into that directory

  • cd temp
    

  • Changes the shell's notion of our current working directory

  • pwd
    

    /home/gvwilson/swc/temp
    

  • No files there yet:

  • ls -a
    

    .  ..
    

• Use the editor of your choice to create a file called earth.txt with the following contents:

  • Name: Earth
    Period: 365.26 days
    Inclination: 0.00
    Eccentricity: 0.02
    

  • Notepad (on Windows) runs in a window of its own
  • Pico (on Unix) takes over the shell window temporarily
  • We'll look at more advanced editing tools for programming in a few lectures
• Easiest way to create a similar file venus.txt is to copy the one we have

  • cp earth.txt venus.txt
    

  • ls -t
    

    venus.txt   earth.txt
    

  • Note: the -t option tells ls to list newest first
• Check the contents of the file using cat (short for “concatenate”)
  • Just prints the contents of a file to the screen

  • cat venus.txt
    

    Name: Earth
    Period: 365.26 days
    Inclination: 0.00
    Eccentricity: 0.02
    

• Edit the file so that it looks like this:

  • Name: Venus
    Period: 224.70 days
    Inclination: 3.39
    Eccentricity: 0.01
    

• Compare the sizes of the two files using wc (for “word count”)

  • wc earth.txt venus.txt
    

      4   9  69 earth.txt
      4   9  69 venus.txt
      8  18 138 total
    

  • Columns show lines, words, and characters
• Comment on this slide

Wildcards

• Some characters (called wildcards) mean special things to the shell
  • * matches zero or more characters
    • So ls *.f77 lists all the Fortran-77 files in a directory

  • wc *.txt
    

      4   9  69 earth.txt
      4   9  69 venus.txt
      8  18 138 total
    

  • ? matches any single character
    • So ls ??.txt lists all the text files with two-letter prefixes
    • And ls ??.* lists all the files with two-letter prefixes, and any extension
  • ~ on its own means “my home directory”
    • I.e., the one I'm in when I first log in
  • ~harry means “Harry's home directory”
• Note: the shell expands wildcards before running commands
  • There's no way for ls to know whether it was invoked as ls *.txt or rm earth.txt venus.txt
• Comment on this slide

Redirecting Input and Output

• A running program is called a process
• By default, every process has three connections to the outside world:
  • Standard input (stdin): connected to the keyboard
  • Standard output (stdout): connected to the screen
  • Standard error (stderr): also connected to the screen
    • Used for error messages

  • Figure 4.6: The Shell Running a Program

• You can tell the shell to connect standard input and standard output to files instead
  • command < inputFile reads from inputFile instead of from the keyboard
    • Don't need to use this very often, because most Unix commands let you specify the input file (or files) as command-line arguments
  • command > outputFile writes to outputFile instead of to the screen
    • Only “normal” output goes to the file, not error messages
  • command < inputFile > outputFile does both

  • Figure 4.7: Redirecting Standard Input and Output

• Example: save number of lines in all text files to words.len

  • $ wc -w *.txt > words.len
    

  • Nothing appears, because output is being sent to the file words.len

  • $ ls -t
    

    words.len       venus.txt       earth.txt
    

  • $ cat words.len
    

      4   9  69 earth.txt
      4   9  69 venus.txt
      3  12  62 words.len
     11  30 200 total
    

• Try typing cat > junk.txt
  • No input file specified, so cat reads from the keyboard
  • Output sent to a file
  • Voila: the world's dumbest text editor
• When you're done, use rm junk.txt to get rid of the file
  • Don't type rm * unless you're really, really sure that's what you want to do…
• Comment on this slide

Pipes

• Suppose you want to use the output of one program as the input of another
  • E.g., use wc -w *.* to count the words in some files, then sort -n to sort numerically
• Option 1: send output of first command to a temporary file, then read from that file

  • wc *.txt > temp
    sort -n < temp
    

• Option 2: use a pipe to connect the two programs
  • Written as "|"
  • Tells the operating system to send what the first program writes to its stdout to the second program's stdin

  • wc -w *.* | sort -n
    

      9 earth.txt
      9 venus.txt
     12 words.len
     30 total
    

  • Figure 4.8: Pipes

  • More convenient (and much less error prone) than temporary files
• Can chain any number of commands together
  • And combine with input and output redirection

  • wc *.txt | sort -n | head -5 > shortest.files
    

• Any program that reads from standard input and writes to standard output can use redirection and pipes
  • Programs that do this are often called filters
  • If you make your programs work like filters, you can easily combine them with others
  • A combinatorial explosion of goodness
• Comment on this slide

Environment Variables

• Like any other program, the shell has variables
• Since they define a user's environment, they are usually called environment variables
  • Usually all upper case
• Type set at the command prompt to get a listing:

  • $ set
    

    ANT_HOME=C:/apache-ant-1.6.2
    BASH=/usr/bin/bash
    COLUMNS=80
    COMPUTERNAME=ISHAD
    HISTFILESIZE=500
    

• Get a particular variable's value by putting a "$" in front of its name
  • E.g., the shell replaces "$HOME" with the current user's home directory
  • Often use the echo command to print this out

    • $ echo $HOME
      

      /home/gvwilson
      

    • Question: why must you type echo $HOME, and not just $HOME?
• To set or reset a variable's value temporarily, use this:

  • $ export VARNAME=value
    

  • Only affects the current shell (and programs run from it)
• To set a variable's value automatically when you log in, set it in ~/.bashrc
  • Remember, "~" is a shortcut meaning “your home directory”
    • For me, right now, ~/.bashrc is /home/gvwilson/.bashrc
• Important environment variables

  • Name	Typical Value	Notes
    HOME	/home/gvwilson	The current user's home directory
    HOMEDRIVE	C:	The current user's home drive (Windows only)
    HOSTNAME	"ishad"	This computer's name
    HOSTTYPE	"i686"	What kind of computer this is
    OS	"Windows_NT"	What operating system is running
    PATH	"/home/gvwilson/bin:/usr/local/bin:/usr/bin:/bin:/Python24/"	Where to look for programs
    PWD	/home/gvwilson/swc/lec	Present working directory (sometimes CWD, for current working directory)
    SHELL	/bin/bash	What shell is being run
    TEMP	/tmp	Where to store temporary files
    USER	"gvwilson"	The current user's ID

    Table 4.1: Important Environment Variables

• Comment on this slide

How the Shell Finds Programs

• The most important of these variables is PATH
  • The search path that tells the shell where to look for programs
  • When you type a command like tabulate, the shell:
    • Splits $PATH on colons to get a list of directories
    • Looks for the program in each directory, in left-to-right order
    • Runs the first one that it finds
• Example
  • PATH is /home/gvwilson/bin:/usr/local/bin:/usr/bin:/bin:/Python24
  • Both /usr/local/bin/tabulate and /home/gvwilson/bin/tabulate exist
  • /home/gvwilson/bin/tabulate will be run when you type tabulate at the command prompt
  • Can run the other one by specifying the path, instead of just the command name
• Warning: it is common to include . in your path
  • This allows you to run a program in the current directory just by typing whatever, instead of ./whatever
  • But it also means you can never be quite sure what program a command will invoke
    • Though you can use the command which program_name, which will tell you
• Common entries in PATH include:
  • /bin, /usr/bin: core tools like ls
    • Note: the word “bin” comes from “binary”, which is geekspeak for “a compiled program”
  • /usr/local/bin: optional (but common) tools, like the gcc
  • $HOME/bin: tools you have built for yourself
    • Remember, $HOME means “the user's home directory”
    • So this is equivalent to ~/bin
• Cygwin does things a little differently
  • Uses the notation /cygdrive/c/somewhere instead of Windows' c:/somewhere
    • The colon in c:/somewhere would clash with the colons in the PATH variable
  • By default, Cygwin treats c:/cygwin as the root of its file system
    • So /home/aturing is a synonym for c:/cygwin/home/aturing
    • Yes, it can be confusing, but remember: we're trying to run one operating system's tools on top of another
• Comment on this slide

Basic Tools

man	Documentation for commands.
cat	Concatenate and display text files.
cd	Change working directory.
chmod	Change file and directory permissions.
clear	Clear the screen.
cp	Copy files and directories.
date	Display the current date and time.
diff	Show differences between two text files.
echo	Print arguments.
env	Show environment variables.
head	Display the first few lines of a file.
ls	List files and directories.
mkdir	Make directories.
more	Page through a text file.
mv	Move (rename) files and directories.
od	Display the bytes in a file.
passwd	Change your password.
pwd	Print current working directory.
rm	Remove files.
rmdir	Remove directories.
sort	Sort lines.
tail	Display the last few lines of a file.
uniq	Remove duplicate lines.
wc	Count lines, words, and characters in a file.
which	locate a command

Table 4.2: Basic Command-Line Tools

• Comment on this slide

Ownership and Permission: Unix

• On Unix, every user belongs to one or more groups
  • The groups command will show you which ones you are in
• Every file is owned by a particular user and a particular group
  • Owner can assign different read, write, and execute permissions to user, group, and others
  • Read: can look at contents, but not modify them
  • Write: can modify contents
  • Execute: can run the file (e.g., it's a program)
• ls -l will show all of this information
  • (Along with the file's size and a few other things)
  • Permissions displayed as three rwx triples
  • “Missing” permissions shown by "-"
  • Example: rw-rw-r-- means “user and group can read and write; everyone else can read; no one can execute”
• Change permissions using chmod
  • Example: chmod u+x something.exe gives the user execute permission to something.exe
  • Example: chmod o-r notes.txt takes away the world's read permission for notes.txt
• Permissions mean something a little different for directories
  • Execute permission means you can “go into” a directory, but does not mean you can read its contents
  • So if a directory called tools has permission rwx--x--x (i.e., owner can do anything, but everyone else only has execute permission), then:
    • If someone other than the owner does ls tools, permission is denied
    • But if there's a useful program called tools/findanswers, other users can still run it
• Comment on this slide

Ownership and Permission: Windows

• Of course, it all works differently on Windows
  • Not better or worse, just differently
• Windows XP uses access control lists (ACLs)
  • Instead of describing users as “file owner, group member, or something else”, ACLs let you specify exactly what any particular user, or set of users, can do to a file, directory, device, etc.
  • Older versions of Windows (such as Windows 95 and Windows 2000) are fundamentally insecure, and shouldn't be used
• Cygwin does its best to make the Windows model look like Unix's
• If you trip over the differences, please consult a system administrator
• Comment on this slide

More Advanced Tools

du	Print the disk space used by files and directories.
find	Find files that match a pattern.
grep	Print lines matching a pattern.
gunzip	Uncompress a file.
gzip	Compress a file.
lpr	Send a file to a printer.
lprm	Remove a print job from a printer's queue.
lpq	Check the status of a printer's queue.
ps	Display running processes.
tar	Archive files.
which	Find the path to a program.
who	See who is logged in.
xargs	Execute a command for each line of input.

Table 4.3: Advanced Command-Line Tools

• Comment on this slide

Exercises

Exercise 4.1:

Suppose you are in your home directory, and ls shows you this:

Makefile        biography.txt   data
enrolment.txt   programs        thesis

What argument(s) do you have to give to ls to get it to put a trailing slash after the names of subdirectories, like this:

Makefile        biography.txt   data/
enrolment.txt   programs/       thesis/

If you run ls data, it shows:

earth.txt       jupiter.txt     mars.txt
mercury.txt     saturn.txt      venus.txt

What command should you run to get the following output:

data/earth.txt          data/jupiter.txt        data/mars.txt
data/mercury.txt        data/saturn.txt         data/venus.txt

What if you want this (note that an extra entry is being displayed):

total 7
drwxr-xr-x    7 someone        0 May  6 08:27 .svn
-rw-r--r--    1 someone     2396 May  6 08:38 earth.txt
-rw-r--r--    1 someone     1263 May  6 08:38 jupiter.txt
-rw-r--r--    1 someone     1015 May  6 08:43 mars.txt
-rw-r--r--    1 someone      946 May  6 08:41 mercury.txt
-rw-r--r--    1 someone     1714 May  6 08:40 saturn.txt
-rw-r--r--    1 someone      881 May  6 08:40 venus.txt

Note: the command will display your user ID, rather than someone. On some machines, the command will also display a group ID. Ignore these differences for the purpose of this question.

Exercise 4.2:

According to the listing of the data directory above, who can read the file mercury.txt? Who can write it (i.e., change its contents or delete it)? When was mercury.txt last changed? What command would you run to allow everyone to edit or delete the file?

Exercise 4.3:

Suppose you want to remove all files whose names (not including their extensions) are of length 3, start with the letter a, and have .txt as extension. What command would you use? For example, if the directory contains three files a.txt, abc.txt, and abcd.txt, the command should remove abc.txt , but not the other two files.

Exercise 4.4:

What does the command cd ~ do? What about cd ~gvwilson?

Exercise 4.5:

What's the difference between the commands cd HOME and cd $HOME?

Exercise 4.6:

Suppose you want to list the names of all the text files in the data directory that contain the word "carpentry". What command or commands could you use?

Exercise 4.7:

Suppose you have written a program called analyze. What command or commands could you use to display the first ten lines of its output? What would you use to display lines 50-100? To send lines 50-100 to a file called tmp.txt?

Exercise 4.8:

The command ls data > tmp.txt writes a listing of the data directory's contents into tmp.txt. Anything that was in the file before the command was run is overwritten. What command could you use to append the listing to tmp.txt instead?

Exercise 4.9:

What command(s) would you use to find out how many subdirectories there are in the lectures directory?

Exercise 4.10:

What does rm *.ch? What about rm *.[ch]?

Exercise 4.11:

What command(s) could you use to find out how many instances of a program are running on your computer at once? For example, if you are on Windows, what would you do to find out how many instances of svchost.exe are running? On Unix, what would you do to find out how many instances of bash are running?

Exercise 4.12:

What do the commands pushd, popd, and dirs do? Where do their names come from?

Exercise 4.13:

How would you send the file earth.txt to the default printer? How would you check it made it (other than wandering over to the printer and standing there)?

Exercise 4.14:

A colleague asks for your data files. How would you archive them to send as one file? How could you compress them?

Exercise 4.15:

The instructor wants you to use a hitherto unknown command for manipulating files. How would you get help on this command?

Exercise 4.16:

You're worried your data files can be read by your nemesis, Dr. Evil. How would you check whether or not he can, and if necessary change permissions so only you can read or write the files?

Exercise 4.17:

You have changed a text file on your home PC, and mailed it to the university terminal. What steps can you take to see what changes you may have made, compared with a master copy in your home directory?

Exercise 4.18:

How would you change your password?

Exercise 4.19:

grep is one of the more useful tools in the toolbox. It finds lines in files that match a pattern and prints them out. For example, assume I have files earth.txt and venus.txt containing lines like this:

Name: Earth
Period: 365.26 days
Inclination: 0.00
Eccentricity: 0.02

If I type grep Period *.txt in that directory, I get:

earth.txt:Period: 365.26 days
venus.txt:Period: 224.70 days

Search strings can use regular expressions, which will be discussed in a later lecture. grep takes many options as well; for example, grep -c /bin/bash /etc/passwd reports how many lines in /etc/passwd (the Unix password file) that contain the string /bin/bash, which in turn tells me how many users are using bash as their shell.

Suppose all you wanted was a list of the files that contained lines matching a pattern, rather than the matches themselves—what flag or flags would you give to grep? What if you wanted the line numbers of matching lines?

Exercise 4.20:

diff finds and displays the differences between two files. It works best if both files are plain text (i.e., not images or Excel spreadsheets). By default, it shows the differences in groups, like this:

3c3,4
< Inclination: 0.00
---
> Inclination: 0.00 degrees
> Satellites: 1

(The rather cryptic header "3c3,4" means that line 3 of the first file must be changed to get lines 3-4 of the second.)

What flag(s) should you give diff to tell it to ignore changes that just insert or delete blank lines? What if you want to ignore changes in case (i.e., treat lowercase and uppercase letters as the same)?

Exercise 4.21:

Suppose you wanted ls to sort its output by filename extension, i.e., to list all .cmd files before all .exe files, and all .exe's before all .txt files. What command or commands would you use?