Terminal tip: Pipe Viewer

A couple of weeks ago, I held a Linux/Unix elementary course. One of the toughest concepts in that course are the concept of pipes and redirect.

I usually begin explaining pipe as "the output of one command becomes input to the next", and show by an example:

 $ zcat pureftpd.log.gz | cut -f1 -d' ' | sort | uniq | wc -l

 1259073

This command reads a ~550MB large compressed pureftpd logfile (from ftp.uio.no), and finds the number of unique visitors. Several commands are linked together by pipe, so the output of one command is input to the next.

However, I received and interesting question: "Which command use the longest time?"

There is no easy way to tell, we can just take an educated guess. However, we can use a handy little unix utility called "Pipe Viewer" to monitor and measure the data going through a pipe. Install from apt:

  $ sudo apt-get install pv

Next, we craft our command above using pv. Since pv behave like cat with respect to input/output, we measure the throughput between each command:

  $ zcat pureftpd.log.gz | pv -cN zcat | cut -f1 -d' ' | \

  > pv -cN cut | sort | pv -cN sort | uniq | pv -cN uniq | \

  > wc -l

As we see from the command, the command that had the slowest throughput was "uniq". Both cut and sort had an impressive 6-7MB/s throughput.

Memory usage by user

A short little script I stumbled across when cleaning my $HOME. I do not think I wrote it myself - at least I can't recall that I did. Quite handy and it's small and compact using the commands ps, awk, sort and head:

ps aux --no-headers | awk '
        {
                name[$1] += $1;
                pros[$1] += $4;
                mem[$1] += $5
        }
        END {
                for (var in name)  {
                        print mem[var]" "var" "pros[var]
                }
        }' | sort -nr | head

The result is a top ten list of the users that consume the most memory:

741012 root 9.3
238152 bjorn 3.8
119380 thomasez 1.2
105332 krav 1.9
100804 lars 0.6
95916 ingvar 1.6
89720 kjetilho 1.5
78044 jo 1
74232 espen 0.8
73460 michael 0.4

There I am - at number five...

Bash prompt with exit status

Let's improve the Bash prompt even further from my last post. I want to see the exit status of the last command in the bash prompt.

All (good written) programs have different exit status depending on how they where terminated. Exit status "0" is equivalent to "I terminated normally", all other exit status codes are the same as "non-normal exit" or "something went wrong". Unfortunate, there is not defined any standard exit status table that can say something about what went wrong given a numeric exit status. That is up to the programmer to decide.

  lars@titan:~$ test 1 -eq 1
  lars@titan:~$ echo $?
  0
  lars@titan:~$ test 1 -eq 2
  lars@titan:~$ echo $?
  1
  lars@titan:~$ notanycommand
  bash: notanycommand: command not found
  lars@titan:~$ echo $?
  127

I know that the exit status is stored in "$?". I use that to colorize my prompt red if the exit status is anything but "0" ("all ok"). In the bash man page, there is a special variable that is exactly what I'm looking for:

PROMPT_COMMAND
  If set, the value is executed as a command prior to issuing each
  primary prompt.

So we create a small function and add it to ~/.bashrc:

function exitstatus {

        EXITSTATUS="$?"
        BOLD="\[\033[1m\]"
        RED="\[\033[1;31m\]"
        OFF="\[\033[m\]"

        if [ "$EXITSTATUS" -eq "0" ]
        then
                PS1="${BOLD}\u@\h:\w\$${OFF} "
        else
                PS1="${BOLD}\u@\h:\w${OFF}${RED}\$${OFF} "
        fi

        PS2="${BOLD}>${OFF} "

}

PROMPT_COMMAND=exitstatus

Fire up a new shell, and every command that has an exit status different than "0" puts a red marker in your prompt:

Bash prompt (on RedHat)

I've been working with a lot of different RHEL-boxes lately, and I've (yet again) been frustrated with the default RedHat Bash-prompt. It is an easy fix, but its tiresome to change every time.

Okay, the "trouble" is this:

  [lars@titan ~]$

This is the default bash prompt. Now, thats fine enough, but lets jump to another directory.:

  [lars@titan lib]$

Where am I now? Which directory is this? Hm?

Since the RedHat prompt only shows the current directory, and NOT the full path, it can be any number of directories:

• /lib
• /usr/lib/
• /usr/src/linux/lib/
• /usr/local/lib/
• /opt/lib
• /var/lib
• ~/lib
• ...

You get the idea? And, if you have the same amount of short-time memory like me, you have to constantly type 'pwd' to check which directory you currently are in. The only nice feature with the default RedHat prompt, is to prevent long wrapping prompt (when the full path gets long).

Luckily, it is easy to change the prompt. Just add this to your ~/.bashrc

  PS1='\u@\h:\w\$ '
  PS2='> '

This gives you:

  lars@dream:~$

And lets now jump to another directory:

  lars@dream:/usr/local/lib$

See? No more 'pwd'! Finally a more sane and useful Bash prompt!

You can find more prompt variables in the Bash manual.

A slightly more fancy variant of the above, is to make the prompt bold:

   BOLD="\[\033[1m\]"
   OFF="\[\033[m\]"
   PS1="${BOLD}\u@\h:\w \$${OFF} "
   PS2="${BOLD}>${OFF} "

Just add it to your ~/.bashrc, and it will look like this:

  lars@titan:~$

You can pimp your prompt with colors and all kinds off information. Read the Bash Prompt HOWTO for more.

Less typing with enviroment variable CDPATH

The BASH shell has several environment variables that can be manipulated. The PATH variable is well known. Another useful variable is CDPATH. As PATH is a list of search paths for commands, so is CDPATH a list of directories used as search path for the "cd" command.

Example: At one ftp server we serve a lot of software, including several of the most popular Linux distributions. The local path to these distributions involves a lot of typing:

  larsks@spheniscus:~$ cd /usit/spheniscus/ftp/linux/
  larsks@spheniscus:/usit/spheniscus/ftp/linux$

From here, I can jump into "slackware/", "centos/", "debian/" and so on. But it's simpler when using CDPATH:

  larsks@spheniscus:~$ export CDPATH="/usit/spheniscus/ftp/linux"
  larsks@spheniscus:~$ cd slackware
  /usit/spheniscus/ftp/linux/slackware
  larsks@spheniscus:/usit/spheniscus/ftp/linux/slackware$

Nice huh?

An even lazier method (involving less typing) is to use alias:

  larsks@spheniscus:~$ alias s="cd /usit/spheniscus/ftp/linux/slackware"
  larsks@spheniscus:~$ s
  larsks@spheniscus:/usit/spheniscus/ftp/linux/slackware$

But then I have to create one alias for each directory. Oh the choices!