Lars Strand's blog: November 2007

SELinux introduces a new access control mechanism in the Linux kernel called "mandatory access control". It has been in the mainline Linux kernel since 2003, and included in RedHat Enterprise Linux 4 (2005). RedHat have been testing SELinux for quite some time through the Fedora releases, where it has been available since Fedora 2 (2004). RedHat is aggressively pushing the development of SELinux and relevant tools forward. From RHEL version 4 to 5 the targeted policy includes more services, added support for a modular policy, (graphical) administrations tools and support for MLS. But what are the performance penalties when running with and without SELinux enabled?

Index

SELinux
1. Access control
2. Security attributes
3. SELinux MAC
4. Security policy
5. AppArmor
Test setup
The tests
1. Apache
2. Postfix
3. MySQL
Conclusion
References

1. SELinux

NSA originally developed SELinux and surprised everyone when they open sourced it.

".. let me assure you that this action by the NSA was the crypto-equivalent of the Pope coming down off the balcony in Rome, working the crowd with a few loaves of bread and some fish, and then inviting everyone to come over to his place to watch the soccer game and have a few beers. There are some things that one just never expects to see, and the NSA handing out source code along with details of the security mechanism behind it was right up there on that list."

-- Larry Loeb, "Uncovering the secrets of SE Linux"

SELinux was quickly embraced by the open source community and RedHat in particular have led the development the last years. Existing first as a set of third party kernel patches, it was later rewritten to use the "Linux Security Modules" (LSM) and is now part of the mainline kernel.

LSM provides no security by itself, but gives a general framework to support access control modules like SELinux. LSM does that by inserting hooks in the kernel code right before the access would have been granted by traditional access control. These "hooks" are just calls to functions that the LSM modules (SELinux) must provide.

1a. Access control
The access control mechanism found in traditional operating systems (Linux, Windows, UNIX, OSX), the users are in control and determine access control. Ordinary users may give or revoke access privileges to their own object (files). This access control scheme is called "discretionary access control" (DAC), since the subjects have discretion over their own objects. Bishop defines DAC as:

"If an individual user can set an access control mechanism to allow or deny access to an object, that mechanism is a discretionary access control (DAC), also called identity-based access control (IBAC)." [Bishop]

SELinux introduces a new access control mechanism called "mandatory access control" (MAC). Here the access control is taken out of the hand of the users and enforced by the system it self. "In general, when systems are built to enforce a security policy independently of user actions, they are described as having mandatory access control" [Anderson]. SELinux supplement Linux with mandatory access control (MAC), since access control is enforced based on a security policy and not user identity alone.

MAC can be defined as: "When a system mechanism controls access to and an individual user cannot alter that access, that control is a mandatory access control (MAC), occasionally called a rule-based access control." [Bishop]

It must be stressed that MAC is not by definition more secure than DAC. It is just two different approaches to access control. The strength of MAC depends on a well-defined security policy. But when MAC is used, the security policy is usually written in accordance with the "principle of least privilege". This principle is best explained using an example: A user reads his email using a program called "mutt". Since "mutt" is executed as a normal user, the "mutt" process has the same privileges to read and write to all the files as the user himself. That means that "mutt" can read the users ssh-keys, change the users password, delete all the users files and so on. Mutt does not need all those privileges in order to function properly, so they can be denied. Again, Bishop gives a clear and concise definition:

"The principle of least privilege states that a subject should be given only those privileges that it need in order to complete its task." [Bishop]

SELinux does not replace the traditional DAC in Linux. SELinux introduces MAC in addition to DAC. All access decisions are first consulted DAC then MAC (SELinux). If an action is denied in DAC, SELinux (MAC) is not consulted and the action is denied. If DAC allows the action, the decision is sent to SELinux for a MAC check.

1b. Security attributes
SELinux uses a combination of an identity model, role-based access control (RBAC) and type enforcement (TE). Where TE is the most important feature. SELinux RBAC authorize each (SELinux) user for a set of roles. Each role is authorized to a set of types.

This is accomplished using four security attributes:

User identity: SELinux has its own user database which are mapped to normal Linux users. The identities are used on both subject and objects. Only a few SELinux users are defined: (can be listed by 'semanage user -l'):
- user_u - normal users.
- system_u - processes started (at boot).
- root - administrator.
Role: Users may enter into different roles. Different roles may enter different domains. For objects (files), this is always object_r.
Type / domain: The "main" attribute in SELinux. Also called the "primary attribute". It is usually only a few users/roles defined, but hundreds of types. There is no difference between "type" and "domain", but "domains" are used when talked about processes and "type" when talking about files. Each process is confined in it own sandbox with restricted access, also called "Type Enforcement".
Category / level: May set category and/or level. Introduced in RHEL 5 and enables Multi Level Security (MLS) or Multi Category Security (MCS).

These four security attributes build up what is called a "security context":

user:role:type:category/level

Security attribute	Name convention	Example name
User	_u	user_u
Role	_r	object_r
Type	_t	unconfined_t
Category / level	(none)	s0:c0

To view the security context of files and/or processes, the option "Z" is used:

$ ls -Z /etc/passwd

-rw-r--r--  root root system_u:object_r:etc_t          /etc/passwd

$ ps Z -C sshd

LABEL                             PID TTY      STAT   TIME COMMAND

system_u:system_r:unconfined_t:SystemLow-SystemHigh 2021 ? Ss   0:00 /usr/sbin/sshd

system_u:system_r:unconfined_t:SystemLow-SystemHigh 16795 ? Ss   0:00 sshd: lars [priv]

system_u:system_r:unconfined_t:SystemLow-SystemHigh 16797 ? S   0:01 sshd: lars@pts/1 

system_u:system_r:unconfined_t:SystemLow-SystemHigh 24700 ? Ss   0:00 sshd: lars [priv]

system_u:system_r:unconfined_t:SystemLow-SystemHigh 24702 ? S   0:00 sshd: lars@pts/0 

1c. SELinux MAC

Lets look closer on the decision making process in the kernel. When a subject (process) wants to access an object (for example a file), it must first be granted by DAC. Then the decision is sent to SELinux via LSM. In SELinux the Policy Enforcment Server does a lookup in the Access Vector Cache (AVC) where earlier subject and objects permission are cached. If the decision is not found in the AVC, the request continues to the Security Server which looks up the security context of the file and consult the policy. Permission is then either denied or granted. The result is cached in the AVC. Se figure below for a graphical illustration:

1d. Security policy
SELinux consists of:

Kernel module. Included in the mainline kernel since 2.6.
Library 'libselinux' used by userspace programs (ls, ps, id, ...).
Administrative tools ("SELinux Management Tool", sestatus, ...).
Security policy.

The policy is written in m4 (same as the config file in Sendmail), compiled and loaded into the kernel at boot time. Writing new or changing existing policy directly might not be intuitive for new users so RedHat has invested a lot of effort into creating user-friendly (graphical) administrative tools. The policy has also been made modular, so it is easier to turn of TE for certain programs.
But the policy does not cover the whole system. In RHEL4 only 15 network based services (apache, bind, ntp, ...) where covered by the policy. The rest of the system ran in an "unconfined_t" domain which is a special domain with no restrictions (same as if SELinux was not running). In RHEL5, the number of programs covered by the policy are over 200. "Covered" here means that the program are confined to its own "domain" with restricted privileges.

In RHEL/Fedora "strict" and "MLS" policy may also be installed, but they are unsupported. In the "strict" policy, every subject and object exists in a specific domain. The "MLS" policy enforces military style security levels and uses the Bell-La Padula model (BLP). The (binary) size of the policies also varies according to how much they cover. The "strict" policy is more than double in size of the "targeted":

Targeted: 1.1 MB
Strict: 2.5 MB
MLS: 2.1 MB

SELinux may run either in "enforcing" or "permissive" mode. In "permissive" mode access control is checked against the security policy but not enforced. Instead warnings are printed to a log file when policy is violated (nice for debugging). When SELinux run in "enforcing" mode, the policy is enforced.
The /selinux pseudo-filesystem gives access to SELinux variables and AVC statistics. I wrote a Munin plugin to monitor the AVC (download). The graphs below shows the load on the AVC. No active services (besides Munin) to the left and running the MySQL benchmark to the right.

When running real-life load or doing benchmarking, several hundred thousand AVC lookups per seconds are preformed.

1e. AppArmor
AppArmor ("Application Armor") is a competing technology to SELinux. It enables MAC in the kernel using LSM, the same as SELinux, but takes a different approach. For instance it uses full path names instead of inode names for file objects.

Immunix created AppArmor as an alternative to SELinux, which was considered to hard to administer. Immunix was later aquired by Novell, and included in Novell Suse. Creating and maintaining AppArmor polices is user friendly, and that has led other Linux distributions, like Ubuntu and Mandriva, to include it in the default install. The overhead using AppArmor is said to be around 2% [Cowan].

In a surprise move, Novell laid of most of their AppArmor devlopers in September 2007 [news.com] [linux-magazine.com]. Making the future of AppArmor more uncertain and depended upon the open source community to continue the development. One indication of popularity can be seen in Google trends.

2. Test setup
Two nodes, connected through a gigabit switch as shown in the illustration below:

Hardware specification given in the tabel below. I turned off CPU stepping on the laptop.

Hardware	Laptop	Workstation	Switch
Model:	Fujitsu Siemens Lifebook S7020D	OptiPlex 745	D-Link DGS-1008D
CPU:	Intel(R) Pentium(R) M 2.00GHz	Intel(R) Core(TM)2 CPU 2.13GHz
RAM:	2 GB	4 GB
Ethernet:	Broadcom NetXtreme BCM5751M PCI Express (Gbps)	Broadcom NetXtreme BCM5754 PCI Express (Gbps)	8 port Gbps
Disk:	SATA: 80GB, Seagate ST98823AS	SATA: 250GB, WDC WD2500JS-75N

Output from "lshw" can be found here: Laptop and Workstation.

The operating system tested was RedHat Enterprise Linux 5 (RHEL5) Server. The laptop ran i386 (32 bits) version, but the workstation ran both i386 and x64 (64 bits) version. The client was running Ubuntu 7.04. So when RHEL5 was tested on the workstation, the laptop ran Ubuntu 7.4 and vice versa.

One little trivia: When testing, I was surprised when I found out that my laptop, with a gigabit interface, could sustain 1Gbps network traffic. Se screenshot from "iptraf" to the right.

3. The Tests
When SELinux is deployed, it is to provide an extra layer of security to existing services. Knowing the performance penalty when running SELinux on common (network based) services was the goal. The first task was to find good benchmark tools. I tested Apache prefork and worker (threaded), Postfix and MySQL.

To flush out different caches (file cache, AVC etc.), the machine was rebootet between each test. All relevant files (log-files, mailboxes etc.) were also blanked before each new test.

3a. Apache
Apache can run using a prefork model, where each request is assigned to a free Apache process. New processes are forked if the number of requests gets high. When running as "worker", Apache uses threads instead of forking. This is not default behavior for Apache on RHEL since PHP is not compatible with this mode. Worker mode is slightly faster than prefork as seen from the graphs below.

The test was used running Apaches own benchmark tool "ab". It was run against the same html-file. For each host, eleven tests with different concurrent connections (1 up to 255) each with 100000 requests was performed. The ab-test script can be downloaded here and the index.html file here.

Using a default (RHEL) httpd.conf, modifying only the number allowed clients (to 256).

This test uses a lot of network traffic, process handling (fork or thread) and disk read (html file) and write (log file).

Below is the result from the prefork results. For each host the average for all 11 tests are shown:

The same results with Apache worker (threaded):

3b. Postfix
Postfix is a popular mail server (MTA) and a rival to Sendmail. It is composed of several daemons, each responsible for performing a specific task (putting the mail in the queue, final delivery to the user's mailbox etc.). Read more about the different components in Postfix when receiving mail here.

The test was performed running Postfix' "smtp-source" against a Postfix mail server. The program is a SMTP test generator which connects to a mail server and send messages to it (sequentially or in parallel). Testing was done from and measured on the client sending 10000 messages four times with different concurrent connections (1 to 1000) (test script). Time taken was taken from the first to the last mail sent.

Using a fairly default (RHEL) main.cf, changing only the listing interface from "localhost" to "all".

This test uses a lot of network traffic, process handling and primarily disk write (mails and logs).

Four tests for each host. Average results shown:

3c. MySQL
MySQL is a popular database. It is multi-threaded and does not fork new processes. The benchmark tool was MySQL own "MySQL Benchmark suite", running all tests ("run-all-tests"). The test was started on the same host as the MySQL-server, so all communication was over a unix-socket. Around 3 million tests done in each run, but these are mostly CPU-bound, which can be seen from the results (low SELinux overhead):

4. Conclusion
There is no "official" number for how much performance penalty SELinux introduces. The Fedora Core FAQ says its hard to measure but states: "When performance was last measured, the impact was around 7% for completely untuned code." Performance depends on how the program behaves, the security policy written for it and the particular usage of that program. We've seen that more CPU bound programs, like the MySQL benchmark, has less SELinux overhead since the process itself spend more time on the CPU and less interaction with other part of the system (disk access, network traffic etc.).

If we combine all the result above, we come up with an average penalty of around 6%. Which is pretty close to 7% stated in the Fedora FAQ.