29 Dec 2011

Terminal tip: Pipe Viewer

A couple of weeks ago, I held a Linux/Unix elementary course. One of the toughest concepts in that course are the concept of pipes and redirect.

I usually begin explaining pipe as "the output of one command becomes input to the next", and show by an example:

 $ zcat pureftpd.log.gz | cut -f1 -d' ' | sort | uniq | wc -l
 1259073

This command reads a ~550MB large compressed pureftpd logfile (from ftp.uio.no), and finds the number of unique visitors. Several commands are linked together by pipe, so the output of one command is input to the next.

However, I received and interesting question: "Which command use the longest time?"

There is no easy way to tell, we can just take an educated guess. However, we can use a handy little unix utility called "Pipe Viewer" to monitor and measure the data going through a pipe. Install from apt:

  $ sudo apt-get install pv

Next, we craft our command above using pv. Since pv behave like cat with respect to input/output, we measure the throughput between each command:

  $ zcat pureftpd.log.gz | pv -cN zcat | cut -f1 -d' ' | \
  > pv -cN cut | sort | pv -cN sort | uniq | pv -cN uniq | \
  > wc -l


As we see from the command, the command that had the slowest throughput was "uniq". Both cut and sort had an impressive 6-7MB/s throughput.

25 Nov 2011

Security architectures in telephony systems

As tradition dictates, before I could defend my Ph.D. dissertation 22th November, I had to give a 45 minutes trial lecture. I was given only the title, and had 14 days to prepare. My title was:

  "The development of security architecture in fixed and mobile telephone systems"

One of the toughest tasks was to interpret the title and limit the scope of the lecture. I discussed with my supervisors and co-researchers and received several tips and relevant references. Then started two intense weeks with study and preparation.

I was satisfied with the disposition and result, and felt comfortable presenting the lecture.

For those interested, the slides can be downloaded here

3 Jun 2011

PhD writing tools

The end is nigh! ...or at least in sight! After four years, my PhD is nearing completion. The plan is to to deliver and defend my thesis sometimes this autumn. In these years, I've used several (free) tools that others might find useful doing the same, or similar, kind of work:

* Zotero - a Firefox-plugin that help me organize papers, citations, web-pages and other resources. All papers/notes are saved in "the cloud" for easy access across multiple terminals (...or Firefoxes). A great tool that enables you to tag, add notes, cross-link papers, add comments to/highlight text in PDFs, etc.

The Zotero Firefox plugin.

* Freemind - a mind-mapping tool. It was the best (open source) mind-mapping tool four years ago - and I believe it still is. Its written in java, so some slowness is included. It has great export functionality as shown in this HTML exported mind-map with the topic "FLOSS development".

Freemind mind-mapping tool.

* Dia - to create professional looking figures for use in papers.

* Dropbox - to sync all my resources (PDFs, papers, presentations, etc) across multiple computers. I might switch to Ubuntu ONE later - but Dropbox works, and it works well.

* OpenOffice (LibreOffice) Draw - to create PhD-posters. Read my howto here.

* I write all my papers in Emacs with the AUCTeX LaTex mode and RefTex to create the TOC speedbar (see screenshot below). For notes I use xpad, and all revision control is done in Subversion (papers, documents, figures, presentations, code).

The desktop manager is Fluxbox with a bunch of key-bindings, bfpager, and gkrellm with bfm-plugin for system monitoring (I like to watch my system resources). Its the same setup I've used for years:

Screenshot of my current workspace: Fluxbox with gkrellm, Emacs, 2x xpads and 2x terminals.

27 Feb 2011

How to get IPv6 on your home network

It has been almost seven years since I last played with IPv6 (link, link). There has been lots of talk of IPv6 lately, and the company I work for has done some real-world IPv6 testing and deployment (check out Tore's IPv6 page: http://fud.no/ipv6/).

Since my local ISP has not deployed IPv6, and probably will not for a long time - I went looking for a "IPv6 tunnel broker". A tunnel broker enables you to tunnel IPv6 traffic over IPv4 to a IPv6 gateway (called "PoP").

So, which tunnel broker to choose from? Wikipedia give me a lot of choices. Several colleagues tipped me of SixXS -- their service is stable and professional, you get a /48 network and they have a PoP here in Oslo they said. Easy choice. 

First some paperwork:

1) First, apply for an account at SixXS:

       http://www.sixxs.net/signup/create/

Be patient - this can take some time since it require manual (human) verification. It took me 8 hours from I applied to my account was accepted.

2) Once you get your username and password, log into the web-interface, and proceed to request a tunnel. Choose "Dynamic NAT-traversing" as type of tunnel. Choose your neares POP, and submit. This process is automated, and it took approx 30 minutes before my request was accepted.

3) When 2) is complete - you have ONE routable IPv6 address. Since we need more than that, we proceed to request a subnet from SixXS webpage. 30 minutes later, my subnet was allocated. 

So I've received from SixXS the following information (not actual IPv6 addresses):
A) IPv6 gateway at SixXS: 2001:FFFF:FFFF:FFF::1/64
B) My local IPv6 gateway: 2001:FFFF:FFFF:FFF::2/64
C) My allocated IPv6 subnet, which is routed to B): 2001:FFFF:EEEE::/48

Next, we're ready to configure/install: 

4) First, install a helper daemon that take care of your IPv6 tunnel automatically. It's called AICCU (Automatic IPv6 Connectivity Client Utility):

  http://www.sixxs.net/tools/aiccu/

On Debian/Ubuntu install it by:

  $ sudo apt-get install aiccu

Enter your SixXS username and password when asked for it. You tunnel interface will be named "sixxs" and enabled (global IPv6 address in red):

  $ ifconfig sixxs
  sixxs     Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 
            inet6 addr: fe80::14d8:eeff:142:2/64 Scope:Link
            inet6 addr: 2001:FFFF:FFFF:FFF::2/64 Scope:Global   
            UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1280  Metric:1
            RX packets:0 errors:0 dropped:0 overruns:0 frame:0
            TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:500
            RX bytes:0 (0.0 B)  TX bytes:144 (144.0 B) 

5) Test your IPv6 connection:

  $ ping6 ipv6.google.com
  PING ipv6.google.com(2a00:1450:8004::93) 56 data bytes
  64 bytes from 2a00:1450:8004::93: icmp_seq=1 ttl=52 time=42.5 ms
  64 bytes from 2a00:1450:8004::93: icmp_seq=2 ttl=52 time=44.0 ms
  64 bytes from 2a00:1450:8004::93: icmp_seq=3 ttl=52 time=43.7 ms
  ... 

Good, we have IPv6 connectivity, but only from one host. We want IPv6 on our whole home network.

6) Since we're given a /48 network, it enables us to have 65536 /64 networks. Which should suffice for most needs. Our network architecture would look like this when complete:



We configure our IPv6 gateway (vallhall-r6) to route IPv6 traffic for our local network. First we need to assign a IPv6 address on the actual interface facing our network. This will be our IPv6 gw address for our network:

  $ sudo ifconfig eth0 inet6 add 2001:FFFF:EEEE::1/48
  $ ifconfig eth0  
  eth0      Link encap:Ethernet  HWaddr 52:54:de:ad:be:ef 
            inet addr:192.168.1.100  Bcast:192.168.1.255  Mask:255.255.255.0
            inet6 addr: fe80::5054:ff:feb6:beef/64 Scope:Link
            inet6 addr: 2001:FFFF:EEEE::1/48 Scope:Global
            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
            RX packets:16849 errors:0 dropped:0 overruns:0 frame:0
            TX packets:13394 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1000
            RX bytes:7336450 (7.3 MB)  TX bytes:7082583 (7.0 MB)

Make it permanent by adding it to:

  # cat /etc/network/interfaces
  ...
  iface eth0 inet6 static
      address 2001:FFFF:EEEE::1      
      netmask 48
      endpoint 2001:FFFF:FFFF:FFF::1  # your SixXS gw
      ttl 64

7) Make sure our gateway route IPv6 traffic:

$ sudo sysctl net.ipv6.conf.all.forwarding=1
  net.ipv6.conf.all.forwarding = 1

Make it permanent, by adding it to:

$ cat /etc/sysctl.conf
  ...
  net.ipv6.conf.all.forwarding=1 

8) You can now configure your network equipment with static IPv6 addresses. Since IPv6 uses hex (and thereby the letters A-F), some words/sentences can be embedded in these addresses:

  baba, babe, bade, bead, beef,
  cede, dace, dada, daff, dead,
  deaf, deed, face, fade, feed, ...

This way, your file-server can have an address like:

  ...:daff:bade:babe

go wild..

9) We don't want to configure all our host manually, so we need some kind of auto-configure. While IPv6 have DHCPv6, (like IPv4's DHCP), a more elegant solution is to use "stateful address autoconfiguration".

The host configure themselves by acquiring a prefix from a local IPv6 router, and combined with the local MAC address, creates a IPv6 address (router advertisements + MAC address of interface = IPv6 address).

We use radvd for sending these "router advertisements". Install and configure:

  $ sudo apt-get install radvd
  $ cat /etc/radvd.conf
  interface eth0 {
      AdvSendAdvert on;
      MinRtrAdvInterval 3;
      MaxRtrAdvInterval 10;
 
      prefix 2001:FFFF:EEEE:aaaa::/64 {
          AdvOnLink on;
                  AdvAutonomous on;

                  # After testing, can be set to 14400
                  AdvPreferredLifetime 30;
                  # After testing, can be set to 86400
                  AdvValidLifetime 30;
          };
  };

Start radvd:

  $ sudo service radvd restart
  Stopping radvd: radvd.
  Starting radvd: radvd. 

10) Excellent! Radvd will now send IPv6 prefix periodically (or when requested by a new client) to your network. All IPv6 capable host should now automatically configure themselves with a IPv6 address.

You should now be able to access IPv6 enabled hosts. Try for example: http://ipv6.google.com 

11) A final warning and advice: ADD A FIREWALL on your IPv6 gateway! Since IPv6 don't use NAT, every hosts that use IPv6 is directly accessable from the Internet. This is a good thing, but it also expose all your IPv6 enabled hosts and their services.

A good starting point for IPv6 firewall:

  https://www.sixxs.net/wiki/IPv6_Firewalling

Good luck!

Troubleshooting tips:

I) Dump IPv6 traffic using tcpdump:

  # tcpdump -i eth0 -vv ip6 or proto ipv6

II) Show your IPv6 routing table:

  # ip -6 r s

II) Check radvd messages

  # radvdump

29 Dec 2010

Convert Word documents (with pictures) to Mediawiki

Update (2011-06-14)! Magnus done a cleaner 'python-only' implementation of this tool - found here: https://github.com/mhagander/word2mediawiki/

This post will explain how to automatically convert MS Word files (with images) to Mediawiki pages. Any filetype OpenOffice supports can be converted.

Short explanation: We use OpenOffice to convert the Word files to wiki-syntax, but some voodoo is needed to fetch and upload any images included in the Word-file (the "voodoo" is depicted yellow in the flowchart below):


More detailed explanation: The perl script word2mediawiki.pl take a Word file as input. After some rudimentary checks, it calls the python script DocumentConverter.py which calls OpenOffice to do the actual conversion. This is done twice; we convert to both .wiki and .xml files. Since the .wiki file DO NOT contain any images (it only adds empty [[Image:]] wiki-tags where the images are supposed to be), we convert to .xml that DO include images. Here the images are base64 encoded, so we parse the .xml file, fetch all base64-images, decode and save as ordinary images files. We re-write the .wiki file to update all empty [[Image:]] wiki-tags with the correct image file just decoded. Finally we upload the original Word file (for reference), all images and create a wiki-page based on the .wiki files using the pywikipediabot. Se example below.

Prerequisite and install:

A) Linux - but may work on other platforms as well (not tested)
B) Install Perl and Python
C) Install the Python-UNO bridge. This enable Python to talk to the OpenOffice API (and do the conversion)
 
  # apt-get install python-uno

D) Install OpenOffice. We run OpenOffice "headless", so X is not required.

E) Install the OpenOffice "Sun Wiki Publisher" extension. This adds support for .wiki and .xml export.

  # unopkg add sun-wiki-publisher.oxt

F) Create a word2mediawiki directory. Download the word2mediawiki.pl script, and the PyODConverter script. Note! I've modified the PyODConverter script to support .wiki and .xml. You can download the modified version below:

  $ mkdir word2mediawiki
  $ cd word2mediawiki
  $ wget http://www.larsstrand.no/code/word2mediawiki/word2mediawiki.pl

G) Install pywikipediabot:

  $ svn co http://svn.wikimedia.org/svnroot/pywikipedia/trunk/pywikipedia

H) Configure pywikipediabot. Use the testbox2_family.py file as template for your Mediawiki installation. The file should be self-explanatory:

  $ cd pywikipedia/families 

Add username and password:

  $ cd ..
  $ cat user-config.py
  # -*- coding: utf-8  -*-
  family = 'testbox2'
  mylang = 'en'
  usernames['testbox2']['en'] = u'Wiki-USERNAME'
  password_file = "user-password"
  $ cat user-password
  ("Wiki-USERNAME", "Wiki-PASSWORD")

I) Test pywikipediabot: 

  $ python ./login.py -force -all
  unicode test: triggers problem #3081100
  Logging in to testbox2:en as Wiki-USERNAME via API.
  Should be logged in now

Great! We're ready for our first test.


Convert:

We find a word-file and execute:

  $ ./word2mediawiki.pl ../Testfile.doc
  #############################
  ## Converting /export/home/lks/tmp/Testfile.doc to .wiki and .xml using soffice..
  #############################
  Converting image: ../converted/Testfile_1.jpg
  Converting image: ../converted/Testfile_2.jpg
  Rewrote wiki page with new Image tag: [[Image:Testfile_1.jpg]]
  Rewrote wiki page with new Image tag: [[Image:Testfile_2.jpg]]
  #############################
  ## Conversion complete: ../converted/Testfile.wiki
  ... >>>> Skipping a bunch of output here.

  ## Uploading the wiki page
  ## Exec: python ./pywikipediabot/pagefromfile.py -start:XZXZ42 -end:YZYZ42 -safe -file:../converted/Testfile.wiki
  Reading '../converted/Testfile.wiki'...
  >>> Testfile <<<
  Logging in to testbox2:en as Wiki-USERNAME via API.
  Should be logged in now
  Sleeping for 8.3 seconds, 2010-12-29 18:03:38
  Creating page [[Testfile]] via API
  End of file.
  ## Conversion and upload complete

Note 1: You might get a warning when the pywikipediabot tries to upload the images/.doc file or create the wiki-page. This can happen if the same image/.doc file already have been uploaded. If the a wiki-page with the same name already exists, the bot issue a warning and abort.

Note 2: When OpenOffice convert the .doc file, it might spew out a bunch of warning and/or error messages. These can be ignored. OpenOffice complains a lot. 

Note 3: If the the script exits with a complain about "Can't connect to soffice on port" - just re-run the script. OpenOffice can be a little slow to start. (It will fork into the background the first time).

Note 4: The conversion is nowhere near perfect, and you might want to look over the wiki-page to ensure correct formatting.

Note 5: The filename of the Word file is used as name of the Wikipage. Example: "Testfile.doc" result in "mediawiki/index.php/Testfile"

Example:

27 Nov 2010

Freezing cold server!

The winter has arrived early in Norway this year. Cold winds from Sibir have brought freezing temperatures all over the country. Its now approx -10C here in Oslo, and the temperatures are expected drop further next week. This is good news for my balcony server!

I use Munin to monitor the temperature sensors in the server. One of the CPU cores is showing a nice 6C. One of the disks records 5C. Hopefully the fans will start spinning again once the temperatures starts rising again in the spring...


Since I'm a little worried what will happen if the temperatures drops below 0C inside the disks, I started the folding client to generate some heat (it uses 100% CPU on all cores). The temperature immediately jumped ~5-10 degrees:


Now I'm ready for the winter!

8 Nov 2010

Great Firewall of China

I'm attending the IETF79 meeting here in Beijing. So far, it has been great. Meeting the people I've only read about, and participating in discussions. In particular, I'm looking forward to the kitten WG meeting (GSS-API authentication) and anything related to SIP, in particular sipcore.

Since this is Beijing, we're behind the Great Firewall of China, also called The Golden Shield. It works, as far as I've read, on three layers:
  1. A rudimentary "DNS block" and/or redirect.
  2. If you access the IP-address directly, it sends a TCP RST effectively tearing down your connection. (You browser responds with a "Connection reset")
  3. Content filtering of HTTP-traffic. Especially targeted at news-articles containing certain sensitive information. If a one or more pre-defined keywords appear in the page, the connection is blocked.
A lots of material have been written about the firewall. And several methods can be used to counter the firewall, like using a proxy or VPN-connection. You can also test if your site is blocked by the firewall.

A couple of DNS lookups of blocked sites from behind the firewall:

  $ cat /etc/resolv.conf
  nameserver 202.106.0.20
  nameserver 202.106.46.151

  $ dig +short www.facebook.com
  $ dig +short www.youtube.com
  youtube-ui.l.google.com.
  youtube-ui-china.l.google.com.
  66.249.89.100
  66.249.89.101
  $ dig +short www.blogspot.com
  blogger.l.google.com.
  72.14.203.191

But IETF's NOC have taken over the hotel network (both wired and wireless) and are currently bypassing the firewall. In cooperation with Tsinghua University, two 1Gbps links connect us to the CERNET (with backup to CSTNet).

A couple of test network has also been deployed. Including a IPv6-only network and a IPv6 network using NAT64.