Beowulf clusters with centos 6.5 virtual machine

Read about Beowulf cluster on wikipedia. Main reference: “Beowulf Howto” by Kurt Swendson

1 Virtual machine installation & network configuration

Download from NUS mirror site Enable VT-x in BIOS/Security/Virtualization Create Redhat x64 virtual machine, attach centos installation image, configure network adapter (as below), boot up virtual machine to begin installation.

After installation of the 1st virtual machine, clone one or more of it with “reinitialize MAC address” enabled. Lastly enable port forwarding on one of them (head node).

1.1 Network

Adapter 1
  • Enable port forwarding for head node:: Example: TCP;; 5679;; 22
Adapter 2
Internal network

Port forwarding is for host to connect to virtual machine via ssh, so port must be ssh default port 22.

One problem that will happen is that the cloned virtual machine’s network adapter configuration will not be updated although the MAC address is changed, by “reinitialize MAC address”. Thus ifconfig shows only the lo adapter but no eth0 and eth1.

In this post, a solution is given as below:

  1. Remove the kernel’s networking interface rules file so that it can be regenerated

    rm -f /etc/udev/rules.d/70-persistent-net.rules

  2. Restart the VM


  3. UPDATE your interface configuration file

    vim /etc/sysconfig/networking/devices/ifcfg-eth0

  4. Remove the HWADDR entry or update it to the new HWADDR for the interface (listed in this file: /etc/udev/rules.d/70-persistent-net.rules).
    • Remove the UUID entry
    • Save and exit the file
  5. Restart the networking service

    service network restart

1.2 Hostname

/etc/hosts. The hostname at command line prompt will change once we change the hostname after and reboot. It allows us to see easily the host that we are in when multiple nodes are connected to via ssh.  beowulf0x localhost localhost.localdomain
... beowulf01 beowulf02


Main reference: How to set up nfs on centos 6 Head node serves as nfs server, which shares its /home directory to all worker node.

yum install nfs-utils 
chkconfig nfs on 
service rpcbind start
service nfs start

Create /etc/exports file and put in the following:

This option allows the client server to both read and write within the shared directory
Sync confirms requests to the shared directory only once the changes have been committed.
This option prevents the subtree checking. When a shared directory is the subdirectory of a larger filesystem, nfs performs scans of every directory above it, in order to verify its permissions and details. Disabling the subtree check may increase the reliability of NFS, but reduce security.
This phrase allows root to connect to the designated directory
exportfs -a

On the client, install

yum install nfs-utils

Append to /etc/fstab and mount

cat >> /etc/fstab     /home   nfs     rw,hard,intr    0 0
mount -a

Docky in Ubuntu 12.04

Weather and gmail docklet in default docky in ubuntu 12.04 is incompatible with the network-manager according to here.

So I built docky 2.2 from source and it works now.

Some dependencies have to meet to make successfully.

I installed mono-complete which may be unneccessary, but it is easier.

sudo apt-get install mono-complete intltool libxml2-parser-perl gnome-sharp2 libgio-cil libgconf2.0-cil libgio-2.0-cil libgnomedesktop2.0-cil-dev libgnome-keyring1.0-cil-dev librsvg2-2.0-cil-dev libdbus1.0-cil-dev libdbus-glib1.0-cil-dev libnotify-cil-dev libwnzk1.0-cil-dev

Hope you enjoy docky in Ubuntu. I also recommend conky manager.

How to achieve simple page layout in reveal.js with org-reveal

There are several ways of exporting org-mode to presentation slides, as described in Writing Non-Beamer presentations in org-mode. reveal.js is a javascript package developed by Hakim El Hattab for making html slides which has rich features and is easy to use. What is even better, Yujie Wen has already made an exported for org-mode: org-reveal, which works really well and is constantly improving.

One thing I miss in MS powerpoint is page layout. I know in html that can be easily achieved but what about in org-mode. I spent an hour on this problem and I am quite satisfied with the result. It looks like this:


org-mode default html block is not recognized by the exporter, but only #+REVEALHTML keyword. We can assign class=”fragment” to a <div> to display each div sequentially.

This is the org-mode file.

#+OPTIONS: num:nil reveal_mathjax:t toc:1
#+REVEAL_THEME: default
#+REVEAL_TRANS: default
#+REVEAL_MATH_JAX_URL: css/MathJax.js
#+REVEAL_ROOT: css/reveal.js
#+ATTR_REVEAL: :frag roll-in
#+OPTIONS: reveal_width:1920 reveal_height:1200

#+REVEAL_HTML: <style type="text/css"> #wrap {width: 80%; margin:0px auto;} #left_col {width: 49% ; float:left; margin: 0.5%; background-color: orange}</style>
#+REVEAL_HTML: <div id="wrap"> <div id="left_col" class="fragment">
#+ATTR_REVEAL: :frag roll-in
1. a
2. b
3. c
4. d
#+REVEAL_HTML: </div> <div id="left_col" class="fragment">
[[file:~/Pictures/blue ubuntu.jpeg]]
#+REVEAL_HTML: </div> <div id="left_col" class="fragment">
A framework for easily creating beautiful presentations using HTML. Check out the live demo.

#+REVEAL_HTML: </div> <div id="left_col" class="fragment">
Org-Reveal exports your Org documents to reveal.js presentations.\ With Org-reveal, you can create beautiful presentations with 3D effects from simple but powerful Org contents.
#+REVEAL_HTML: </div> </div>

Connect to juniper webvpn in 64-bit ubuntu

This post by Scott in 2011 has explained in great details how to connect to Juniper VPN from 64-bit linux. It still works now for ubuntu 13.04 64-bit, but things did change a little bit, which I feel it may be useful to document here. Besides I will probably keep this post updated as new releases of ubuntu coming along.

In 32-bit linux Juniper webvpn just works, but not 64-bit ones. And most likely IT department, or computer center do not support linux.

1 Oracle java

Juniper webvpn works with Oracle java, but not openjdk which is the default java in ubuntu. We can download 64-bit jdk from Oracle website here.

Refer to the following code snippet for installation.

if [[ ! -d /usr/lib/jvm ]]; then
    sudo mkdir /usr/lib/jvm
sudo tar xvf "$DIR"/jdk-7u*-linux-x64.tar.gz -C /usr/lib/jvm/
sudo mv /usr/lib/jvm/jdk1.7.0_?? /usr/lib/jvm/jdk1.7.0
sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk1.7.0/bin/java" 1
sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/lib/jvm/jdk1.7.0/bin/javac" 1
sudo update-alternatives --install "/usr/bin/javaws" "javaws" "/usr/lib/jvm/jdk1.7.0/bin/javaws" 1
sudo update-alternatives --install "/usr/bin/javah" "javah" "/usr/lib/jvm/jdk1.7.0/bin/javah" 1
sudo update-alternatives --install "/usr/bin/jar" "jar" "/usr/lib/jvm/jdk1.7.0/bin/jar" 1
sudo chmod a+x /usr/bin/java 
sudo chmod a+x /usr/bin/javac 
sudo chmod a+x /usr/bin/javaws
sudo update-alternatives --config java
sudo update-alternatives --config javac
sudo update-alternatives --config javaws
if [[ ! -d ~/.mozilla/plugins/ ]]; then
    mkdir ~/.mozilla/plugins/
ln -s /usr/lib/jvm/jdk1.7.0/jre/lib/amd64/ ~/.mozilla/plugins/

2 Login to webvpn

Login to webvpn web interface from firefox or chromium/chrome. We may see in chromium/chrome notification like “Java(TM) was blocked because it is out of date.”. Click “Run this time”; while firefox may ask you to update java, click “later”.

The following files will be downloaded to ~/.junipernetworks

META-INF  ncLinuxApp.jar  whitelist.txt           ncdiag    ncsvc           ssl.crt             NC.jar    version.txt

The last time I saw only ncLinuxApp.jar. So I uncompressed it and make ncsvc executable.

jar xf ncLinuxApp.jar
chmod +x ncsvc

Then we can build 32-bit ncui executable.

sudo apt-get install ia32-libs gcc-multilib
gcc -m32 -Wl,-rpath,`pwd` -o ncui

Next get webvpn host SSL certificate:

echo | openssl s_client -connect (your host):443 2>&1 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | openssl x509 -outform der > ssl.crt

To run ncui, we need to provide 3 parameters, host name, cookie, ssl certificate. cookie can be different every time we log in, it is a string started with DSID. To get the cookie:

  1. chrome/chromium: in url box javascript:alert(document.cookie), enter
  2. firefox: ctrl+shift+k to get a javascript console, type alert(document.cookie), enter

Last, we can run ncui, e.g.

sudo ./ncui -h hostname -c DSID=217f78c497e6731b0895daa130fbdaac -f ssl.crt

Ubuntu 13.10 tweaks

1 ibus configuration and shuangpin in sunpinyin

Input methods configuration is very different from the previous ubuntu versions. It seems that there are now two places to configure input mehods text entry settings (in notification panel on right corner) and ibus-setup. I have been using sunpinyin’s shuangpin and ibus-zhengma for some years, so they are the first ones to install to a new os. Then I tried to configure sunpinyin to shuangpin using ibus-setup, an error was raised for missing python module ibus. We must install python-ibus package with apt-get to be able to configure it.

2 More tweaks to come

This is my 2nd day of using ubuntu 13.10 x8664 and so far the above are the one issues that I encountered.

ess + eldoc + ssh + X11 forwarding + R running in screen

Running interactive R on server is a daily routine for many bio-statistician. But your productivity can be limited in an unproductive environment like in the terminal.

  1. We want to see plots, maybe also html help.
  2. Connection is broken very often.
  3. Function arguments. Although R can show arguments in terminal, that fills the screen very quickly.

Terminal is not good enough. Emacs is an excellent text editor; ess is a emacs package that provides comprehensive support for statistical languages like R, S, etc in emacs. The solution to the above problem would be:

  1. X11 forwarding for plots and html help.
  2. screen
  3. eldoc doesn’t work in ssh session yet. But I found a work-around.

1 Enabling X11 forwarding with Tramp

We are editing a R script on a remote host with Tramp, while running an interative R session on the remote host. We would like to see plot and probably html help files, which requires x11 forwarding. To enable x11 forwarding in a shell we can do the following:

ssh -XC user@host

where -X is for x11 forwarding and -C is for compression (optional)

But we cannot type options when lauching Tramp. The solution is to write those options in ~/.ssh/config as following:

Host *
Compression yes
ForwardX11 yes

We can specify a particular hostname if we like, by replacing * with the host name.

Function hints work in both R console and script buffer.

1.1 Alternative R

We can have different versions of R installed, and we want to launch the correct one automatically from script buffer.

Use the following code to set to your chosen R executable:

(setq inferior-R-program-name "~/bin/R")

2 Complete setup: ess + X11 forwarding + screen

The objective is to have a persistent R session in a screen session running on remote host. I want to edit local or remote R script while being able to send command to the R session interatively, and also being able to see plots from R, no matter which computer I am using.

ess works nicely with ssh but not with ansi-term. eldoc does not work with ess-remote for me right now.

  1. In ssh, screen key-bindings are not recognized. Put in init.el so that we can detach screen session by C-l C-a C-l C-d.
;; used to send screen keybindings to shell in emacs
(define-key shell-mode-map (kbd "C-l") (lambda (seq) (interactive "k") (process-send-string nil seq)))
(define-key inferior-ess-mode-map (kbd "C-l") (lambda (seq) (interactive "k") (process-send-string nil seq)))

  1. In order to use screen in ssh, we must use xterm.
TERM=xterm ## give clearing capabilities to shell-mode

Refer to VINH NGUYEN’s post for more details about screen in emacs with ansi-term.

2.1 X11 forwarding from screen and R session in screen session

X11 forwarding depends on the following environment variables, e.g.

  • SSH_CLIENT= 4671 22
  • SSH_TTY=/dev/pts/11
  • SSH_AUTH_SOCK=/tmp/ssh-zIn16635/agent.16635
  • SSH_CONNECTION= 4671 22
  • DISPLAY=localhost:13.0

SSH_AUTH_sock is always empty for me. But it does not cause any problem.

screen session will inherit those variables from the shell in which the screen session is spawned. If you ssh from a different client, these environment must be changed in the screen session. There is a script written by Sam that does it for you, check here. The basic idea is to save the environment variables in remote shell to a file, which later can be sourced by screen session.

In a R session inside a screen session, set DISPLAY environment variable same as DISPLAY in shell.


2.2 Issues

TAB completion of function arguments does not work in R session in ssh. Upload ESSR.R to server and source(file = ‘ESSR.R’).

Eldoc does not work in M-x ess-remote session or R script buffer. First run ESSR.R. Run the following block in remote R buffer by C-x C-e for each line

;; arguments cache
(ess-process-put 'funargs-cache (make-hash-table :test 'equal))
(ess-process-put 'funargs-pre-cache nil)

2.3 Summary of workflow

Put in init.el to be able to send key to terminal from ssh.

;; used to send screen keybindings to shell in emacs
(define-key shell-mode-map (kbd "C-l") (lambda (seq) (interactive "k") (process-send-string nil seq)))
(define-key inferior-ess-mode-map (kbd "C-l") (lambda (seq) (interactive "k") (process-send-string nil seq)))

  1. (for new R session)From terminal ssh to host, create screen session and launch R in the screen session. C-z to send R to background. Detach from screen.
  2. In emacs M-x ssh, next enter user@host, and password.
  3. In ssh, enter TERM=xterm, for screen to work.
  4. sh grabssh, save environment variables for X11 forwarding.
  5. screen -r and run bin/fixssh to update screen environment variable for X11 forwarding. If you are in R, C-l C-z to send R to background first.
  6. fg, to send R to foreground.
  7. source(‘ESSR.R’)
  8. M-x ess-remote to attach the current process to ess.


(ess-process-put 'funargs-cache (make-hash-table :test 'equal))
(ess-process-put 'funargs-pre-cache nil)

The following configuration may be useful, put in init.el.

;; eldoc in echo area
(setq ess-eldoc-abbreviation-style 'normal)
(setq eldoc-echo-area-use-multiline-p t)