Advocating Tcl/Tk for GUI development

I am really surprised of very few uses of Tcl/Tk for GUI building. It’s a super-simple programming language, you can learn the syntax in only one day if you are accustomed to programming in other languages. Portable across Unix/Win/Mac OS. The Tk toolkit allows you to build GUIs directly by defining the hierarchical relations between GUI elements (widgets) and attaching event-response functions with parameters. It is an object-based rather than object-oriented system and this preserves simplicity, easy of learning and maintaining. There is no need for a graphical GUI builder too, everything can be scripted with much less code to write compared to other solutions. I’d like to see a desktop environment fully based on Tcl/Tk, at least on Linux. I really do not like the complexity of Qt and Gtk. GUI development should not be done in system languages like C and alike, but in high-level scripting languages. There is no need to scale up, every GUI is used only by one user. Sun decided to market Java, a system language, but without any operating system based on it, so any advantage of being low level is just a disadvantage. Using Java for some high-level task like GUI building is certainly possible but very far from optimal. But without starting a language war, we should just recognize no language is best at everything. This is why is often worth using more than one language in the same project. Tcl/Tk can be embedded in various system languages, so the GUI can be implemented in Tcl/Tk and the rest in the hosting language, if more appropriate. I wish there were more Tcl/Tk users and more support, but I must acknowledge the reality: there are many languages in computer science, some generic, some specific, and they are often misused. Programmers are responsible to make the right technical choices. If managers do that without considering alternatives, then you have a disaster: a lot of money lost and development headaches, which could be avoided.

Building a cheap software sound mixer with Linux

You will need at least two USB mics. Use the alsamixer command. Press F6 to select the sound card and F4 to adjust capture volumes. E.g. condenser microphones should be operated at lower volume than dynamic ones, because they are more sensitive to noise. To start and stop recording from both sources at practically the same time by using the following shell script:

#!/bin/sh
arecord -q -D hw:2,0 -t wav -f s16_le -c1 -r44100 track1.wav &
arecord -q -D hw:3,0 -t wav -f s16_le -c1 -r44100 track2.wav &
read -n1 -r -p "Press any key to stop recording..."
kill $(jobs -p)

replace hw:3,0 and hw:2,0 with your actual audio sources or add more. You can list all audio sources with:

arecord -l

Device names may change order depending on the USB connection order, so for example if you are recording vocals and guitar, you will have to check which track will contain which by playing each track separately. This is something you will want to do anyway to check how the recording from each mic went.

To mix, use Audacity: for each track, select a silence part and use Effect/Noise Reduction/Get Noise Profile. Then type J K to select the whole track and again Effect/Noise Reduction but this time hit OK to reduce noise for that track only based on the profile. You have to do this for each track separately. Then use the gain knob to reduce or increase volume of each track. Select all tracks by clicking on each track panel while you keep shift pressed then go to Tracks/Mix and Render. In the same menu, you can also mix and render to another track, if you do not want to loose the original ones.

Playing with MariaDB (MySQL) replication

MariaDB is an open-source replacement for MySQL. MariaDB keeps a so-called binary log of transactions, which keeps track of all updates to the database. It is used for database restoration and replication, but in this tutorial we will focus on replication only. Keeping a binary log also proves useful if you need the ability to do point in time recovery from your latest backup, but we will not see that.

Note: I am using Arch Linux and following the official MariaDB documentation for replication.

Master configuration

master # pacman -S mariadb

Optional - for mytop to work:
master # pacman -S perl-dbd-mysql perl-term-readkey

Optional - auto completion of table, field names, etc. in the client
master # sed -ri -e 's/^no-auto-rehash/auto-rehash/' /etc/mysql/my.cnf

master # systemctl start mysqld

I recommend answering yes to all questions and setting a MySQL root password:
master # mysql_secure_installation

master # systemctl restart mysqld

Try connection:

master # mysql -p


Here is courses.mysqldump, a very simple relational database:

DROP DATABASE IF EXISTS courses;
CREATE DATABASE courses;
USE courses;
DROP TABLE IF EXISTS offering, course, instructor;

CREATE TABLE course (
  id int(10) unsigned NOT NULL AUTO_INCREMENT,
  title varchar(255) NOT NULL,
  PRIMARY KEY (id)
);

CREATE TABLE instructor (
  id int(10) unsigned NOT NULL AUTO_INCREMENT,
  name varchar(255) NOT NULL,
  PRIMARY KEY (id)
);

CREATE TABLE offering (
  course_id int(10) unsigned NOT NULL,
  instructor_id int(10) unsigned NOT NULL,
  PRIMARY KEY (course_id, instructor_id),
  FOREIGN KEY (course_id) REFERENCES course(id)
    ON DELETE CASCADE ON UPDATE CASCADE,
  FOREIGN KEY (instructor_id) REFERENCES instructor(id)
    ON DELETE CASCADE ON UPDATE CASCADE
);

Then insert some sample data:


master # mysql -p courses
MariaDB [courses]> INSERT INTO course VALUES (1,'Basic Linux Programming');
MariaDB [courses]> INSERT INTO instructor VALUES (1, 'Antonio Bonifati');
MariaDB [courses]> INSERT INTO offering VALUES (1,1);

The binary log is active by default:

master # cat /var/lib/mysql/mysql-bin.index
./mysql-bin.000001
./mysql-bin.000002
./mysql-bin.000003
./mysql-bin.000004
master # file /var/lib/mysql/mysql-bin.000001
/var/lib/mysql/mysql-bin.000001: MySQL replication log
It's default size is 1 GB (max_binlog_size):

master # mysqld --verbose --help 2>/dev/null | grep ^max-binlog-size
max-binlog-size 1073741824

As you can see in /etc/mysql/my.cnf, by default MariaDB is configured as a master and mixed binary-logging format is used:


# Replication Master Server (default)
# binary logging is required for replication
log-bin=mysql-bin
# binary logging format - mixed recommended
binlog_format=mixed

On the master, we also need to create an account that will be used by the slave to connect and start replicating. We can reuse a normal user account, and just grant the REPLICATION SLAVE permission, but it is more secure to have a dedicated account (e.g. repl) with this privilege only:

master # mysql -p -e "GRANT REPLICATION SLAVE ON *.* TO 'repl'@'SLAVE_IP' IDENTIFIED BY 'REPL_PWD'; FLUSH PRIVILEGES"


Remember to replace SLAVE_IP with the slave IP address before issuing the above command.

Each master or slave in the same replicating group must have a unique server_id. E.g. I will leave the master server_id at its default value (1) and give a different value to the slave (e.g. 2):

master # mysqld --verbose --help 2>/dev/null | grep ^server-id

server-id 1

Slave configuration

Since I do not have another machine for tests, I have installed another copy of Arch Linux in Virtualbox to serve as a slave. The host is the master server itself (my laptop). On the slave, I installed and secured MariaDB exactly as done before for the master.

Then assign the slave a 32-bit number different from the master (e.g. 2). You do this by editing /etc/mysql/my.cnf, commenting the line

server-id = 1

and uncommenting a similar one a few lines below:

# server-id = 2

Also, since this is a slave, disable the binary log by commenting the following lines:

log-bin=mysql-bin
binlog_format=mixed

Actually you can leave this lines (or luncomment another line log-bin=mysql-bin just below in the section related to slave configuration), if you want to have a chained replication setup. That is this slave will write to a binary log any data modifications that are received from the replication master, in order to act as a master to one or more other slaves.

Another reason to enable binary logging on the slave is to enable incremental backups using the slave.

This is not our case and to save disk space and speed up the slave replication, we will not write a binary log there. I have also deleted the existent binary logs:

slave # rm /var/lib/mysql/mysql-bin.*

You can also purge binary logs before a certain date with a query.


After you save edits to /etc/mysql/my.cnf, you need to restart mysql:


slave # systemctl restart mysqld

We now need to copy the data from master to slave. Let's pretend the master server is busy and there are data manipulation queries like INSERT and UPDATE going on, how do we make sure the data copy is consistent (not half new and half old)?

The solution for InnoDB (the default storage engine in MariaDB) is to use the --single-transaction option of mysqldump. It works by starting a transaction statement to the server before dumping data. Because of the isolation property of transactions, the dump will always be consistent (unless your application contains ALTER, CREATE or similar data definition instructions, which is unlikely). If your tables are big add a --quick option to retrieve one table row at a time rather than retrieving the whole row set and buffering it in memory before writing it out.

Temporary allow master to connect to slave as root:
slave # mysql -p -e "GRANT ALL PRIVILEGES ON *.* TO 'root'@'MASTER_IP' IDENTIFIED BY 'TEMP_PWD'; FLUSH PRIVILEGES"

master # mysqldump -p --master-data --single-transaction -B courses | mysql -h SLAVE_IP -p'TEMP_PWD'
Drop privileges and root access from the master to the slave:
slave # mysql -p -e "DROP USER 'root'@'MASTER_IP'"



If you had more than one database to copy, you would simply add other database names after "courses". You can also run the data copy command the other way around, that is on the slave. Left as an exercise for the reader.


The --master-data options causes the dump to include a query like:

--
-- Position to start replication or point-in-time recovery from
--

CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000004', MASTER_LOG_POS=5815;

This tells the slave the position in the binary log of the master server to start replication from.

In general, for InnoDB tables, mysqldump's --single-transaction option is the way of making an online backup, e.g. locally for all databases:

# mysqldump -A --single-transaction all_databases.sql

Now issue this query on the slave, to tell it what the master is and what account to use for replication:

slave # mysql -p -e "CHANGE MASTER TO
  MASTER_HOST='MASTER_IP',
  MASTER_USER='repl',
  MASTER_PASSWORD='REPL_PWD'"


You can optionally override the default value for MASTER_CONNECT_RETRY (86400). This is the number of times that the slave tries to connect to the master before giving up. A value of 0 means “infinite”; the slave attempts to connect forever.

Replication has not started yet. You have to issue:

slave # mysql -p -e "START SLAVE"

You can now try to add a record in a table of the courses database in the master and you will see the same appears on the slave.


On the slave use:

slave # mysql -p -e "SHOW SLAVE STATUS \G"
Enter password:
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.14.51
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000006
Read_Master_Log_Pos: 245
Relay_Log_File: mysqld-relay-bin.000006
Relay_Log_Pos: 529
Relay_Master_Log_File: mysql-bin.000006
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 245
Relay_Log_Space: 1108
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 1
to see the state of replication, while on the master:


master # mysql -p -e "SHOW MASTER STATUS \G"
Enter password:
*************************** 1. row ***************************
File: mysql-bin.000006
Position: 245
Binlog_Do_DB:
Binlog_Ignore_DB: Note: during replication, a slave server creates several logs that hold the binary log events relayed from the master to the slave, and to record information about the current status and location within the relay log. There are three types of logs used in the process by the slave, one relay log and two status logs: 

The relay log (e.g. mysqld-relay-bin.000006 above for the slave) consists of the events read from the binary log of the master and written by the slave I/O thread. Events in the relay log are executed on the slave as part of the SQL thread.

Note you can convert and display a binary log file (including a relay log, since they have the same structure as a regular binary log) in text mode using mysqlbinlog, e.g.:

slave # mysqlbinlog /var/lib/mysql/mysqld-relay-bin.000006

Anyway, only SQL statements logged are readable in this output (if any). Row changes are not easy to interpret.

The master info log (master.info) is plain text and contains status and current configuration information for the slave's connection to the master. This log holds information on the master host name, login credentials, and coordinates indicating how far the slave has read from the master's binary log. Since it contains the password in clear, this file is only readable by the mysql user and any user in the mysql group (by default none) and of course by root:

slave # cat /var/lib/mysql/master.info
18
mysql-bin.000006
245
192.168.14.51
repl
REPL_PWD
3306
60
0





0
1800.000

0


0




Values in master.info can be changed using an SQL query: CHANGE MASTER TO. It allows to set the master host to use and the position in the binary log to start replication from (e.g. the one you get from a SHOW MASTER STATUS query on the master):

slave # mysql -p -e "CHANGE MASTER TO
    MASTER_HOST=MASTER_IP,
    MASTER_USER='repl',
    MASTER_PASSWORD=REPL_PWD,
    MASTER_LOG_FILE='mysql-bin.000004',
    MASTER_LOG_POS=2012"

You better not set these options in my.cnf like:


master-host = MASTER_IP
master-user =  repl
master-password = REPL_PWD

because that is currently not supported (it will probably be in the future, I hope).


The relay log info log is also simple text and holds status information about the execution point within the slave's relay log.

slave # cat /var/lib/mysql/relay-log.info
./mysqld-relay-bin.000006
529
mysql-bin.000006
245

04With that background, here is an explanation of some of the fields in the output of SHOW SLAVE STATUS.

Field Relay_Master_Log_File (mysql-bin.000006) is the name of the binlog on the master containing the last SQL statement successfully executed on the slave. It is not the name of a relay log on the slave, but a binary log on the master.

master # mysqlbinlog /var/lib/mysql/mysql-bin.000006

Exec_Master_Log_Pos (245) is the position in the Relay_Master_Log_File that the slave SQL thread has executed up to. So in your example, the slave db has executed all statements up until binlog mysql-bin.000006 pos 245 on the master db. You see this value is the same you see in SHOW MASTER STATUS in the Position column.


The tuple (Relay_Master_Log_File, Exec_Master_Log_Pos) expresses the coordinates in the master binary log indicating how far the slave SQL thread has executed events received from that log.


What is the difference between Relay_Master_Log_File and Master_Log_File? (Master_Log_file, Read_Master_Log_Pos): Coordinates in the master binary log indicating how far the slave I/O thread has read events from that log.

Finally (Relay_Log_File, Relay_Log_Pos) is the coordinates in the slave relay log indicating how far the slave SQL thread has executed the relay log. These correspond to the preceding coordinates, but are expressed in slave relay log coordinates rather than master binary log coordinates.



until_log_pos is really only used if you start your slave with the query "START SLAVE UNTIL master_log_pos = integer". That syntax will replicate up to that position (it will be the exec_master_log_pos) and then stop. You would normally only do this if you wanted to replicate to a specific point, but no further (like if the next statement is an accidental table drop or something). The value of until_log_pos is 0 when it is not specified, so in our case means that replication will just keep moving forward.

You can now try to shut down MySQL on the slave:

slave # systemctl stop mysqld

Insert a new course on the master:

master # mysql -p courses
MariaDB [courses]> INSERT INTO course VALUES (3, "C Programming");
MariaDB [courses]> INSERT INTO offering VALUES (3, 1);

Then start the slave again:

slave # systemctl start mysqld

And check you have this new data:

slave # mysql -p courses
MariaDB [courses]> SELECT * FROM course WHERE id=3;
+----+---------------+
| id | title         |
+----+---------------+
| 3  | C Programming |
+----+---------------+


MariaDB [courses]> SELECT * FROM offering;
+-----------+---------------+
| course_id | instructor_id |
+-----------+---------------+
|         1 |             1 |
|         3 |             1 |
+-----------+---------------+


So replication resumes automatically. You usually won't disconnect a slave on purpose, but there is a case where this makes sense: if you want to back up the database consistently. In this case you disconnect the slave and take the backup from it. You can simply copy or rsync all files in /var/lib/mysql/* recursively. This ensures all database will be backed up at the same instant in time.


What happens if you write to the slave, which is supposed to be used as read-only? Go on and make a query like this:

slave # mysql -p -e "INSERT INTO instructor VALUES(2,'Duffy Duck')" courses

You see there is nothing to prevent you from writing to the slave if you connect with a user that has such privilege:

slave # mysql -p -e "SELECT * FROM instructor" courses
Enter password:
+----+------------------+
| id | name             |
+----+------------------+
|  1 | Antonio Bonifati |
|  2 | Duffy Duck       |
+----+------------------+

Normally you would prevent writes to the slave by not having users with CREATE, UPDATE, etc. permission, but if you really want to be sure that no user other than those with the SUPER privileges and the replication can write, set the read_only global variable:


slave # mysqld --verbose --help 2>/dev/null | grep ^read-only
read-only FALSE

You can set it permanently in /etc/mysql/my.cnf (under the [mysqld] section):

read_only = 1

read_only is a dynamic variable, meaning a change does not require a full restart of the server with systemctl restart mysqld. It's value can be changed dynamically while the server is running, so to make effective the change to my.cnf, you either restart or just do:

slave # mysql -p -e 'FLUSH TABLES WITH READ LOCK; SET GLOBAL read_only = ON; SHOW VARIABLES LIKE "read_only"'

FLUSH TABLES WITH READ LOCK locks all tables to make sure that nobody can write to databases (e.g. by doing an INSERT or UPDATE). Once the lock is acquired, we can safely set the server as read-only for clients. The lock will be automatically releases at the end of the session, that is when the mysql command returns.

After that try:

slave # mysql -p -e "INSERT INTO instructor VALUES(3,'Wiley E. Coyote')" courses

you see that it succeeds, because you are root. But if you create another user with write access and no SUPER privilege, you will get an error:

slave # mysql -p -e 'GRANT ALL PRIVILEGES ON courses.* TO ant@localhost IDENTIFIED BY "ant_pwd"'



slave # mysql -uant -p -e "INSERT INTO instructor VALUES(4,'Porky Pig')" courses
ERROR 1290 (HY000) at line 1: The MariaDB server is running with the --read-only option so it cannot execute this statement

So it is your responsibility to make sure the slave databases are consistent. MySQL does not care. It provides replication only, not cloning.

Transparent double SSH connections and issuing the same command to multiple machines at once

Goal: Connect directly to any final host you have an account on, using a jumphost and typing your passphrase only once at each reboot of your client machine.

On your client, generate your SSH keys one off. Please choose to encrypt your private key using a passphrase:

client$ ssh-keygen -t dsa -C "$(whoami)@$(hostname)-$(date -I)"
client$ ssh-copy-id username_on_jumphost@jumphost_fully_qualified_name

Now install keychain if you do not have it already and enable it for your local user by adding an alias to your .bashrc:

client$ echo "alias ssh='eval \$(/usr/bin/keychain --eval --agents ssh -Q --quiet ~/.ssh/id_dsa) && ssh'" >>~/.bashrc

Source .bashrc or reopen your terminal to make sure this alias is defined. Next step is to configure ssh for easy connection to our hosts. Create ~/.ssh/config if it does not exist else add to it:

ControlMaster auto
ControlPath /home/YOUR_LOCAL_USERNAME/.ssh/tmp/%h_%p_%r

Host jumphost
  ForwardAgent yes
  Hostname jumphost_fully_qualified_name
  User YOUR_USER_NAME_ON_JUMPHOST

Host ...
  ForwardAgent yes
  User YOUR_USER_NAME_ON_ALL_HOSTS
  ProxyCommand ssh -q jumphost nc -q0 %h 22

Where ... is a blank-separated list of all host names reachable from the jumphost. You can use wildcards, e.g. host*

Next, create a temporary directory to keep track of all connections, so that ssh can reuse a connection to the jumphost multiple times and logins are visibly faster:

client$ mkdir ~/.ssh/tmp

You can now login to every host from your client with one short ssh command, e.g.:

client$ ssh host1

client$ ssh host2

Goal: execute the same command(s) or edit the same file on multiple hosts, but type stuff only once.

You may also find useful to install a utility like Parallel SSH on either your client or the jumphost. Parallel SSH allows you to execute commands on multiple hosts in parallel, using only one command. See this tutorial and remember that on ubuntu pssh is called parallel-ssh. E.g. here is how to get a description of linux distribution installed on multiple hosts:

jumphost$ parallel-ssh -h host_list -i lsb_release -d

Where host_list is just a file containing IPs or host names, one per line. Note we use the option -i rather than -P else output looks messy.

Another interesting utility is clusterssh. You install it locally and it will open multiple terminals. You can edit the same file on multiple hosts: your input goes to all terminals if directed to a small empty control windows. If focus is on a specific terminal, it will only go to that terminal. Of course, it is up to you to make sure these two mechanisms are used correctly:

client$ cssh host1 host2

Count the number of lines added to a log file in a time interval

I needed to estimate how many lines are added to a log file in 10 minutes. THe idea was to start from a command like tail -f file | wc -l, but that never ends, because wc never sees an end of file and gets blocked when reading on the read size of the pipe if the pipe is empty.

So what I need to do is just to kill tail after a certain time (e.g. 10 minutes). Now, if I run tail -f file | wc -l in background by adding a & at the end, that becomes one job for the shell and I can only use kill %1 to kill both the tail and the wc process. But that way I will not get any output from wc.

The solution was to save the pid of the tail process in a file e.g. tail.pid and then kill tail only by its ID. To save the ID we just output variable $! to descriptor number 3. $! expands to the process ID of the most recently executed background (asynchronous) command, if tail succeeds, it will be its PID. Before tail & echo is executed we make user descriptor number 3 is opened and redirected to a file (this is what 3>tail.pid does):

$ ( tail -f trace.130524.txt & echo $! >&3 ) 3>tail.pid | wc -l &
$ sleep 600 && kill $(

The command substitution $(< file) is just a faster replacement for $(cat file).

Reserved disk space on the root partition in Linux

In Linux by default a certain percentage of space (by default 5%, but you can change that with the -m option to tune2fs) is reserved on a root partition. So if you run df on an almost full filesystem, you may see about 100% used, but still there is a difference between the size and the total used space! This is because the percentage refers to the unreserved space.

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 193G 183G 288M 100% /
...
Why is some space put aside? Consider this reserved space can only be used by processes running with the root privileges. So the purpose is to allow some of the process to continue running for a while and using more disk space even if some other non-privileged processes gone crazy fill all the non-reserved space. If you have a very big drive, you may want to reduce the percentage of reserved space to avoid waste. E.g. 5% of 1 TB (one terabyte) is about 51 GB, but 5% of 6 TB is a whopping 307 GB, probably too much reserved space, if you have a root partition that big.

If you are in dire straits you may want to unreserve this space so it can be used by non-root processes:

# tune2fs -m 0 /dev/sda3

Then you suddenly have a 5% more free space:

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 193G 183G 11G 95% /
...

but beware that the filesystem may get fragmented badly without any space reserved. ext4 suffers less from this problem, so it is recommended that you remount the root partition as ext4, if you are still using ext3. This is safe to do and does not require a conversion (of course not all ext4 features will be available). Just replace ext3 with ext4 in /etc/fstab for your root partition (/) and reboot. You can safely revert to ext3 the same way. Anyway, you cannot live long with an almost full partition. You need to make more space available and/or cleanup.

Nagios

Warning: I do not like Nagios. The fact is I do not like frameworks. They only impose structure, usually too much structure and policies and mostly only that, with very little features. You loose true flexibility, which comes only from Turing-complete programming languages. So you better use libraries and not frameworks. Frameworks are for non-programmers or mediocre programmers. Whenever you need to do something not supported by the framework, you will end up fighting it. No framework designer can foretell all users needs and no framework can be as programmable and generic as a programming language. Anyway, here it is how to setup this monitoring framework, if you really have to. If you are not a programmer you may find it useful. But if you are a programmer and have time, you better use sysstat to collect data and cook up you own monitoring solution.

Installation
# yaourt -S nagios nagios-plugins
# htpasswd -c /etc/nagios/htpasswd.users nagiosadmin

# cp /etc/nagios/cgi.cfg.sample /etc/nagios/cgi.cfg
# cp /etc/nagios/resource.cfg.sample /etc/nagios/resource.cfg
# cp /etc/nagios/nagios.cfg.sample /etc/nagios/nagios.cfg
# cp /etc/nagios/objects/commands.cfg.sample /etc/nagios/objects/commands.cfg
# cp /etc/nagios/objects/contacts.cfg.sample /etc/nagios/objects/contacts.cfg
# cp /etc/nagios/objects/localhost.cfg.sample /etc/nagios/objects/localhost.cfg
# cp /etc/nagios/objects/templates.cfg.sample /etc/nagios/objects/templates.cfg
# cp /etc/nagios/objects/timeperiods.cfg.sample /etc/nagios/objects/timeperiods.cfg

# cat >>/etc/httpd/conf/httpd.conf

# Nagios
Include "conf/extra/nagios.conf"

# PHP
Include "conf/extra/php5_module.conf"
^D
# cp /etc/webapps/nagios/apache.example.conf /etc/httpd/conf/extra/nagios.conf

# usermod -G nagios -a http

# pacman -Sy apache php-apache gd
# sed -ri '/^(open_basedir = )/ s,$,:/etc/webapps:/usr/share/nagios,' /etc/php/php.ini

Add LoadModule php5_module modules/libphp5.so to /etc/httpd/conf/httpd.conf

# systemctl start httpd
# systemctl status nagios

Go to http://localhost/nagios
login as nagiosadmin

Edit /etc/nagios/objects/contacts.cfg and change 30@localhost with your email address. Log file is /var/nagios/nagios.log

How to create a plugin


I want to create a plugin file named check_something. Any plugin should return the following exit codes:

OK—0—service works properly
WARNING—1—service is in warning state
CRITICAL—exit code 2—service is in critical state
UNKNOWN—exit code 3—service is in unknown state

If an error/exception happens, you better return UNKNOWN, because this is usually the best thing to suggest to nagios in this case. Returning WARNING on error/exception is not appropriate. E.g. if the plugin user chooses to disable notifications for WARNINGS, he will not know that the plugin is actually not working.

Before exiting you should print a line with the format:


SOMETHING OK/WARNING/CRITICAL/UNKNOWN: ...

Standard plugins are installed into /usr/share/nagios/libexec/. You can put yours there too, but I have decided to use a separate directory /usr/local/share/nagios/libexec/ In /etc/nagios/objects/commands.cfg you have to choose a command_name (unique nickname) for your plugin and define the full path of the executable:

define command{
         command_name check_something
         command_line /usr/local/share/nagios/libexec/check_something
}

This way you register the plugin with nagios. Then to add this check to a host, e.g. localhost, edit file /etc/nagios/objects/localhost.cfg and add there a new definition of a service, e.g.:

define service{
         use local-service
         host_name localhost
         service_description My Plugin
         check_period 24x7
         contact_groups admins
         notification_options c,r
         check_command check_something
}


Here the value of check_command must match the command_name defined above. A command is just a command: you can even differently named commands  that call the same plugin of another command but with different options. Or you can define a command with some parameters. Use the placeholders $ARG1$, $ARG2$, ... in the command_line directive above and check_something!arg1!arg2!... is the syntax to pass parameters.


Plugins can be implemented in any language. For most tasks bash is appropriate, especially if there is no much processing to do, since it makes easy to interface with all unix commands. Here is a template for a Nagios plugin written using bash:

#!/bin/bash

# Plugin description.


## Processing parameter code.
OPTIND=1


# Define and initialize vars storing parameters
warnlvl=value1
critlvl=value2

...


help() {

  cat <
usage: $(basename $0) [-w warnlvl] [-c critlvl] [file]
-w warnlvl  description [value1]
-c critlvl  description [value2]
file        description [value3]
HELP
}


while getopts "h?a:b:c" opt; do
  case "$opt" in
  h|\?)
    help
    exit 0
    ;;
  w)
    # Example check for a numeric parameter
    if [[ $OPTARG != *[!0-9]* ]]; then
      warnlvl=$OPTARG
    else
      help
    fi
    ;;
  c)
    critlvl=$OPTARG
    ;;
  *)
    help
    exit 1
    ;;
  esac
done
shift $((OPTIND-1))

[ "$1" = "--" ] && shift

# Process $1 as the file name
if [ -z "$1" ]; then
  file=default_file_name

# Gather an integer lvl describing the state of something...

if [ $lvl -gt $critlvl]; then
  echo "CHECKNAME CRITICAL: description $lvl > $critlvl"
elif [ $lvl -gt $warnlvl]; then
  echo "CHECKNAME WARNING: description $lvl > $warnlvl"
else
  echo "CHECKNAME OK: description $lvl"
fi
Rif.
https://wiki.archlinux.org/index.php/Nagios
http://users.telenet.be/mydotcom/howto/nagios
http://www.ibm.com/developerworks/aix/library/au-nagios

Make and play transcripts of terminal sessions

This is great for teaching programming or Unix usage, for producing "live" documentation for some common tasks. You may also want to make a transcript of a terminal session if you are planning to develop a script to automate the task later on.

It is advisable to save timing information along with the transcript so that you can reply it.

Record:
$ script -t transcript.tm transcript.txt

Play (you better use the same type of terminal you recorded from):
$ scriptreplay -t transcript.tm transcript.txt

scriptreplay can't do without a timing file. If you have forgot the -t option, an acceptable way to display the script could be by disabling printing of raw control characters in less using the usual caret notation (e.g. ^C):

$ less -r transcript.txt

but this won't work well with programs who clear the screen like vi(1).

Faster cipher in OpenSSH

The default cipher used with ssh and scp version 1 (3des) is very secure but slow. Version 2 improved on that with support for more ciphers and by default the fastest are used. I wanted to find out what cipher is the fastest. Warning: change the cipher only if strong security is unimportart (e.g. when transferring between two servers in a trusted LAN).

In the test script below I created a half GB binary file with random content on my Linux laptop:

laptop $ dd if=/dev/urandom of=ciphertest.data bs=1M count=512
laptop $ hexdump ciphertest.data |head
0000000 7de0 ce1a 6468 b677 31f4 e899 4271 ee91
0000010 c103 1fdf 886b b91f edf6 f05b 59a3 ec03
0000020 2f6d 47bf 92d4 d0df b695 1217 ddfe edfe
0000030 7f98 f65e e69c 94b0 5113 f66d 608a 7b49
0000040 6750 21ea ebe6 2e54 4ff1 e3c5 ac56 9ae8
0000050 f186 99a1 7c8f f9c7 95c3 8dc1 26d3 3014
0000060 a0ec 139a 62df e07c 69db 9008 7775 75dd
0000070 9009 4e56 9f5c cc2f 6ebd 08ce 5c45 e2b0
0000080 f8a6 5c08 a143 ea81 d966 416f e5b0 88c8
0000090 2eb0 0b1c 8cf9 fc35 7131 36ee 1ee4 0958

then I transfer this file to an idle VM hosted by my same PC using all ciphers available. To avoid typing the password many times, I set up key-based authentication:

$ ssh-keygen
...
$ ssh-copy-id ant@192.168.14.70

Here is the script ciphertest.sh:

#!/bin/bash
# Measures speed of different SSH ciphers.
# Before, you may want to run:
# $ ssh-keygen
# $ ssh-copy-id $USER_HOST
# to save time on typing passwords. If your key is encrypted with a passphrase
# you better fire up an ssh agent or you will have to type the passphrase many times.
# $ eval $(ssh-agent)
# $ ssh-add ~/.ssh/id_dsa

# You can find this list in ssh_config(5) CIPHERS=(3des-cbc aes128-cbc aes192-cbc aes256-cbc aes128-ctr aes192-ctr \
aes256-ctr aes128-gcm@openssh.com aes256-gcm@openssh.com arcfour128 \
arcfour256 arc‐four blowfish-cbc cast128-cbc)
USER_HOST="ant@192.168.14.70"

TMPFILE=$(mktemp)
echo -n "Generating random file. Please wait... "
dd if=/dev/urandom of=$TMPFILE bs=1M count=512
echo "done!"

for cypher in "${CIPHERS[@]}"; do
echo $cypher
scp -c $cypher $TMPFILE "$USER_HOST:ciphertest.data"
echo
done
ssh "$USER_HOST" rm ciphertest.data

rm $TMPFILE

and here are the results:

$ ./ciphertest.sh
Generating random file. Please wait... 512+0 records in
512+0 records out
536870912 bytes (537 MB) copied, 37.1943 s, 14.4 MB/s
done!
3des-cbc
tmp.pmPSfoUGqT 100% 512MB 13.5MB/s 00:38

aes128-cbc
tmp.pmPSfoUGqT 100% 512MB 46.6MB/s 00:11

aes192-cbc
tmp.pmPSfoUGqT 100% 512MB 46.6MB/s 00:11

aes256-cbc
tmp.pmPSfoUGqT 100% 512MB 42.7MB/s 00:12

aes128-ctr
tmp.pmPSfoUGqT 100% 512MB 51.2MB/s 00:10

aes192-ctr
tmp.pmPSfoUGqT 100% 512MB 46.6MB/s 00:11

aes256-ctr
tmp.pmPSfoUGqT 100% 512MB 46.6MB/s 00:11

aes128-gcm@openssh.com
tmp.pmPSfoUGqT 100% 512MB 42.7MB/s 00:12

aes256-gcm@openssh.com
tmp.pmPSfoUGqT 100% 512MB 42.7MB/s 00:12

arcfour128
tmp.pmPSfoUGqT 100% 512MB 51.2MB/s 00:10

arcfour256
tmp.pmPSfoUGqT 100% 512MB 46.6MB/s 00:11

arcfour
tmp.pmPSfoUGqT 100% 512MB 51.2MB/s 00:10

blowfish-cbc
tmp.pmPSfoUGqT 100% 512MB 32.0MB/s 00:16

cast128-cbc
tmp.pmPSfoUGqT 100% 512MB 32.0MB/s 00:16

Installing text-mode Arch Linux in Virtualbox

First install Virtualbox. An installer for Windows is available on the Virtualbox site download page. In another Arch Linux box:

host # pacman --noconfirm -S virtualbox
host # gpasswd -a ant vboxusers
host # echo vboxdrv >/etc/modules-load.d/virtualbox.conf

Replace ant with your username, and use a newly open terminal to run Virtualbox:

master $ Virtualbox &

Then New, Name: Arch Linux Tests, 256 MB, Dynamically allocated virtual HD, 8GB.

Then head to the Arch Linux download page and download latest image, e.g. using Torrent. Machine, Settings, Storage, Controller IDE, Empty, click on the small CD-ROM button on the right hand side, Choose a Virtual CD/DVD disk file, point to archlinux-2013.05.01-dual.iso. Start, Boot Arch Linux (x86_64). Make sure your host computer stays connected to the Internet, because some commands you issue on the guest will need a working network connection.

If you do not have a US keyboard issue:

guest # loadkeys keymap

where keymap is the value of KEYMAP you find in this table row for your country, or using this command (ignore the .map.gz extension and use only the file name as the keymap variable value):

guest # ls /usr/share/kbd/keymaps/i386/qwerty

Then:

guest # ntpd -qg
guest # hwclock -w

guest # cgdisk /dev/sda

New, accept all defaults, then Write and confirm by typing yes. Quit.

guest # mkfs.ext4 /dev/sda1
guest # mount /dev/sda1 /mnt
guest # pacstrap /mnt base base-devel
guest # genfstab -U -p /mnt >> /mnt/etc/fstab
guest # arch-chroot /mnt pacman --noconfirm -S syslinux gptfdisk
guest # arch-chroot /mnt /bin/bash
guest # mkinitcpio -p linux

guest # echo "archlinux" > /etc/hostname
You may choose a different hostname than archlinux.

guest # echo "KEYMAP=keymap" > /etc/vconsole.conf

where keymap is that chosen before (e.g. us).

guest # ln -s /usr/share/zoneinfo/Europe/Rome /etc/localtime

Change Europe and Rome if you are not in Italy.

guest # sed -ri -e 's/^#(en_US.UTF-8)/\1/' /etc/locale.gen
guest # locale-gen
guest # echo LANG=en_US.UTF-8 > /etc/locale.conf
guest # hwclock --systohc --utc

guest # passwd

Set a root password.

guest # syslinux-install_update -iam
guest # sed -ri -e 's/sda3/sda1/' /boot/syslinux/syslinux.cfg

guest # exit
guest # umount /mnt
guest # poweroff

Machine, Settings, Storage, right click on archlinux-2013.05.01-dual.iso, Remove Attachment, Remove, Machine, Start. Login as root.

guest # useradd -m -g users -s /bin/bash ant
guest # passwd ant

guest # systemctl enable dhcpcd@enp0s3.service
guest # systemctl start dhcpcd@enp0s3
guest # pacman --noconfirm -S virtualbox-guest-utils
guest # echo -e 'vboxguest\nvboxsf\nvboxvideo' >/etc/modules-load.d/virtualbox.conf
guest # modprobe -a vboxguest vboxsf vboxvideo

Console mouse support:
guest # pacman --noconfirm -S gpm
guest # systemctl start gpm.service
guest # systemctl enable gpm.service

Synchronize clock with host:
guest # systemctl enable vboxservice.service
guest # systemctl start vboxservice.service

Optional: ability to locate files:
guest # pacman --noconfirm mlocate

Optional: text mode browser

guest # pacman --noconfirm elinks

Optional: Support for host-only and bridged network interface:
guest # pacman --noconfirm -S net-tools
host #  pacman --noconfirm -S virtualbox-host-modules
host # modprobe -a vboxnetadp vboxnetflt vboxpci
host # echo -e 'vboxnetadp\nvboxnetflt\nvboxpci' >>/etc/modules-load.d/virtualbox.conf
host # pacman --noconfirm -S net-tools

Optional: install yaourt
guest # pacman --noconfirm -S wget yajl
guest # su - ant
guest $ cd /tmp
guest $ wget https://aur.archlinux.org/packages/pa/package-query/package-query.tar.gz
guest $ tar zxf package-query.tar.gz
guest $ cd package-query
guest $ makepkg -c
guest $ su -c 'pacman --noconfirm -U package-query-*.tar.gz'
guest $ cd ..
guest $ wget  https://aur.archlinux.org/packages/ya/yaourt/yaourt.tar.gz
guest $ tar zxf yaourt.tar.gz
guest $ cd yaourt
guest $ makepkg -c
guest $ exit
guest # cd /tmp/yaourt
guest # pacman --noconfirm -U yaourt-1.3-*.tar.xz

Optional: ssh access (useful if you have to copy and paste a lot of commands between host and guest or viceversa). Requires bridge interface, see above.
guest # pacman --noconfirm -S openssh
guest # systemctl start sshd
guest # systemctl enable sshd.service
guest # poweroff
Settings,Network,Attached to,Bridged Adapter,Start.

Ref.
https://wiki.archlinux.org/index.php/Installation_Template
https://wiki.archlinux.org/index.php/Installation_Guide
https://wiki.archlinux.org/index.php/Beginners'_Guide#Hostname
https://wiki.archlinux.org/index.php/Virtualbox
https://wiki.archlinux.org/index.php/Yaourt

Recovering a deleted file in Linux

As long as there is still a process holding a file open, a deleted file can still be recovered.

Here we simulate the process. In one terminal do:

$ cd /tmp/
$ cat>deleted_file
some thoughts
go here

^D
$ tail -f deleted_file
some thoughts
go here

Since -f blocks this terminal, open another one and do:

$ cd /tmp/
$ lsof deleted_file
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
tail 32664 ant 3r REG 0,29 22 583000 /tmp/deleted_file
$ rm -f deleted_file
$ lsof deleted_file
lsof: status error on deleted_file: No such file or directory
...
$ lsof | grep deleted_file
tail 32664 ant 3r REG 0,29 22 583000 /tmp/deleted_file (deleted)
$ cat /proc/32664/fd/3
some thoughts
go here
$ cp /proc/32664/fd/3 deleted_file
or
$ cat /proc/32664/fd/3 >deleted_file
$ cat deleted_file
some thoughts
go here

You can only reliably recover deleted files that are still open though, e.g. typically database or log files.

KVM Tutorial on Ubuntu Server

Alternatives

One alternative is LXC (Linux Containers), which is a lightweight virtualization method to run multiple virtual units (containers, akin to chroot). KVM has much better isolation than LXC, but the latter is more performant.

Installation

configuration files policy

A useful convention, before editing a configuration file for the first time, is to make a copy of it in the SAME directory where it is located but with a .bak-default extension added. I have authored a script /opt/bak/sbin/bak which backs up the configuration into /root/bak-HOSTNAME-HOSTID.tgz (only readable by root). This convention also allows to see what changed in each configuration file with respect to the default version shipped by upstream, so you can know what has been changed at a glance:

# vimdiff -o file.conf file.conf.bak-default

Bare OS

Use the 64-bit alternate install CD from the LTS Ubuntu release download page. Be sure to press F4 and select "Install a command-line system" in order to get a minimal installation and preferably select expert mode. Set up LVM Disk to partion sda. LVM provides VMs with direct filesystem access, speeding up disk I/O considerably. Allocate enough space for logical partitions in the volume group: a swap partition (e.g. equal to the amount of memory) and use the rest of the space as a single root partition (about at least 5GB). Bet sure to install openssh-server as an additional package. Set the system clock to UTC. Re-enter the BIOS and reset the boot orderif you changed it in order to boot from the DVD.

Network

After first boot, configure network (static IP and bridge interface so that guests can have full LAN access).

apt-get install bridge-utils

Change these lines in /etc/network/interfaces:

auto eth0 iface eth0 inet dhcp

to:

auto eth0 iface eth0 inet manual auto br0 iface br0 inet static         address SERVER_IP_ADDRESS_EG_192.168.1.1         netmask NETMASK_EG_255.255.255.0         network NETWORK_EG_192.168.1.0         broadcast BROADCAST_EG_192.168.1.255         gateway GATEWAY_EG_192.168.1.254         dns-nameservers DNS_EG_192.168.1.254         bridge_ports eth0         bridge_fd 9         bridge_hello 2         bridge_maxage 12         bridge_stp off

bridge_fd is the forwarding delay for interfaces joining the bridge. It's how long it'll be before the interface will be able to do anything. During this time the bridge will be discovering other bridges and checking that no loops are created. For a better description and the reason for it you need to read up on spanning tree protocol. See brctl(8) for an introduction. Anyway the spanning tree protocol is turned off by bridge_stp off, since there could not be any loops for this topology. See bridge-utils-interfaces(5)
After this change you can either reboot:

# reboot

or just reload the network:

# /etc/init.d/networking restart Running /etc/init.d/networking restart is deprecated because it may not enable again some interfaces Reconfiguring network interfaces...                                             ssh stop/waiting ssh start/running, process 2292 Waiting for br0 to get ready (MAXWAIT is 20 seconds). ssh stop/waiting ssh start/running, process 2559                                                                             [ OK ]

Reconnect and check the bridge has been created:

# ifconfig # brctl show # brctl show br0

Note virbr0 is another bridge created automatically when installing KVM, but that's only for NAT connectivity of VMs, so br0 still needs to be created.

KVM installation

Make sure there is hardware support for virtualization:

# apt-get install cpu-checker # kvm-ok INFO: /dev/kvm exists KVM acceleration can be used

Install the virtualization shell (virsh) and other programs for Kernel-based virtualization:

# apt-get install libvirt-bin

Add the user used to manage VMs to this group and be sure to log out and log back in for new privileges to be effective or the current user:

# adduser root libvirtd # adduser root kvm

If you want to create Ubuntu-based VMs as described below, install this handy script too:

# apt-get install python-vm-builder

Usage

Managing the VMs

Use virsh, the virtual shell:

# virsh --connect qemu:///system

Get a list of available commands:

virsh # help

Show running VM:

virsh # list

Show all VM:

virsh # list --all

Update VM configuration (to be done before the first boot and at every configuration edit):

virsh # define /etc/libvirt/qemu/vm1.xml

Start/shutdown/suspend/resume/pull the power plug of a VM:

virsh # start/shutdown/suspend/resume/destroy vm1

First login (e.g. with mandatory password change):

# ssh MY_LOGIN_NAME@VM_IP_ADDRESS password: TEMPORARY_PASSWORD

Change your password now and login again.

Remove a VM:

virsh # shutdown vm1 virsh # undefine vm1

Edit configuration, e.g. change cores, memory, etc:

# virsh edit vm1

Here is a guide about interpreting CPU load values, which can help to decide how many cores are needed.

Mount/change a CD-ROM iso (keep ISO images into /var/lib/libvirt/images):

# virsh attach-disk vm1 /path/to/image.iso hdc --driver file --type cdrom --mode readonly

Remove a CD-ROM iso:

# virsh attach-disk vm1 " " hdc --driver file --type cdrom --mode readonly

Creating an LVM-Based Ubuntu VM (headless installation)

In this example I am creating a simple Ubuntu JeOS machine named lemon, with 2 GB memory, dual core and a 200G disk mapped to one logical volume. For more details see vmbuilder(1):

# lvcreate -n lemon -L 421888 VOLUME_GROUP # vmbuilder kvm ubuntu --libvirt qemu:///system --suite=precise --flavour=virtual --arch=amd64 --mirror=http://de.archive.ubuntu.com/ubuntu -o --ip=IP_EG_192.168.1.2 --gw=GATEWAY_EG_192.168.1.254 --mask=NETMASK_EG_255.255.255.0 --dns=DNS_EG_192.168.1.252 --user=YOUR_USERNAME --name="YOUR NAME" --pass=YOUR_PASSWORD --addpkg=acpid --addpkg=openssh-server --mem=2048 --hostname=lemon --bridge=br0 --raw=/dev/VOLUME_GROUP/lemon --part=/tmp/vmbuilder.partition

If you do not specify neither --rootsize nor --swapsize nor, vmbuilder will default to 4 GB and 1 GB respectively and the rest will be left as free space, this is probably not what you want. For advanced partition schemas you better use --part.

option	meaning
kvm	hypervisor
ubuntu	distro
--libvirt qemu:///system	Needed if you want to use Virsh to manage your virtual machines
--arch	i386 or amd64
--mem	virtual RAM in megabyte
--bridge	bridge interface to connect the VM to
--ip	static IP address
--gw	gateway address
--mask	netmask
--dns	DNS server IP
--raw	raw device to create the partitions in, e.g. an LVM logical volume
--rootsize	Root FS size, default 4096. Ignored if --part is used.
--swapsize	Swap FS size, default 1024. Ignored if --part is used.
--part	text file containing partition layout, e.g.: root 10240 swap 2048 /data 409600

# virsh start lemon

Cloning a VM

First, make sure the VM is shut down:

# virsh list --all  Id Name                 State ----------------------------------   3 vm1                     running

Supposing we want to clone vm1, we first have to stop it:

# virsh shutdown vm1

Wait a bit for the machine to shut down, then you can check its state:

# virsh list --all  Id Name                 State ----------------------------------   - vm1                     shut off

Once it has been shut down, you can clone it, in this example to a VM called vm1clone:

# lvcreate -L1024G -n vm1clone VOLUME_GROUP_NAME # virt-clone -o vm1 -n vm1clone -f /dev/VOLUME_GROUP_NAME/vm1clone

If you are cloning from a smaller to a bigger virtual disk, you need to extend both the partition and the filesystem in order for the guest OS to make use of all space. First, use virt-manager to log in to the VM and re-create the partition using fdisk, then reboot and resize it, as explained here:

# fdisk /dev/vda p d 1 n p 1 p w # reboot # resize2fs /dev/vda1

Anyway, the ext3/4 online resize algorithm can be slow if your partition is big. A slightly faster approach to clone a small VM into a big disk is to start from an empty filesystem and copy the directory structure through the virtual network. But formatting a big partition (e.g. 1 terabyte) can also be slow, do not expect miracles. See also this serverfault page about speeding up formatting for large partitions.

Snapshotting a VM

Please refer to the Ubuntu Wiki for background informations and details. Here is an example of snapshotting the vm1 VM and mount the snapshot read-only (e.g. for backup purposes):

# lvcreate -s -n vm1-snap -L 2G VOLUME_GROUP/vm1

  Logical volume "vm1-snap" created

# kpartx -av /dev/VOLUME_GROUP/vm1-snap

add map VOLUME_GROUP-vm1--snap1 (252:11): 0 2147472747 linear /dev/VOLUME_GROUP/vm1-snap 63

# mkdir -p /vm-mnt/vm1-snap

# mount -t ext4 -o ro /dev/mapper/VOLUME_GROUP-vm1--snap1 /vm-mnt/vm1-snap

... now modify vm1 and see that vm1-snap remains the same ...

You can see how much space the snapshot is using:

# lvs

  LV           VG    Attr   LSize   Origin  Snap%  Move Log Copy%  Convert

  ...

  vm1          volum owi-ao   1.00t

  vm1-snap     volum swi-ao   2.00g wm1       0.01

  ...

# umount /vm-mnt/vm1-snap

# kpartx -d /dev/VOLUME_GROUP/vm1-snap

# lvremove /dev/VOLUME_GROUP/vm1-snap




Creating an LVM-based CentOS VM (headless installation)

This is rather specific. I needed to run the old CentOS 5.5 in a VM. We will create a new logical volume for holding ISO images to install OSs from, format it using XFS (which is optimized for large files) and mount in /var/lib/libvirt/images/. We will also download an old Cent OS 5.5 image from multiple sites, using a download accelator:

# lvcreate -n images -L 20g VOLUME_GROUP   Logical volume "images" created # lvs   LV     VG    Attr   LSize  Origin Snap%  Move Log Copy%  Convert   images volum -wi-a- 20.00g                                        root   volum -wi-ao  5.28g                                        swap_1 volum -wi-a- 31.97g # apt-get install xfsprogs # mkfs.xfs -b size=4096 /dev/VOLUME_GROUP/images # cd /etc # cp fstab fstab.bak-default # echo '/dev/mapper/VOLUME_GROUP-images /var/lib/libvirt/images/ xfs defaults,noatime,nodev,nosuid,noexec 0 99' >>fstab # mount /var/lib/libvirt/images/ # apt-get install aria2 # aria2c http://download.filesystems.org/linux/centos/CentOS-5.5-i386-bin-DVD.iso http://mirror.teklinks.com/centos/5.5/isos/i386/CentOS-5.5-i386-bin-DVD.iso http://media.kombinasi.net/~download/iso/centos/CentOS-5.5-i386-bin-DVD/CentOS-5.5-i386-bin-DVD.iso ftp://ftp.sandy.ru/pub/Linux/CentOS/CentOS-5.5-i386-bin-DVD/CentOS-5.5-i386-bin-DVD.iso ftp://nephi.unice.fr/linux/centos/distrib/5.5/CentOS-5.5-i386-bin-DVD.iso

Now let's create an LVM-based CentOS 5.5 VM:

# lvcreate -L12G -n centos-5.5 VOLUME_GROUP # apt-get install virtinst # virt-install -n centos-5.5 -r 1024 --vcpus=2 --disk path=/dev/VOLUME_GROUP/centos-5.5 -c /var/lib/libvirt/images/CentOS-5.5-i386-bin-DVD.iso --graphics vnc --noautoconsole --os-type linux --os-variant=rhel5.4 --network=bridge:br0 Starting install... Creating domain...                                                             |    0 B     00:00     Domain installation still in progress. You can reconnect to the console to complete the installation process.

Enable root login b/c it is needed to connect via virt-manager from your client machine and complete the graphical installation:

# sudo passwd root

On your client PC start:

$ virt-manager

then File, Add Connection... etc. Once connected select the connection by name in the main windows and click Open to start a VNC session (you will be asked for the host root password again). Once the installation is completed, the VM will be shutdown. Right click on it and select Run, then hit the Open button again to see its console.

Host reboot

Every now and then, e.g. because of system updates or for stability reasons, the host OS will need to be rebooted. Unfortunately that implies all the guest OSs must be restarted too. Unfortunately, you cannot pause a guest system and resume its execution state after a reboot of the host. This feature is unfortunately not implemented in KVM. To mitigate this problem, you can schedule a nightly reboot of the host using this script, which first shuts down all guests (starting shutdown requests in parallel to reduce overall downtime), and checks their state before rebooting the host:

#!/bin/bash
# Safely stops all VMs before rebooting the system.
# Useful to schedule a nightly reboot with cron(8).
# See: https://lists.ubuntu.com/archives/ubuntu-server/2011-May/005663.html

for vm in $(virsh -q list | awk '{ print $2 }'); do
{
virsh shutdown $vm;

# WAIT until the machine is really powered off
until [ "$(virsh -q list --all | awk '{if ($2 == "$vm") print $3" "$4}')" = 'shut off' ]; do
sleep 3
done
} &
done

# Wait until all the machines are powered off.
wait

# Now all VMs are shutted down.
reboot "$@"
I called this script kvmsafe-reboot and it is implemented just as a reboot(8) wrapper, so it should be used instead of reboot. Please remember to set the 'autostart' flag on all domains you want to come up after rebooting:
# for i in vm1 vm2 vm3...; do virsh autostart $i; done
Also make sure every VM has the acpid daemon running.

Monitoring host and/or VM performance

There are many tools to do this. A simple text-mode solution is sysstat. You can also install a graphs generator:

# apt-get install zenity gnuplot xsltproc
# cd /usr/local/bin
# gunzip -c /usr/sysstat/examples/sargraph.gz >sargraph
# chmod 755 sargraph

To use it, from your client machine type:
$ ssh -X host_machine sargraph

Look at the sar(1) manual page for a clear explanation of the various output parameters.

Backups

In my Github repository I am sharing a script to back up the system and some of the VMs on an external USB disk, e.g. put it in /usr/local/sbin/snapbak.sh It's configuration has to be in /usr/local/etc/snapbak.conf (or change the path at the beginning of snapbak.sh) E.g. you can configure here the VMs to backup. You may want to label the USB disk (e.g. as SNAPBAKUSB) so that if the device name changes, it can still be mounted automatically:
# e2label /dev/sdc1 SNAPBAKUSB

then add this line to /etc/fstab:
# echo 'LABEL=SNAPBAKUSB /mnt/hdd-usb ext3 noauto,defaults 0 0' >>/etc/fstab
Not: After a warm reboot, the USB disk was not recognized and cannot be mounted, so backups will fail unless one goes in the server room and manually disconnects and reconnects the disk. I have added usb_storage to /etc/modules to solve this problem.

References

• Ubuntu Official Documentation: Virtualization
• Ubuntu Documentation, Advanced Installation, Logical Volume Manager (LVM)
• Ubuntu Wiki, Logical Volume Management (LVM)
• KVM/qemu - use LVM volumes directly without image file?