Here's a screenshot of the final result to give you an idea of what you will learn to code.
Tutorial
Hello people! In this post I will show how to code your own static blog in HTML5 and begin blogging by minutes. This is for HTML beginners and a step-by-step, learn-by-doing tutorial. Yeah, there are many ready-to-use blog platforms that do not require you to hand-code everything, including this, Blogger. But making your own blog from scratch is a good way to learn about HTML, CSS and how web sites are made.
You are encouraged to try out all the code snippets in your browser to see how they render and what the effect of each tag is. Copy and paste each sample in a simple PLAIN text editor, that is Notepad, not Word and save the result as index.html, then click on it and it will be opened with your browser. After each modification, save the new version in Notepad and hit your browser's reload button to make it aware of the code change and redisplay the page.
Let's start from an empty HTML page:
Now, what is a blog? It's a collection of posts (articles), displayed from the most recent one. That's the page "main content" - posts - but there is also a page header with the name of the blog and a tagline.
You choose how many posts to put in any page, from one to many, depending on how long they are. Now, let's detail each of the posts content. Post have a title and consist usually of paragraphs of text, but you may also want to post a video or an image and you can intermingle text and images/video as you wish. Each post has a publication date that we chose to put in the article footer:
<!DOCTYPE html>
<html>
<head>
<meta name="generator" content=
"HTML Tidy for HTML5 for Linux version 5.2.0">
<title>YOURNAME's Blog</title>
</head>
<body>
<header>
<p>My daily thoughts...</p>
<h1>YOURNAME's Blog</h1>
</header>
<main>
<article>
<h1>Text post</h1>
<p>First paragraph... ends here.</p>
<p>Another paragraph.</p>
<footer>
<p>Posted <time datetime="2016-11-20T18:45+01:00">on 2016/11/20 at
6:45pm</time></p>
</footer>
</article>
<h1>A picnic on the lake</h1>
<article><video src="vids/picnic.mp4" controls preload>
<p><a href="vids/picnic.mp4">Download my picnic video</a>.</p>
</video>
<footer>
<p>Posted <time datetime="2016-11-19T13:34+01:00">on 2016/11/19 at
1:34pm</time></p>
</footer>
</article>
<article>
<h1>My pets</h1>
<p><img src="imgs/fido.jpg" alt="Fido">This is my dog Fido. He
needs my attention all the time!</p>
<p><img src="imgs/alice.jpg" alt="Alice">And this is my kitty,
Alice! Do not disturb her when she's sleeping...</p>
<footer>
<p>Posted <time datetime="2016-11-17T10:03+01:00">on 2016/11/17 at
10:03am</time></p>
</footer>
</article>
</main>
</body>
</html>
Note the IMG tag alt attribute is for the benefit of visually impaired people, search engines or users that have disabled image loading in the browser, e.g. to save bandwidth on slow connections. It defines a replacement (ALternate Text) for images, which will be shown if the image doesn't exist or is deleted too. You may want to describe the picture better, e.g. "My kitty-cat Alice sleeping on the sofa", instead of just "Alice". For those browsers that cannot display embedded videos, a download link is provided inside the VIDEO tag. Note also to keep things tidy we collect all images in a imgs/ subfolder and likewise on videos.
It is a good idea to use the TIME tag to provide a machine-readable format for the date and time. This is a format standardized by ISO and includes a numeric timezone - that +01:00 part, which means 1 hour ahead of Greenwich standard time.
We then need to add navigation links. The first time you want to create a new page, you copy index.html to 2.html and delete articles so to get an empty blog page you can add new articles to. You also rename index.html to 1.html. After the closing tag, you add a page footer with a navigation menu in both files. In the same footer you can also fit a copyright notice, if you wish. Then you create index.html as a link to 2.html, which is the latest page.
In 1.html:
You now probably got what to do when adding a third page. Remember to update the file link, making index.html point to 3.html and activate the "Newer Posts" clickable link in 2.html to point to 3.html. If all pages are linked together like this, visitors will be able to browse through your blog as it were a book to leaf through and you will make sure a search engine will index your entire blog. You only need to add the index.html page address in the "Add site to index" service of the search engine or make sure your blog is linked by another already-indexed site.
As you have seen, with HTML you define the logical structure of the page, but not its visual appearance. If you display 2.html with a browser now, you will see it looks pretty dull, because the default formatting styles for tags are basic and simple. To spice up your blog, you need to change the presentation style of the structure you defined with HTML. This requires you to have a bit of the typesetter's creativity and is done with another language, called CSS (Cascading Style Sheets). You will put the style definitions in a separate file so that they will be shared amongst all pages. In all pages you will need to add a tag to link to the stylesheet. Nest that into the HEAD tag:
Now in blog.css you will add style rules one by one and reload the blog page in the browser to check the result. E.g. I wish a kind of golden yellow background for the main content (posts):
main {
background-color: gold;
}
Fortunately, you can simply use plain-English color names. I also want to have some space between posts' text and the borders of the main page content. It's a good idea, while you are coding a style, to turn on the display of the border of each element, so you can actually see the structure of your page. This is the purpose of the first rule with * meaning any tag name (any HTML element). I chose a dotted border because a solid one clutters the page too much:
"em" is rough relative typographical measure, meaning the size of an "M" in the current font, so 2em means 2 times the size of the current font. You probably notice the first article's title is not at the same vertical and horizontal distance from the main content borders. The vertical distance is more. Why is that? You should now each element has not only padding space inside the borders that delimit it, but also margin space outside the borders that separate it from other elements. By default, an h1 element has such a top and bottom margin space. So to make distances even, we just need to set to zero out the top margin of the h1 element, but of the first article only. In CSS lingo this is done like so:
article:first-of-type h1 {
margin-top: 0;
}
Let's say we want some words to be highlighted in the article text. There is a STRONG tag, which gives strong emphasis to the words it surronds. Let's wrap your beloved pets' names with that. You will see they are rendered bold. I only want to change the emphasised text color from default black to red:
strong {
color: red;
}
By default images are put inline with the text surrounding them, with the image bottom aligned to the bottom of the text line that contains it. Make the sentences about your pets very long, reload the page and you will see what I mean. Unfortunate default behaviour! We rather want to wrap text around the image. Here is how to tell the browser to do that for every image inside a paragraph:
p img {
float: left;
}
We chose to float images to the left - you can also use "right". But look what may happen: if text around an image is not long enough to fill up all the the image height, whatever follows will flow to the right of the image too. We better not let cats and dogs run into each other :-)
So we need a way to clear the wrapping and tell the browser to return to normal, default flow. Thinking about it, we may have many short paragraphs of text that should wrap around an image. So we better put the IMG tag outside, before the first P tag that should wrap around it. Then we will tell the browser to stop wrapping text when a new image or anything else we do not want to wrap around the image follows. How to do that? There is a special element with no semantic associated, called DIV for DIVersion, which you should use when you need to wrap two or more elements for styling purposes only, and, I repeat, there is other meaning attached. By giving the DIV a class name, you can easily style all DIVs with the same class name in the CSS.
So here's the correct HTML and CSS for inserting images between text, wrapping the text at the left or right of an image, with added space between them and even a nifty 1-pixel image border added. Everything inside the div will be wrapped around the image, but what follows the div, whatever it is, won't. E.g. the selector div.right_img + * means to apply a style rule to all (*) sibling tags following (+) DIVs with the right_img class (div.right_img). For these we do not want any floating elements to the left or the right (clear:both).
Now that you can format illustrated books (give it a try), we can refine the overall fonts. We want the characters of the main text to be a bit bigger than the default. This applies to the paragraphs inside an article, not the P in the header, so we can't just write P as a selector. "article p" means any p nested inside an article (a "child of" article). Larger here means larger than the font size of the parent element (article), which is the default one, so a bit larger than the browser default.
I also want smaller, italicized article footers, because they are less important than the text itself and I do not want them to stand up too much. And small-caps font for the post titles. In a small-caps font, all lowercase letters are converted to uppercase letters. However, the converted uppercase letters appears in a smaller font size than the original uppercase letters in the text.
Let's add a thick black border to tell the main content apart from the page header and footer. Change the top rule for main so that it reads as follows:
And embellish the page footer with colors... Note that after setting a red background, the default bluish link color is no more appropriate: it does not contrast well on darkred. So now we change both visited and unvisited link colors for the footer. We also wish to center the page footer content, which contains the NAVigation menu and a copyright note. They are both text paragraphs, taking up all the horizontal space available, so we just need to center the text inside their boxes:
body > footer {
background-color: darkred;
color: white;
padding: 1em;
}
body > footer p {
text-align: center;
}
body > footer a:link, body > footer a:visited {
color: gold;
}
Note we want to style only the footer at the end of the page, not the ones belonging to each post. body > footer means footer that is the direct child of body and thus excludes those FOOTERs that are the direct child of article. > means direct child, not descendant. The latter could be child, nephew, grand-nephew, etc. Note we add some padding space to the footer, which also makes it taller.
Our last CSS feat is the styling of the site headline. The artist instinct in me wants that buzzword - "blog" - in very big bold red text and the owner name just above it in smaller all capitals. The tagline to the left, bigger than the name but smaller "blog". Since both the blog owner name and the word "blog" are part of the H1 header, in order to style them separatedly, we need the SPAN tag. This tag has no structural meaning, much like DIV, but while the latter is a "block-level" element, SPAN is an "inline-level" element. Nah, just big words to say that SPAN applies to a portion of text without breaking it with a newline before and after itself. You can't nest a DIV inside an H1 because that would break the H1 element in two, which does not make much sense to do. Change your HTML in both files so that the site title looks like this:
Some tags are given an ID attribute - guess what - to identify them. We do not use a CLASS attribute, though we could. An ID is more appropriate when there is one and only one tag to style. ID values must be unique throughout all the document. IDs are referred in the CSS with a hash mark before the chosen nickname. In order to center two block elements side-by-side, I had turned them to inline-block, so that for the purpose of presentation, they behave just like inline elements. You may want to try the effect of the added and/or modified CSS rules one by one, for a better understanding:
HTML and CSS are not easy. Everything should be easier. It wouldn't be difficult to accomplish the goal of keeping structure and style separated using a what-you-see-is-what-you-get (WYSIWYG) editor without writing any line of code. The main reason why this is hard to do with the current web standard is the way CSS works and is defined. CSS is a tricky language and not very general and powerful enough for being easily generated by a GUI. The great diversity of devices capable of displaying web pages can make it harder to make the page look the same everywhere, but a stricter standard would have helped to achieve that. Unfortunately we have to cope with what we have. As a consequence web development remains difficult, slow and quite expensive. Not within any internet user's reach, as it should be.
Rationale: I was a computer expert for over 10 years, but I was never been able to find and intelligent job, whereby I could apply what I know about computer science. I have always been ordered about by people much more stupid than me, who wanted me to use the wrong tool for the job, just like knocking a nail with a chainsaw or worse with your bare hands. Lately I become an independent organic farmer and, surprisingly, I can finally apply all I know about computers, although there aren't any in my farm!
One application is a circular queue to differentiate between fresh and old eggs. My chickens now lay about 5-6 eggs per day. I only eat two eggs a day and put the rest aside to make sweets or for possible guests.
I always want to eat eggs on a FIFO basis: First egg In, First egg Out, meaning I always use the oldest one, so to avoid some eggs are eaten too late and go bad. I reused an egg cardboard of an egg carton I took from the supermarket and numbered each of its eggcups, up-bottom and left-to-right.
Starting from 1, I keep adding eggs to the end of the queue and take ones to eat from the start. Initially, the queue head is the lowest number filled by an egg, while its tail is the highest number, but when you reach the end of the queue - 30 in my case - this is not true anymore. E.g. in this picture
the oldest egg is number 20 and egg number 1 comes just after egg number 30! This is why it is called a "circular" queue: although the cardboard shape is squared, you have to imagine its egg cups as arranged in a circle, so that 1 really comes after 30 physically. Queued eggs all make one single circular piece of "snake", which changes its length and goes around the circle clockwise or counterclockwise as eggs are added and taken, depending on your convention.
Now look at this picture:
Here I have 29 eggs and only one eggcup free, which happens to be number 26. Oldest egg is thus number 27 - this is the one I would eat first, while latest laid is 25. You can't add another egg in eggcup 26, since if you do it, you would not be able to tell fresh from old eggs, that is the beginning and end of the queue. If you put some markers, you could fill the queue completely, but for the sake of simplicity, I prefer to lower the queue capacity by 1 rather than having to use head and tail markers. This way queue head and tail is located visually, with no need for markers. Of course, your mind implements an algorithm to do it. If you are a programmer, can you formalize it in a programming language?
BTW If you want to store more than 30 eggs, of course you need another cardboard. You can just have two separate pieces or cardboard and remember which is the first one, or you can glue them together so that you have a 30*2=60-egg circular queue or even more if you have so many eggs to store - and chickens to feed!
The bottom line: chickens are more intelligent than IT managers. Oh well, do not take it personally if you are one of them.
MariaDB is an open-source replacement for MySQL. MariaDB keeps a so-called binary log of transactions, which keeps track of all updates to the database. It is used for database restoration and replication, but in this tutorial we will focus on replication only. Keeping a binary log also proves useful if you need the ability to do point in time recovery from your latest backup, but we will not see that.
Optional - for mytop to work: master # pacman -S perl-dbd-mysql perl-term-readkey
Optional - auto completion of table, field names, etc. in the client master # sed -ri -e 's/^no-auto-rehash/auto-rehash/' /etc/mysql/my.cnf
master # systemctl start mysqld
I recommend answering yes to all questions and setting a MySQL root password: master # mysql_secure_installation
master # systemctl restart mysqld Try connection:
master # mysql -p
Here is courses.mysqldump, a very simple relational database: DROP DATABASE IF EXISTS courses; CREATE DATABASE courses; USE courses; DROP TABLE IF EXISTS offering, course, instructor;
CREATE TABLE course ( id int(10) unsigned NOT NULL AUTO_INCREMENT, title varchar(255) NOT NULL, PRIMARY KEY (id) );
CREATE TABLE instructor ( id int(10) unsigned NOT NULL AUTO_INCREMENT, name varchar(255) NOT NULL, PRIMARY KEY (id) );
CREATE TABLE offering ( course_id int(10) unsigned NOT NULL, instructor_id int(10) unsigned NOT NULL, PRIMARY KEY (course_id, instructor_id), FOREIGN KEY (course_id) REFERENCES course(id) ON DELETE CASCADE ON UPDATE CASCADE, FOREIGN KEY (instructor_id) REFERENCES instructor(id) ON DELETE CASCADE ON UPDATE CASCADE ); Then insert some sample data:
master # mysql -p courses MariaDB [courses]> INSERT INTO course VALUES (1,'Basic Linux Programming'); MariaDB [courses]> INSERT INTO instructor VALUES (1, 'Antonio Bonifati'); MariaDB [courses]> INSERT INTO offering VALUES (1,1); The binary log is active by default: master # cat /var/lib/mysql/mysql-bin.index ./mysql-bin.000001 ./mysql-bin.000002 ./mysql-bin.000003 ./mysql-bin.000004 master # file /var/lib/mysql/mysql-bin.000001 /var/lib/mysql/mysql-bin.000001: MySQL replication log It's default size is 1 GB (max_binlog_size): master # mysqld --verbose --help 2>/dev/null | grep ^max-binlog-size max-binlog-size 1073741824 As you can see in /etc/mysql/my.cnf, by default MariaDB is configured as a master and mixed binary-logging format is used:
# Replication Master Server (default) # binary logging is required for replication log-bin=mysql-bin # binary logging format - mixed recommended binlog_format=mixed On the master, we also need to create an account that will be used by the slave to connect and start replicating. We can reuse a normal user account, and just grant the REPLICATION SLAVE permission, but it is more secure to have a dedicated account (e.g. repl) with this privilege only: master # mysql -p -e "GRANT REPLICATION SLAVE ON *.* TO 'repl'@'SLAVE_IP' IDENTIFIED BY 'REPL_PWD'; FLUSH PRIVILEGES"
Remember to replace SLAVE_IP with the slave IP address before issuing the above command.
Each master or slave in the same replicating group must have a unique server_id. E.g. I will leave the master server_id at its default value (1) and give a different value to the slave (e.g. 2): master # mysqld --verbose --help 2>/dev/null | grep ^server-id
server-id 1
Slave configuration
Since I do not have another machine for tests, I have installed another copy of Arch Linux in Virtualbox to serve as a slave. The host is the master server itself (my laptop). On the slave, I installed and secured MariaDB exactly as done before for the master.
Then assign the slave a 32-bit number different from the master (e.g. 2). You do this by editing /etc/mysql/my.cnf, commenting the line
server-id = 1
and uncommenting a similar one a few lines below:
# server-id = 2
Also, since this is a slave, disable the binary log by commenting the following lines:
log-bin=mysql-bin
binlog_format=mixed
Actually you can leave this lines (or luncomment another line log-bin=mysql-bin just below in the section related to slave configuration), if you want to have a chained replication setup. That is this slave will write to a binary log any data modifications that are received from the replication master, in order to act as a master to one or more other slaves.
Another reason to enable binary logging on the slave is to enable incremental backups using the slave.
This is not our case and to save disk space and speed up the slave replication, we will not write a binary log there. I have also deleted the existent binary logs:
After you save edits to /etc/mysql/my.cnf, you need to restart mysql:
slave # systemctl restart mysqld
We now need to copy the data from master to slave. Let's pretend the master server is busy and there are data manipulation queries like INSERT and UPDATE going on, how do we make sure the data copy is consistent (not half new and half old)?
The solution for InnoDB (the default storage engine in MariaDB) is to use the --single-transaction option of mysqldump. It works by starting a transaction statement to the server before dumping data. Because of the isolation property of transactions, the dump will always be consistent (unless your application contains ALTER, CREATE or similar data definition instructions, which is unlikely). If your tables are big add a --quick option to retrieve one table row at a time rather than retrieving the whole row set and buffering it in memory before writing it out.
Temporary allow master to connect to slave as root:
slave # mysql -p -e "GRANT ALL PRIVILEGES ON *.* TO 'root'@'MASTER_IP' IDENTIFIED BY 'TEMP_PWD'; FLUSH PRIVILEGES"
master # mysqldump -p --master-data --single-transaction -B courses | mysql -h SLAVE_IP -p'TEMP_PWD'
Drop privileges and root access from the master to the slave:
slave # mysql -p -e "DROP USER 'root'@'MASTER_IP'"
If you had more than one database to copy, you would simply add other database names after "courses". You can also run the data copy command the other way around, that is on the slave. Left as an exercise for the reader.
The --master-data options causes the dump to include a query like:
--
-- Position to start replication or point-in-time recovery from
--
CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000004', MASTER_LOG_POS=5815;
This tells the slave the position in the binary log of the master server to start replication from.
In general, for InnoDB tables, mysqldump's --single-transaction option is the way of making an online backup, e.g. locally for all databases:
# mysqldump -A --single-transaction all_databases.sql
Now issue this query on the slave, to tell it what the master is and what account to use for replication:
slave # mysql -p -e "CHANGE MASTER TO
MASTER_HOST='MASTER_IP',
MASTER_USER='repl',
MASTER_PASSWORD='REPL_PWD'"
You can optionally override the default value for MASTER_CONNECT_RETRY (86400). This is the number of times that the slave tries to connect to the master before giving up. A value of 0 means “infinite”; the slave attempts to connect forever.
Replication has not started yet. You have to issue:
slave # mysql -p -e "START SLAVE" You can now try to add a record in a table of the courses database in the master and you will see the same appears on the slave.
On the slave use:
slave # mysql -p -e "SHOW SLAVE STATUS \G" Enter password: *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 192.168.14.51 Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000006 Read_Master_Log_Pos: 245 Relay_Log_File: mysqld-relay-bin.000006 Relay_Log_Pos: 529 Relay_Master_Log_File: mysql-bin.000006 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 245 Relay_Log_Space: 1108 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 to see the state of replication, while on the master:
master # mysql -p -e "SHOW MASTER STATUS \G" Enter password: *************************** 1. row *************************** File: mysql-bin.000006 Position: 245 Binlog_Do_DB: Binlog_Ignore_DB: Note: during replication, a slave server creates several logs that hold the binary log events relayed from the master to the slave, and to record information about the current status and location within the relay log. There are three types of logs used in the process by the slave, one relay log and two status logs: The relay log (e.g. mysqld-relay-bin.000006 above for the slave) consists of the events read from the binary log of the master and written by the slave I/O thread. Events in the relay log are executed on the slave as part of the SQL thread.
Note you can convert and display a binary log file (including a relay log, since they have the same structure as a regular binary log) in text mode using mysqlbinlog, e.g.:
slave # mysqlbinlog /var/lib/mysql/mysqld-relay-bin.000006 Anyway, only SQL statements logged are readable in this output (if any). Row changes are not easy to interpret. The master info log (master.info) is plain text and contains status and current configuration information for the slave's connection to the master. This log holds information on the master host name, login credentials, and coordinates indicating how far the slave has read from the master's binary log. Since it contains the password in clear, this file is only readable by the mysql user and any user in the mysql group (by default none) and of course by root: slave # cat /var/lib/mysql/master.info 18 mysql-bin.000006 245 192.168.14.51 repl REPL_PWD 3306 60 0
0 1800.000
0
0
Values in master.info can be changed using an SQL query: CHANGE MASTER TO. It allows to set the master host to use and the position in the binary log to start replication from (e.g. the one you get from a SHOW MASTER STATUS query on the master):
slave # mysql -p -e "CHANGE MASTER TO
MASTER_HOST=MASTER_IP,
MASTER_USER='repl',
MASTER_PASSWORD=REPL_PWD,
MASTER_LOG_FILE='mysql-bin.000004',
MASTER_LOG_POS=2012"
because that is currently not supported (it will probably be in the future, I hope).
The relay log info log is also simple text and holds status information about the execution point within the slave's relay log. slave # cat /var/lib/mysql/relay-log.info ./mysqld-relay-bin.000006 529 mysql-bin.000006 245
04With that background, here is an explanation of some of the fields in the output of SHOW SLAVE STATUS.
Field Relay_Master_Log_File (mysql-bin.000006) is the name of the binlog on the master containing the last SQL statement successfully executed on the slave. It is not the name of a relay log on the slave, but a binary log on the master.
Exec_Master_Log_Pos (245) is the position in the Relay_Master_Log_File that the slave SQL thread has executed up to. So in your example, the slave db has executed all statements up until binlog mysql-bin.000006 pos 245 on the master db. You see this value is the same you see in SHOW MASTER STATUS in the Position column.
The tuple (Relay_Master_Log_File, Exec_Master_Log_Pos) expresses the coordinates in the master binary log indicating how far the slave SQL thread has executed events received from that log.
What is the difference between Relay_Master_Log_File and Master_Log_File? (Master_Log_file, Read_Master_Log_Pos): Coordinates in the master binary log indicating how far the slave I/O thread has read events from that log.
Finally (Relay_Log_File, Relay_Log_Pos) is the coordinates in the slave relay log indicating how far the slave SQL thread has executed the relay log. These correspond to the preceding coordinates, but are expressed in slave relay log coordinates rather than master binary log coordinates.
until_log_pos is really only used if you start your slave with the query "START SLAVE UNTIL master_log_pos = integer". That syntax will replicate up to that position (it will be the exec_master_log_pos) and then stop. You would normally only do this if you wanted to replicate to a specific point, but no further (like if the next statement is an accidental table drop or something). The value of until_log_pos is 0 when it is not specified, so in our case means that replication will just keep moving forward.
You can now try to shut down MySQL on the slave:
slave # systemctl stop mysqld
Insert a new course on the master:
master # mysql -p courses MariaDB [courses]> INSERT INTO course VALUES (3, "C Programming"); MariaDB [courses]> INSERT INTO offering VALUES (3, 1);
Then start the slave again:
slave # systemctl start mysqld
And check you have this new data:
slave # mysql -p courses MariaDB [courses]> SELECT * FROM course WHERE id=3; +----+---------------+ | id | title | +----+---------------+ | 3 | C Programming | +----+---------------+
So replication resumes automatically. You usually won't disconnect a slave on purpose, but there is a case where this makes sense: if you want to back up the database consistently. In this case you disconnect the slave and take the backup from it. You can simply copy or rsync all files in /var/lib/mysql/* recursively. This ensures all database will be backed up at the same instant in time.
What happens if you write to the slave, which is supposed to be used as read-only? Go on and make a query like this:
slave # mysql -p -e "INSERT INTO instructor VALUES(2,'Duffy Duck')" courses
You see there is nothing to prevent you from writing to the slave if you connect with a user that has such privilege:
slave # mysql -p -e "SELECT * FROM instructor" courses Enter password: +----+------------------+ | id | name | +----+------------------+ | 1 | Antonio Bonifati | | 2 | Duffy Duck | +----+------------------+ Normally you would prevent writes to the slave by not having users with CREATE, UPDATE, etc. permission, but if you really want to be sure that no user other than those with the SUPER privileges and the replication can write, set the read_only global variable:
slave # mysqld --verbose --help 2>/dev/null | grep ^read-only read-only FALSE You can set it permanently in /etc/mysql/my.cnf (under the [mysqld] section): read_only = 1 read_only is a dynamic variable, meaning a change does not require a full restart of the server with systemctl restart mysqld. It's value can be changed dynamically while the server is running, so to make effective the change to my.cnf, you either restart or just do: slave # mysql -p -e 'FLUSH TABLES WITH READ LOCK; SET GLOBAL read_only = ON; SHOW VARIABLES LIKE "read_only"' FLUSH TABLES WITH READ LOCK locks all tables to make sure that nobody can write to databases (e.g. by doing an INSERT or UPDATE). Once the lock is acquired, we can safely set the server as read-only for clients. The lock will be automatically releases at the end of the session, that is when the mysql command returns. After that try: slave # mysql -p -e "INSERT INTO instructor VALUES(3,'Wiley E. Coyote')" courses you see that it succeeds, because you are root. But if you create another user with write access and no SUPER privilege, you will get an error: slave # mysql -p -e 'GRANT ALL PRIVILEGES ON courses.* TO ant@localhost IDENTIFIED BY "ant_pwd"'
slave # mysql -uant -p -e "INSERT INTO instructor VALUES(4,'Porky Pig')" courses ERROR 1290 (HY000) at line 1: The MariaDB server is running with the --read-only option so it cannot execute this statement
So it is your responsibility to make sure the slave databases are consistent. MySQL does not care. It provides replication only, not cloning.
The binomial coefficient (n choose k) is very simple to understand, if explained by means of an example.
For instance if I have 52 cards and want to count all possible ways to choose two cards from the deck, I reason like this: the first card I can obviously
choose in 52 ways. For each choice of the first card I have 52-1=51
ways to choose the second one, so in total there are 52*51=2652 ways, if I
care about order. If I do not, I have to divide by two, because 2 are
all the possible ways to permute each set of two cards. Therefore I
have 52*51/2=1326. This is the value of "52 choose 2".
Now, what is "52 choose 3"? Same reasoning: 52*51*50 gives me all
combinations, also those differing for order only. If I want to consider
all combinations having the same three cards but in different order as
the same combination, I have to divide that figure by the number of these
permutations.
Now, in how many ways can two elements permute? 2 ways (AB, BA). And 3
elements? It's 3*2 ways: ABC, ACB, BAC, BCA, CAB, CBA. You see I put A
as the first element, then permuted the remaining two (B and C). Then I put B as the first and permuted the other two (A and B)... So 52 choose 3 is
52*51*50/(3*2).
What is n choose k? It's n*(n-1)*..*(n-k+1)/(k*(k-1)*...*2)
It looks like a complicated formula, but you can see it works for n=52 and
k=2,3 or any other value of n and k. Well, not really any. Obviously, for this to make sense n-k+1 must be non-negative and non-zero, that is n-k+1>0 or n>k-1. Since k and n are integer, n>k-1 means n>=k (e.g. if k=5, n>5-1 or n>4 means n>=5, given that n is also integer). The formula does not make sense if n Mathematicians call k*(k-1)*...*2*1 (I have added a harmless *1) the "factorial of k". It's just a specific word for the product of an integer by all the integers that come before it, up to 1. Factorial comes from factor, indeed it is a big or long sequence of factors. If you denote the factorial by a bang, you can write the above formula in a more compact way: n*(n-1)*..*(n-k+1)/k! If not still satisfied, you can put n! at the numerator, but then to compensate you have to divide by (n-k)!, so you have n!/((n-k)!k!) or n!/(k!(n-k)!) and this is the most compact form.
Goal: Connect directly to any final host you have an account on, using a jumphost and typing your passphrase only once at each reboot of your client machine.
On your client, generate your SSH keys one off. Please choose to encrypt your private key using a passphrase:
Source .bashrc or reopen your terminal to make sure this alias is defined. Next step is to configure ssh for easy connection to our hosts. Create ~/.ssh/config if it does not exist else add to it:
ControlMaster auto
ControlPath /home/YOUR_LOCAL_USERNAME/.ssh/tmp/%h_%p_%r
Host jumphost
ForwardAgent yes
Hostname jumphost_fully_qualified_name
User YOUR_USER_NAME_ON_JUMPHOST
Where ... is a blank-separated list of all host names reachable from the jumphost. You can use wildcards, e.g. host*
Next, create a temporary directory to keep track of all connections, so that ssh can reuse a connection to the jumphost multiple times and logins are visibly faster:
client$ mkdir ~/.ssh/tmp
You can now login to every host from your client with one short ssh command, e.g.:
client$ ssh host1
client$ ssh host2
Goal: execute the same command(s) or edit the same file on multiple hosts, but type stuff only once.
You may also find useful to install a utility like Parallel SSH on either your client or the jumphost. Parallel SSH allows you to execute commands on multiple hosts in parallel, using only one command. See this tutorial and remember that on ubuntu pssh is called parallel-ssh. E.g. here is how to get a description of linux distribution installed on multiple hosts:
Where host_list is just a file containing IPs or host names, one per line. Note we use the option -i rather than -P else output looks messy.
Another interesting utility is clusterssh. You install it locally and it will open multiple terminals. You can edit the same file on multiple hosts: your input goes to all terminals if directed to a small empty control windows. If focus is on a specific terminal, it will only go to that terminal. Of course, it is up to you to make sure these two mechanisms are used correctly:
I needed to estimate how many lines are added to a log file in 10 minutes. THe idea was to start from a command like tail -f file | wc -l, but that never ends, because wc never sees an end of file and gets blocked when reading on the read size of the pipe if the pipe is empty.
So what I need to do is just to kill tail after a certain time (e.g. 10 minutes). Now, if I run tail -f file | wc -l in background by adding a & at the end, that becomes one job for the shell and I can only use kill %1 to kill both the tail and the wc process. But that way I will not get any output from wc.
The solution was to save the pid of the tail process in a file e.g. tail.pid and then kill tail only by its ID. To save the ID we just output variable $! to descriptor number 3. $! expands to the process ID of the most recently executed background (asynchronous) command, if tail succeeds, it will be its PID. Before tail & echo is executed we make user descriptor number 3 is opened and redirected to a file (this is what 3>tail.pid does):
$ ( tail -f trace.130524.txt & echo $! >&3 ) 3>tail.pid | wc -l & $ sleep 600 && kill $( The command substitution $(< file) is just a faster replacement for $(cat file).
In Linux by default a certain percentage of space (by default 5%, but you can change that with the -m option to tune2fs) is reserved on a root partition. So if you run df on an almost full filesystem, you may see about 100% used, but still there is a difference between the size and the total used space! This is because the percentage refers to the unreserved space.
# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 193G 183G 288M 100% / ...
Why is some space put aside? Consider this reserved space can only be used by processes running with the root privileges. So the purpose is to allow some of the process to continue running for a while and using more disk space even if some other non-privileged processes gone crazy fill all the non-reserved space. If you have a very big drive, you may want to reduce the percentage of reserved space to avoid waste. E.g. 5% of 1 TB (one terabyte) is about 51 GB, but 5% of 6 TB is a whopping 307 GB, probably too much reserved space, if you have a root partition that big.
If you are in dire straits you may want to unreserve this space so it can be used by non-root processes:
# tune2fs -m 0 /dev/sda3
Then you suddenly have a 5% more free space:
# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 193G 183G 11G 95% / ...
but beware that the filesystem may get fragmented badly without any space reserved. ext4 suffers less from this problem, so it is recommended that you remount the root partition as ext4, if you are still using ext3. This is safe to do and does not require a conversion (of course not all ext4 features will be available). Just replace ext3 with ext4 in /etc/fstab for your root partition (/) and reboot. You can safely revert to ext3 the same way. Anyway, you cannot live long with an almost full partition. You need to make more space available and/or cleanup.
Warning: I do not like Nagios. The fact is I do not like frameworks. They only impose structure, usually too much structure and policies and mostly only that, with very little features. You loose true flexibility, which comes only from Turing-complete programming languages. So you better use libraries and not frameworks. Frameworks are for non-programmers or mediocre programmers. Whenever you need to do something not supported by the framework, you will end up fighting it. No framework designer can foretell all users needs and no framework can be as programmable and generic as a programming language. Anyway, here it is how to setup this monitoring framework, if you really have to. If you are not a programmer you may find it useful. But if you are a programmer and have time, you better use sysstat to collect data and cook up you own monitoring solution.
Go to http://localhost/nagios
login as nagiosadmin
Edit /etc/nagios/objects/contacts.cfg and change 30@localhost with your email address. Log file is /var/nagios/nagios.log
How to create a plugin
I want to create a plugin file named check_something. Any plugin should return the following exit codes: OK—0—service works properly WARNING—1—service is in warning state CRITICAL—exit code 2—service is in critical state UNKNOWN—exit code 3—service is in unknown state If an error/exception happens, you better return UNKNOWN, because this is usually the best thing to suggest to nagios in this case. Returning WARNING on error/exception is not appropriate. E.g. if the plugin user chooses to disable notifications for WARNINGS, he will not know that the plugin is actually not working. Before exiting you should print a line with the format:
SOMETHING OK/WARNING/CRITICAL/UNKNOWN: ... Standard plugins are installed into /usr/share/nagios/libexec/. You can put yours there too, but I have decided to use a separate directory /usr/local/share/nagios/libexec/ In /etc/nagios/objects/commands.cfg you have to choose a command_name (unique nickname) for your plugin and define the full path of the executable: define command{ command_name check_something command_line /usr/local/share/nagios/libexec/check_something } This way you register the plugin with nagios. Then to add this check to a host, e.g. localhost, edit file /etc/nagios/objects/localhost.cfg and add there a new definition of a service, e.g.: define service{ use local-service host_name localhost service_description My Plugin check_period 24x7 contact_groups admins notification_options c,r check_command check_something }
Here the value of check_command must match the command_name defined above. A command is just a command: you can even differently named commands that call the same plugin of another command but with different options. Or you can define a command with some parameters. Use the placeholders $ARG1$, $ARG2$, ... in the command_line directive above and check_something!arg1!arg2!... is the syntax to pass parameters.
Plugins can be implemented in any language. For most tasks bash is appropriate, especially if there is no much processing to do, since it makes easy to interface with all unix commands. Here is a template for a Nagios plugin written using bash: #!/bin/bash
# Plugin description.
## Processing parameter code. OPTIND=1
# Define and initialize vars storing parameters warnlvl=value1 critlvl=value2
while getopts "h?a:b:c" opt; do case "$opt" in h|\?) help exit 0 ;; w) # Example check for a numeric parameter if [[ $OPTARG != *[!0-9]* ]]; then warnlvl=$OPTARG else help fi ;; c) critlvl=$OPTARG ;; *) help exit 1 ;; esac done shift $((OPTIND-1)) [ "$1" = "--" ] && shift # Process $1 as the file name if [ -z "$1" ]; then file=default_file_name # Gather an integer lvl describing the state of something... if [ $lvl -gt $critlvl]; then echo "CHECKNAME CRITICAL: description $lvl > $critlvl" elif [ $lvl -gt $warnlvl]; then echo "CHECKNAME WARNING: description $lvl > $warnlvl" else echo "CHECKNAME OK: description $lvl" fi
Rif. https://wiki.archlinux.org/index.php/Nagios http://users.telenet.be/mydotcom/howto/nagios http://www.ibm.com/developerworks/aix/library/au-nagios
This is great for teaching programming or Unix usage, for producing "live" documentation for some common tasks. You may also want to make a transcript of a terminal session if you are planning to develop a script to automate the task later on.
It is advisable to save timing information along with the transcript so that you can reply it.
Record:
$ script -t transcript.tm transcript.txt
Play (you better use the same type of terminal you recorded from):
$ scriptreplay -t transcript.tm transcript.txt
scriptreplay can't do without a timing file. If you have forgot the -t option, an acceptable way to display the script could be by disabling printing of raw control characters in less using the usual caret notation (e.g. ^C):
$ less -r transcript.txt
but this won't work well with programs who clear the screen like vi(1).
The default cipher used with ssh and scp version 1 (3des) is very secure but slow. Version 2 improved on that with support for more ciphers and by default the fastest are used. I wanted to find out what cipher is the fastest. Warning: change the cipher only if strong security is unimportart (e.g. when transferring between two servers in a trusted LAN).
In the test script below I created a half GB binary file with random content on my Linux laptop:
laptop $ dd if=/dev/urandom of=ciphertest.data bs=1M count=512 laptop $ hexdump ciphertest.data |head 0000000 7de0 ce1a 6468 b677 31f4 e899 4271 ee91 0000010 c103 1fdf 886b b91f edf6 f05b 59a3 ec03 0000020 2f6d 47bf 92d4 d0df b695 1217 ddfe edfe 0000030 7f98 f65e e69c 94b0 5113 f66d 608a 7b49 0000040 6750 21ea ebe6 2e54 4ff1 e3c5 ac56 9ae8 0000050 f186 99a1 7c8f f9c7 95c3 8dc1 26d3 3014 0000060 a0ec 139a 62df e07c 69db 9008 7775 75dd 0000070 9009 4e56 9f5c cc2f 6ebd 08ce 5c45 e2b0 0000080 f8a6 5c08 a143 ea81 d966 416f e5b0 88c8 0000090 2eb0 0b1c 8cf9 fc35 7131 36ee 1ee4 0958
then I transfer this file to an idle VM hosted by my same PC using all ciphers available. To avoid typing the password many times, I set up key-based authentication:
$ ssh-keygen ... $ ssh-copy-id ant@192.168.14.70
Here is the script ciphertest.sh:
#!/bin/bash # Measures speed of different SSH ciphers. # Before, you may want to run: # $ ssh-keygen # $ ssh-copy-id $USER_HOST # to save time on typing passwords. If your key is encrypted with a passphrase # you better fire up an ssh agent or you will have to type the passphrase many times. # $ eval $(ssh-agent) # $ ssh-add ~/.ssh/id_dsa # You can find this list in ssh_config(5)CIPHERS=(3des-cbc aes128-cbc aes192-cbc aes256-cbc aes128-ctr aes192-ctr \ aes256-ctr aes128-gcm@openssh.com aes256-gcm@openssh.com arcfour128 \ arcfour256 arc‐four blowfish-cbc cast128-cbc) USER_HOST="ant@192.168.14.70" TMPFILE=$(mktemp) echo -n "Generating random file. Please wait... " dd if=/dev/urandom of=$TMPFILE bs=1M count=512 echo "done!" for cypher in "${CIPHERS[@]}"; do echo $cypher scp -c $cypher $TMPFILE "$USER_HOST:ciphertest.data" echo done ssh "$USER_HOST" rm ciphertest.data rm $TMPFILE and here are the results: $ ./ciphertest.sh Generating random file. Please wait... 512+0 records in 512+0 records out 536870912 bytes (537 MB) copied, 37.1943 s, 14.4 MB/s done! 3des-cbc tmp.pmPSfoUGqT 100% 512MB 13.5MB/s 00:38 aes128-cbc tmp.pmPSfoUGqT 100% 512MB 46.6MB/s 00:11 aes192-cbc tmp.pmPSfoUGqT 100% 512MB 46.6MB/s 00:11 aes256-cbc tmp.pmPSfoUGqT 100% 512MB 42.7MB/s 00:12 aes128-ctr tmp.pmPSfoUGqT 100% 512MB 51.2MB/s 00:10 aes192-ctr tmp.pmPSfoUGqT 100% 512MB 46.6MB/s 00:11 aes256-ctr tmp.pmPSfoUGqT 100% 512MB 46.6MB/s 00:11 aes128-gcm@openssh.com tmp.pmPSfoUGqT 100% 512MB 42.7MB/s 00:12 aes256-gcm@openssh.com tmp.pmPSfoUGqT 100% 512MB 42.7MB/s 00:12 arcfour128 tmp.pmPSfoUGqT 100% 512MB 51.2MB/s 00:10 arcfour256 tmp.pmPSfoUGqT 100% 512MB 46.6MB/s 00:11 arcfour tmp.pmPSfoUGqT 100% 512MB 51.2MB/s 00:10 blowfish-cbc tmp.pmPSfoUGqT 100% 512MB 32.0MB/s 00:16 cast128-cbc tmp.pmPSfoUGqT 100% 512MB 32.0MB/s 00:16
First install Virtualbox. An installer for Windows is available on the Virtualbox site download page. In another Arch Linux box:
host # pacman --noconfirm -S virtualbox
host # gpasswd -a ant vboxusers
host # echo vboxdrv >/etc/modules-load.d/virtualbox.conf
Replace ant with your username, and use a newly open terminal to run Virtualbox:
master $ Virtualbox &
Then New, Name: Arch Linux Tests, 256 MB, Dynamically allocated virtual HD, 8GB.
Then head to the Arch Linux download page and download latest image, e.g. using Torrent. Machine, Settings, Storage, Controller IDE, Empty, click on the small CD-ROM button on the right hand side, Choose a Virtual CD/DVD disk file, point to archlinux-2013.05.01-dual.iso. Start, Boot Arch Linux (x86_64). Make sure your host computer stays connected to the Internet, because some commands you issue on the guest will need a working network connection.
If you do not have a US keyboard issue:
guest # loadkeys keymap
where keymap is the value of KEYMAP you find in this table row for your country, or using this command (ignore the .map.gz extension and use only the file name as the keymap variable value):
guest # ls /usr/share/kbd/keymaps/i386/qwerty
Then:
guest # ntpd -qg
guest # hwclock -w
guest # cgdisk /dev/sda
New, accept all defaults, then Write and confirm by typing yes. Quit.
Optional: ability to locate files:
guest # pacman --noconfirm mlocate
Optional: text mode browser
guest # pacman --noconfirm elinks
Optional: Support for host-only and bridged network interface:
guest # pacman --noconfirm -S net-tools
host # pacman --noconfirm -S virtualbox-host-modules
host # modprobe -a vboxnetadp vboxnetflt vboxpci
host # echo -e 'vboxnetadp\nvboxnetflt\nvboxpci' >>/etc/modules-load.d/virtualbox.conf
host # pacman --noconfirm -S net-tools
Optional: install yaourt
guest # pacman --noconfirm -S wget yajl
guest # su - ant
guest $ cd /tmp
guest $ wget https://aur.archlinux.org/packages/pa/package-query/package-query.tar.gz
guest $ tar zxf package-query.tar.gz
guest $ cd package-query
guest $ makepkg -c
guest $ su -c 'pacman --noconfirm -U package-query-*.tar.gz'
guest $ cd ..
guest $ wget https://aur.archlinux.org/packages/ya/yaourt/yaourt.tar.gz
guest $ tar zxf yaourt.tar.gz
guest $ cd yaourt
guest $ makepkg -c
guest $ exit
guest # cd /tmp/yaourt
guest # pacman --noconfirm -U yaourt-1.3-*.tar.xz
Optional: ssh access (useful if you have to copy and paste a lot of commands between host and guest or viceversa). Requires bridge interface, see above.
guest # pacman --noconfirm -S openssh
guest # systemctl start sshd
guest # systemctl enable sshd.service
guest # poweroff
Settings,Network,Attached to,Bridged Adapter,Start.
