January 29, 2014

Configure Nagios Monitoring

Prerequisite

Before you begin verify you server and client configuration. From the server test client connection.

$ /usr/lib64/nagios/plugins/check_nrpe -H <client_ip_or_hostname>
NRPE v2.14

To setup server, see http://magnus-k-karlsson.blogspot.se/2014/01/install-nagios-core-35-on-rhel-6-from.html.

And for client, see http://magnus-k-karlsson.blogspot.se/2014/01/install-nagios-agent-nrpe-on-rhel-6.html

Client Nagios Plugins Configuration

First decide what you want to monitor. Then find a plugin that do the job. The standard plugins are documented here (http://nagios-plugins.org/doc/man/index.html).

You could also get help documentation for a specific plugin by executing it with option '--help'.

$ /usr/lib64/nagios/plugins/check_disk --help
check_disk v1.4.16 (nagios-plugins 1.4.16)

After you have tested it and is satisfied, you now define it as a nagios command.

$ vi /etc/nrpe.d/commands.cfg
...
command[check_sda2_disk]=/usr/lib64/nagios/plugins/check_disk -w 20% -c 15% -p /dev/sda2
...

Restart nrpe to let new configuration take effect.

$ service nrpe restart
Shutting down nrpe:                                        [  OK  ]
Starting nrpe:                                             [  OK  ]

Server Nagios Plugins Configuration

From the server test that you can call the client nrpe command.

$ /usr/lib64/nagios/plugins/check_nrpe -H <client_ip_or_hostname> -c check_sda2_disk

If this did not work, go back to the prerequisites and verify you server and client installation. If good, lets continue and define the nrpe command on the server.

$ vi /etc/nagios/conf.d/server1.domain.local.cfg
...
# 'check_nrpe' command definition
define command{
        command_name    check_nrpe
        command_line    $USER1$/check_nrpe -H <client_ip_or_hostname> -c $ARG1$
        }
...

Then we need to define a service that will run this command.

$ vi /etc/nagios/conf.d/server1.domain.local.cfg
...
define service{
        use                             generic-service         ; Name of service template to use
        host_name                       yourhostname.domain.local
        service_description             Checks partition sda2 at 20 proc and 15 proc
        check_command                   check_nrpe!check_sda2_disk
        }
...

Finally you will need to define a host, if you have not already done that.

$ vi /etc/nagios/conf.d/server1.domain.local.cfg
...
define host{
        use                     linux-server            ; Name of host template to use
                                                        ; This host definition will inherit all variables that are defined
                                                        ; in (or inherited by) the linux-server host template definition.
        host_name               yourhostname.domain.local
        alias                   yourhostname.domain.local
        address                 192.168.1.117
        }
...

Before we restart the nagios server, you can test you new configuration

$ nagios -v /etc/nagios/nagios.cfg 

Nagios Core 3.5.1
...
Total Warnings: 0
Total Errors:   0
..

If everything is fine, you can now restart you nagios server.

Open you nagios web admin and login in and check you new service.

January 21, 2014

The NTP Daemon on RHEL 6

Installation

Check that NTP services is started at boot.

$ chkconfig ntpdate --list
ntpdate         0:off 1:off 2:off 3:off 4:off 5:off 6:off

If not automatically started.

$ chkconfig ntpdate on

If NTP service is not running, start it.

$ service ntpdate start

Configuration

Default configuration is almost always sufficient.

$cat /etc/ntp.conf
# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).
...
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server 0.rhel.pool.ntp.org iburst
server 1.rhel.pool.ntp.org iburst
server 2.rhel.pool.ntp.org iburst
server 3.rhel.pool.ntp.org iburst
...

Test

Check system clock against a reliable source.

$ date
Tue Jan 21 07:07:06 CET 2014

January 13, 2014

Install Nagios Agent NRPE on RHEL 6 from EPEL Repository

Introduction

The Nagios Plugins are the components that do the actual monitoring work. They are all bash scripts and are located in:

$ ll /usr/lib64/nagios/plugins/

To make the plugins/agents talk with server/Nagios Core you use the NRPE (Nagios Remote Plugin Executor).

Agent/Client Installation

Install the EPEL (Extra Packages for Enterprise Linux) repository. [https://fedoraproject.org/wiki/EPEL]

$ rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

Install NRPE and all Nagios plugins.

$ yum install nrpe nagios-plugins-all

Agent/Client Configuration

Add the Nagios Core server IP adress.

$ vi /etc/nagios/nrpe.cfg
...
allowed_hosts=127.0.0.1,192.168.122.93
...

Start the nrpe service or restart if you have previously started it, to let the new configuration take effects.

$ service nrpe restart

The NRPE is using port 5666 (see /etc/nagios/nrpe.cfg) to communicate with the server, so we need to open that port in the firewall. Below is the current settings in iptables for the client computer we are trying to monitor.

$ iptables -vnL --line-numbers
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1        0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
2    53027   71M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED 
3        7   588 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
4        1    60 ACCEPT     tcp  --  *      *       192.168.122.0/24     0.0.0.0/0           state NEW tcp dpt:22 
5        0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:21 
6        1    60 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0           LOG flags 0 level 4 
7        1    60 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable 

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1        0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable 

Chain OUTPUT (policy ACCEPT 31259 packets, 3978K bytes)
num   pkts bytes target     prot opt in     out     source               destination  

Open tcp port 5666 for incoming traffic.

$ iptables -I INPUT 6 -m state --state NEW -p tcp --dport 5666 -j ACCEPT

Nagios Core Server Configuration

Install the nagios nrpe plugin on the server.

$ yum install nagios-plugins-nrpe

Then we need to enable it. Add the below command to the end of the file.

$ vi /etc/nagios/objects/commands.cfg
...
define command{
    command_name check_nrpe
    command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

Before we can proceed, we now need to understand how Nagios arrenge items in it's admin GUI.

If you look at the selected menu items to the right, I have selected:

  • Hosts
  • Services
  • Host Groups
  • Service Groups

Now if we open the main nagios configuration file, you will see a similiar structure of the configuration files.

$ vi /etc/nagios/nagios.cfg
...
# OBJECT CONFIGURATION FILE(S)
# These are the object configuration files in which you define hosts,
# host groups, contacts, contact groups, services, etc.
# You can split your object definitions across several config files
# if you wish (as shown below), or keep them all in a single config file.

# You can specify individual object config files as shown below:
cfg_file=/etc/nagios/objects/commands.cfg
cfg_file=/etc/nagios/objects/contacts.cfg
cfg_file=/etc/nagios/objects/timeperiods.cfg
cfg_file=/etc/nagios/objects/templates.cfg

# Definitions for monitoring the local (Linux) host
cfg_file=/etc/nagios/objects/localhost.cfg

# Definitions for monitoring a Windows machine
#cfg_file=/etc/nagios/objects/windows.cfg

# Definitions for monitoring a router/switch
#cfg_file=/etc/nagios/objects/switch.cfg

# Definitions for monitoring a network printer
#cfg_file=/etc/nagios/objects/printer.cfg


# You can also tell Nagios to process all config files (with a .cfg
# extension) in a particular directory by using the cfg_dir
# directive as shown below:

#cfg_dir=/etc/nagios/servers
#cfg_dir=/etc/nagios/printers
#cfg_dir=/etc/nagios/switches
#cfg_dir=/etc/nagios/routers

cfg_dir=/etc/nagios/conf.d
...

You can open the /etc/nagios/objects/localhost.cfg and compare how items are arrenged in the web admin GUI.

Now when we have got a basic understanding of the internal configuration structure we are going to put our new configuration file in /etc/nagios/conf.d/.

$ vi /etc/nagios/conf.d/virtual1.example.com.cfg

###############################################################################
#
# HOST DEFINITION
#
###############################################################################

# Define a host for the remote machine

define host{
        use                     linux-server            ; Name of host template to use
                                                        ; This host definition will inherit all variables that are defined
                                                        ; in (or inherited by) the linux-server host template definition.
        host_name               virtual1.example.com
        alias                   virtual1.example.com
        address                 192.168.122.196
        }

###############################################################################
#
# SERVICE DEFINITIONS
#
###############################################################################

define service{
        use                             generic-service         ; Name of service template to use
        host_name                       virtual1.example.com
        service_description             Current Load
        check_command                   check_nrpe!check_load
        }

Finally restart nagios service and watch you new Host and Service in the web admin GUI.

If you run into problems, open the default nagios log file.

$ less /var/log/nagios/nagios.log

And RHEL default log file. And also read mine previous blog how to configure iptables and logging. [http://magnus-k-karlsson.blogspot.se/2014/01/configure-iptables-for-ftp-server-vsftp.html]

$ less /var/log/messages

Install Nagios Core 3.5 on RHEL 6 from EPEL Repository

Introduction

The Nagios is one of the most poplur product for monitoring IT Infrastructure, such as servers, network and services.

The system overview of the Nagios modules:

  • Nagios Core - The Server with Monitor GUI.
  • NRPE (Nagios Remote Plugin Executor) - The communication handler between server and client/agent.
  • Nagios Plugins - Client or Agent that do the actual monitoring, e.g. check server load or if service is running.

Remember Nagios is not supported by Red Hat and installing Nagios from EPEL repository, is on you own risk. But if you think of the alternative, which is installing Nagios from source and compiling it with gcc is not an attractive alternative. Since then you have to manually uphold the patch level you want in production. And beside you do not want to have the gcc compiler installed in production, due to security risks.

For manual installation see http://nagios.sourceforge.net/docs/3_0/quickstart-fedora.html.

Installation

Install the EPEL (Extra Packages for Enterprise Linux) repository. [https://fedoraproject.org/wiki/EPEL]

$ rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

Install the Nagios Core and all plugins.

$ yum install nagios nagios-plugins-all

Start Nagios Server.

$ service nagios start

Start Nagios Server Web GUI.

$ service httpd start

Nagios Core Web GUI

http://localhost/nagios

username: nagiosadmin
password: nagiosadmin

To change the default password for admin, see /etc/httpd/conf.d/nagios.conf.

$ htpasswd /etc/nagios/passwd nagiosadmin

Reference for htpasswd - Manage user files for basic authentication. [http://httpd.apache.org/docs/2.2/programs/htpasswd.html]

Configuration Files

The Nagios main configuration file.

$ /etc/nagios/nagios.cfg

The Nagios Core Admin Server is built with CGI scripts.

$ /etc/nagios/cgi.cfg

HTTPD configuration for Nagios Core

$ /etc/httpd/conf.d/nagios.conf

Appendix. RPM Dependency and Installation

$ repoquery -q --requires --plugins nagios
/bin/sh
chkconfig
group(nagios)
httpd
initscripts
libc.so.6(GLIBC_2.8)(64bit)
libcrypt.so.1()(64bit)
libdl.so.2()(64bit)
libdl.so.2(GLIBC_2.2.5)(64bit)
libgcc_s.so.1()(64bit)
libgcc_s.so.1(GCC_3.0)(64bit)
libgcc_s.so.1(GCC_3.3.1)(64bit)
libgd.so.2()(64bit)
libjpeg.so.62()(64bit)
libm.so.6()(64bit)
libm.so.6(GLIBC_2.2.5)(64bit)
libnsl.so.1()(64bit)
libperl.so()(64bit)
libpng12.so.0()(64bit)
libpthread.so.0()(64bit)
libpthread.so.0(GLIBC_2.2.5)(64bit)
libresolv.so.2()(64bit)
libutil.so.1()(64bit)
libz.so.1()(64bit)
mailx
nagios-common
perl(:MODULE_COMPAT_5.10.1)
php
rtld(GNU_HASH)
user(nagios)
$ yum install nagios
...
==============================================================================================================================================================
 Package                              Arch                          Version                                 Repository                                   Size
==============================================================================================================================================================
Installing:
 nagios                               x86_64                        3.5.1-1.el6                             epel                                        1.2 M
Installing for dependencies:
 gd                                   x86_64                        2.0.35-11.el6                           rhel-x86_64-server-6                        142 k
 libXpm                               x86_64                        3.5.10-2.el6                            rhel-x86_64-server-6                         51 k
 nagios-common                        x86_64                        3.5.1-1.el6                             epel                                         17 k
 php                                  x86_64                        5.3.3-27.el6_5                          rhel-x86_64-server-6                        1.1 M
 php-cli                              x86_64                        5.3.3-27.el6_5                          rhel-x86_64-server-6                        2.2 M
 php-common                           x86_64                        5.3.3-27.el6_5                          rhel-x86_64-server-6                        525 k

Transaction Summary
==============================================================================================================================================================
Install       7 Package(s)
$ yum install nagios-plugins-all
...
==============================================================================================================================================================
 Package                                    Arch                     Version                                     Repository                              Size
==============================================================================================================================================================
Installing:
 nagios-plugins-all                         x86_64                   1.4.16-10.el6                               epel                                    14 k
Installing for dependencies:
 fping                                      x86_64                   2.4b2-10.el6                                epel                                    31 k
 lm_sensors                                 x86_64                   3.1.1-17.el6                                rhel-x86_64-server-6                   123 k
 nagios-plugins                             x86_64                   1.4.16-10.el6                               epel                                   200 k
 nagios-plugins-breeze                      x86_64                   1.4.16-10.el6                               epel                                    14 k
 nagios-plugins-by_ssh                      x86_64                   1.4.16-10.el6                               epel                                    35 k
 nagios-plugins-cluster                     x86_64                   1.4.16-10.el6                               epel                                    29 k
 nagios-plugins-dhcp                        x86_64                   1.4.16-10.el6                               epel                                    35 k
 nagios-plugins-dig                         x86_64                   1.4.16-10.el6                               epel                                    34 k
 nagios-plugins-disk                        x86_64                   1.4.16-10.el6                               epel                                    69 k
 nagios-plugins-disk_smb                    x86_64                   1.4.16-10.el6                               epel                                    17 k
 nagios-plugins-dns                         x86_64                   1.4.16-10.el6                               epel                                    35 k
 nagios-plugins-dummy                       x86_64                   1.4.16-10.el6                               epel                                    26 k
 nagios-plugins-file_age                    x86_64                   1.4.16-10.el6                               epel                                    15 k
 nagios-plugins-flexlm                      x86_64                   1.4.16-10.el6                               epel                                    16 k
 nagios-plugins-fping                       x86_64                   1.4.16-10.el6                               epel                                    35 k
 nagios-plugins-game                        x86_64                   1.4.16-10.el6                               epel                                    31 k
 nagios-plugins-hpjd                        x86_64                   1.4.16-10.el6                               epel                                    33 k
 nagios-plugins-http                        x86_64                   1.4.16-10.el6                               epel                                    76 k
 nagios-plugins-icmp                        x86_64                   1.4.16-10.el6                               epel                                    38 k
 nagios-plugins-ide_smart                   x86_64                   1.4.16-10.el6                               epel                                    31 k
 nagios-plugins-ircd                        x86_64                   1.4.16-10.el6                               epel                                    16 k
 nagios-plugins-ldap                        x86_64                   1.4.16-10.el6                               epel                                    33 k
 nagios-plugins-load                        x86_64                   1.4.16-10.el6                               epel                                    31 k
 nagios-plugins-log                         x86_64                   1.4.16-10.el6                               epel                                    15 k
 nagios-plugins-mailq                       x86_64                   1.4.16-10.el6                               epel                                    18 k
 nagios-plugins-mrtg                        x86_64                   1.4.16-10.el6                               epel                                    31 k
 nagios-plugins-mrtgtraf                    x86_64                   1.4.16-10.el6                               epel                                    31 k
 nagios-plugins-mysql                       x86_64                   1.4.16-10.el6                               epel                                    41 k
 nagios-plugins-nagios                      x86_64                   1.4.16-10.el6                               epel                                    31 k
 nagios-plugins-nt                          x86_64                   1.4.16-10.el6                               epel                                    36 k
 nagios-plugins-ntp                         x86_64                   1.4.16-10.el6                               epel                                    48 k
 nagios-plugins-ntp-perl                    x86_64                   1.4.16-10.el6                               epel                                    18 k
 nagios-plugins-nwstat                      x86_64                   1.4.16-10.el6                               epel                                    39 k
 nagios-plugins-oracle                      x86_64                   1.4.16-10.el6                               epel                                    16 k
 nagios-plugins-overcr                      x86_64                   1.4.16-10.el6                               epel                                    33 k
 nagios-plugins-perl                        x86_64                   1.4.16-10.el6                               epel                                    14 k
 nagios-plugins-pgsql                       x86_64                   1.4.16-10.el6                               epel                                    33 k
 nagios-plugins-ping                        x86_64                   1.4.16-10.el6                               epel                                    36 k
 nagios-plugins-procs                       x86_64                   1.4.16-10.el6                               epel                                    67 k
 nagios-plugins-real                        x86_64                   1.4.16-10.el6                               epel                                    32 k
 nagios-plugins-rpc                         x86_64                   1.4.16-10.el6                               epel                                    16 k
 nagios-plugins-sensors                     x86_64                   1.4.16-10.el6                               epel                                    14 k
 nagios-plugins-smtp                        x86_64                   1.4.16-10.el6                               epel                                    70 k
 nagios-plugins-snmp                        x86_64                   1.4.16-10.el6                               epel                                    70 k
 nagios-plugins-ssh                         x86_64                   1.4.16-10.el6                               epel                                    32 k
 nagios-plugins-swap                        x86_64                   1.4.16-10.el6                               epel                                    32 k
 nagios-plugins-tcp                         x86_64                   1.4.16-10.el6                               epel                                    38 k
 nagios-plugins-time                        x86_64                   1.4.16-10.el6                               epel                                    32 k
 nagios-plugins-ups                         x86_64                   1.4.16-10.el6                               epel                                    35 k
 nagios-plugins-users                       x86_64                   1.4.16-10.el6                               epel                                    29 k
 nagios-plugins-wave                        x86_64                   1.4.16-10.el6                               epel                                    14 k
 net-snmp-utils                             x86_64                   1:5.5-49.el6                                rhel-x86_64-server-6                   173 k
 postgresql-libs                            x86_64                   8.4.18-1.el6_4                              rhel-x86_64-server-6                   201 k
 qstat                                      x86_64                   2.11-9.20080912svn311.el6                   epel                                   162 k

Transaction Summary
==============================================================================================================================================================
Install      55 Package(s)
...

January 8, 2014

Virtualization with KVM on RHEL 6

Install KVM

$ yum groupinstall "Virtualization" "Virtualization Client" "Virtualization Platform" "Virtualization Tools"

Verify Installation

Check that kvm kernel modules are loaded. There should be two: kvm and either kvm_intel or kvm_amd.

$ lsmod | grep kvm
kvm_intel              54285  0 
kvm                   332980  1 kvm_intel

If kvm_intel or kvm_amd is not loaded, try to load it. E.g. kvm_intel.

$ modprobe kvm_intel

In my case, I recieved an error. And in the general RHEL log file, i recieved the following.

$ cat /var/log/messages
Jan  4 20:39:23 tester2 kernel: kvm: disabled by bios

So for my HP Elite 7500 I had to enable Virtualization in the BIOS:

  1. Computer Setup
  2. Security
  3. System Security
  4. Virtualization Technology (VTx/VTd): Enable
  5. Save and reboot

NOTE: REALLY TRY TO FIX THE PROBLEM OF NOT HAVING LOADED THE KVM_INTEL OR KVM_AMD, OTHERWISE WILL THINGS GO REALLY SLOW.

Start KVM

$ service libvirtd start

Prerequisite RHEL DVD

The easiest way to distribute the RHEL 6 DVD is to actually use the ISO. To make it accessible for the KVM, we place it under libvirt default images folder.

$ cp rhel-server-6.5-x86_64-dvd.iso /var/lib/libvirt/images/

Finally we need to restore the SELinux type for the ISO file.

$ restorecon -F /var/lib/libvirt/images/rhel-server-6.5-x86_64-dvd.iso

Install Guest Virtual Machine Graphically

The easiest way is to do it graphically with Virtual Machine Manager

$ virt-manager
If you had a kickstart file, you could here press TAB and enter the URI for the kickstart file.

Make sure you auto starts the default network. Otherwise you have to manually start it, every time you boot.

Since this is a virtual server, you can fill the entire virtual disk with RHEL.

A nice feature before writing the new partition table to disk is to preview. You do that with the option at the bottom.

Again the RHEL installation will stretch the entire virtual disk, so cleaning the Master Boot Record, MBR, is no problem. But if you were to install a system along with other OS, you must have good strategy how these OS can work alongside and be booted.

The most common installation is for server is Minimal and for client it is Desktop.

Install Guest Virtual Machine CLI

$ virt-install --prompt
What is the name of your virtual machine? 
virtual1.example.com
How much RAM should be allocated (in megabytes)? 
2048
What would you like to use as the disk (file path)? 
/var/lib/libvirt/images/virtual1.example.com.img
How large would you like the disk (/var/lib/libvirt/images/virtual1.example.com.img) to be (in gigabytes)? 
10
What is the install CD-ROM/ISO or URL? 
/var/lib/libvirt/images/rhel-server-6.5-x86_64-dvd.iso

Install Guest Virtual Machine CLI with Kickstart

You can automate the entire installation process of new virtual guest in line, with the help of a kickstart file.

To do this you need to extract the ISO file, for example an FTP server. On the FTP server

$ mkdir /media/rhel6

$ mount -o loop /home/magnus/Downloads/rhel-server-6.5-x86_64-dvd.iso /media/rhel6

$ cp -ar /media/rhel6/. /var/ftp/pub/pub/inst
$ virt-install --name virtual2.example.com --ram 2048 --vcpus 2 --disk path=/var/lib/libvirt/images/virtual2.example.com.img,size=20 --location ftp://192.168.1.3/pub/inst --extra-args "ks=ftp://192.168.1.3/pub/minimal-installation-ks.cfg"

Manage Virtual Machine

List all virtual machines.

$ virsh list --all
 Id    Name                           State
----------------------------------------------------
 2     virtual2.example.com           running

Auto start of virtual machine.

$ virsh autostart virtual2.example.com
Domain virtual2.example.com marked as autostarted

Disable auto start of virtual machine.

$ virsh autostart --disable virtual2.example.com
Domain virtual2.example.com unmarked as autostarted

Stop virtual machine.

$ virsh destroy virtual2.example.com
Domain virtual2.example.com destroyed

Start virtual machine.

$ virsh start virtual2.example.com
Domain virtual2.example.com started

Remove Virtual Machine

$ virsh destroy virtual2.example.com

$ rm -f /var/lib/libvirt/images/virtual2.example.com.img

$ rm -f /etc/libvirt/qemu/virtual2.example.com.xml

January 7, 2014

Configure iptables for FTP Server, vsftp

Server Installation

$ yum install vsftpd

$ service vsftpd start

Now the ftp server is ready to be used. Lets create a simple text file in the root of the ftp server.

$ echo "Test" > /var/ftp/pub/test.txt

Configure iptables

Existing iptables rules.

$ iptables -vnL --line-numbers
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1        0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
2    12592   18M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED 
3        0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
4        0     0 ACCEPT     tcp  --  *      *       192.168.122.0/24     0.0.0.0/0           state NEW tcp dpt:22 
5        0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable 

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1        0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable 

First we will add a LOG operation just before the last line in INPUT of iptables that rejects the incoming traffic.

$ iptables -I INPUT 5 -j LOG

Then we try to list the root of the ftp server. Which will fail, because we have not opened the firewall ftp port and which you will see in the log file.

$ tail -f /var/log/messages
...
Jan  7 21:13:59 virtual1 kernel: IN=eth0 OUT= MAC=52:54:00:51:e4:07:52:54:00:00:99:4a:08:00 SRC=192.168.122.1 DST=192.168.122.196 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=53779 DF PROTO=TCP SPT=38835 DPT=21 WINDOW=14600 RES=0x00 SYN URGP=0

Now lets open the ftp port in the firewall. We will insert the rule just before the log rule.

$ iptables -I INPUT 5 -m state --state NEW -p tcp --dport 21 -j ACCEPT

Now lets try to list the root again, which will fail. Now look at the log.

$ tail -f /var/log/messages
...
Jan  7 21:21:49 virtual1 kernel: IN=eth0 OUT= MAC=52:54:00:51:e4:07:52:54:00:00:99:4a:08:00 SRC=192.168.122.1 DST=192.168.122.196 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=18564 DF PROTO=TCP SPT=39862 DPT=29736 WINDOW=14600 RES=0x00 SYN URGP=0

You might now wonder why ftp is trying to open port 29736 and the reason is that vsftp is using passive ports to communicate. To fix this we need to add to add ftp filter rules to iptables. The relevant filter rules are found in the kernel module filter directory.

$ ls /lib/modules/`uname -r`/kernel/net/ipv4/netfilter/*ftp*
/lib/modules/2.6.32-431.el6.x86_64/kernel/net/ipv4/netfilter/nf_nat_ftp.ko
/lib/modules/2.6.32-431.el6.x86_64/kernel/net/ipv4/netfilter/nf_nat_tftp.ko

Now add those two filters two iptables

$ vi /etc/sysconfig/iptables-config

Now save your iptables new rules and restart iptables.

$ service iptables save && service iptables restart

And finally try to list the content of the root in the server again and this should work.

Scanning network for open ports with nmap command

Whenever you are configuring a firewall, you want to make sure afterwards that the firewall really is configured properly and to do that is to scan it. Below follows two great links that helped me.

http://www.cyberciti.biz/tips/linux-scanning-network-for-open-ports.html
 
http://www.cyberciti.biz/networking/nmap-command-examples-tutorials/