DIY: How to build your own host-based IDS (HIDS) using OSSEC

This is the first of some posts that’ll walk you through setting up, tuning and maintaining your very own host-based intrusion detection system. This will provide you with a powerful security monitoring platform to help identify and protect your servers and networks from potential threats. These could include an external attacker trying to brute force a valid set of credentials on one of your Internet facing servers, or a colleague creating a domain account with excessive privileges.

Let’s start off by defining what I mean by a host-based intrusion detection system or the more commonly known acronym, HIDS. A HIDS is an application that monitors the internals of a computer system, such as configuration files and log files, constantly looking for malicious activity or policy violations and if it detects such an event then it will fire an alert, such as sending an email.

In this guide we’ll be looking at the software OSSEC. This freely available IDS currently maintained by Trend Micro can be rolled out across a heterogeneous network, scales well, and provides both passive and active monitoring in the following key areas:

• integrity monitoring
• log monitoring
• rootcheck
• process monitoring

While an IDS is usually known to just watch passively and if it detects an attack it will produce an associated alert, OSSEC can also work actively, so when an attack is detected it can take action such as blocking that particular IP address. This type of reactive security technology is often classed as an Intrusion Prevention System (IPS) and can be very useful in thwarting a potential attack.

The OSSEC architecture in its basic form consists of a OSSEC server that is used to manage as well as collate and analyse data received from the connected agents. An agent is any computer system (e.g. Windows 2008) that has the OSSEC agent software installed and configured to communicate back to the central OSSEC server.

Installing the OSSEC Server

The server component of this platform needs to run on a UNIX based OS that includes gnu make, gcc and libc to compile the software. We are choosing to compile OSSEC from source for maximum flexibility as we’ll discover later in the blog series.

In this instance to keep things simple I am going to use the well-known Linux distribution known as Ubuntu. In this tutorial I’ll be using Ubuntu Server 14.04.4 LTS which is freely available and well supported.

Once you’ve got a minimal Ubuntu server up and running with a static IP address configured and Internet access (this should be restricted to just the Ubuntu update servers once installation is complete), we can install the pre-requisites needed to compile OSSEC. To do this, run the following from the command line:

Once completed we can download and extract the latest version of OSSEC (make sure it’s the latest version):

You should now be able to change to the newly expanded installation directory and run the installation script:

You’ll now be guided through the installation process. Select the defaults values where possible (these are shown in the square brackets), and choose “server” when prompted and provide a suitable e-mail address for server alerts to be sent. Note that you will need to allow the OSSEC server to relay email via the designated email server.

After the script completes you should have a working OSSEC server, so let’s now start the OSSEC server with the following command:

You should then see the following output:

At this point we should have a working OSSEC server, however it won’t be able to detect activity on other systems unless we install an agent (that acts a remote sensor).

Installing the OSSEC Agent on Windows

The OSSEC agent software is available for both Windows and UNIX systems, and is fairly straight forward to configure.

It is worth noting that the agent will communicate with the server over UDP port 1514, so make sure communication is permitted at the network level.

First let’s download the Windows binary onto the server you wish to monitor using the link below:
https://bintray.com/artifact/download/ossec/ossec-hids/ossec-agent-win32-2.8.3.exe

On running this executable, you’ll be guided through the installation where you can accept the default values.

After clicking finish, you should have the OSSEC Agent Manager displayed:

As the message above suggests, we need to configure an authentication key. To do this, hop back to the OSSEC server and run the following command:

You’ll be presented with a menu that you’ll be getting very familiar with:

Let’s add the details for the agent we just installed by selecting ‘A’. You’ll then be prompted for an arbitrary server name, and then an IP address. Assuming it’s a static IP, add that, otherwise you can add an IP range such as ‘192.168.0.0/24’.
This just configures where OSSEC data for that agent is expected to originate.

Finally use the default agent ID by hitting the ‘return’ key and select the ‘y’ key to confirm adding.

Now you’ll be back at the main menu, and you can extract the authentication key that is needed by the Windows server by selecting ‘E’. Then type in the ID, for example ‘001’ to extract the base64 encoded authentication key. You’ll need to copy and paste into the “OSSEC Agent Manager” window on the Windows server as well as typing in the OSSEC server’s IP address, and then click ‘Save’. You’ll get a confirmation response like this:

After first installing the OSSEC agent, it won’t be running, so let’s sort that out by clicking the ‘Manage’ menu, then ‘Start OSSEC’. It’s worth mentioning that I found the ‘Status’ of the agent didn’t update immediately!

Now jump back to the server, and let’s review the status OSSEC agent using the following command:

You should now see the Windows server is showing “Active” that means the OSSEC agent was able to successfully authenticate and communicate with the OSSEC server:

Let’s test that everything works by purposely generating an alert. One of the many rules enabled by default is generating an alert when a user logs into a system for the very first time. So let’s do just that by RDPing onto the Windows server.

This causes that rule to fire and an alert to be generated and in turn an email sent:

As you’ll discover with your new OSSEC platform, there’s a large number of predefined rules enabled with the default configuration that we’ll need to tune.

Next time we’ll dive into OSSEC rules, looking at how they work and how we can tweak the default ones to filter out the stuff we don’t care about and prioritise the alerts we do, and even look at creating our own.