Checking the Splunk Process from Nagios

Now its time to have Nagios check to make sure that splunk is running. For version 3 of Splunk there was a app / plugin you could get for Splunk that would work with Nagios. It appears to be gone. But I did find a snippet that some one posted here. Several things have changed so that script doesn’t work 100% out by cut and paste, but it was an excellent jumping off point and it took very few modifications to get going. Due to the fact that there is a copy right on this script, then I can’t put it here with out permission. But I will note that you can do the same thing almost by using the default nagios check_procs command.

So copy that script as check_splunk and stick it in your libexec folder. If your playing along with my setup thats: /opt/local/libexec/nagios

Once you have it downloaded you can ./check_splunk ports or procs
I didn’t worry about checking or trying to edit the search portion as for what I’m doing I don’t really need it right now, but I will revisit it if the need arises.

Now you have the script, its time todo the normal nagios setup stuff.
1. Add it to your commands
vi /opt/local/etc/nagios/objects/commands.cfg

define command {
command_name check_splunk
command_line $USER1$/check_splunk $ARG1$
}

2. Add it to your localhost
vi /opt/local/etc/nagios/objects/localhost.cfg

define service{
use generic-service
host_name localhost
service_description Splunk Port
check_command check_splunk!ports
}

define service{
use generic-service
host_name localhost
service_description Splunk Procs
check_command check_splunk!procs
}

Now restart nagios and you should be good to go

Adding Cacti Logs to Splunk

This basically the same process as adding the nagios logs, but I’ll put it up anyway.

1. Click Manager in the upper right-hand corner of Splunk Web.
2. Under System configurations, click Data Inputs.
3. Click Files and directories.
4. Click New to add an input
I choose Monitor a file or directory
6. Specify the path to the file:
With my setup it is:
/Library/WebServer/Documents/cacti/log/cacti.log
7. Under Host Heading
I choose constant value
8. Under Source Type
I choose Automatic
9. Click Save

Thats it now your cacti logs show up in splunk.

Checking the OSSEC Processes from Nagios

There are several OSSEC processes that are running at once. So I’ll add some simple process checking to nagios to make sure I know they are running. One note is that if you haven’t enabled the ossec-csyslogd to run (I did that to talk to splunk) then you won’t need that one. So here we go, its easy.

1. Add it to your commands (Note I’m adding a new command for this instead of using the command already there so I can pass some different information, plus I like to keep all my modifications separate to make things easier to reproduce on other boxes)

vi /opt/local/etc/nagios/objects/commands.cfg

# ‘check_ossec_procs’ command definition
define command{
command_name check_ossec_procs
command_line $USER1$/check_procs -c $ARG1$ -C ARG2$
}

2. Add it to your localhost
vi /opt/local/etc/nagios/objects/localhost.cfg

define service{
use generic-service
host_name localhost
service_description OSSEC csyslogd
check_command check_ossec_procs!1:1!ossec-csyslogd
}

define service{
use generic-service
host_name localhost
service_description OSSEC maild
check_command check_ossec_procs!1:1!ossec-maild
}

define service{
use generic-service
host_name localhost
service_description OSSEC execd
check_command check_ossec_procs!1:1!ossec-execd
}

define service{
use generic-service
host_name localhost
service_description OSSEC analysisd
check_command check_ossec_procs!1:1!ossec-analysisd
}

define service{
use generic-service
host_name localhost
service_description OSSEC logcollector
check_command check_ossec_procs!1:1!ossec-logcollector
}

define service{
use generic-service
host_name localhost
service_description OSSEC monitord
check_command check_ossec_procs!1:1!ossec-monitord
}

Now just reload nagios and you should be able to tell if you ossec process is there or not.

Adding OSSEC Alerts to Splunk

Next up I want to add my OSSEC Alerts to Splunk. This is slightly more complicated then adding the nagios logs, but well document. The main part of this comes from the OSSEC Wiki Here. And the rest from the forums. But I’ll put it all here for my reference.

1. Edit your ossec.conf (If you’ve installed it like I have its located at: /var/ossec/etc/ossec.conf)
add the following block:
<syslog_output>
<server>172.10.2.3</server>
<port>10002</port>
</syslog_output>

2. Enable syslog_output module and restart OSSEC:
#/var/ossec/bin/ossec-control enable client-syslog
#/var/ossec/bin/ossec-control restart

On restart you’ll see ossec-csyslogd starting up.

On the Splunk Side
1. Goto Manager
2. Goto Data Inputs
3. On UDP click Add New
4. On my setup the UPD Port is 10002
5. Set sourcetype is Manual
6. Source type is ossec
7. Save

Now since things have changed in Splunk 4 the rest of the wiki entry doesn’t help. But there is more information on this Forum Thread.

So download that file and extract it to your splunk directory as stated. Restart splunk and bingo your OSSEC alerts plus lots of nice menu options to access that data.

Adding Nagios Logs to Splunk

Now that we have all these systems working correctly under OSX its time to start making them work together a little.

First up I want to add the nagios logs to splunk. This is very easy, you can get this off the splunk site here. But I’ll recap exactly what I did for my setup here.

1. Click Manager in the upper right-hand corner of Splunk Web.
2. Under System configurations, click Data Inputs.
3. Click Files and directories.
4. Click New to add an input
I choose Monitor a file or directory
6. Specify the path to the file:
With my setup it is:
/opt/local/var/nagios/nagios.log
7. Under Host Heading
I choose constant value
8. Under Source Type
I choose Automatic
9. Click Save

Thats it now your nagios logs show up in splunk. Pretty easy stuff.