This is certainly not a topic that hasn’t been covered elsewhere on the internet. But, In this article I’d like to show you a slightly different take on the setup.

But first, a bit of history as to why I’ve decided to use the vMA + Splunk Light Forwarder.

Up until recently we have been pointing all our vSphere ESX and ESXi hosts directly at a Splunk installation. As this is a free installation we soon run over the 500mb/day quota. I very much wondered if there was a way to filter the syslogs prior to them hitting the Splunk instance so as to reduce the size if the resultant indexes on Splunk. Not to subvert any licensing, but just to be able to control what logs are going to the Splunk. By using the Splunk light forwarder on the vMA it is possible to select which files are forwarded to the main instance of Splunk.

I stumbled on the ‘vMA + Splunk Awesomeness’ article (links at bottom of article) and Andy was doing some jiggery pokery in the inputs.conf on the Splunk light forwarder that was exactly what I was looking for. More on this later.

So, with the help of the three sites mentioned at the bottom of this article I set about setting up a vMA, Vilogger and Splunk Light Forwarding.

INSTALLATION AND CONFIGURATION OF THE vMA

• Install the vMA appliance – I won’t be covering the install, so here is a link to the documentation at VMWare
• BEFORE powering up, set the hard disk size to 15GB

• Resize the /var/log

sudo fdisk -l

sudo fdisk /dev/sda

n,p,3,enter,t,3,8e,w <- commands to be entered into fdisk

sudo fdisk -l

sudo reboot

sudo pvcreate /dev/sda3

sudo vgextend VolGroup00 /dev/sda3

sudo vgdisplay | grep -i free

sudo lvextend -L+10G /dev/VolGroup00/var

sudo resize2fs -p /dev/VolGroup00/var

df -h

reboot

sudo bash

vifp listservers

vilogger enable –server <ESXi hostname or IP> –numrotation 10 –maxfilesize 10 –collectionperiod 10

/etc/init.d/vmware-vilogd status

/etc/init.d/vmware-vilogd start

ls -lrt /var/log/vmware/<ESX/i hostname or IP>

vilogger updatepolicy –server <ESX/i hostname or IP> –numrotation 20 –maxfilesize 10 –collectionperiod 10

• Download the latest Splunk here. Make sure you pick the Linux distribution (32 bit)
• Copy the file to the vMA, I used WinSCP as I was using my Windows 7 machine.
• Get yourself a root bash prompt

sudo bash

• Install the Splunk into the vMA

rpm -i splunk-XXXXXX-.rpm

• Start the Splunk installation

/opt/splunk/bin/splunk start

• Change the web port. This step isn’t necessary, but I had an issue with connecting to port 8000 from my Windows 7 machine.

/opt/splunk/bin/splunk set web-port 8080

/opt/splunk/bin/splunk restart

• Enable Splunk at boot

/opt/splunk/bin/splunk enable boot-start

• Change the default local Splunk admin password

/opt/splunk/bin/splunk edit user admin -password NEW_PASSWORD -roles admin -auth admin:changeme

• Enable the Splunk Light Forwarder

/opt/splunk/bin/splunk enable app SplunkLightForwarder

• Add the log directory on the vMA that we want to Splunk monitor and some other configuration on how we deal with the logs.

vi /opt/splunk/etc/system/local/inputs.conf

[default]
host = vma.<fqdn>

[monitor:///var/log/vmware/]
blacklist = vima
recursive = no
host_regex = /var/log/vmware/(\S+\/)

[monitor:///var/log/vmware/]
whitelist = messages.log
whitelist = vmkwarning.log
#blacklist = \.(gz|bz2|z|zip|log.[0-9])$
recursive = yes
disabled = false
host_regex = /var/log/vmware/(\S+\/)

• The above config will blacklist the vMA machine from being logged. It will also only forward messages.log and vmkwarning.log to the main Splunk instance.

• Add the forward server (main Splunk instance)

/opt/splunk/bin/splunk add forward-server <Splunk-server-IP_or_FQDN>:9997 -auth admin:<password>

•  Restart Splunk

/opt/splunk/bin/splunk restart

Logs are now flowing in from the added ESX/i hosts and will be forwarded to a Splunk instance listed in the forward server configuration. The last thing to do now is configure the main Splunk instance to receive the logs from the vMA

CONFIGURING THE SPLUNK RECEIVER

• Fire up a browser to your main Splunk installation (http://<Splunk hostname or IP>:8000) and login
• Go to the Manager (top right)
• Go to Forwarding and receiving in the Data panel
• Click Configure receiving
• Select New and add port 9997
• Click back to search in the top left corner

TROUBLESHOOTING TIPS

Coming soon!

LINKS

The SLOG - Simon Long’s excellent blog - Enabling Vilogger
ESX Virtualisation Mag - vMA + Splunk article by Andy Grant
Resizing the vMA - VMWare site
Splunk for Linux installation documentation - Splunk documentation

vMA FIREWALL

http://www.vladan.fr/how-to-configure-vma-as-a-destination-of-esxi-log-files/

http://splunk-base.splunk.com/answers/8917/configured-but-inactive-splunk-2-splunk-forwards