Security Onion 1.1 for Splunk

README notes w/ bonus comments for Version 1.1

I’ve added an input for Bro’s capture_loss.log which now displays on the SOstat Security Onion monitor in a time chart paired with Snort packet loss. To enable this log in Bro edit:

/usr/local/share/bro/site/local.bro

and add the following:

@load misc/capture-loss

You’ll have to check and install Bro for the change to get loaded.

sudo broctl
check
install
exit

and you’re done. It takes a few before the first logged event will show so give it a bit before you worry if it’s working. (The main reason I added this aside from the value is it will likely be standard in SO down the road. It’s completely optional and if I didn’t tell you about it you’d be none the wiser unless you did have it turned up, in which case you’d be pleasantly surprised.)

I also tweaked the sguild inputs to exclude “{URL” events. This data is already being consumed via bro_http so it should cut down on the licensing volume. (This will save you a ton of indexing volume and alone is worthy of updating!)

Monitors Dashboard

  • Returned misc-activity to the Sguil panel. (I’d yanked it due to the volume of URL events, but since we’re leaving those to bro_http, it’s value returns.)
  • Added date/time and raw event to drill down display for the FTP Args panel.

GeoIP

  • A drop down list has been added to GeoIP allowing you to search GeoIP by sourcetype which should reduce query times for more targeted views. The map also now includes drill down capabilities with results appearing below the map when selected. (Neatest update. Much more efficient than relying on all connections and lets you get geo visibility into each sourcetype.)

Mining

  • Added drill down to the time chart panels for HTTP and SMTP mining

(The following additions bring a little asset and vulnerability management to the game via two dashboards: PADS [passive asset detection] and  Bro’s Known Knowns.)

  • Added a PADS dashboard (similar to HTTP and SMTP mining) searchable by Name, Classification, Source IP, Source Port, Destination IP, Destination Port, Protocol, and Severity.
  • Added a Known Knowns dashboard. Includes: Known Services; Known Software searchable by Name and Type; and Known Certs searchable by Country, Common Name, Certificate Issuer Subject, Location, Organization, Organizational Unit, Port Number, State, and Certificate Subject.

PADS

  • Created an event type for PADS in addition to the PADS Mining dashboard.

SOstat

  • Updated SOstat SO to include Bro capture loss in addition to Snort packet loss. (Also improved the packet/capture loss displays to be more “deployment friendly” tracking by host or sensor.)

Enjoy!

Few screenshots for @remor:

1.1 GeoIP1.1 Known Knowns

1.1 PADS

Security Onion 1.0 for Splunk

Today I give you version 1.0 of the Security Onion app for Splunk. While previous releases aimed at core functionality and essentially getting the data in with rather minimalistic views, version 1.0 brings a bit of a overhaul to several dashboards and welcomes Splunk Visualizations to the party.

For starters Overview now shows Sguil events by classification. The Overview dashboard is intended to be a quick pulse check of the environment and in this case I think leveraging classifications provides a cleaner heartbeat, so to speak.

Squil events by name has moved to the Monitors dashboard and in further effort to reduce clutter, I’ve excluded OSSEC and misc-activity classified events.

The Mining dashboards are where the major overhauls have occurred. The General Mining dashboard now contains a couple drop down lists to provide greater access with fewer concurrent searches. HTTP and SMTP filename searches have been consolidated as well.

But where things start to get cool is the Interesting URI Values panel:

The drop down list allows you to search Bro HTTP detected URI values for matching regex values. Due to the transitory nature of URI values in attacks, I’ve setup the drop down list to populate via an external lookup from a .csv file. The file is located at

/opt/splunk/etc/apps/securityonion/lookups/mining_uri_values.csv

and contains two fields: value and label, where value is a regex value that will be
searched for in URIs and label is the name you want to appear in the drop down.

The idea is to enable users to perform historical searching against newer active malicious URIs to identify potential victims retroactively or find similar patterns. I’ll likely be posting an updated copy of the file periodically as well, so suggestions for additions are welcome.

HTTP and SMTP Mining also get a lot cleaner in this release, going from a multiple panel view to a drop down list with table view of events and a time chart. HTTP Mining:

and SMTP Mining:

Last but not least, the previous incarnation of SOstat is now broken up into two views: SOstat Security Onion, for service status and Snort/Snorby details, and SOstat *nix, for details about the system (top, ifconfig, NSM log archive listing, etc). I’ve also added (as a proof of concept and because it’s really cool) a Security Onion Data Flow view that leverages the Splunk Visualizations app. I hope to do more with this in the future as I think it has a lot of potential, especially for visualizing replayed events or monitoring a host in real time.

For more details on the Splunk Visualization app and a video demo, check out http://metasplunk.com/projects/particle. It does require an Adobe Flash upgrade on a Security Onion build (achieved easily via Ubuntu Software Center) and can be browser intensive, so be warned.

I hope you enjoy the changes and wish you good luck finding evil!

Querying CIF Data From Splunk

Collective Intelligence Framework (CIF) is a feed parser that brings a vast wealth of collective threat intel to your fingertips. (Update: Kyle Maxwell has posted a great introduction to CIF at http://overhack.wordpress.com/2012/05/07/introduction-to-the-collective-intelligence-framework/.)  There’s a client app (perl or python, which is being rewritten presently) and browser plug-ins for Firefox and Chrome to make running queries simple. If you have access to a CIF server, it’s very easy to incorporate CIF queries into Splunk’s event and field menus. For Security Onion app users, this feature is coming, but you’ll need to edit the workflow to configure your CIF server and api key. Once configured you’ll see the item added to the event menu:

and/or the field menu:

The search results will open in a new window, and if there were any matches you’ll see something like this:

To add a workflow menu item in Splunk, go to Manager > Fields > Workflow actions and reference the screenshot below. If you’re a Security Onion user, you’ll need to enter the IP or hostname of your CIF server and a valid API key.

In the Security Onion app for Splunk the dest_ip and domain fields can be queried via the fields menu. If you prefer to edit/create via terminal:

CIF provides an enormous amount of intelligence for very little time, money or effort on the end users part and it’s future is looking very bright due to it’s flexibility in parsing content and the ease of interacting with data from other applications like Splunk. Once the python client is released I’ll look at building more correlation with CIF directly into the Security Onion dashboards.

 

Splunking the Onion

Five months brewing, I’m pleased and excited (and mostly ready to spend some time watching playoff hockey) to announce the initial release of Security Onion for Splunk has been submitted to Splunkbase (should appear here once approved). I’ve talked about Security Onion before, but in brief it’s an incredibly powerful compilation of open source network security monitoring tools that is so easy to configure and deploy it probably makes some commercial vendors blush. I started playing with Splunk about 6 months ago and it didn’t take long to realize what a powerful combination it could make with Security Onion.

Splunkbase, Splunk’s repository of community contributed apps, already had some apps relevant to Security Onion, but they were tool specific, like Splunk for OSSEC and Splunk for Snort. I wanted to create a broader canvas, pulling together events from as many layers of the onion as possible. I also wanted a better way to visualize Bro IDS logs for event correlation and mining events for incidents.

I want to emphasize this has been a learning process. The last 5 months have been spent learning Splunk and figuring out how I could improve access to the data from Security Onion tools I have spent the previous year learning. I have a lot to learn on both fronts and hope this app will ultimately reflect that as it matures. While some of the Splunk views have been tested with lots of data, the majority of testing has been done on my home install or with sample packet captures. So if you give it a spin, keep that in mind and please let me know if you run into any issues that need to be addressed for broader deployments.

So without further ado…splunking the onion.

Overview

The Overview provides visibility into the current status of your Security Onion (hereafter referred to as SO) deployment. The first panel displays recent Sguil events with total count of events and a sparkline to show the event trend over time, sortable by name or total number of events. The second panel leverages Splunk for OSSEC to show recent OSSEC events, followed by a timechart of Bro IDS notice events in panel three. From these three panels you can get incredible insight into what has been going on in your environment and the fourth panel pulls everything together to give you a timechart of all SO log activity.

Splunk provides some neat drilldown capabilities and I’ve included them on most of the panels throughout the app. Clicking on any result will provide a drilldown menu at the bottom of the panel showing specific event data. In the screenshot below, I’ve drilled down on SSL::Invalid_Server_Cert events from the Bro Notice panel and Bro SSL events in the Activity by Sourcetype panel. Additionally, the time range picker’s associated with each panel allow you to adjust your visibility over time on a panel by panel basis (notice the Sguil Events are displaying the last 4 hours of activity, while OSSEC is displaying the last 7 days).

Monitors

The Monitors view takes us a step deeper into the SO event data. Six panels give us quick access to Sguil Events by Priority, Bro Weird events, Bro SSL Validation Status Messages, Bro HTTP Status Messages, Bro Dynamic Protocol Detection events and FTP arguments monitored.

As with the Overview, drilldowns are in play. In the screenshot below you can see how easy it is to piece together the bits of an incident for targeting your investigation. In this case, I’d be heading over to the IR Search view and giving 192.168.10.128 a deeper look (which we’ll do momentarily).

GeoIP

The GeoIP view includes a geographical representation of events accompanied by an alphabetical listing of those events by destination country. As you can see from the screenshot, selecting a country from the list on the left will provide a drilldown listing of Bro Connection events to that country.

IR Search

Now let us go take a look at 192.168.10.128. The IR Search allows you to search by source IP, destination IP, source or destination ports and gives a holistic view of activity based on your search parameters.

For starters, we see Bro Notice and Sguil events specific to the IP address being investigated paired with a breakdown of all activity seen from our source IP by sourcetype in a timechart, followed by Bro Weird events, SSL Validation Status Messages and HTTP MIME Types associated with the host.

Continuing down the page, we see HTTP Filenames and FTP Arguments detected by Bro IDS, Bro Dynamic Protocol Detections and a grand finale of all events, line by line, of activity from our suspect IP.

Web Monitor

The Web Monitor view is an attempt at expanding the “reuse” potential of SO collected data, by providing a basic, high level view of HTTP related activity being monitored. Geared less towards investigative analysis, it can be used to supplement existing web filter capabilities, or for home or small office deployments it can provide great awareness of HTTP activity of monitored hosts. Two pie charts show activity by source and geoip, followed by the top source and destination IPs, and domains.

Mining

In using Bro IDS, I’ve noticed some events can be direct indicators of activity, while others more indirect leading you towards potential issues or activity that is otherwise suspicious. The Mining view attempts to ease the process of identifying items of interest with six table panels with drilldown capability: Bro Weird Events (anomolous activity), Bro Dynamic Protocol Detection Events, Bro HTTP Status Messages, Bro SSL Validation Status Messages, Bro HTTP Filenames Detected (exe, bat, cmd, pdf, jar, swf, doc, xls, sh) and FTP Arguments detected.

The Bro HTTP Filenames Detected can often be a dead giveaway of suspicious activity, but others can be more subtle. For example, deploying an SO sensor in front of critical web servers gives visibility into HTTP Status Messages, so you can quickly and easily monitor Forbidden or Not Authorized attempts to access web sites. Similarly, Bro SSL Validation Status Messages can help you monitor untrusted or expired certs.

OSSEC

The OSSEC menu leverages the Splunk for OSSEC app, which is a pre-requisite for Security Onion for Splunk. I won’t go into too many details on it. While I leverage Splunk for OSSEC for the data inputs and field extractions to tie into Bro IDS and Sguil, the OSSEC app itself is very well done and provides excellent visibility, via dashboards, reports and searches, all canned and ready to be consumed directly from the SO app menu.

Snorby/Squert

These menu items are designed to link directly to Snorby and Squert. Unfortunately, I’ve not figured out a way to dynamically generate the links based on the SO server hostname, so they point to http://localhost:<port> by default. I’ve included instructions in the README on how to modify those to fit your deployment.

Long term, I do not plan on trying to mature this app to replace the other tools included in SO. They all have unique value and use cases: Sguil is great for analysts; Snorby is great for Snort; Squert provides a good high level view. If anything I hope to improve the integration over time.

SOstat

SO includes a status monitoring script called “sostat” for verifying the status of agents, collecting disk and performance data, and stats related to Sguil and Snorby. I actually delayed the release of the app until I could get this piece integrated as the importance of monitoring your SO deployment is as significant as the data it’s collecting.

The Server/Service Status panel shows the last two status changes. The first service status listed is the status when SOstat last checked. If there is a second status listed for the same service you can see when it last changed. In the screenshot below you’ll see ossec_agent (sguil) with a status of FAIL on 4/13 @ 9:33. Towards the bottom of the list you’ll see it listed again with a status of OK on 4/11 @ 10:20.

SOstat also monitors Snort packet drop percent, disk status, interfaces, system info (via top), Snorby top 50 events and top events from the previous day, and directory listings for the NSM log archive.

What’s Next?

  • GeoIP – add tracking by event type, name and severity.
  • SOstat – implement Bro status monitoring; trending charts where applicable, like disk utilization.
  • Optimize searches and views for efficiency.
  • Suricata – incorporate and test.
  • Sideview Utils for Splunk – I’m going to start exploring where this add-on can benefit views and the interactivity of the Security Onion app.

If you find any bugs, have suggestions on how to improve the app or have ideas for correlation searches that could help improve the usability, effectiveness and efficiency of detecting and investigating events, please feel free to contact me at brad@eyeis.net or via Twitter: @bradshoop.