- Tweaked Sguil indexing to prevent Bro URL data from being duplicated into Splunk via sguild.log.
- Monitors dashboard field name drop down selections added to all panels.
- General Mining dashboard added panels for Bro SSH logs and Bro HTTP TLDs (top level domains). Also added drop down options for Bro FTP and IRC panels.
- Squil Mining has been updated and improved.
- Syslog Mining dashboard added for Bro Syslog.
- An Event Workflow was added for searching Splunk for events by src_ip.
- …and last but not least:
- CIF Dashboards!
For the CIF integration to work you need access to a CIF (Collective Intelligence Framework – http://code.google.com/p/collective-intelligence-framework/) server in order to export query results to a .csv file for the external lookups.
The CIF integration leverages three files created by exporting CIF query results to .csv format. If you don’t have access to a CIF server but want a sense of how the app works, I’ve provided three sample files in the lookups folder. Visit http://testmyids.com via a browser and you should see results in the CIF dashboards.
IMPORTANT – By including these 3 sample files (that contain NO actual CIF data; just fields and a test/sample entry) we prevent Splunk from throwing lookup table errors for those who won’t use the CIF capability. The drawback is updating the Security Onion app will overwrite existing .csv files, so remember to backup the lookups/*.csv files prior to future updates or plan to recreate new CIF .csv files.
The three CIF queries used for development and testing were:
cif -q infrastructure -s medium -c 85 -p csv -O infrastructure.csv
cif -q url -s medium -c 85 -p csv -O url.csv
cif -q domain -s medium -c 65 -p csv -O domain.csv
Once you’ve created these files you have to do two things:
- Edit each file removing the “# ” (hash followed by space) from the first row of each file.
- Copy the files to /opt/splunk/etc/apps/securityonion_cif/lookups/
I highly recommend scripting the CIF export, file manipulation and copy/move process.
In the first release there are two dashboards:
- IP Correlation checks dest_ip addresses from the Bro Connection logs indexed against the CIF infrastructure.csv export.
- Domain/MD5 Malware Correlation checks Bro HTTP domains and executable files downloaded MD5 hashes against CIF.
Both dashboards work the same in that you’ll first see a list of IP addresses with an event count. Clicking on an IP will provide CIF related details around the match as well as a list of all activity to the dest_ip address. From there you can use the Event workflow menu to splunk deeper.
I’ll try to post some screenshots and more details soon!
Here are a couple screenshots of the CIF dashboards. When they load, the IP list in the top right shows all the IP matches in the CIF infrastructure.csv file that appeared in the Bro connection logs. A click will load some CIF details (impact, severity, confidence, etc) with the matching events below. From the events view, you can then use the Events Workflow menu (blue square with white arrow) to pivot to a search for activity from the the source IP, full CIF queries, Robtex and DShield lookups. OSINT at it’s finest!
The Domain/Malware MD5 dashboard works the same way. The only real difference is it’s matching Bro HTTP logs against both the domains.csv and url.csv. I wanted to match full urls from Bro against the url.csv, but I’ve run into a few technical difficulties. Bro logs domain and uri as separate fields. I can make Splunk index them as a single field for a direct match lookup against url.csv but that would equal more indexed data. So until I can figure out a better way to do that, we’re matching against the CIF malware_md5 field. Bro hashes executable files downloaded via HTTP by default and this allows us to limit the query to only bro_http fields that actually have an MD5 value.
The key distinction with the Domain/Malware MD5 dashboard is the presence of the cif_malware_md5 field in the CIF Details panel. If that field is populated with a hash value, then you have had a hit on a positive malicious executable download. If there is no hash value, then you have a domain name hit.