BigSnarf blog

Infosec FTW

Category Archives: Thoughts

Finding attackers with Neo4J

Algebird Monoids for IP Addresses and counts in Scala

import com.twitter.algebird.Operators._


case class IPRecord(val ipAddress: String, val number: Int) extends Ordered[IPRecord] {
 def compare(that: IPRecord): Int = {
   val c = this.number - that.number
   if (c == 0) this.ipAddress.compareTo(that.ipAddress) else c
 }
}


val oneOneOneOne = IPRecord("1.1.1.1", 67391)
val twoTwoTwoTwo = IPRecord("2.2.2.2", 48013573)
val threeThreeThreeThree = IPRecord("3.3.3.3", 6470)
val fourFourFourFour = IPRecord("4.4.4.4", 731)

val topIPAddress: Max[IPRecord] = Max(oneOneOneOne) + Max(twoTwoTwoTwo) + Max(threeThreeThreeThree) + Max(fourFourFourFour)
assert(topIPAddress.get == twoTwoTwoTwo)

Algebirds

algebirds

MetaDATA BigDATA

metaDATA

bigDATA

Using machine learning for anomaly detection is not new but …

Algebird for Infosec Analytics

Security Big Data Analytics Solutions

Building a system that can do full context PCAP for a single machine is trivial, IMHO compared to creating predictive algorithms for analyzing PCAP traffic.  There are log data search solutions like Elasticsearch, GreyLog2, ELSA, Splunk and Logstash that can help you archive and dig through the data.

My favorite network traffic big data solution (2012) is PacketPig. In 2014 I noticed another player named packetsled. I found this nice setup by AlienvaultSecurity OnionBRO IDS is a great network security IDS etc distro. I have seen one called xtractr, MR for forensics. Several solutions exist and PCAP files can be fed to the engines for analysis. I think ARGUS  and Moloch (PCAP Elasticsearch) have a place here too, but I haven’t tackled it yet. There’s a DNS Hadoop presentation from Endgame clairvoyant-squirrel. There’s also openfpc and streamDB. There are some DNS tools like passivedns. ELSA is another tool.

I started using PCAP to CSV conversion perl program, and written my own sniffer to csv in scapy. Super Timelines are being done in python too. Once I get a PCAP file converted to csv, I load it up to HDFS via HUE. I also found this PCAP visualization blog entry by Raffael Marty.

I’ve stored a bunch of csv network traces and did analysis using HIVE and PIG queries. It was very simple. Name the columns and query each column looking for specific entries. Very labour intensive. Binary analysis on Hadoop.

I’m working on a MapReduce library that uses machine learning to classify attackers and their network patterns. As of 2013, there are a few commercial venders like IBM and RSA which have added Hadoop capability to their SIEM product lines. Here is Twitters logging setup. In 2014 I loaded all the csv attack data into CDH4 cluster with Impala query engine. I’m also looking at writing pandas dataframes to Googles Big Query. As of 2014 there are solutions on hadoop for malware analysis , forensics , DNS data mining. Cisco has released their OpenSOC here http://www.slideshare.net/JamesSirota/cisco-opensoc

The biggest advantage with all these systems will be DATA ENRICHMENT. Feeding and combining data to turn a weak signal into actionable insights.

There are a few examples of PCAP ingestion with open source tools like Hadoop:

First one I found was P3:

The second presentation I found was Wayne Wheelers – SherpaSurfing and https://github.com/sherpasurfing/SHERPASURFING:

The third I found was https://github.com/RIPE-NCC/hadoop-pcap:

The fourth project I found was presented at BlackHatEU 2012 by PacketLoop and https://github.com/packetloop/packetpig:

Screen Shot 2012-11-30 at 11.15.22 AM

DNS tshark example

Screen Shot 2014-03-13 at 2.41.31 PM

tshark -i en1 -nn -e http://dns.qry.name  -E separator=”;” -T fields port 53

tshark -i en1 -R “dns” -T pdml | tee dns_log.xml

Skip Lists, Min-Sketches and Sliding Hyperloglog for detection DDOS and Port Scans

Kibana, 2 Node ElasticSearch Cluster, and Python in 15 minutes

Screen Shot 2014-02-26 at 11.55.30 PM

Screen Shot 2014-02-27 at 12.03.54 AM Screen Shot 2014-02-27 at 12.03.29 AM

  1. Download Kibana git clone https://github.com/elasticsearch/kibana.git
  2. Download ElasticSearch wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.0.1.tar.gz
  3. python -m SimpleHTTPServer 8000
  4. Load Apache log data using pyelasticsearch and IPython
  5. Query logs

Screen Shot 2014-02-27 at 1.42.54 PM

http://demo.kibana.org/#/dashboard

Follow

Get every new post delivered to your Inbox.

Join 41 other followers