BigSnarf blog

Infosec FTW

Table Flip


Intrusion Detection approaches for Anomaly Detection still relies on the Analyst not Software

Typical approaches for Anomaly Detection

  1. Statistical anomaly detection using 90th and 99th percentile T-Digest Algorithm, Time Series Analysis, Heavy Hitters, TopK
  2. Distance based methods like SimHash and LSH on features
  3. Rule-based detection using Data Mining (geoLocation, login behaviors per day, workstation, time)
  4. Signature-based detection using Snort and BRO
  5. Model based AD built on tons of features for DNS traffic, Users, Servers
  6. Change Detection
  7. Machine Learning

Typical approaches for Analyst ad-hoc query detection

  1. Visual Analysis
  2. Alert investigation
  3. Correlation Analysis
  4. Search
  5. SQL
  6. Time Series Analysis
  7. Graph Processing Queries

Malware Detection with Algebird LSH

Detection of polymorphic malware variants by identifying features based on static/dynamic analysis and using Locality-sensitive hashing (LSH) data structure for comparisons. Enrich? Geo? Host?

Couple papers?

Brute force comparison. Return distinct matches above threshold.

.flatMap { case (_, malwareIdSet) =>
      for {
        (malwareId1, sig1) <- malwareIdSet
        (malwareId2, sig2) <- malwareIdSet
        sim = minHasher.similarity(sig1, sig2)
         if (malwareId1 != malwareId2 && sim >= targetThreshold)
      } yield (malwareId1, malwareId2)

T-Digest Algebird

Redis Analytics

Algebird So Hot Right Now

Screen Shot 2015-10-04 at 5.42.19 PM

Scala monoids monads implicits type classes

Just some code exploring Algebird, Akka HTTP, Serialization, and storing in Redis:

PCAP – Logs – Kafka -Kinesis – Compute – Storage

When my Scala code compiles with no errors


AWS Lambda in Scala

Compile Apache Spark with Kinesis Support


Get every new post delivered to your Inbox.

Join 52 other followers