BigSnarf blog

Infosec FTW

STL for anomaly detection

Monitor Cloudtrail logs with this Python Slackbot – Who is using AWS Console?

Cloudtrail Log Analyzer Python3 Aysnc Await

Table Flip


Intrusion Detection approaches for Anomaly Detection still rely on the Analyst not Software

Typical approaches for Anomaly Detection

  1. Statistical anomaly detection using 90th and 99th percentile T-Digest Algorithm, Time Series Analysis, Heavy Hitters, TopK
  2. Distance based methods like SimHash and LSH on features
  3. Rule-based detection using Data Mining (geoLocation, login behaviors per day, workstation, time)
  4. Signature-based detection using Snort and BRO
  5. Model based AD built on tons of features for DNS traffic, Users, Servers
  6. Change Detection
  7. Machine Learning

Typical approaches for Analyst ad-hoc query detection

  1. Visual Analysis
  2. Alert investigation
  3. Correlation Analysis
  4. Search
  5. SQL
  6. Time Series Analysis
  7. Graph Processing Queries

Malware Detection with Algebird LSH

Detection of polymorphic malware variants by identifying features based on static/dynamic analysis and using Locality-sensitive hashing (LSH) data structure for comparisons. Enrich? Geo? Host?

Couple papers?

Brute force comparison. Return distinct matches above threshold.

.flatMap { case (_, malwareIdSet) =>
      for {
        (malwareId1, sig1) <- malwareIdSet
        (malwareId2, sig2) <- malwareIdSet
        sim = minHasher.similarity(sig1, sig2)
         if (malwareId1 != malwareId2 && sim >= targetThreshold)
      } yield (malwareId1, malwareId2)

T-Digest Algebird

Redis Analytics

Algebird So Hot Right Now

Screen Shot 2015-10-04 at 5.42.19 PM

Scala monoids monads implicits type classes

Just some code exploring Algebird, Akka HTTP, Serialization, and storing in Redis:

PCAP – Logs – Kafka -Kinesis – Compute – Storage


Get every new post delivered to your Inbox.

Join 53 other followers