BigSnarf blog

Infosec FTW

Monthly Archives: August 2014

Monitoring JVM


Scala REPL in Notebook

Screen Shot 2014-08-26 at 10.29.31 PM

Scalaz in the REPL

First run sbt:


Then issue these commands:

set scalaVersion := “2.10.0”
set libraryDependencies += “org.scalaz” %% “scalaz-core” % “7.0.0”
set initialCommands += “import scalaz._, Scalaz._”
session save

Algebird Monoids for IP Addresses and counts in Scala

import com.twitter.algebird.Operators._

case class IPRecord(val ipAddress: String, val number: Int) extends Ordered[IPRecord] {
 def compare(that: IPRecord): Int = {
   val c = this.number - that.number
   if (c == 0) this.ipAddress.compareTo(that.ipAddress) else c

val oneOneOneOne = IPRecord("", 67391)
val twoTwoTwoTwo = IPRecord("", 48013573)
val threeThreeThreeThree = IPRecord("", 6470)
val fourFourFourFour = IPRecord("", 731)

val topIPAddress: Max[IPRecord] = Max(oneOneOneOne) + Max(twoTwoTwoTwo) + Max(threeThreeThreeThree) + Max(fourFourFourFour)
assert(topIPAddress.get == twoTwoTwoTwo)

Simple Apache Auth Log Processing with Spark job

Screen Shot 2014-08-03 at 10.07.22 PM
Screen Shot 2014-08-03 at 10.13.14 PM



Simple Spark Job for processing Apache auth.log for Invalid user login attempts and Failed password counts
./bin/spark-submit --class "SimpleApp" --master local[4] target/scala-2.10/simple-project_2.10-1.0.jar

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

object SimpleApp {
 def main(args: Array[String]) {
 val logFile = "/Users/antigen/Downloads/sanitized_log/auth.log" 
 val conf = new SparkConf().setAppName("SimpleApacheLogProcessing Application")
 val sc = new SparkContext(conf)
 val logData = sc.textFile(logFile, 2).cache()
 val numAs = logData.filter(line => line.contains("Invalid user")).count()
 val numBs = logData.filter(line => line.contains("Failed password")).count()
 println("Lines with INVALID USER: %s, Lines with FAILED PASSWORD: %s".format(numAs, numBs))

Code, Folder Structure, simple.sbt, and packaged jar files here:

Data Science Stack

Screen Shot 2014-08-02 at 10.47.09 PM

Finally got Algebird and Apache Log Parsing libraries into my Apache Spark REPL