BigSnarf blog

Infosec FTW

Category Archives: Tools

Monitoring JVM

BuS2MHPCMAAm-fk

Scala REPL in Notebook

Screen Shot 2014-08-26 at 10.29.31 PM

Simple Apache Auth Log Processing with Spark job

Screen Shot 2014-08-03 at 10.07.22 PM
Screen Shot 2014-08-03 at 10.13.14 PM

/* 

SimpleApp.scala

Simple Spark Job for processing Apache auth.log for Invalid user login attempts and Failed password counts
./bin/spark-submit --class "SimpleApp" --master local[4] target/scala-2.10/simple-project_2.10-1.0.jar

 */
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

object SimpleApp {
 def main(args: Array[String]) {
 val logFile = "/Users/antigen/Downloads/sanitized_log/auth.log" 
 val conf = new SparkConf().setAppName("SimpleApacheLogProcessing Application")
 val sc = new SparkContext(conf)
 val logData = sc.textFile(logFile, 2).cache()
 val numAs = logData.filter(line => line.contains("Invalid user")).count()
 val numBs = logData.filter(line => line.contains("Failed password")).count()
 println("Lines with INVALID USER: %s, Lines with FAILED PASSWORD: %s".format(numAs, numBs))
 }
}


Code, Folder Structure, simple.sbt, and packaged jar files here:

https://github.com/bigsnarfdude/SimpleApp

Data Science Stack

Screen Shot 2014-08-02 at 10.47.09 PM

Finally got Algebird and Apache Log Parsing libraries into my Apache Spark REPL

Screen Shot 2014-08-02 at 9.06.57 AM

Apache Spark 1.0.1 – SQL – Load from files, JSON and arrays

D3.js – Crossfilter – Chart Porn

IPython Notebook, Apache Spark, Honeynet Forensic Challenge 10

Hello World – Analysis of StackOverflow dataset using Apache Spark Job Server

Screen Shot 2014-06-13 at 8.32.54 PM

curl --data-binary @target/scala-2.10/spark-jobserver-examples_2.10-1.0.0.jar localhost:8090/jars/sparking
curl 'localhost:8090/jars'
curl 'localhost:8090/contexts'
curl -X POST 'localhost:8090/contexts/users-context'
curl -X POST 'localhost:8090/jobs?appName=sparking&classPath=sparking.jobserver.GetOrCreateUsers&context=users-context'
curl 'localhost:8090/jobs/<insertJobNumberHere>'

https://github.com/bigsnarfdude/spark-jobserver-examples

Cisco enters the big data security analytics with OpenSOC

Screen Shot 2014-06-13 at 4.39.41 PM

Follow

Get every new post delivered to your Inbox.

Join 40 other followers