BigSnarf blog

Infosec FTW

Open Source Monitoring Tools

Open source monitoring projects

  • statsd is a node.js network daemon that listens for metrics and aggregates them for transfer into another service such as Graphite.
  • Graphite stores time-series data and displays them in graphs through a Django web application.
  • Bucky measures the performance of a web application from end user’s browsers and sends that data back to the server for collection.
  • Sensu is an open source monitoring framework written in Ruby but applicable to any programming language web application.
  • Graph Explorer by Vimeo is a Graphite-based dashboard with added features and a slick design.
  • PacketBeat sniffs protocol packets. Elasticsearch then allows developers to search the collected data and visualize what’s happening inside their web application using the Kibana user interface.
  • Munin is a client plugin-based monitoring system that sends monitoring traffic to the Munin node where the data can be analyzed and visualized. Note this project is written in Perl so Perl 5 must be installed on the node collecting the data.

Happy Pancake Stack

Brute Force D3.js Visualization

Running Apache Spark EMR and EC2 scripts on AWS with read write S3

Video Demo of Spark on EMR

Other posts I did learning EMR

https://bigsnarf.wordpress.com/2014/10/22/process-logs-with-kinesis-s3-apache-spark-on-emr-amazon-rds/

https://bigsnarf.wordpress.com/2015/01/05/apache-spark-1-0-0-emr-via-command-line/

Script to launch you own cluster on EC2

Spark Cluster Build Output for EC2

Commands to experiment with Spark Shell and read write to S3

Output for Simple Word Count job on EMR

Screen Shot 2015-01-20 at 4.42.11 PM

Links to Apache Spark and Collection of Spark EMR Posts

Rsyslog Remotes sending to ElasticSearch and Kibana

Apache Spark 1.0.0 EMR via command line

Facebook Security At Scale 2014 videos

 

Incident Response -> Digital Forensics

First IR book I got my hands on above. My first computer conference was http://www.first.org/. It’s no wonder I’m a Blue Team kinda person. Here is my common process IR braindump.

  • Test your process? What automation is gonna help IR -> DF?
  • Finding the trail …
  • What kind of system are you looking at?
  • Do you have remote access? Root access?
  • What evidence do you wish to collect from the system?
  • Remote acquisition of HDD, Ram or interactive query?
  • VM? Cloud? Local? Remote Office? Laptop? OS?
  • Custom tools? Audit scripts? Forensics Automation?
  • pslist, malfind, filescan, sockets, connscan, and connections with Volatility Framework?
  • Who’s talking to badguy.com?
  • Did you check your DNS logs?
  • Did you check your Firewall logs?
  • Did you check the outbound proxy?
  • Who’s internal address is on 10.187.10.112?
  • What is that computer?
  • Who logged into that computer?
  • Did you check the DHCP logs?
  • Is that a static assignment?
  • OMG that’s a Windows XP box?
  • When was that last patched? Was it phished?
  • Is it exploited?
  • What did anti-virus or host based IDS say?
  • Facebook OSQuery installed on box?
  • GRR installed? MIG?
  • Can I remote image the HDD? Can I remote image the Mem?
  • What’s the baseline on the box?
  • Any sysinternal tools on the box for process monitoring baselines?
  • Files changed? Local logs?
  • Did Windows XP box at 10.187.10.112 communicate with anyone?
  • Did this box do anything? Scans? Talk to any systems locally? Exfil?
  • Did you check the proxies?
  • Did you check the mapped drives?
  • Did you check the SIEM? NTOP? Logs?
  • What are the IOCs? C&C communications?
  • Surricata? Bro? Snort? Security Onion? MIDAS?
  • Are there PCAPs? Network packet logging?
  • Moloch? Packet Pig? DNS Pig? PCAP-RPC? MHN?
  • Is it time for forensics? Do you need to contain? Observe?
  • Remote or local acquisition? Mem or HDD?
  • Process extraction? File extraction? PySSDeep?
  • What tools you gonna use? Volatility? Bulk extractor? Cuckoo? YARA? DNS Graph? Communications graph?
  • Correlate to threat intelligence? Situational awareness? Vulns? Activate other processes? Protect data?
  • What box do I got to get to next? Follow the white rabbit?

Top 25 Programming-Code Links that get clicked every day on this blog

Misc Algebird

Follow

Get every new post delivered to your Inbox.

Join 50 other followers