BigSnarf blog

Infosec FTW

Category Archives: Framework

PCAP – Logs – Kafka -Kinesis – Compute – Storage

Open Source Monitoring Tools

Open source monitoring projects

  • statsd is a node.js network daemon that listens for metrics and aggregates them for transfer into another service such as Graphite.
  • Graphite stores time-series data and displays them in graphs through a Django web application.
  • Bucky measures the performance of a web application from end user’s browsers and sends that data back to the server for collection.
  • Sensu is an open source monitoring framework written in Ruby but applicable to any programming language web application.
  • Graph Explorer by Vimeo is a Graphite-based dashboard with added features and a slick design.
  • PacketBeat sniffs protocol packets. Elasticsearch then allows developers to search the collected data and visualize what’s happening inside their web application using the Kibana user interface.
  • Munin is a client plugin-based monitoring system that sends monitoring traffic to the Munin node where the data can be analyzed and visualized. Note this project is written in Perl so Perl 5 must be installed on the node collecting the data.

Open Source Intelligence Feeds

Source Location Notes Various malware trackers.
AdBlock AdBlock pattern matches
AlienVault AlienVault’s IP reputation database. Blacklists.
AVG Site Safety Report Site safety checker.
Bing Scraping but future version to also use API. Blacklists. Look up username availability on popular sites.
DNS Your configured DNS server. Defaults to your local DNS but can be configured to whatever IP address you supply SpiderFoot.
Facebook Scraping but future version to also use API.
Google Scraping but future version to also use API.
Google+ Scraping but future version to also use API.
Google Safe Browsing Site safety checker.
LinkedIn Scraping but future version to also use API. Blacklists. Blacklists. Blacklists.
McAfee SiteAdvisor Site safety checker.
NameDroppers Blacklists.
OpenBL Blacklists.
PasteBin Achieved through Google scraping.
PGP Servers PGP public keys.
PhishTank Identified phishing sites.
Project Honeypot Blacklists. API key needed.
SANS ISC Internet Storm Center IP reputation database.
SHODAN API key needed.
SORBS Blacklists.
SpamHaus Blacklists.
ThreatExpert Blacklists.
TOR Node List Domains/IPs used by malware.
UCEPROTECT Blacklists.
Whois Various Whois

Statistical Analysis

Data collection: We will use data from a large national survey that was de- signed explicitly with the goal of generating statistically valid infer- ences about the U.S. population.

Descriptive statistics: We will generate statistics that summarize the data concisely, and evaluate different ways to visualize data.

Exploratory data analysis: We will look for patterns, differences, and other features that address the questions we are interested in. At the same time we will check for inconsistencies and identify limitations.

Hypothesis testing: Where we see apparent effects, like a difference be- tween two groups, we will evaluate whether the effect is real, or whether it might have happened by chance.

Estimation: We will use data from a sample to estimate characteristics of the general population.



Vincent Vega d3.js in python charts are super simple for pandas dataframes

Graphing different website user experiences

graph5 graph4 graph3




User experience (UX) involves a person’s emotions about using a particular productsystem or service. User experience highlights the experiential, affective, meaningful and valuable aspects of human-computer interaction and product ownership. Additionally, it includes a person’s perceptions of the practical aspects such as utility, ease of use and efficiency of the system. User experience is subjective in nature because it is about individual perception and thought with respect to the system. User experience is dynamic as it is constantly modified over time due to changing circumstances and new innovations.


Metrics platitudes or just the Fogg behaviour grid applied to startups

d3.js mixedtape tutorials – creators gotta create

Bulk processing memory, network traces and HDD using fuzzy hashing and sdhash

Cloudera Impala for Real Time Queries in Hadoop

Machine Learning – LinkedIn profile matcher based on Skills tags

Screen Shot 2013-01-03 at 10.45.58 AM

Linkedin Profiles 4,2, and 1 matched to ‘jQuery’ etc. tags.

Linkedin Profiles 5 and 4 matched to ‘Data Analysis’ etc. tags


Get every new post delivered to your Inbox.

Join 50 other followers