BigSnarf blog

Infosec FTW

Category Archives: Framework

PCAP – Logs – Kafka -Kinesis – Compute – Storage

Open Source Monitoring Tools

Open source monitoring projects

  • statsd is a node.js network daemon that listens for metrics and aggregates them for transfer into another service such as Graphite.
  • Graphite stores time-series data and displays them in graphs through a Django web application.
  • Bucky measures the performance of a web application from end user’s browsers and sends that data back to the server for collection.
  • Sensu is an open source monitoring framework written in Ruby but applicable to any programming language web application.
  • Graph Explorer by Vimeo is a Graphite-based dashboard with added features and a slick design.
  • PacketBeat sniffs protocol packets. Elasticsearch then allows developers to search the collected data and visualize what’s happening inside their web application using the Kibana user interface.
  • Munin is a client plugin-based monitoring system that sends monitoring traffic to the Munin node where the data can be analyzed and visualized. Note this project is written in Perl so Perl 5 must be installed on the node collecting the data.

Open Source Intelligence Feeds

Source Location Notes
abuse.ch http://www.abuse.ch Various malware trackers.
AdBlock https://easylist-downloads.adblockplus.org/easylist.txt AdBlock pattern matches
AlienVault https://reputation.alienvault.com AlienVault’s IP reputation database.
Autoshun.org http://www.autoshun.org Blacklists.
AVG Site Safety Report http://www.avgthreatlabas.com Site safety checker.
Bing http://www.bing.com Scraping but future version to also use API.
Blocklist.de http://lists.blocklist.de Blacklists.
Checkusernames.com http://www.checkusernames.com Look up username availability on popular sites.
DNS Your configured DNS server. Defaults to your local DNS but can be configured to whatever IP address you supply SpiderFoot.
DomainTools http://www.domaintools.com
DroneBL http://www.dronebl.org
Facebook http://www.facebook.com Scraping but future version to also use API.
FreeGeoIP http://freegeoip.net
Google http://www.google.com Scraping but future version to also use API.
Google+ http://plus.google.com Scraping but future version to also use API.
Google Safe Browsing http://www.google.com/safebrowsing Site safety checker.
LinkedIn http://www.linkedin.com Scraping but future version to also use API.
malc0de.com http://malc0de.com Blacklists.
malwaredomainlist.com http://www.malwaredomainlist.com Blacklists.
malwaredomains.com http://www.malwaredomains.com Blacklists.
McAfee SiteAdvisor http://www.siteadvisor.com Site safety checker.
NameDroppers http://www.namedroppers.org
Nothink.org http://www.nothink.org Blacklists.
OpenBL http://www.openbl.org Blacklists.
PasteBin http://www.pastebin.com Achieved through Google scraping.
PGP Servers http://pgp.mit.edu/pks/ PGP public keys.
PhishTank http://www.phishtank.org Identified phishing sites.
Project Honeypot http://www.projecthoneypot.org Blacklists. API key needed.
RIPE/ARIN http://stat.ripe.net/
Robtex http://www.robtex.com
SANS ISC http://isc.sans.edu Internet Storm Center IP reputation database.
SHODAN http://www.shodanhq.com API key needed.
SORBS http://www.sorbs.net Blacklists.
SpamHaus http://www.spamhaus.org Blacklists.
ThreatExpert http://www.threatexpert.com Blacklists.
TOR Node List http://torstatus.blutmagie.de
TotalHash.com http://www.totalhash.com Domains/IPs used by malware.
UCEPROTECT http://www.uceprotect.net Blacklists.
VirusTotal http://www.virustotal.com
Whois Various Whois
Yahoo http://www.yahoo.com
Zone-H http://www.zone-h.org

Statistical Analysis

Data collection: We will use data from a large national survey that was de- signed explicitly with the goal of generating statistically valid infer- ences about the U.S. population.

Descriptive statistics: We will generate statistics that summarize the data concisely, and evaluate different ways to visualize data.

Exploratory data analysis: We will look for patterns, differences, and other features that address the questions we are interested in. At the same time we will check for inconsistencies and identify limitations.

Hypothesis testing: Where we see apparent effects, like a difference be- tween two groups, we will evaluate whether the effect is real, or whether it might have happened by chance.

Estimation: We will use data from a sample to estimate characteristics of the general population.

 

Links

Vincent Vega d3.js in python charts are super simple for pandas dataframes

Graphing different website user experiences

graph5 graph4 graph3

graph2

graph1

 

User experience (UX) involves a person’s emotions about using a particular productsystem or service. User experience highlights the experiential, affective, meaningful and valuable aspects of human-computer interaction and product ownership. Additionally, it includes a person’s perceptions of the practical aspects such as utility, ease of use and efficiency of the system. User experience is subjective in nature because it is about individual perception and thought with respect to the system. User experience is dynamic as it is constantly modified over time due to changing circumstances and new innovations.

http://en.wikipedia.org/wiki/User_experience

 

Metrics platitudes or just the Fogg behaviour grid applied to startups

d3.js mixedtape tutorials – creators gotta create

Bulk processing memory, network traces and HDD using fuzzy hashing and sdhash

Cloudera Impala for Real Time Queries in Hadoop

Machine Learning – LinkedIn profile matcher based on Skills tags

Screen Shot 2013-01-03 at 10.45.58 AM

Linkedin Profiles 4,2, and 1 matched to ‘jQuery’ etc. tags.

Linkedin Profiles 5 and 4 matched to ‘Data Analysis’ etc. tags

https://github.com/bigsnarfdude/machineLearning/tree/master/linkedin

Follow

Get every new post delivered to your Inbox.

Join 50 other followers