BigSnarf blog

Infosec FTW

Category Archives: Framework

Statistical Analysis

Data collection: We will use data from a large national survey that was de- signed explicitly with the goal of generating statistically valid infer- ences about the U.S. population.

Descriptive statistics: We will generate statistics that summarize the data concisely, and evaluate different ways to visualize data.

Exploratory data analysis: We will look for patterns, differences, and other features that address the questions we are interested in. At the same time we will check for inconsistencies and identify limitations.

Hypothesis testing: Where we see apparent effects, like a difference be- tween two groups, we will evaluate whether the effect is real, or whether it might have happened by chance.

Estimation: We will use data from a sample to estimate characteristics of the general population.

 

Links

Vincent Vega d3.js in python charts are super simple for pandas dataframes

Graphing different website user experiences

graph5 graph4 graph3

graph2

graph1

 

User experience (UX) involves a person’s emotions about using a particular productsystem or service. User experience highlights the experiential, affective, meaningful and valuable aspects of human-computer interaction and product ownership. Additionally, it includes a person’s perceptions of the practical aspects such as utility, ease of use and efficiency of the system. User experience is subjective in nature because it is about individual perception and thought with respect to the system. User experience is dynamic as it is constantly modified over time due to changing circumstances and new innovations.

http://en.wikipedia.org/wiki/User_experience

 

Metrics platitudes or just the Fogg behaviour grid applied to startups

d3.js mixedtape tutorials – creators gotta create

Bulk processing memory, network traces and HDD using fuzzy hashing and sdhash

Cloudera Impala for Real Time Queries in Hadoop

Machine Learning – LinkedIn profile matcher based on Skills tags

Screen Shot 2013-01-03 at 10.45.58 AM

Linkedin Profiles 4,2, and 1 matched to ‘jQuery’ etc. tags.

Linkedin Profiles 5 and 4 matched to ‘Data Analysis’ etc. tags

https://github.com/bigsnarfdude/machineLearning/tree/master/linkedin

Here is definitely something that will be part of the bigsnarf technology stack

 

iPython Notebook pandas data analysis of web logs and auth logs

Get code here:

https://github.com/dgleebits/PythonSystemAdminTools/blob/master/pandasAuthLogAnalysis.ipynb

Get sample attack data set here:

http://honeynet.org/files/sanitized_log.zip

Thanks to Vincent for testing the code and helping out with the screenshots.

Influences

http://pixlcloud.com/

Follow

Get every new post delivered to your Inbox.

Join 50 other followers