Data Scientist Roles

http://www.fastcolabs.com/3008620/lessons-crash-course-data-science

Data Science Bootcamp

http://www.bigdive.eu/

THE IDEA BEHIND BIG DIVE IS TO BOOST THE GROWTH OF A NEW GENERATION OF DEVELOPERS.

A street-fighting gym where high value datasets are the raw material in the hands of a bunch of ambitious smart geeks tutored and mentored by experts in three key areas: Development, Visualization and Data Science.

Formatting code in iPython

http://nbviewer.ipython.org/5385520

Extracting features out of web logs to identify Human vs. Robot

Classifying traffic intensity and temporary differences in access

1. Total pages request per IP address
2. Percentage of images requested
3. Percentage of binaries requested like pdf
4. Total request for robots.txt
5. Percentage of HTML pages requested
6. Percentage of text files requested
7. Percentage of zip files requested
8. Percentage of video files requested
9. Bounce rate
10. Session time
11. Standard deviation between clicks
12. Percentage of night time requests
13. Percentage of errors
14. Percentage of garbage requests
15. Percentage of GETS
16. Percentage of POSTS
17. Percentage of HEAD
18. URL traversal
19. Depth of URL traversal
20. Pathlength
21. Referrer
22. User Agents
23. IP Address location
24. Known crawler IP addresses
25. Repeated requests
26. Average time between clicks
27. OS badges
28. ARIN registration
29. ASN analysis
30. Geolocation

Chart SQL joins

 

.

Security Data Visualization

Tableau Public is for anyone who wants to tell stories with interactive data on the web. It’s delivered as a service which allows you to be up and running overnight. With Tableau Public you can create amazing interactive visuals and publish them quickly, without the help of programmers or IT.

The Premium version of Tableau Public is for organizations that want to enhance their websites with interactive data visualizations. There are higher limits on the size of data you can work with. And among other premium features, you can keep your underlying data hidden.

Why tell stories with data? Because interactive content drives more page views and longer dwell time. Industry experts have cited figures showing that the average reading time of a web page with an interactive visual is 4, 5 or 6 times that of a static web page.

http://www.tableausoftware.com/products/public

Feature Extraction Network Packets Machine Learning

Scikit-learn (sklearn) is an established, open-source machine learning library, written in Python with the help of NumPy, SciPy and Cython.

Scikit-learn is very user friendly, has a consistent API, and provides extensive documentation. Its implementation is high quality due to strict coding standards and high test coverage.  Behind sklearn is a very active community, which is steadily improving the library.

• How to perform scalable text feature extraction with the Hashing Trick

Feature Extraction of the following features from each network packet

1. Ethernet Size
2. Ethernet Destination
3. Ethernet Source
4. Ethernet Protocol
5. IP header length
6. IP Time To Live
7. IP Protocol
8. IP Length
9. IP Type of Service
10. IP Source
11. IP Destination
12. TCP Source Port
13. TCP Destination Port
14. UDP Source Port
15. UDP Destination Port
16. UDP Length
17. ICMP Type
18. ICMP Code

Other potential feature extractions from packets could be:

1. Duration of the connection
2. Connection Starting Time
3. Connection Ending Time
4. Number of packets from src to dst
5. Number of packets from dst to src
6. Number of bytes from src to dst
7. Number of byte from dst to src
8. Number of Fragmented packets
9. Number of ACK packets
10. Number of retransmitted packets
11. Number of pushed packets
12. Number of SYN packets
13. Number of FIN packets
14. Number of TCP header flags
15. Number of Urgent packets
16. Number of sequence packets

Network traffic type features:

1. Per src IP to set(all dst IP) per minute, hour, day, month, year
2. Per src IP to set(all dst same Port) per minute, hour, day, month, year
3. Per src IP to set(all dst to different Ports) per minute, hour, day, month, year
4. Per src IP to set (all dst per protocal like SYN, FIN, ACK) per minute, hour, day, month, year
5. All reverse stats from dst to src for items 1-4
6. Conversations per IP per minute, hour, day, month, year
7. Conversations based on protocol or flag, per MHDY

Vectorizing a large text corpus with the hashing trick

http://nbviewer.ipython.org/urls/raw.github.com/bigsnarfdude/machineLearning/master/Vectorizing.ipynb

My thoughts on building a security data analytics practice in an organization

Build People, Processes and Policies

1. Gather all the questions that need to be answered
2. Select team members
3. Develop data preparation workflows
4. Select data preparation tools (python, bash, hadoop)
5. Develop how you want to consume and present data to users and consumers
6. Select data presentation tools (tableau, ipython notebooks, d3.js)
7. Develop the experimentation workflow (tools etc)
8. Observe and analyze experiment outcomes (gotta build stuff / POC)
9. Build data products and optimize (POC => WIP => Prod1.0 => Prod2.0)
10. Train anyone and everyone to love you data products
11. Build data products you love

Analysis of the current environment

1. What questions need to be asked? What questions need to be answered? Who need these answers? How fast?
2. Where is your data now, how is it stored, who controls it, how do you get access
3. Are you getting the right kinds of data? Is it in the format you want? Is the systems in place answering 90% of questions?
4. Consider instrumenting everything
5. Consider storing all the data in one place. Figure out how to protect and monitor access.
6. Need data to feed the algorithms to feed the peoples questions
7. You need to store the data then you can process from unstructured to structured data
8. Consume the data you have first before building
9. Plan on keeping all the data forever
10. Build data products for self service, exploration and experimentation. “Data Lovefest”
11. Make tools for everyone, including yourself
12. Build for analytical applications that encourage consumption

 

Update: Mature DS Shops

The laboratory. To succeed with the data lab, companies must create an open, questioning, collaborative environment. They must nurture a critical mass of data scientists and provide them access to lots of data, state-of-the-art tools, and time to dream up and work through hundreds of hypotheses — most of which will not yield insight.

The factory. The work of creating a product or service from an insight, figuring out how to deliver and support it, scaling up to do so, dealing with special cases and mistakes, and doing so at profit is beyond the scope of the lab. It calls for a sense of urgency; discipline and coordination; project plans and schedules; and higher levels of automation and repeatability. The work requires many more people with a wider variety of skill sets, a more rigid environment, and different sorts of metrics.

http://blogs.hbr.org/cs/2013/04/two_departments_for_data_succe.html

If you torture the data long enough, it will confess!

“If you torture the data long enough, it will confess.” – Ronald Coase

@BigsnarfDude tweet - April 3, 2013

What side you on? Blue Team or Red Team? OSS Security Distros

REMnux < SIFT Kit < Security Onion < IPCop > Samurai WTF > BackTrack > Kali

• http://zeltser.com/remnux/
• http://computer-forensics.sans.org/community/downloads
• http://securityonion.blogspot.ca/
• http://www.ipcop.org/
• http://samurai.inguardians.com/
• http://www.backtrack-linux.org/
• http://www.kali.org/

Dude where’s my naive bayes?

A naive Bayes classifier is a simple probabilistic classifier based on applying Bayes’ theorem with strong (naive) independence assumptions. A more descriptive term for the underlying probability model would be “independent feature model”.  An overview of statistical classifiers is given in the article on Pattern recognition.

http://en.wikipedia.org/wiki/Naive_Bayes_classifier

https://github.com/bigsnarfdude/machineLearning/blob/master/mason_vs_sklearn_naive_bayes.py