BigSnarf blog

Infosec FTW

Monthly Archives: January 2013

The emergence of a new security role – Security Data Analytics Engineer

1 Comment Posted by Security Dude on January 31, 2013

Security Data Analytics Engineer shall be responsible for carrying out engineering tasks to deliver a clustered computing environment. The engineer shall design and build large-scale security data analytics platforms, using open source software and tools, Cloud based tools and COTS technologies. The engineer shall establish a security data analytics system that produces manageable, actionable intelligence from massive streams of a structured and semi-structured security data.

This is a broad engineering role which requires years of defensive security experience, automating data feeds from different sources, and encompasses building of the core frameworks and platforms to deal with the complexities of ingesting, storing, and manipulating masses of data in real-time.

This engineering role will research and analyse large volumes of data by applying advanced analytical tools and methodologies, build data analytic pipelines, build data processing pipelines, and drive analytical reports to security analysts and investigators for situational awareness.

The reports and analytics dashboards provide analysts and investigators the ability to identify, process, and comprehend critical elements of information about what is happening.

Job Qualifications:

Software engineering, machine learning, data mining, modelling users, modelling attackers, data visualization, big data, data analytics, investigations, ETL, data munging, data wrangling, pipeline automation, Information Security, DFIR. 25 percent Infosec, 25 percent DFIR, 25 percent business knowledge, 25 percent analytics expertise, 25 percent technological capabilities and 25 percent visualization.

Potential Example of a Big Data Security Data Analytics system:

Twitter Analytics
Job description liberated from a variety of job roles
wordle wordcloud text

Thoughts

Cleaning tweetstream or twitter archive

Leave a comment Posted by Security Dude on January 31, 2013

import string

DEFAULT_STOP_WORDS = [ 

  'a', 'about', 'also', 'am', 'an', 'and', 'any', 'are', 'as', 'at', 'be',

  'but', 'by', 'can', 'com', 'did', 'do', 'does', 'for', 'from', 'had',

  'has', 'have', 'he', "he'd", "he'll", "he's", 'her', 'here', 'hers',

  'him', 'his', 'i', "i'd", "i'll", "i'm", "i've", 'if', 'in', 'into', 'is',

  'it', "it's", 'its', 'just', 'me', 'mine', 'my', 'of', 'on', 'or', 'org',

  'our', 'ours', 'she', "she'd", "she'll", "she's", 'some', 'than', 'that',

  'the', 'their', 'them', 'then', 'there', 'these', 'they', "they'd",

  "they'll", "they're", 'this', 'those', 'to', 'us', 'was', 'we', "we'd", 

  "we'll", "we're", 'were', 'what', 'where', 'which', 'who', 'will', 'with',

  'would', 'you', 'your', 'yours',

  ]

def clean_data(data):

  for char in string.punctuation:

    data = data.replace(char, "")

  return data

def clean_stop_words(data):

  for word in DEFAULT_STOP_WORDS:

    data = data.replace(word, "")
  return data

Tools