BigSnarf blog

Infosec FTW

Category Archives: Thoughts

Cosine Similarity in Spark

package org.apache.spark.examples.mllib

import scopt.OptionParser

import org.apache.spark.SparkContext._
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.linalg.distributed.{MatrixEntry, RowMatrix}
import org.apache.spark.{SparkConf, SparkContext}

/**
 * Compute the similar columns of a matrix, using cosine similarity.
 *
 * The input matrix must be stored in row-oriented dense format, one line per row with its entries
 * separated by space. For example,
 * {{{
 * 0.5 1.0
 * 2.0 3.0
 * 4.0 5.0
 * }}}
 * represents a 3-by-2 matrix, whose first row is (0.5, 1.0).
 *
 * Example invocation:
 *
 * bin/run-example mllib.CosineSimilarity \
 * --threshold 0.1 data/mllib/sample_svm_data.txt
 */
object CosineSimilarity {
  case class Params(inputFile: String = null, threshold: Double = 0.1)
    extends AbstractParams[Params]

  def main(args: Array[String]) {
    val defaultParams = Params()

    val parser = new OptionParser[Params]("CosineSimilarity") {
      head("CosineSimilarity: an example app.")
      opt[Double]("threshold")
        .required()
        .text(s"threshold similarity: to tradeoff computation vs quality estimate")
        .action((x, c) => c.copy(threshold = x))
      arg[String]("<inputFile>")
        .required()
        .text(s"input file, one row per line, space-separated")
        .action((x, c) => c.copy(inputFile = x))
      note(
        """
          |For example, the following command runs this app on a dataset:
          |
          | ./bin/spark-submit  --class org.apache.spark.examples.mllib.CosineSimilarity \
          | examplesjar.jar \
          | --threshold 0.1 data/mllib/sample_svm_data.txt
        """.stripMargin)
    }

    parser.parse(args, defaultParams).map { params =>
      run(params)
    } getOrElse {
      System.exit(1)
    }
  }

  def run(params: Params) {
    val conf = new SparkConf().setAppName("CosineSimilarity")
    val sc = new SparkContext(conf)

    // Load and parse the data file.
    val rows = sc.textFile(params.inputFile).map { line =>
      val values = line.split(' ').map(_.toDouble)
      Vectors.dense(values)
    }.cache()
    val mat = new RowMatrix(rows)

    // Compute similar columns perfectly, with brute force.
    val exact = mat.columnSimilarities()

    // Compute similar columns with estimation using DIMSUM
    val approx = mat.columnSimilarities(params.threshold)

    val exactEntries = exact.entries.map { case MatrixEntry(i, j, u) => ((i, j), u) }
    val approxEntries = approx.entries.map { case MatrixEntry(i, j, v) => ((i, j), v) }
    val MAE = exactEntries.leftOuterJoin(approxEntries).values.map {
      case (u, Some(v)) =>
        math.abs(u - v)
      case (u, None) =>
        math.abs(u)
    }.mean()

    println(s"Average absolute error in estimate is: $MAE")

    sc.stop()
  }
}

Bunch of Security Skillz from SANS

File Carving and Data Extraction
The candidate will demonstrate an understanding of stream-based data carving and extraction, using tools like those contained in The Sleuth Kit.

Filesystem Structure and Analysis
The candidate will demonstrate an understanding of FAT and NTFS filesystems, and the ability to recover and analyze evidence from file system layers, including the data storage layer, metadata layer, and filename layer.

Forensic Image Acquision, Preservation, and Handling
The candidate will demonstrate an understanding of how and when to collect, document, and handle logical and physical images from file systems for the purpose of performing evidence analysis, and how to preserve evidence integrity.

Incident Response and Forensic Framework
The candidate will demonstrate an understanding of the steps in the enterprise incident response process, attack progression, the role of risk components in developing security intelligence, and the duties of a forensic analyst.

Indicators of Compromise and Malware Detection
The candidate will demonstrate an understanding of techniques to uncover indicators of compromise on a system, detect malware, and counter anti-forensic actions.

Timeline Analysis
The candidate will demonstrate an understanding of creating timelines and supertimelines using data from multiple sources, and will be able to analyze timelines to identify relevant events.

Volatile Data Analysis
The candidate will demonstrate an understanding of identifying abnormal processes, network connections, and other malicious artifacts stored in volatile evidence, using tools like Redline and Volatility.

Volatile Data Preservation and Collection
The candidate will demonstrate an understanding of how and when to collect volatile data from a system, and how to document and preserve the integrity of volatile evidence.

Windows File System Artifacts
The candidate will demonstrate the ability to extract and analyze artifacts from Windows-specific elements, including volume shadow copies and restore points.

Common Network Protocols
The candidate will demonstrate and understanding of the behavior, security risks and controls of common network protocols.

Encryption and Encoding
The candidate will demonstrate an understanding of techniques and practices used to encode and encrypt common network traffic and common attacks on these controls.

NetFlow Analysis and Attack Visualization
The candidate be familiar with the use of NetFlow data and information sources to identify network attacks.

Network Anaysis Tool and Usage
The candidate will be familiar with open source packet analysis tools and their purpose to effectively filter and rebuild data streams for analysis.

Network Architecture
The candidate will be be familiar with the process to design and deploy a network employing diverse transmission and collection technologies.

Network Protocol Reverse Engineering
The candidate will be familiar with the tools and techniques required to analyze diverse protocols and data traversing a network environment.

Open Source Network Security Proxies
The candidate will demonstrate an understanding of the architecture, deployment, benefits and weaknesses of network security proxies, common log formats and flow of data in a network environment.

Security Event and Incident Logging
The candidate be familiar with diverse log formats, protocols and the security impact of the event generating processes. They will demonstrate an understanding of the configuration and deployment strategies to secure and position logging aggregators and collection devices throughout a network environment.

Wireless Network Analysis
The candidate will be familiar with the process to identify and control the risks associated with wireless technologies, protocols and infrastructure.

Analysis of Malicious Document Files
The candidate will be able to demonstrate an understanding of the tools and techniques used to analyze malicious document files.

Analyzing Protected Executables
The candidate will demonstrate an understanding of the techniques malware authors employ to protect malicious software from being analyzed, and the corresponding malware analysis techniques.

Analyzing Web-Based Malware
The candidate will be able to demonstrate an understanding of the tools and techniques used to analyze web-based malware

Common Windows Malware Characteristics in x86 Assembly
The candidate will demonstrate an understanding of the common malware characteristics, as seen when statically examining malicious x86 assembly code.

In-Depth Analysis of Malicious Browser Scripts
The candidate will demonstrate an understanding of the skills needed to analyze complex web-based malicious software employing browser scripts.

In-Depth Analysis of Malicious Executables
The candidate will demonstrate an understanding of advanced methods for examining malicious software to uncover additional details about its functionality.

Malware Analysis Using Memory Forensics
The candidate will be able to demonstrate an understanding of using Windows memory forensics techniques to analyze malware threats.

Malware Code and Behavioral Analysis Fundamentals
The candidate will be able to demonstrate an understanding of the tools and techniques used to conduct code and behavioral analysis of malware, including buidling a lab environment and the use of debuggers, disassemblers, sniffers, and other useful tools.

Windows x86 Assembly Code Concepts for Reverse-Engineering
The candidate will demonstrate an understanding of the core concepts associated with reverse-engineering malware at the x86 assembly level in Windows.

AJAX
The candidate will demonstrate an understanding of AJAX technology and its known weaknesses

Automated Web Application Vulnerability Scanners
The candidate will demonstrate familiarity with automated tools used to find web application vulnerabilities and their distinguishing features.

Cross Site Scripting and Attack Frameworks
The candidate will demonstrate an understanding of the types of XSS attacks and XSS attack frameworks that can be utilized during a pen test

Programming Fundamentals
The candidate will demonstrate familiarity with modern web-based languages including Javascript with Ajax, and Python

Reconnaissance
The candidate will demonstrate comprehension of techniques used to conduct reconnaissance using available information.

Scanning and Mapping
The candidate will demonstrate an understanding of mapping and scanning web applications and servers, including port scanning, identifying services and configurations, spidering, application flow charting and session analysis.

Session Tracking and SSL
The candidate will demonstrate comprehension of session tracking and SSL/TLS use in modern web communications as well as the attacks that can leverage flaws in session state

SQL Injection
The candidate will demonstrate an understanding of how to perform SQL injection attacks and how to identify SQL injection vulnerabilities in applications

Understanding the Web and HTTP
The candidate will demonstrate an understanding of the fundamentals web applications and their architecture and a thorough comprehension of the HTTP protocol

Web App Pen Test Methodology and Reporting
The candidate will demonstrate comprehension of the typical methods and components used during a web application penetration test

Advanced IDS Concepts
Demonstrate an understanding of IDS tuning methods and correlation issues (e.g., snort, bro)

Application Protocols
The candidate will demonstrate knowledge, skill, and ability relating to application layer protocol dissection and analysis including HTTP, SMTP, and various Microsoft protocols

Concepts of TCP/IP and the Link Layer
The candidate will understand the the TCP/IP communications model and link layer operations

DNS
The candidate will demonstrate a thorough understanding of how DNS works for both legitimate and malicious purposes.

Fragmentation
The candidate will demonstrate comprehension of how fragmentation works through theory and packet capture examples, as well as the concepts behind fragmentation-based attacks.

IDS Fundamentals and Initial Deployment (e.g., snort, bro)
Understand architecture, benefits/weaknesses, and configuration options of common IDS systems. Demonstrate ability to configure and deploy IDS (e.g., snort, bro)

IDS Rules (e.g., snort, bro)
Create effective IDS (e.g., snort, bro) rules to detect varied types of malicious activity

IP Headers
The candidate will demonstrate the ability to dissect IP packet headers and analyze them for normal and anomalous values that may point to security issues

IPv6
The candidate will demonstrate knowledge, skill and ability relating to the analysis of IPv6 as well as issues involving IP6 over IPv4.

Network Architecture and Event Correlation
The candidate will demonstrate competence with issues relating to IDS/IPS management, network architecture as it pertains to intrusion detection, and event correlation and management

Network Traffic Analysis and Forensics
The candidate will demonstrate the ability to analyze real traffic and associated artifacts: malicious, normal and application traffic; and demonstrate the ability to discern malicious traffic from false positives.

Packet Engineering
The candidate will demonstrate knowledge, skill, and ability relating to packet engineering and manipulation including packet crafting, OS fingerprinting, and IDS Evasion/Insertion

Silk and Other Traffic Analysis Tools
The candidate will demonstrate the ability to use Silk and other tools to perform network traffic and flow analysis

TCP
The candidate will understand TCP communications as well as expected responses to given stimuli at this layer

Tcpdump Filters
The candidate will demonstrate the skill and ability to craft tcpdump filters that match on given criteria.

UDP and ICMP
The candidate will demonstrated the ability to analyze both UDP and ICMP packets and recognize common issues

Wireshark Fundamentals
The candidate will demonstrate the knowledge, skills, and abilities associated with traffic analysis using wireshark from an intermediate to high degree of proficiency.

Backdoors & Trojan Horses
The candidate will demonstrate a detailed understanding of how Backdoors are used to gain access to systems, and how to defend systems.

Buffer Overflows
The candidate will demonstrate an understanding of what a buffer overflow is, how they are created, and how to defend against them. Additionally, candidates will demonstrate a high-level understanding of how attackers use common tools to create and maintain a backdoor on a compromised system.

Covering Tracks: Networks
The candidate will demonstrate an understanding of how attackers use tunneling and covert channels to cover their tracks on a network, and the strategies involved in defending against them.

Covering Tracks: Systems
The candidate will demonstrate an understanding of how attackers hide files and directories on Windows and Linux hosts and how they attempt to cover their tracks.

Denial of Service Attacks
The candidate will demonstrate a comprehensive understanding of the different kinds of Denial of Service attacks and how to defend against them.

Exploiting Systems using Netcat
The candidate will demonstrate an understanding of how to properly use the Netcat utility and how to defend against it.

Format String Attacks
The candidate will demonstrate a comprehensive understanding of how format string attacks work and how to defend against them.

Incident Handling Overview and Preparation
The candidate will demonstrate an understanding of what Incident Handling is, why it is important, and an understanding of best practices to take in preparation for an Incident.

Incident Handling Phase 2 Identification
The candidate will demonstrate an understanding of important strategies to gather events, analyze them, and determine if we have an incident.

Incident Handling Phase 3 Containment
The candidate will demonstrate an understanding of high-level strategies to prevent an attacker from causing further damage to the victim after discovering the incident.

Incident Handling: Recovering and Improving Capabilities
The candidate will demonstrate an understanding of the general approaches to get rid of the attacker’s artifacts on compromised machines, the general strategy to safely restore operations, and the importance of the incident report and “lessons learned” meetings.

IP Address Spoofing
The candidate will demonstrate an understanding of what IP Spoofing is, the three different types of spoofing, and strategies to defend against it.

Network Sniffing
The candidate will know what network sniffing is, how to use common sniffing tools, and how to defend against sniffers.

Password Attacks
The candidate will demonstrate a detailed understanding of the three methods of password cracking.

Reconnaissance
The candidate will demonstrate an understanding of public and open source reconnaissance techniques.

Rootkits
The candidate will demonstrate an understanding of how user-mode and kernel-mode rootkits operate, what their capabilities are and how to defend against them.

Scanning: Host Discovery
The candidate will demonstrate an understanding of the tools and techniques used for host discovery on wired and wireless networks.

Scanning: Network and Application Vulnerability scanning and tools
The candidate will demonstrate an understanding of the fundamentals of network and application vulnerability scanners, common commercial and open source tools, and how to defend against them.

Scanning: Network Devices (Firewall rules determination, fragmentation, and IDS/IPS evasion)
The candidate will demonstrate an understanding of how to use Firewalk to determine firewall policies, the general principles of IP fragmentation attacks, why they are used, as well as the ability to identify them.

Scanning: Service Discovery
The candidate will demonstrate an understanding of the tools and techniques used for network mapping, port scanning, and passive fingerprinting techniques and how to defend against them.

Session Hijacking, Tools and Defenses
The candidate will demonstrate an understanding of the definition of session hijacking, the two methods commonly used and why it is effective. Additionally, the candidate will demonstrate an understanding of how to identify common hijacking tools and the strategies to prepare for, identify and contain hijacking attacks.

Types of Incidents
The candidate will demonstrate an understanding of multiple types of incidents, including espionage, unauthorized use, intellectual property, and insider threats and apply strategies to prevent or address these cases.

Virtual Machine Attacks
The candidate will demonstrate an understanding of the virtual machine environment from an attackers perspective, including targets and detection, and how to defend against threats.

Web Application Attacks
The candidate will demonstrate an understanding of the value of the Open Web Application Security Project (OWASP), as well as different Web App attacks such as account harvesting, SQL injection, Cross-Site Scripting and other Web Session attacks.

Worms, Bots & Bot-Nets
The candidate will demonstrate a detailed understanding of what worms, bots and bot-nets are, and how to protect against them.

Advanced Password Attacks
The candidate will be able to use additional methods to attack password hashes and authenticate.

Attacking Password Hashes
The candidate will be able to obtain and attack password hashes and other password representations.

Command Shell vs. Terminal Access
The candidate will know the benefits, limitations, and distinguishing characteristics of command shell and terminal access.

Enumerating Users
The candidate will be able to enumerate users through different methods.

Exploitation Fundamentals
The candidate will be able to demonstrate the fundamental concepts associated with the exploitation phase of a pentest.

General Web Application Probing
The candidate will be able to use tools and proxies to understand and exploit web application weaknesses.

Initial Target Scanning
The candidate will be able to conduct port, operating system and service version scans and analyze the results.

Metasploit
The candidate will be able to use and configure the Metasploit Framework at an intermediate level.

Moving Files with Exploits
The candidate will be able to use exploits to move files between remote systems.

Password Attacks
The candidate will understand types of password attacks, formats, defenses, and the circumstances under which to use each password attack variation. The candidate will be able to conduct password guessing attacks.

Pen-testing Foundations
The candidate will be able to demonstrate the fundamental concepts associated with pen-testing.

Pen-testing Process
The candidate will be able to utilize a process-oriented approach to pentesting and reporting.

Pen-Testing via the Command Line
The candidate be able to use advanced Windows command line skills during a pen test.

Reconnaissance
The candidate will understand the fundamental concepts of reconnaissance and will understand how to obtain basic, high level information about the target organization and network, often considered information leakage, including but not limited to technical and non technical public contacts, IP address ranges, document formats, and supported systems.

Scanning for Targets
The candidate will be able to use the appropriate technique to scan a network for potential targets.

Vulnerability Scanning
The candidate will be able to conduct vulnerability scans and analyze the results.

Web Application Attacks
The candidate will be able to utilize common web application attacks.

Wireless Crypto and Client Attacks
The candidate will be able to utilize wireless cryptographic and client attacks including but not limited to hijacking and key attacks.

Wireless Fundamentals
The candidate will understand the fundamental concepts associated with wireless networks.

Analyzing Network and Wireless Design
The candidate will demonstrate familiarity with network design principles and decisions, and with basic wireless security issues.

Creating and Auditing a Rulebase
The candidate will demonstrate an understanding of building and verifying firewall rulebases that serve the needs of the business and map to security policy.

Firewall Assessment and Penetration Testing
The candidate will demonstrate a thorough understanding of assessing and validating the security of a firewall.

Host-Based Detection and DLP
The candidate will demonstrate understanding of the capabilities of HIDS and HIPS, and be familiar with DLP techniques.

Incident Detection and Analysis
The candidate will demonstrate a basic understanding of detecting incidents, intrusions, and preserving evidence.

IOS and Router Security
The candidate will demonstrate understanding of the basics of Cisco IOS and router hardening through applying ACLs

IPv6 and ICMPv6
The candidate will understand the basics of IP and ICMP version 6.

Log Collection and Analysis
The candidate will demonstrate understanding of techniques for centralizing log collection and analyzing firewall logs.

NAT and Proxies
The candidate will demonstrate understanding of transparent, non-transparent, and reverse proxy functionality, and the four standard implementations of NAT.

Netfilter iptables
The candidate will understand the features and configuration of the free firewall, Netfilter.

Network Access Control
The candidate will be familiar with Network Access Control theory.

Network-Based Intrusion Detection
The candidate will demonstrate an understanding of signature-based network intrusion detection.

Packet Filters and Inspection
The candidate will demonstrate an understanding of how static and stateful packet filters work.

Packet Fragmentation
The candidate will demonstrate an understanding of how fragmentation works and fragmentation-based attacks.

Perimeter Concepts and IP Fundamentals
The candidate will demonstrate a thorough understanding of the IP header, and basic perimeter concepts including services, firewalls, and layered security.

Securing Hosts and Services
The candidate will demonstrate an understanding of the principles, tools, and techniques for securing and hardening hosts and services.

TCP/IP Protocols
The candidate will demonstrate a thorough understanding of TCP, UDP and ICMP.

VPN Design and Auditing
The candidate will demonstrate an understanding of VPN authentication, encryption and placement techniques.

VPN Implementation
The candidate will demonstrate an understanding of IPSEC, SSL and SSH as VPN technologies.

802.11
The candidate will be familiar with the collection of standards, drafts and recommendations collectively known as the 802.11 specification.

802.11 Fuzzing Attacks
The candidate will be able to perform basic fuzzing attacks.

Bluetooth
The candidate will be familiar with the structure, uses and weaknesses of Bluetooth.

Bridging the Air Gap
The candidate will be able to use a compromised wireless system to further compromise a wired network.

DECT
The candidate will be familiar with DECT including how to attack and secure it.

DoS on Wireless Networks
The candidate will be able to perform and defend against common DoS attacks.

EAP and Cipher Suit Selection
The candidate demonstrate the ability to select the proper type authentication and encryption method for a given use.

Hotspots
The candidate will be familiar with the identification and auditing of hotspots.

LEAP
The candidate will be familiar with LEAP including how to attack and secure it.

Other Wireless Attacks
The candidate will be familiar attacks against other types of wireless devices.

PEAP
The candidate will be familiar with PEAP including how to attack and secure it.

Rogue Networks
The candidate will understand how to identify and protect against rogue networks.

Securing and Configuring Wireless Clients
The candidate will demonstrate an understanding of techniques used to manage client systems and the related wireless parameters.

Sniffing Wireless
The candidate will be capable of capturing wireless traffic.

TKIP
The candidate will be familiar with TKIP including how to attack and secure it.

WEP
The candidate will be familiar with Wep including how to attack and secure it.

Wireless Basics
The candidate will be familiar with common wireless threats as well as current wireless network standards.

WLAN Auditing Methodologies
The candidate will be familiar with the basics of auditing wireless networks.

WLAN Intrusion Detection Technology
The candidate will be familiar with the use of IDS systems as related to wireless networks.

WPA2
The candidate will be familiar with WPA2 including how to attack and secure it.

Zigbee
The candidate will be familiar with Zigbee including how to attack and secure it.

Accessing the Network
The candidate will demonstrate an understanding of how to bypass network access control systems.

Advanced Fuzzing Techniques
The candidate will be able to develop custom fuzzing test sequences using the Sulley framework.

Advanced Stack Smashing
The candidate will demonstrate an understanding of how to write advanced stack overflow exploits against canary-protected programs and ASLR.

Crypto for Pen Testers
The candidate will be able to attack and exploit common weaknesses in cryptographic implementations.

Escaping Restricted Environments
The candidate will demonstrate an understanding of restricted environments in Linux and Windows, Desktop restriction techniques, as well as tools and techniques for bypassing them.

Exploiting the Network
The candidate will demonstrate an understanding of how to exploit common vulnerabilities in modern networks attacking client systems and common network protocols.

Fuzzing Introduction and Operation
The candidate will demonstrate an understanding of the benefits and practical application of protocol fuzzing to identify flaws in target software systems.

Introduction to Memory and Dynamic Linux Memory
The candidate will demonstrate a basic understanding of X86 processor architecture, Linux memory management, assembly and the linking and loading process.

Introduction to Windows Exploitation
The candidate will demonstrate an understanding of Windows constructs required for exploitation and the most common OS and Compile-Time Controls.

Manipulating the Network
The candidate will demonstrate an understanding of how to manipulate common network systems to gain escalated privileges and the opportunity to exploit systems.

Network Boot Attacks
The candidate will be able to attack and exploit common weaknesses in network boot environments, including DHCP, BOOTP, and PXE.

Python and Scapy For Pen Testers
The candidate will demonstrate an understanding of the ability to read and modify Python scripts and packet crafting using Scapy to enhance functionality as required during a penetration test.

Shellcode
The candidate will demonstrate the ability to write shellcode on the Linux operating system, and demonstrate an understanding of the Windows shellcode methodology.

Smashing the Stack
The candidate will demonstrate an understanding of how to write basic exploits against stack overflow vulnerabilities.

Windows Overflows
The candidate will demonstrate an understanding of how to exploit Windows vulnerabilities on the stack, and bypass memory protections.

AIDE
The candidate will demonstrate the ability to configure AIDE and utilize it to detect intrusions.

Apache
The candidate will be able to demonstrate an understanding of how to securely setup and configure an Apache server

Best Practices for Kernel Tuning and Warning Banners
The candidate will demonstrate the ability to implement best-practice configurations for kernel parameters and warning banners.

Boot Services
The candidate will be able to demonstrate an understanding of disabling unnecessary services at boot time

Chroot()
The candidate will understand chroot() strengths and weaknesses, how to configure services in a chroot() environment and how to configure scponly with chroot().

DNS- BIND
The candidate will understand DNS and be able to implement best practices for secure BIND configuration.

DNSSec
The candidate will demonstrate a fundamental understanding of DNSSec and how it provides increased security over DNS.

Evidence Collection and Preservation
The candidate will demonstrate the ability to collect evidence from compromised hosts and maintain the chain of custody in a forensically sound manner.

Forensic Analysis
The candidate will demonstrate the ability to conduct rudimentary forensic analysis on images of compromised systems.

Forensic Preparation and Incident Handling
The candidate will demonstrate understanding of the IR process as well as preparations and planning especially where forensics is concerned.

Host Based Firewalls – iptables
The candidate will demonstrate an understanding of firewall configuration in general and basic iptables commands used to create a host based firewall.

Intro to Forensics
The candidate will demonstrate an understanding of the forensic process and fundamental forensic concepts.

OS Install and Patching
The candidate will demonstrate an under standing of secure OS installation options and patching techniques.

Physical, User Account, and Password Access Control
The candidate will demonstrate an understanding of physical security issues and fundamental user account/password security with respect to Linux/Unix systems.

SE Linux
The candidate will demonstrate the ability to use basic SELinux commands understand how SELinux can be used to create and enforce a security policy

Sendmail
The candidate will demonstrate the ability to set-up and configure sendmail securely.

SSH
The candidate will be able to securely configure SSH to perform numerous Unix security/sysadmin tasks

Stack Smashing
The candidate will understand how traditional memory based attacks work.

Sudo
The candidate will demonstrate the ability to configure sudo for access control in Linux/Unix environments.

Syslog-NG
The candidate will demonstrate the ability to configure a centralized logging server using syslog-ng.

Unix Logging
The candidate will understand how to configure logging features native to Unix environments.

Using Akka for backpressure to Spark

FlowBot

Screen Shot 2014-10-25 at 1.26.17 PM

Screen Shot 2014-10-25 at 12.25.02 AM

Screen Shot 2014-10-25 at 12.09.30 AM

Commands

  • incident response checklists
  • memory capture
  • routine live memory analysis reporting
  • elasticsearch netflows
  • asset listing
  • users listing
  • users login histories
  • remote login histories
  • password changes histories
  • patch management
  • last contact firewall
  • last contact SIEM
  • software inventories
  • graph analysis netflows
  • ports listing
  • connection listing
  • stats. history
  • anomaly detection
  • dns resolution
  • passive dns lookup
  • PCAP analysis
  • osquery lookups
  • ASN
  • malware
  • blacklists
  • whitelists
  • google
  • long running history
  • zip functionality and hosting
  • ssdeep
  • cuckoo sandbox submission
  • md5
  • google safebrowsing
  • Carbon Black
  • NetWitness Pivot Query
  • RSA NetWitness
  • RSA Security Analytics graphs
  • url void
  • safe search
  • malware domain search
  • centralops
  • bit9 md5
  • virustotal
  • dns
  • asn
  • netflow
  • internal search history
  • betweeness and centrality measures
  • mathy anomaly detection
  • ML AD
  • command lookup
  • gifs
  • emoticons
  • weather
  • jokes
  • compliments
  • new
  • horoscope
  • help
  • movie quotes
  • limericks
  • daily standup reminder
  • vacay pics
  • cute pics
  • corgis
  • cat
  • youtube
  • calendar
  • fortune cookie
  • message of the day
  • password generator
  • plugins like hubbot

 

How it works:

  1. Private message to FlowBot
  2. A text pattern detected in any message
  3. HTTP response triggered when there is a match

Things that can be done:

  1. Executing a shell command
  2. Executing something on a remote server
  3. HTTP get
  4. Listing of data sources

 

Links

 

Apache Spark now on AWS-EMR from S3

Joining firewall and geolocation log data with Apache Spark

val format = new java.text.SimpleDateFormat("yyyy-MM-dd")
case class Register (d: java.util.Date, uuid: String, cust_id: String, lat: Float, lng: Float)
case class Click (d: java.util.Date, uuid: String, landing_page: Int)

val reg = sc.textFile("geoLocation.tsv").map(_.split("\t")).map(
 r => (r(1), Register(format.parse(r(0)), r(1), r(2), r(3).toFloat, r(4).toFloat))
 )

val clk = sc.textFile("dnsEntry.tsv").map(_.split("\t")).map(
 c => (c(1), Click(format.parse(c(0)), c(1), c(2).trim.toInt))
 )

reg.join(clk).take(2)

reg.join(clk).toDebugString

Metrics vs Analytics vs Data Mining vs Machine Learning

Metrics    ->    Analytics    ->    Data Mining    ->    Machine Learning

1. Metrics is simple measurements of performance. Aggregates and counts of raw data.

2. Analytics is the ability to slice and dice metrics and aids in the the discovery and communication of meaningful patterns in data. Includes manual segmentation and filtering of data to discover patterns.

3. Data mining uses a computer to automatically discover patterns in larger dataset that the average human can manage. It aids in the discovery of meaningful patterns in data.

4. Machine learning users a computer to automatically discovering patterns in larger datasets. ML can also learn about new patterns and discover unknown patterns.  ML aids in the discovery of meaningful patterns and unknown patterns.

 

Me learning Scala, Akka, Spark, JVM, Intellij, sbt, Java, maven, ant, Functional programming – Meat grinding knob turner

Screen Shot 2014-09-29 at 11.58.29 AM

I thought this photo was really cool, then I found a better one

8812567121_25d8d9ae53_o

real-mountain-biking-225395

Finding attackers with Neo4J

Follow

Get every new post delivered to your Inbox.

Join 46 other followers