BigSnarf blog

Infosec FTW

Anomaly Detection Python T-Digest

https://www.usenix.org/system/files/conference/hotcloud14/hotcloud14-vallis.pdf

Screen Shot 2016-05-01 at 12.16.13 AM

Parameterized anomaly detection settings

 

Event correlation is a technique for making sense of a large number of events and pinpointing the few events that are really important in that mass of information. This is accomplished by looking for and analyzing relationships between events.

Cloudtrail Dashboard

awsServiceEventTrackingScreenshot

Get Moar Data or Tough Luck

ml_map.png

scikit-learn.org/stable/_static/ml_map.png

Lambda Architecture – Redis/Postgres

Lambda-architecture-illustration

 

Speed Layer (Blue):

Query (Green Output):

Batch Layer (Purple):

  • S3 or Postgres
  • Luigi/Cro
  • Prediction Models Trained
  • Postgres

TensorFlow and Kaggle

Screen Shot 2016-04-17 at 12.02.46 AM

Screen Shot 2016-04-17 at 12.04.35 AM

TensorFlow explained

Anomaly detection with Bayesian networks

Anomaly detection, also known as outlier detection, is the process of identifying data which is unusual. I have been using basic python Markov Chains or more complex python MCMC.

Anomaly detection can also be used to detect unusual time series. Bayesian networks are well suited for anomaly detection, because they can handle high dimensional data, which humans find difficult to interpret.

One typical way we can use data visualizations to identify some anomalies and these are clearly visible by plotting individual variables. More often anomalies are far more subtle, and are based on the interaction of many variables.

detect

Screen Shot 2016-04-10 at 9.12.48 AMScreen Shot 2016-04-10 at 9.07.25 AM

Here is a nice notebook on python mcmc:

I haven’t read the previous blog post on FFT. There are lots of time series analysis.

An interesting method for detection of patterns is using “Shape Search”:

Screen Shot 2016-04-10 at 8.58.20 AM

 

But I think there are interesting things using signal processing as well for AD like Median Filter.

http://docs.scipy.org/doc/scipy-0.16.0/reference/generated/scipy.ndimage.filters.median_filter.html

https://github.com/bugra/pydata-sv-2014

http://probcomp.csail.mit.edu/bayesdb/satellites-notebook.html

Real Time Analytics with PostgreSQL

Screen Shot 2016-04-02 at 10.38.53 PMScreen Shot 2016-03-29 at 8.18.46 AM

Screen Shot 2016-04-02 at 8.48.39 AM

 

RF, SVM, KNN ensembles training

Screen Shot 2016-03-26 at 12.24.10 AM

Spark OLAP

Follow

Get every new post delivered to your Inbox.

Join 53 other followers