Cloudtrail Dashboards

CloudTrail Monitoring and Alerting

• http://www.tenable.com/blog/amazon-cloudtrail-monitoring-with-securitycenter-continuous-view
• https://www.cloudlytics.com/product/index/aws-cloudtrail-log-analytics
• http://githublemming.github.io/
• https://twitter.com/BigsnarfDude/status/709270243202347010
• http://logz.io/blog/integrate-aws-cloudtrail-elk-stack/

Are Overview Reports helpful?

• Top 10 Events
• Top 10 Locations
• Top 10 Access Keys
• Top 10 Services
• Top 10 Ip Addresses
• Unauthorized Access

Activity reports helpful?

• List of Instances
• Errors
• Instance Activities

Audit reports helpful?

• List Users
• List Keys
• Access Keys Used
• Locations Used
• User activities
• User patterns

Machine Learning and Data Mining helpful?

• https://bigsnarf.wordpress.com/2016/05/11/identify-compromised-logins/

Get Moar Data or Tough Luck

scikit-learn.org/stable/_static/ml_map.png

Lambda Architecture – Redis/Postgres

 

Speed Layer (Blue):

• Queue: http://python-rq.org/, Kafka/Kinesis
• Speed Layer Processing:  http://www.celeryproject.org/
• Prediction Models Deployed
• Random Write:  Postgres, CMS, TopK, HLL
• https://bigsnarf.wordpress.com/2016/03/29/analytics-with-postgres/

Query (Green Output):

• Flask/Postgres/Aggregate Data serving
• Luigi/Cron Data Processing
• See results of predictions, real time aggregates, and historical queries
• https://gist.github.com/bigsnarfdude/6f9e1296862f9190eb9045c949441a9b

Batch Layer (Purple):

• S3 or Postgres
• Luigi/Cro
• Prediction Models Trained
• Postgres

 

Newer research into lambda with http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf

TensorFlow and Kaggle

TensorFlow explained

https://github.com/bigsnarfdude/TensorFlow-Examples

http://playground.tensorflow.org/#activation=sigmoid&regularization=L1&batchSize=30&dataset=xor&regDataset=reg-plane&learningRate=0.03&regularizationRate=0.001&noise=30&networkShape=8,4&seed=0.87276&showTestData=true&discretize=false&percTrainData=30&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification

 

Anomaly detection with Bayesian networks

Anomaly detection, also known as outlier detection, is the process of identifying data which is unusual. I have been using basic python Markov Chains or more complex python MCMC.

https://www.quantstart.com/articles/Markov-Chain-Monte-Carlo-for-Bayesian-Inference-The-Metropolis-Algorithm

Anomaly detection can also be used to detect unusual time series. Bayesian networks are well suited for anomaly detection, because they can handle high dimensional data, which humans find difficult to interpret.

One typical way we can use data visualizations to identify some anomalies and these are clearly visible by plotting individual variables. More often anomalies are far more subtle, and are based on the interaction of many variables.

Here is a nice notebook on python mcmc:

• http://bugra.github.io/work/notes/2014-04-26/outlier-detection-markov-chain-monte-carlo-via-pymc/

I haven’t read the previous blog post on FFT. There are lots of time series analysis.

An interesting method for detection of patterns is using “Shape Search”:

 

But I think there are interesting things using signal processing as well for AD like Median Filter.

http://docs.scipy.org/doc/scipy-0.16.0/reference/generated/scipy.ndimage.filters.median_filter.html

https://github.com/bugra/pydata-sv-2014

http://probcomp.csail.mit.edu/bayesdb/satellites-notebook.html