May 22, 2015
Posted by on
Apache Spark Use Cases
Our specific use case
Kinesis gets raw logs
Spark Streaming does the counting
Two Tables Created, One for Kinesis Log Position and the Second for Aggregates
DynamoDB stores the aggregations
May 15, 2015
Posted by on
Returns a list of metric values based on a set of criteria. Also returns a set of all tag names and values that are found across the data points.
The time range can be specified with absolute or relative time values. Absolute time values are in milliseconds. Relative time values are specified as an integer duration and a unit. Possible unit values are “milliseconds”, “seconds”, “minutes”, “hours”, “days”, “weeks”, “months”, and “years”. For example, “5 hours” means that metric values submitted 5 hours ago will be returned. The end time is optional. If no end time is specified, the end time is assumed to be now (the current date and time).
The results of the query can be grouped together.There are three ways to group the data; by tags, by a time range, and by value. Grouping is done with the groupBy or groupByKey which is an array of one or more groupers.
Aggregators perform an operation on data points and down samples. For example, you could sum all data points that exist in 5 minute periods.
Aggregators can be combined together. For example, you could sum all data points in 5 minute periods then average them for a week period.
It is possible to filter the data returned by specifying a tag. The data returned will only contain data points associated with the specified tag. Filtering is done using the “tags” property.