Build a real time data streaming system with Amazon Kinesis Data Stream and Kinesis Agent

Lab Details

This lab shows you how to host a sample website on the EC2 Linux instance using apache and how to collect the real-time logs for AWS S3.

This lab will be practiced using Kinesis Data Streams and Kinesis Agent.

Duration: 1 hour and 30 minutes

AWS Region: US East (N. Virginia). us-east-1

Introduction

Amazon Kinesis Data Streams

Data streaming technology allows customers to access, process, and analyze large amounts of data from many sources.

Kinesis data streams are a scalable, durable and reliable real-time streaming data service.

Kinesis data streams are a sequence of data records that can be written to or read from in real-time.

Pricing for data streams is per-shard.

Components

Data record - A unit of data that Kinesis Data Stream stores.

Data stream - A group of data records. Data records within a data stream can be broken down into shards.

Retention period – Length of the time Data records can be accessed from streams. Kinesis data streams store records for 24 hours default and up to 365 day.

Kinesis Client Library – Ensures there is a record processor running for each shard.

Producer converts data records into shards.

Data records are obtained from shards by a consumer

Shard - A sequence of data records that are part of a stream.

You can have more than one shard. When creating the data stream, it is necessary to specify how many shards are required.

The total capacity of a stream is the sum all its shards.

Ingest rate per shard - 1 MB or 1,000 messages per second.

Data read rate per shard - 2 MB per second.

Partition Key – Used to group data within a stream by shard.

The stream records can be sent directly to services such as ElasticSearch, Redshift, and S3, Redshift. Instead of creating consumer apps.

Amazon Kinesis Agent

Amazon Kinesis agent, a Java software program that allows you to easily collect and send data directly to Kinesis Data Firehose, is available.

An agent monitors files continuously and sends new data via your Kinesis Data Firehose delivery stream.