Skip to content

Commit fdb9c5e

Browse files
committed
Fixed style issues with kinesis guide
1 parent 036d219 commit fdb9c5e

File tree

1 file changed

+37
-20
lines changed

1 file changed

+37
-20
lines changed

docs/streaming-kinesis-integration.md

Lines changed: 37 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -51,52 +51,67 @@ A Kinesis stream can be set up at one of the valid Kinesis endpoints with 1 or m
5151

5252
</div>
5353
</div>
54-
`streamingContext`: StreamingContext containg an application name used by Kinesis to tie this Kinesis application to the Kinesis stream
5554

56-
`[Kinesis stream name]`: The Kinesis stream that this streaming application receives from
55+
- `streamingContext`: StreamingContext containg an application name used by Kinesis to tie this Kinesis application to the Kinesis stream
5756

58-
*Points to remember:*
57+
- `[Kinesis stream name]`: The Kinesis stream that this streaming application receives from
58+
- The application name used in the streaming context becomes the Kinesis application name
59+
- The application name must be unique for a given account and region.
60+
- The Kinesis backend automatically associates the application name to the Kinesis stream using a DynamoDB table (always in the us-east-1 region) created during Kinesis Client Library initialization.
61+
- Changing the application name or stream name can lead to Kinesis errors in some cases. If you see errors, you may need to manually delete the DynamoDB table.
5962

60-
- The application name used in the streaming context becomes the Kinesis application name
61-
- The application name must be unique for a given account and region.
62-
- The Kinesis backend automatically associates the application name to the Kinesis stream using a DynamoDB table (always in the us-east-1 region) created during Kinesis Client Library initialization.
63-
- Changing the application name or stream name can lead to Kinesis errors in some cases. If you see errors, you may need to manually delete the DynamoDB table.
6463

65-
`[endpoint URL]`: Valid Kinesis endpoints URL can be found [here](http://docs.aws.amazon.com/general/latest/gr/rande.html#ak_region).
64+
- `[endpoint URL]`: Valid Kinesis endpoints URL can be found [here](http://docs.aws.amazon.com/general/latest/gr/rande.html#ak_region).
6665

67-
`[checkpoint interval]`: The interval (ie. Duration(2000) = 2 seconds) at which the Kinesis Client Library saves its position in the stream. For starters, set it to the same as the batch interval of the streaming application.
66+
- `[checkpoint interval]`: The interval (e.g., Duration(2000) = 2 seconds) at which the Kinesis Client Library saves its position in the stream. For starters, set it to the same as the batch interval of the streaming application.
6867

69-
`[initial position]`: Can be either `InitialPositionInStream.TRIM_HORIZON` or `InitialPositionInStream.LATEST` (see Kinesis Checkpointing section and Amazon Kinesis API documentation for more details).
68+
- `[initial position]`: Can be either `InitialPositionInStream.TRIM_HORIZON` or `InitialPositionInStream.LATEST` (see Kinesis Checkpointing section and Amazon Kinesis API documentation for more details).
7069

71-
3. **Deploying:** Package `spark-streaming-flume_{{site.SCALA_BINARY_VERSION}}` and its dependencies (except `spark-core_{{site.SCALA_BINARY_VERSION}}` and `spark-streaming_{{site.SCALA_BINARY_VERSION}}` which are provided by `spark-submit`) into the application JAR. Then use `spark-submit` to launch your application (see [Deploying section](streaming-programming-guide.html#deploying-applications) in the main programming guide).
70+
71+
3. **Deploying:** Package `spark-streaming-kinesis-asl_{{site.SCALA_BINARY_VERSION}}` and its dependencies (except `spark-core_{{site.SCALA_BINARY_VERSION}}` and `spark-streaming_{{site.SCALA_BINARY_VERSION}}` which are provided by `spark-submit`) into the application JAR. Then use `spark-submit` to launch your application (see [Deploying section](streaming-programming-guide.html#deploying-applications) in the main programming guide).
7272

7373
*Points to remember at runtime:*
74+
7475
- Kinesis data processing is ordered per partition and occurs at-least once per message.
76+
7577
- Multiple applications can read from the same Kinesis stream. Kinesis will maintain the application-specific shard and checkpoint info in DynamodDB.
78+
7679
- A single Kinesis stream shard is processed by one input DStream at a time.
80+
7781
<p style="text-align: center;">
78-
<img src="img/streaming-kinesis-arch.png"
79-
title="Spark Streaming Kinesis Architecture"
80-
alt="Spark Streaming Kinesis Architecture"
81-
width="60%" />
82-
<!-- Images are downsized intentionally to improve quality on retina displays -->
83-
</p>
82+
<img src="img/streaming-kinesis-arch.png"
83+
title="Spark Streaming Kinesis Architecture"
84+
alt="Spark Streaming Kinesis Architecture"
85+
width="60%"
86+
/>
87+
<!-- Images are downsized intentionally to improve quality on retina displays -->
88+
</p>
89+
8490
- A single Kinesis input DStream can read from multiple shards of a Kinesis stream by creating multiple KinesisRecordProcessor threads.
91+
8592
- Multiple input DStreams running in separate processes/instances can read from a Kinesis stream.
93+
8694
- You never need more Kinesis input DStreams than the number of Kinesis stream shards as each input DStream will create at least one KinesisRecordProcessor thread that handles a single shard.
95+
8796
- Horizontal scaling is achieved by adding/removing Kinesis input DStreams (within a single process or across multiple processes/instances) - up to the total number of Kinesis stream shards per the previous point.
97+
8898
- The Kinesis input DStream will balance the load between all DStreams - even across processes/instances.
99+
89100
- The Kinesis input DStream will balance the load during re-shard events (merging and splitting) due to changes in load.
101+
90102
- As a best practice, it's recommended that you avoid re-shard jitter by over-provisioning when possible.
91-
- Each Kinesis input DStream maintains its own checkpoint info. See the Kinesis Checkpointing section for more details.
103+
104+
- Each Kinesis input DStream maintains its own checkpoint info. See the Kinesis Checkpointing section for more details.
105+
92106
- There is no correlation between the number of Kinesis stream shards and the number of RDD partitions/shards created across the Spark cluster during input DStream processing. These are 2 independent partitioning schemes.
93107

94108
#### Running the Example
95109
To run the example,
96110

97111
- Download Spark source and follow the [instructions](building-with-maven.html) to build Spark with profile *-Pkinesis-asl*.
98112

99-
mvn -Pkinesis-asl -DskipTests clean package
113+
mvn -Pkinesis-asl -DskipTests clean package
114+
100115

101116
- Set up Kinesis stream (see earlier section) within AWS. Note the name of the Kinesis stream and the endpoint URL corresponding to the region where the stream was created.
102117

@@ -126,8 +141,10 @@ To run the example,
126141
This will push 1000 lines per second of 10 random numbers per line to the Kinesis stream. This data should then be received and processed by the running example.
127142

128143
#### Kinesis Checkpointing
129-
- Each Kinesis input DStream periodically stores the current position of the stream in the backing DynamoDB table. This allows the system to recover from failures and continue processing where the DStream left off.
144+
- Each Kinesis input DStream periodically stores the current position of the stream in the backing DynamoDB table. This allows the system to recover from failures and continue processing where the DStream left off.
145+
130146
- Checkpointing too frequently will cause excess load on the AWS checkpoint storage layer and may lead to AWS throttling. The provided example handles this throttling with a random-backoff-retry strategy.
147+
131148
- If no Kinesis checkpoint info exists when the input DStream starts, it will start either from the oldest record available (InitialPositionInStream.TRIM_HORIZON) or from the latest tip (InitialPostitionInStream.LATEST). This is configurable.
132149
- InitialPositionInStream.LATEST could lead to missed records if data is added to the stream while no input DStreams are running (and no checkpoint info is being stored).
133150
- InitialPositionInStream.TRIM_HORIZON may lead to duplicate processing of records where the impact is dependent on checkpoint frequency and processing idempotency.

0 commit comments

Comments
 (0)