ADD: 자료 보충

kyungjunleeme · Nov 28, 2024 · 363b9b9 · 363b9b9
1 parent d645062
commit 363b9b9
Show file tree

Hide file tree

Showing 36 changed files with 2,212 additions and 36 deletions.
diff --git a/GURU/고승범.md b/GURU/고승범.md
@@ -0,0 +1,9 @@
+
+https://www.popit.kr/author/peter5236
+
+1개씩 씹어 먹어 볼까?
+
+
+
+
+
diff --git a/KIP/KIP-392_Allow consumers to fetch from closest replica.pdf b/KIP/KIP-392_Allow consumers to fetch from closest replica.pdf
diff --git a/KIP/KIP-879 copy.md b/KIP/KIP-879 copy.md
diff --git a/KIP/KIP-879.md b/KIP/KIP-879.md
diff --git a/KIP/KIP-879_Multi-level Rack Awareness.pdf b/KIP/KIP-879_Multi-level Rack Awareness.pdf
diff --git a/KIP/KIP-392.md → KIP/Kafka Client-side Assignment Proposal.md b/KIP/KIP-392.md → KIP/Kafka Client-side Assignment Proposal.md
diff --git a/KIP/Kafka Client-side Assignment Proposal.pdf b/KIP/Kafka Client-side Assignment Proposal.pdf
diff --git a/README.md b/README.md
@@ -51,6 +51,10 @@ https://www.yes24.com/Product/Goods/118397432
 ### Confluent Document
 - https://docs.confluent.io/kafka/overview.html
 
+### 카프카 활용 사례
+- https://techblog.woowahan.com/17386/?fbclid=IwY2xjawGvWTtleHRuA2FlbQIxMQABHT7cUG57I0xphRt_-GJkb91oFgCc0vgX_DuyM6fpXthHzve9-Gs2T_0xeg_aem_BIbRnX4qZ80ybfnqHDQ07w
+- 
+
 
 ### apache mailing list
 - https://issues.apache.org/jira/projects/KAFKA/issues
@@ -66,7 +70,10 @@ https://kafka.apache.org/39/javadoc/org/apache/kafka/clients/consumer/ConsumerPa
 ### 동작을 이해하기 위해 참고하면 좋은 파이썬 테스트 코드들
 - https://github.com/apache/kafka/blob/trunk/tests/kafkatest/tests/streams/streams_broker_compatibility_test.py  # kafka
 - https://github.com/dpkp/kafka-python/blob/master/test/test_consumer.py  # kafka-python
+- https://cwiki.apache.org/confluence/display/KAFKA/tutorial+-+set+up+and+run+Kafka+system+tests+with+ducktape  # 유투브 영상 볼수 없으나, ducktape라는 라이브러리 확인 필요
 
+###
+- 모니터링 도구 비교(https://devocean.sk.com/blog/techBoardDetail.do?ID=163980)
 
 ### 후원
 - https://okky.kr/  # 음료비 지원 및 개발 생태계 활성화 기여
diff --git a/assets/confluent_주키퍼_환경변수.png b/assets/confluent_주키퍼_환경변수.png
diff --git a/assets/개괄변수.png b/assets/개괄변수.png
diff --git a/assets/도커_카프카_환경변수.png b/assets/도커_카프카_환경변수.png
diff --git a/chapter01/241122_평일반_경준.md b/chapter01/241122_평일반_경준.md
@@ -0,0 +1,64 @@
+1.1 단방향 insert
+1.2 trace -> 포맷이 달라진것, 
+
+redis hashing key (클러스터링 환경에서도 절대 터지면 안되는 용도). hashing key
+
+sticky hashing, 로드밸런서, 고정이된다.
+
+redis, 로드 밸런서, hashing]
+
+컨트롤러는 클러스터안의 현재 작동중인 브로중 하나가 자동으로 선정된다. -> 주키퍼에 
+
+rabitmq
+
+저자를 통해서 책의 의도를 파악한다.
+
+```
+2024-11-23 00:25:23 [2024-11-22 15:25:23,630] INFO [GroupMetadataManager brokerId=1] Finished loading offsets and group metadata from __consumer_offsets-14 in 4 milliseconds for epoch 0, of which 4 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,630] INFO [GroupMetadataManager brokerId=1] Finished loading offsets and group metadata from __consumer_offsets-23 in 3 milliseconds for epoch 0, of which 3 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,630] INFO [GroupMetadataManager brokerId=1] Finished loading offsets and group metadata from __consumer_offsets-38 in 3 milliseconds for epoch 0, of which 3 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,630] INFO [GroupMetadataManager brokerId=1] Finished loading offsets and group metadata from __consumer_offsets-8 in 3 milliseconds for epoch 0, of which 3 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,630] INFO [GroupMetadataManager brokerId=1] Finished loading offsets and group metadata from __consumer_offsets-45 in 3 milliseconds for epoch 0, of which 3 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,630] INFO [GroupMetadataManager brokerId=1] Finished loading offsets and group metadata from __consumer_offsets-15 in 3 milliseconds for epoch 0, of which 3 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,630] INFO [GroupMetadataManager brokerId=1] Finished loading offsets and group metadata from __consumer_offsets-30 in 3 milliseconds for epoch 0, of which 3 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,630] INFO [GroupMetadataManager brokerId=1] Finished loading offsets and group metadata from __consumer_offsets-0 in 3 milliseconds for epoch 0, of which 3 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,630] INFO [GroupMetadataManager brokerId=1] Finished loading offsets and group metadata from __consumer_offsets-35 in 3 milliseconds for epoch 0, of which 3 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,630] INFO [GroupMetadataManager brokerId=1] Finished loading offsets and group metadata from __consumer_offsets-5 in 3 milliseconds for epoch 0, of which 3 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,630] INFO [GroupMetadataManager brokerId=1] Finished loading offsets and group metadata from __consumer_offsets-20 in 3 milliseconds for epoch 0, of which 3 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,630] INFO [GroupMetadataManager brokerId=1] Finished loading offsets and group metadata from __consumer_offsets-27 in 3 milliseconds for epoch 0, of which 3 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,630] INFO [GroupMetadataManager brokerId=1] Finished loading offsets and group metadata from __consumer_offsets-42 in 3 milliseconds for epoch 0, of which 3 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,630] INFO [GroupMetadataManager brokerId=1] Finished loading offsets and group metadata from __consumer_offsets-12 in 3 milliseconds for epoch 0, of which 3 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,630] INFO [GroupMetadataManager brokerId=1] Finished loading offsets and group metadata from __consumer_offsets-21 in 3 milliseconds for epoch 0, of which 3 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,630] INFO [GroupMetadataManager brokerId=1] Finished loading offsets and group metadata from __consumer_offsets-36 in 3 milliseconds for epoch 0, of which 3 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,630] INFO [GroupMetadataManager brokerId=1] Finished loading offsets and group metadata from __consumer_offsets-6 in 3 milliseconds for epoch 0, of which 3 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,630] INFO [GroupMetadataManager brokerId=1] Finished loading offsets and group metadata from __consumer_offsets-43 in 3 milliseconds for epoch 0, of which 3 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,631] INFO [GroupMetadataManager brokerId=1] Finished loading offsets and group metadata from __consumer_offsets-13 in 4 milliseconds for epoch 0, of which 3 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,631] INFO [GroupMetadataManager brokerId=1] Finished loading offsets and group metadata from __consumer_offsets-28 in 4 milliseconds for epoch 0, of which 4 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,765] INFO [GroupCoordinator 1]: Dynamic member with unknown member id joins group schema-registry in Empty state. Created a new member id sr-1-ec4b7ed2-7b69-4ed8-9b41-4e0b084126e2 and request the member to rejoin with this id. (kafka.coordinator.group.GroupCoordinator)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,769] INFO [GroupCoordinator 1]: Preparing to rebalance group schema-registry in state PreparingRebalance with old generation 0 (__consumer_offsets-29) (reason: Adding new member sr-1-ec4b7ed2-7b69-4ed8-9b41-4e0b084126e2 with group instance id None; client reason: need to re-join with the given member-id: sr-1-ec4b7ed2-7b69-4ed8-9b41-4e0b084126e2) (kafka.coordinator.group.GroupCoordinator)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,771] INFO [GroupCoordinator 1]: Stabilized group schema-registry generation 1 (__consumer_offsets-29) with 1 members (kafka.coordinator.group.GroupCoordinator)
+2024-11-23 00:25:23 [2024-11-22 15:25:23,779] INFO [GroupCoordinator 1]: Assignment received from leader sr-1-ec4b7ed2-7b69-4ed8-9b41-4e0b084126e2 for group schema-registry for generation 1. The group has 1 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator)
+2024-11-23 00:25:26 [2024-11-22 15:25:26,104] INFO [Controller id=1] Processing automatic preferred replica leader election (kafka.controller.KafkaController)
+2024-11-23 00:25:26 [2024-11-22 15:25:26,104] TRACE [Controller id=1] Checking need to trigger auto leader balancing (kafka.controller.KafkaController)
+2024-11-23 00:25:26 [2024-11-22 15:25:26,111] DEBUG [Controller id=1] Topics not in preferred replica for broker 1 HashMap() (kafka.controller.KafkaController)
+2024-11-23 00:25:26 [2024-11-22 15:25:26,112] TRACE [Controller id=1] Leader imbalance ratio for broker 1 is 0.0 (kafka.controller.KafkaController)
+```
+
+
+
+Consumers maintain membership in a consumer group and ownership of the parti‐
+tions assigned to them by sending heartbeats to a Kafka broker designated as the
+group coordinator (this broker can be different for different consumer groups). The
+heartbeats are sent by a background thread of the consumer, and as long as the con‐
+sumer is sending heartbeats at regular intervals, it is assumed to be alive.
+
+상태머신
+
+https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61312126#KafkaClientsideAssignmentProposal-CoordinatorStateMachine
+
+
+https://dealicious-inc.github.io/2021/04/12/state-machine.html
+
+https://youngminz.netlify.app/posts/managing-state-using-finite-state-machine
+
+
diff --git a/chapter01/경준.md b/chapter01/경준.md
@@ -74,6 +74,7 @@ cf) 제가 직접 겪은 프로그래머스 면접 기출 문제
 - 카프카 서버를 브로커라고 부름
 
 ### **1.3.5 고성능**
+cf) https://www.confluent.io/blog/kafka-producer-and-consumer-internals-4-consumer-fetch-requests/
 
 ### <1> Sequential I/O (순차적 입출력)
 

diff --git a/chapter02/241124_주말반.md b/chapter02/241124_주말반.md
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,9 @@

		https://www.popit.kr/author/peter5236

		1개씩 씹어 먹어 볼까?