안녕하세요 오늘은 베스핀글로벌 D&A 한제호 님이 작성해 주신 ‘Kinesis’에 대해 알아보겠습니다.
궁금하신 부분이 있으시면 댓글을 달아주세요 🙂
1. overview
1-1. What is Streaming?
- 스트리밍 데이터는 수천개의 데이터 소스에서 연속적으로 생성되는 데이터로서 일반적으로 데이터 레코드를 작은 크기로 동시에 전송하는 방식
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-116.png?resize=885%2C330&ssl=1)
1-2. Data Service in AWS
- AWS 에 존재하는 데이터 처리 서비스
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-117.png?resize=972%2C399&ssl=1)
1-3. Kinesis overview
- 스트림 데이터를 수집,처리,분석 가능한 서비스
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-118.png?resize=904%2C390&ssl=1)
1-4. Kinesis 리전표
- 리전별 Kinesis 서비스 GA 내역
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-119.png?resize=862%2C344&ssl=1)
2. Kinesis Data Stream
2-1. Kinesis Data Streams
- 대량의 스트림 데이터를 가능한 완전 관리형 서비스
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-120.png?resize=840%2C404&ssl=1)
2-2. Kinesis Data Streams Architecture
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-121.png?resize=930%2C460&ssl=1)
2-3. Kinesis Data Streams – 제약 사항
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-122.png?resize=794%2C377&ssl=1)
2-4. Kinesis Data Streams- 리샤딩
- CLI 또는 SDK를 통해 샤드에 대해 리샤딩 가능
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-123.png?resize=989%2C416&ssl=1)
2-5. Kinesis Data Streams – Agent, API, KCL, KPL
- Agent, API, KPL, KCL을 통해 Producer 및 Consumer를 구성할 수 있음
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-124.png?resize=1003%2C413&ssl=1)
2-6. Kinesis Data Streams – Consumer 주요 API (1/2)
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-125.png?resize=978%2C429&ssl=1)
2-6. Kinesis Data Streams- Consumer 주요 API (2/2)
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-126.png?resize=984%2C403&ssl=1)
2-7. Kinesis Data Streams – Standard consumers
- AWS SDK, KCL을 통해 구현 가능
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-127.png?resize=964%2C403&ssl=1)
2-8. Kinesis Data Streams – Enhanced fan-out consumers
- AWS SDK, KCL 2.0을 통해 구현 가능
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-128.png?resize=982%2C407&ssl=1)
3. Kinesis Firehose
3-1. Kinesis Firehose
- 대량의 스트림 데이터를 다양한 AWS 서비스와 연계가능한 완전 관리형 서비스
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-129.png?resize=935%2C400&ssl=1)
3-2. Kinesis Firehose Architecture
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-130.png?resize=918%2C465&ssl=1)
3-3. Kinesis Firehose – 제약 사항 (1/2)
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-133.png?resize=972%2C414&ssl=1)
3-3. Kinesis Firehose – 제약 사항 (2/2)
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-132.png?resize=839%2C331&ssl=1)
3-4. Kinesis Firehose – Data Flow (1/2)
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-134.png?resize=927%2C465&ssl=1)
3-4. Kinesis Firehose – Data Flow (2/2)
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-135.png?resize=930%2C463&ssl=1)
3-5. Kinesis Firehose – 주요 설정 화면
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-136.png?resize=1013%2C471&ssl=1)
4. Kinesis Analytics
4-1. Kinesis Analytics
- 실시간 스트리밍 데이터를 Read, Process, Deliver 할 수 있는 완전 관리형 서비스
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-137.png?resize=832%2C430&ssl=1)
4-2. Kinesis Analytics Architecture
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-138.png?resize=930%2C465&ssl=1)
4-3. Kinesis Analytics – 제약 사항
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-139.png?resize=885%2C381&ssl=1)
4-4. Kinesis Analytics – 윈도우 모드 쿼리
- 지정된 시간 내에 수집된 이벤트들을 대상으로 집계를 수행하기 위해 윈도우 모드를 지원함
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-140.png?resize=999%2C355&ssl=1)
4-5. Kinesis Analytics – 예제 – Architecture
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-141.png?resize=985%2C439&ssl=1)
4-5. Kinesis Analytics – 예제 – Client Application
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-142.png?resize=884%2C430&ssl=1)
4-5. Kinesis Analytics – 예제 – Source 구성
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-143.png?resize=962%2C446&ssl=1)
4-5. Kinesis Analytics – 예제 – Application 처리
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-144.png?resize=1007%2C464&ssl=1)
4-5. Kinesis Analytics – 예제 – Destination 구성
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-145.png?resize=1014%2C458&ssl=1)
4-5. Kinesis Analytics – 예제 – Athena를 통한 S3 조회
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-146.png?resize=974%2C465&ssl=1)
4-6. Kinesis Analytics – 연계 Case
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-147.png?resize=931%2C440&ssl=1)
5. Appendix
5-1. Appendix. 서비스 비교
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-148.png?resize=850%2C445&ssl=1)
5-2. Appendix. 사례
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-149.png?resize=554%2C287&ssl=1)
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-150.png?resize=462%2C273&ssl=1)
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-151.png?resize=933%2C477&ssl=1)
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-152.png?resize=898%2C279&ssl=1)
5-3. Appendix. Kinesis Video Stream 예제
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-153.png?resize=651%2C378&ssl=1)
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-154.png?resize=925%2C415&ssl=1)
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-155.png?resize=994%2C468&ssl=1)
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-156.png?resize=935%2C465&ssl=1)
![](https://i0.wp.com/bespin-wordpress-bucket.s3.ap-northeast-2.amazonaws.com/wp-content/uploads/2023/07/1-157.png?resize=1018%2C450&ssl=1)
감사합니다 🙂