<h2 id="heading-project-overview">Project Overview</h2>
This project demonstrates a real-time data pipeline for stock market data. It integrates Apache Kafka for data streaming and multiple AWS services for data storage and querying.
<h3 id="heading-key-features">Key Features:</h3>
<ul>
<li>Real-time data ingestion using Kafka Producer.
</li>
<li>Data storage in AWS S3 via Kafka Consumer.
</li>
<li>Data cataloging using AWS Glue Crawler.
</li>
<li>Data querying with AWS Athena for analysis.
</li>
</ul>
<h2 id="heading-architecture">Architecture</h2>
<ol>
<li>EC2 Instance: Hosts Kafka and Zookeeper.
</li>
<li>Kafka Producer: Generates stock market data.
</li>
<li>Kafka Consumer: Consumes data and sends it to S3.
</li>
<li>S3 Bucket: Stores real-time data.
</li>
<li>AWS Glue Crawler: Creates data catalog from S3.
</li>
<li>AWS Athena: Queries data for insights.
</li>
</ol>
<h2 id="heading-technologies-used">Technologies Used:</h2>
<ul>
<li>Apache Kafka
</li>
<li>AWS S3
</li>
<li>AWS Glue Crawler
</li>
<li>AWS Athena
</li>
<li>EC2 Instance
</li>
</ul>
<h2 id="heading-conclusion">Conclusion:</h2>
This pipeline ensures efficient handling of real-time data for analysis and decision-making in financial markets.

## Project Overview

This project demonstrates a real-time data pipeline for stock market data. It integrates Apache Kafka for data streaming and multiple AWS services for data storage and querying.

### Key Features:

* **Real-time data ingestion** using Kafka Producer.
    
* **Data storage** in AWS S3 via Kafka Consumer.
    
* **Data cataloging** using AWS Glue Crawler.
    
* **Data querying** with AWS Athena for analysis.
    

## Architecture

1. **EC2 Instance**: Hosts Kafka and Zookeeper.
    
2. **Kafka Producer**: Generates stock market data.
    
3. **Kafka Consumer**: Consumes data and sends it to S3.
    
4. **S3 Bucket**: Stores real-time data.
    
5. **AWS Glue Crawler**: Creates data catalog from S3.
    
6. **AWS Athena**: Queries data for insights.
    

## Technologies Used:

* Apache Kafka
    
* AWS S3
    
* AWS Glue Crawler
    
* AWS Athena
    
* EC2 Instance
    

## Conclusion:

This pipeline ensures efficient handling of real-time data for analysis and decision-making in financial markets.

Built a real-time data pipeline for stock market data. It integrates Apache Kafka for data streaming and multiple AWS services for data storage and querying

Real-Time Stock Market Data Pipeline

Real-Time Stock Market Data Pipeline with Kafka and AWS

Blogs focused on cloud data engineering, including database management, ETL processes, data warehousing, and cloud-based data solutions using tools like Snowflake, AWS, SQL, Python and more.
