Skip to content
All projects

Data, 2025

Real-time stock market pipeline

A streaming data pipeline that pushes live stock market events through Kafka on AWS and makes them queryable with SQL in Athena.

Source code

Real-time stock events

Streaming

Overview

An end-to-end data engineering project on real-time stock market data, built to learn how streaming systems are put together in the cloud.

The problem

Batch jobs can't keep up with markets that change every second. The goal was a pipeline that ingests events as they happen and makes them available for analysis right away.

What I did

  1. 1.Wrote a Python producer that simulates a live stock market feed from historical index data.
  2. 2.Ran Apache Kafka on an Amazon EC2 instance to stream the events.
  3. 3.Built a consumer that writes each event to Amazon S3.
  4. 4.Used an AWS Glue crawler to discover the schema automatically and register it in the Glue Data Catalog.
  5. 5.Queried the live data with SQL in Amazon Athena.

Results

  • A working streaming pipeline from producer to queryable table, with no manual schema management.

Next project

Alliance IT ticketing system