Tutorial: Build a pipeline to join streams of real time data

Hands-on tutorial

This GitHub repository contains a sample application to demonstrate the related concepts and provides step-by-step guide to setup and run the end to end demo. It showcases how to build a data enrichment pipeline with streaming joins using a combination of Azure Event Hubs for data ingestion, Azure SQL Database for storing reference data, Azure Stream Analytics for data processing and Azure Cosmos DB for storing “enriched” data.

  • Azure Event Hubs (Input Data source) — ingests raw orders data
  • Azure SQL Database (Reference Data source) — stores reference customer data
  • Azure Stream Analytics (Stream Processing) — joins the stream of orders data from Azure Event Hubs with the static reference customers data
  • Azure Cosmos DB (Output data source) — acts as a “sink” to store enriched orders info

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Abhishek Gupta

Abhishek Gupta

Principal Developer Advocate at AWS | I ❤️ Databases, Go, Kubernetes