Data Engineering

A Scalable Data Engineering Solution By Hijir Della Wirasti Project Overview This project was conducted as part of a Data Engineering initiative with ID/X Partners, focusing on Data Warehouse Development, ETL Optimization, and SQL Analytics using MSSQL and Talend. The goal was to enhance data integration and streamline reporting for a financial services client by […]

For this Data Engineer project, I implemented a testing workflow for Airflow DAGs. The project involves creating more than 5 tables in MySQL, extracting data to PostgreSQL, and managing the pipeline with dynamic DAGs. Jobs are scheduled to run every 2 hours between 9 AM and 9 PM on Fridays, twice a month (week 1 and week 3), at the 15th minute.

This Data Engineer project creates an ETL pipeline using Airflow. Data is extracted from CSV sources, saved as CSV in the `data/` folder, and loaded into SQLite. The `BranchOperator` selects the extraction task. Includes DAGs, staging folder, SQLite results, and screenshots of the pipeline.

This Data Engineering project involves collecting streaming data using Apache Kafka, performing real-time ETL with Apache Spark Streaming, and validating data. The project includes creating a purchasing event producer, building a streaming job to aggregate daily purchases, and outputting cumulative purchase totals using PySpark.

This project sets up a Kafka cluster using Docker with one broker and Kafka UI (Bitnami or Confluent). A producer generates random events every 5 seconds, and a consumer processes them. The project includes configuring partitions, replication, and consumer groups.

A web scraping program to extract data from the website https://toscrape.com/ and save the scraped data into a file.