For this Data Engineer project, I implemented a testing workflow for Airflow DAGs. The project involves creating more than 5 tables in MySQL, extracting data to PostgreSQL, and managing the pipeline with dynamic DAGs. Jobs are scheduled to run every 2 hours between 9 AM and 9 PM on Fridays, twice a month (week 1 and week 3), at the 15th minute.
Automated ETL Workflow with Airflow – MySQL to PostgreSQL Read More »
This Data Engineer project creates an ETL pipeline using Airflow. Data is extracted from CSV sources, saved as CSV in the `data/` folder, and loaded into SQLite. The `BranchOperator` selects the extraction task. Includes DAGs, staging folder, SQLite results, and screenshots of the pipeline.
ETL Pipeline with Apache Airflow: Extract, Transform, and Load Data into SQLite Read More »
This SQL project explores a movie database using queries to find characters, analyze genres, and determine actor involvement. Key features include identifying the most popular genre, highest and lowest rated movies, and actors in multiple films. SQL joins, grouping, and aggregate functions are utilized to gain insights efficiently.
SQL Movie Database Analysis: Genres, Ratings & Actors Read More »
This data science project predicts car selling prices using machine learning based on features like engine size, fuel type, and ownership history. It includes data preprocessing, EDA, feature engineering, model training, and evaluation.
Predicting Car Selling Prices with Machine Learning Read More »
In this case study, I performed a detailed analysis of retail product sales data to extract meaningful insights using Python. This article walks through the process, the techniques employed, and the key findings.
In this case study, I explore how to predict Airbnb prices in Sydney using various features such as location, property attributes, and host characteristics. By building a machine learning model, I aim to uncover the key drivers behind listing prices and help hosts optimize their pricing strategy.
Predicting Airbnb Prices in Sydney Read More »
This Data Engineering project involves collecting streaming data using Apache Kafka, performing real-time ETL with Apache Spark Streaming, and validating data. The project includes creating a purchasing event producer, building a streaming job to aggregate daily purchases, and outputting cumulative purchase totals using PySpark.
Streaming Data Pipeline Read More »
This project sets up a Kafka cluster using Docker with one broker and Kafka UI (Bitnami or Confluent). A producer generates random events every 5 seconds, and a consumer processes them. The project includes configuring partitions, replication, and consumer groups.
A web scraping program to extract data from the website https://toscrape.com/ and save the scraped data into a file.
Predicting YouTube video views using regression models with data analysis and preprocessing on the YouTube Statistics dataset. Involves feature engineering, model selection (Linear Regression, Random Forest), and evaluation using RMSE and R2 metrics to ensure prediction accuracy. Code, Queries & Documentation Find the complete code, and documentation on my GitHub: https://github.com/hijirdella/YouTube-Supervised-Learning
YouTube Supervised Learning Read More »
