Applications

Airflow Installation on AWS EC2

Are you looking for a hassle-free way to set up Apache Airflow on your Amazon Web Services (AWS) Elastic Compute Cloud (EC2) instance? Look no further! In this article, we'll walk you through the process of installing Airflow using a simple script that can be added as part of the user data while launching an EC2 instance.

Updated 22 Jun, 2024

MLflow Installation on AWS EC2

Are you looking for a comprehensive guide on how to install the MLflow tracking server in an AWS EC2 instance? Look no further! In this article, we will walk you through the process of setting up an MLflow tracking server in an EC2 instance, including creating an S3 bucket and assigning an IAM role.

Updated 22 Jun, 2024

Kafka Connect to AWS S3 Sink

In this article we will learn how to write Kafka event messages to AWS S3 using Kafka Connect. We will use Amazon S3 Sink Connector to write the messages as Parquet files to our S3 Datalake. Also we will write the Kafka Tombstone records to a separate file to handle downstream delete operations.

Updated 07 Aug, 2023

Kafka Connect Debezium Source to AWS S3 Sink

In this article we will learn how to write Kafka event messages from Debezium Source Database Topics to AWS S3 using Kafka Connect. We will use Amazon S3 Sink Connector to write the messages as Parquet files to our S3 Datalake. Also we will write the Kafka Tombstone records to a separate file to handle downstream delete operations.

Updated 07 Aug, 2023

Kafka installation on AWS EC2

In this article, we will explore how to deploy a Confluent Kafka cluster using Docker-Compose. We will create a comprehensive configuration file that includes the necessary services and dependencies to provision a fully functional Kafka cluster in AWS EC2 instance for a Demo or PoC use case.

Updated 23 Jun, 2024

Setup Apache Spark with Jupyter Notebook on MacOS

Are you interested in exploring the world of big data and machine learning? Look no further! In this article, we'll take you through a quick and easy guide to installing and configuring Apache Spark with Jupyter Notebook on your MacOS device.

Updated 05 Jul, 2024

Change Data Capture from PostgreSQL to Kafka

As data becomes increasingly critical to businesses, the need to capture and process changes in real-time has never been more important. In this article, we'll explore how to read changed data from a PostgreSQL database and write it to a Kafka topic as event messages using Debezium's PostgreSQL CDC Source Connector.

Updated 02 Jul, 2024

Change Data Capture from MySQL to Kafka

As data becomes increasingly critical to businesses, the need to capture and process changes in real-time has never been more important. In this article, we'll explore how to read changed data from a MySQL database and write it to a Kafka topic as event messages using Debezium's MySQL CDC Source Connector.

Updated 02 Jul, 2024

Change Data Capture from MSSQL Server to Kafka

As data becomes increasingly critical to businesses, the need to capture and process changes in real-time has never been more important. In this article, we'll explore how to read changed data from a MSSQL Server and write it to a Kafka topic as event messages using Debezium's SQL Server CDC Source Connector.

Updated 02 Jul, 2024

Change Data Capture from MongoDB to Kafka

As data becomes increasingly critical to businesses, the need to capture and process changes in real-time has never been more important. In this article, we'll explore how to read changed data from a MongoDB Server and write it to a Kafka topic as event messages using Debezium's MongoDB CDC Source Connector.

Updated 02 Jul, 2024

Change Data Capture from Oracle to Kafka

As data becomes increasingly critical to businesses, the need to capture and process changes in real-time has never been more important. In this article, we'll explore how to read changed data from a Oracle database and write it to a Kafka topic as event messages using Confluent Oracle CDC Source Connector.

Updated 03 Jul, 2024