Airflow

Containers Spark

Apache Spark Streaming in K8s with ArgoCD & Spark Operator

Here’s an interesting talk Albert Franziu Cros on a CI/CD setup composed by a Spark Streaming job in K8s consuming from Kafka. Over the last year, we have been moving from a batch processing jobs setup with Airflow using EC2s to a powerful & scalable setup using Airflow & Spark in K8s. The increasing need […]

Read More