CI/CD for Data Engineers

Tom Lous
17 min readFeb 26, 2021

Reliably Deploying Scala Spark containers for Kubernetes with Github Actions

One of the most under-appreciated parts of software engineering is actually deploying your code. There is al lot of focus on building highly scalable data pipelines, but in the end your code has to ‘magically’ transferred from a local machine to a deployable piece of pipeline in the cloud.

van Bree — Le Friedland

--

--

Tom Lous

Freelance Data & ML Engineer | husband + father of 2 | #Spark #Scala #ZIO#BigData #ML #Kafka #Airflow #Kubernetes | Shodan Aikido