CI/CD for Data Engineers

Reliably Deploying Scala Spark containers for Kubernetes with Github Actions

One of the most under-appreciated parts of software engineering is actually deploying your code. There is al lot of focus on building highly scalable data pipelines, but in the end your code has to ‘magically’ transferred from a local machine to a deployable piece of pipeline in the cloud.

van Bree — Le Friedland

--

--

--

Freelance Data & ML Engineer | husband + father of 2 | #Spark #Scala #ZIO#BigData #ML #Kafka #Airflow #Kubernetes | Shodan Aikido

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

READ/DOWNLOAD^ Medical Coding Online 2012 for Step

Solutions to the testing challenges when working agile at scale

Getting started with MongoDB and Go on Azure

All You Need To Know About VPS Hosting & Its Top 6 Features

Pytest: How to mock the built-in open()

How to Write Unit Test in Go

New CKA/CKAD v1.19.0 Exam Notes

Error Handling in Kafka Consumer for API Calls

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Tom Lous

Tom Lous

Freelance Data & ML Engineer | husband + father of 2 | #Spark #Scala #ZIO#BigData #ML #Kafka #Airflow #Kubernetes | Shodan Aikido

More from Medium

from the above table we can easily understand that 1st offset will process the 24 rows(4+12+08)…

About reading raw json files in spark

SQOOP Architecture and Commands

Job Orchestration on Databricks with interdependent tasks