Building an open source Scala gRPC/REST HTTP Proxy for Kafka (I)

Day 0 of the journey on building my first Scala GraalVM ZIO application

Johannes Adam Simon Oertel - The Holy Grail

Scala ecosystem

I’ve been working with Scala for quite some time, but as most Scala devs know there is this separation between the Scala Spark crowd (California School) vs the Scala Functional Programming crowd (Glasgow School).
Even as a member of the former one, I’ve always tried to make my a FP as possible. However, truth be told, the Spark API does not require that level of strictness and even feels like you are being punished for trying to do it the pure typed way.

Problem statement

In my work as a data engineer, I do come across problems that don’t require a distributed data solution (Spark, Flink, Hadoop, Kafka Streaming, etc) but more of a mundane issue: Getting data into Kafka.

Day 0

The best part of any project: The clean slate. Frodo leaving the Shire, Harry going to Hogwarts. It’s all new, exciting and what could go wrong. #foreshadowing?

lazy val prokzio = (project in file("."))
.aggregate(
util,
service
)
.settings(
crossScalaVersions := Nil,
publish / skip := true
)

lazy val util = (project in file("util"))
.settings(
commonSettings,
testSettings,
scalafmtSettings,
name := "util"
)

lazy val service = (project in file("service"))
.enablePlugins(JavaAppPackaging, GraalVMNativeImagePlugin)
.settings(
name := "service",
Compile / mainClass := Some("xyz.graphiq.prokzio.service.HelloWorld"),
commonSettings,
testSettings,
scalafmtSettings,
graalSettings
)
.dependsOn(
util
)
# add this to ~/.zshrc
export GRAALVM_HOME="/Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/"
# allow to execute graalvm
sudo xattr -r -d com.apple.quarantine /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0
# install native-image tool
/Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/bin/gu install native-image
# install espresso (do I need this?)
/Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/bin/gu install espresso
# add to jenv
jenv add /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home
# enable export
jenv enable-plugin export
# set jenv
jenv local graalvm64-11.0.10
# Before running sbtexec $SHELL -l
sbt service/graalvm-native-image:packageBin
./service/target/graalvm-native-image/service

Conclusion Day 0

At the end I’m happy I have something to show for. A simple hello world in zio, build by GraalVM tools and run locally on my mac.

Freelance Data & ML Engineer | husband + father of 2 | #Spark #Scala #BigData #ML #DeepLearning #Airflow #Kubernetes | Shodan Aikido

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store