Building an open source Scala gRPC/REST HTTP Proxy for Kafka (I)

Day 0 of the journey on building my first Scala GraalVM ZIO application


Caveat: Normally I write posts about how I tackled a problem and present the solution on a silver platter.
In this series I’d like you to take you on a journey of exploration. I have no idea how fast this will progress or where it will end up. I will try to work at least 1 day a week on this.

There are way better tutorials write ups for each of these tools (zio, kafka, api, graalvm), but I just want to show that even people working for years in the software engineering field still don’t know everything, have to learn and are beginners when broaching new terrains.
Follow along at your own risk ;-)

Johannes Adam Simon Oertel - The Holy Grail

Scala ecosystem

I’ve been working with Scala for quite some time, but as most Scala devs know there is this separation between the Scala Spark crowd (California School) vs the Scala Functional Programming crowd (Glasgow School).
Even as a member of the former one, I’ve always tried to make my a FP as possible. However, truth be told, the Spark API does not require that level of strictness and even feels like you are being punished for trying to do it the pure typed way.

Problem statement

In my work as a data engineer, I do come across problems that don’t require a distributed data solution (Spark, Flink, Hadoop, Kafka Streaming, etc) but more of a mundane issue: Getting data into Kafka.

In all tools I used there are great connectors for this, but when running applications have to send data connecting to Kafka can be cumbersome. That’s why we need a API endpoint that is easy to use to POST messages on via REST or gRPC and that handles the interaction with Kafka for us, including validating the messages with a schema registry before publishing them.

Of course this already exists, most notably , but I’d like to build my own using Scala libraries I don’t use often. My wish is to build this for both Scala 2 & 3, using the framework and running it on

I’ve a lot of other ideas for features, libraries and build tools I’d like to try out, but this seems like quite the challenge already.

Day 0

The best part of any project: The clean slate. Frodo leaving the Shire, Harry going to Hogwarts. It’s all new, exciting and what could go wrong. #foreshadowing?

is born! It’s Apache 2 license, so you can’t sue me for bad naming.

First a little struggle to get my build.sbt in the shape I wanted.
I do always set them up as multiproject, because adding that later is always more painful and having a util module will allow to centralise some abstractions.

lazy val prokzio = (project in file("."))
crossScalaVersions := Nil,
publish / skip := true

lazy val util = (project in file("util"))
name := "util"

lazy val service = (project in file("service"))
.enablePlugins(JavaAppPackaging, GraalVMNativeImagePlugin)
name := "service",
Compile / mainClass := Some("xyz.graphiq.prokzio.service.HelloWorld"),

Then I had to decide what basic Zio “Hellow World”-esque program I wanted to deploy, I opted to go with the basic one from the Z

Finally the implementation of GraalVM. I’ve never worked with many other JDK’s before and to be honest I’ve never paid a lot of attention to it, since working with Spark tied you to JDK8 this was always a basic given in each project.

I needed to install GraalVM locally & all tools needed to build my native image.
brew install --cask graalvm/tap/graalvm-ce-java11

# add this to ~/.zshrc
export GRAALVM_HOME="/Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/"
# allow to execute graalvm
sudo xattr -r -d /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0
# install native-image tool
/Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/bin/gu install native-image
# install espresso (do I need this?)
/Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/bin/gu install espresso
# add to jenv
jenv add /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home
# enable export
jenv enable-plugin export
# set jenv
jenv local graalvm64-11.0.10
# Before running sbtexec $SHELL -l

So in the end I have something that actually could build my scala project

sbt service/graalvm-native-image:packageBin

However I need to disable --no-fallback other wise my program fails when executing. This means that the binary still requires a JDK somewhere for reflections. And also the --static flag is a big no-no on mac. I need to learn more about GraalVM, but even though the application will only run in *nix environments, I’d like to see if I can make it work for mac as well at least.


Conclusion Day 0

At the end I’m happy I have something to show for. A simple hello world in zio, build by GraalVM tools and run locally on my mac.

Next I really want to smooth out the GraalVM builds. They feel flaky and I wan to focus on building code later. And probably setup some basic CI/CD to build the binary and wrap & publish a docker image.

Let’s see what the next day will bring…

Freelance Data & ML Engineer | husband + father of 2 | #Spark #Scala #BigData #ML #DeepLearning #Airflow #Kubernetes | Shodan Aikido

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store