Building an open source Scala gRPC/REST HTTP Proxy for Kafka (II)

Tom Lous
6 min readMar 17, 2021

Day 1 of the journey on building my first Scala GraalVM ZIO application

» Day 0
» Day 1

Continued from Day 0

The Iron Rolling Mill — Adolph Menzel

Day 1: Getting builds under control

Now that we have a basic project setup and somewhat building, we need to stabilise it a bit more, so we can spend all our future energy on actually writing the code. This means CI/CD, dockerising, versioning, but more important getting the binary stable on *nix and mac environments. This means diving deeper into GraalVM and it’s available build tools. So far we (the native packager) only used the native-imagetool from graal, but we need a way to start defining the scala/java reflections via the native-image-agent
Fortunately reflections are not encouraged in Scala and many libraries have few, but few is not the same as none.

Local & remote

As we’ve seen we can build the native image locally (on mac) if we do all these prerequisite steps, but I just discovered this awesome sbt library, sbt-native-image, that basically does everything for you, downloading all binaries using coursier, without needing to install & run graal and subsequent tools project local. This would have saved me a lot of time, but the code is not wasted. The biggest strength is also the biggest drawback of this library. It does everything for you, but leaves you with little control over the end product, especially if the goal is dockerising the end product.

So for now the approach is twofold, and why not. We keep both the sbt-native-packager and sbt-native-image plugins and use each of their strengths.

We do define some basic graalvm settings that can be used by both approaches. I don’t know if we’ll keep all these settings as is, but for now it works

val jvmVersion = "11"
val graalVersion = "21.0.0.2"
val baseGraalOptions = Seq(
"--verbose",
"--no-fallback",
"--no-server",
"--install-exit-handlers",
"--allow-incomplete-classpath",
"--enable-http",
"--enable-https",
"--enable-url-protocols=https,http",
"--initialize-at-build-time",
"--report-unsupported-elements-at-runtime"…
Tom Lous

Freelance Data & ML Engineer | husband + father of 2 | #Spark #Scala #ZIO#BigData #ML #Kafka #Airflow #Kubernetes | Shodan Aikido