Building an open source Scala gRPC/REST HTTP Proxy for Kafka (II)

Day 1 of the journey on building my first Scala GraalVM ZIO application

The Iron Rolling Mill — Adolph Menzel

Day 1: Getting builds under control

Now that we have a basic project setup and somewhat building, we need to stabilise it a bit more, so we can spend all our future energy on actually writing the code. This means CI/CD, dockerising, versioning, but more important getting the binary stable on *nix and mac environments. This means diving deeper into GraalVM and it’s available build tools. So far we (the native packager) only used the native-imagetool from graal, but we need a way to start defining the scala/java reflections via the native-image-agent
Fortunately reflections are not encouraged in Scala and many libraries have few, but few is not the same as none.

Local & remote

As we’ve seen we can build the native image locally (on mac) if we do all these prerequisite steps, but I just discovered this awesome sbt library, sbt-native-image, that basically does everything for you, downloading all binaries using coursier, without needing to install & run graal and subsequent tools project local. This would have saved me a lot of time, but the code is not wasted. The biggest strength is also the biggest drawback of this library. It does everything for you, but leaves you with little control over the end product, especially if the goal is dockerising the end product.

val jvmVersion = "11"
val graalVersion = "21.0.0.2"
val baseGraalOptions = Seq(
"--verbose",
"--no-fallback",
"--no-server",
"--install-exit-handlers",
"--allow-incomplete-classpath",
"--enable-http",
"--enable-https",
"--enable-url-protocols=https,http",
"--initialize-at-build-time",
"--report-unsupported-elements-at-runtime",
"-H:+RemoveSaturatedTypeFlows",
"-H:+ReportExceptionStackTraces",
"-H:-ThrowUnsafeOffsetErrors",
"-H:+PrintClassInitialization"
)
lazy val graalLocalSettings = Seq(
nativeImageVersion := graalVersion,
nativeImageJvm := s"graalvm-java$jvmVersion",
nativeImageOptions ++= baseGraalOptions,
nativeImageOutput := file("output") / name.value
)
Hello, world from thread: zio-default-async-1!
Hello, world from thread: zio-default-async-2!
lazy val graalDockerSettings = Seq(
GraalVMNativeImage / containerBuildImage := GraalVMN ativeImagePlugin
.generateContainerBuildImage(s"ghcr.io/graalvm/graalvm-ce:java$jvmVersion-$graalVersion")
.value,
graalVMNativeImageOptions ++= baseGraalOptions ++ Seq(
"--static"
),
...
)
lazy val baseImage = "alpine:3.13.1"
lazy val dockerBasePath = "/opt/docker/bin"
lazy val graalDockerSettings = Seq(
...
dockerBaseImage
:= baseImage,
dockerEntrypoint := Seq(dockerBinaryPath.value),
dockerChmodType := DockerChmodType.Custom("ugo=rwX"),
dockerAdditionalPermissions += (DockerChmodType.Custom(
"ugo=rwx"
), dockerBinaryPath.value),
mappings in Docker := Seq(
((target in GraalVMNativeImage).value / name.value) -> dockerBinaryPath.value
),
...
)

CI/CD

Next we want to get a basic pipeline running that does all these steps on push and also publish the resulting docker image to github’s container registry.

def publishNativeDocker(project: Project): ReleaseStep =
ReleaseStep(
action = { beginState: State =>
val extracted = Project.extract(beginState)
Seq(
(state: State) => extracted.runTask(packageBin in GraalVMNativeImage in project, state),
(state: State) => extracted.runTask(publish in Docker in project, state)
).foldLeft(beginState) {
case (newState, runTask) => runTask(newState)._1
}
}
)
lazy val graalDockerSettings = Seq(
...
dockerRepository
:= sys.env.get("DOCKER_REPOSITORY"),
dockerAlias := DockerAlias(
dockerRepository.value,
dockerUsername.value,
s"$baseName/$baseName-${name.value}".toLowerCase,
Some(version.value)
),
dockerUpdateLatest := true,
dockerUsername := sys.env.get("DOCKER_USERNAME").map(_.toLowerCase)
)
on:
push:
branches: ['**']
tags: [v*]
jobs:
ci:
runs-on: ubuntu-latest
steps:
...
- name: Checkout code
uses: actions/checkout@v2
- name: Set Java / Scala
uses: olafurpg/setup-scala@v10
with:
java-version: 11
- name: Import GPG key & set Git config
id: import_gpg
uses: crazy-max/ghaction-import-gpg@v3
with:
gpg-private-key: ${{ secrets.GPG_PRIVATE_KEY }}
passphrase: ${{ secrets.GPG_PASSWORD }}
git-user-signingkey: true
git-commit-gpgsign: true
git-tag-gpgsign: true
git-push-gpgsign: false
- name: Docker Login
uses: azure/docker-login@v1
with:
login-server: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Test, Bump & Deploy
run: sbt bumpSnapshot
env:
DOCKER_REPOSITORY: ghcr.io
DOCKER_USERNAME: ${{ github.repository_owner }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
shell: bash

Conclusion Day 1

Day 1 was the longest day so far. It was way more work than I anticipated, especially switching between the two plugins, coming to the conclusion to keep them both (for now) and leverage each individual strength.

Freelance Data & ML Engineer | husband + father of 2 | #Spark #Scala #BigData #ML #DeepLearning #Airflow #Kubernetes | Shodan Aikido

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store