🔥🔥🔥 Hot off the press! 🔥🔥🔥
After months of development behind the scenes, I’m thrilled to announce the latest release of FlashPipe – major version v3.0.0. It is now publicly available from Docker Hub, ready to be incorporated into your SAP Integration Suite CI/CD pipelines.
This release marks a significant change to FlashPipe since it was first released more than two years ago. With the docker pull count hitting 24k and rising, it was time to revisit the implementation. The outcome was a reimplementation of the entire solution from scratch in a different programming language, Go (a.k.a. Golang).
Why Go?
Single development environment and context
FlashPipe was initially implemented using a combination of Unix Bash scripts and Groovy classes. There were various points of context switching which sometimes made it tricky to develop. Additionally, testing (especially unit testing) was challenging when the Bash scripts were involved.
Switching to Go enabled both parts of the solution to be implemented in a single development environment. This significantly eased both development and testing of the solution. To begin with, the testing coverage of the solution is already at 75%!
An elegant CLI
Creating a Command-line Interface (CLI) has always been a goal for FlashPipe. Due to the reasons mentioned above, it was not so feasible with the previous approach.
In comes Go, a programming language that is ideal for creating CLIs. With Cobra (used in popular CLIs like GitHub CLI and Kubernetes), it was easy to introduce an elegant CLI.
Great performance at a small size
One of the key strengths of Go is its speed. It is very fast to compile and has a quick startup and speedy execution time. The price to pay? Minimal, as it compiles to a small self-contained binary (17MB for FlashPipe) and can be executed without additional installation of libraries, runtimes or dependencies.
In contrast, it makes Java Virtual Machine (JVM) based applications seem bloated and sluggish with its overhead of dependencies and runtime installation, and slow startup times.
A new language and a cute mascot!
Lastly, and my personal favourite ❤️ – an opportunity to learn a new and much loved programming language.
🚀 Enough talk, what’s new? 🚀
Let’s now look at some of the new aspects brought about by the reimplementation.
Did I mention the new CLI?
Powered by Cobra, FlashPipe CLI has the look and feel of a modern and easy-to-use CLI. It provides a list of all the available commands together with the corresponding flags available for each of them. Besides using it in a CI/CD pipeline, you can even run it locally using Docker.
The following image shows a preview of running FlashPipe CLI using Docker and a few examples of the usage of the CLI commands.
Another cool aspect of the CLI is the ability to pass input values to the commands using different methods. This capability comes from another popular Go library, Viper, which supports the following methods:
- Command line flags
- Environment variables
- Configuration file
The specific details of each command can be found in the FlashPipe’s CLI documentation.
Support for new artifact types
While the 😎 coolest feature 😎 in this new release is the new CLI, the 🔥 hottest feature 🔥 must be the support for additional artifact types – Script Collection, Message Mapping, and Value Mapping. Version 3.0.0 delivers this much requested feature across all the available commands – update, deploy, sync, and snapshot.
The following image shows an example of the sync command being executed via CLI. The four different artifact types are then downloaded from the tenant and stored in the Git repository.
Due to the lack of an update functionality in the SAP provided API for Value Mapping, updates to value mapping artifacts (from Git to tenant) are implemented as a delete followed by a create. This minor drawback means that the history of the artifact will not be retained in the tenant.
Docker image optimisation
One of the goals of the reimplementation was the reduction of the Docker image size. As the image is pulled from Docker Hub for each execution in a CI/CD pipeline, reduction of the image size automatically contributes to a faster pipeline execution.
I achieved this using a minimalist Debian image, and adding on just the bare minimum required to get FlashPipe to work. The results were staggering!
- 23% of the size of the previous full image (i.e. engswee/flashpipe:2.7.1)
- 38% of the size of the previous lite image (i.e. engswee/flashpipe:2.7.1-lite)
- Initialisation of the container in a CI/CD pipeline takes less than half the time it used to
With the consolidation to just a single image per release (i.e. no more full vs lite images), a rolling tag named latest is now available. This is useful in non-productive pipelines so that the latest version of FlashPipe is always used.
Speedy execution 🏎️💨
FlashPipe in Go is blazing fast! Did you notice the speediness implied in both FlashPipe and Go’s logos? It’s a marriage made in heaven! 😉
An execution of the update artifact command that typically used to take 13-15 seconds, now takes only 3-4 seconds!
The following screenshot shows a pipeline that updates multiple artifacts from a Git repository to the tenant. The overall runtime improved from around 3 minutes to just under 1 minute – a 66% reduction in runtime!
A few tradeoffs
Unfortunately, in the process of reimplementing the solution, the following tradeoffs were made.
The most noticeable one is that the Neo environment is no longer supported. This is inline with the announcement of the sunset of the Neo environment. Furthermore, the dwindling availability of Neo tenants makes it challenging to test and ensure FlashPipe works correctly on them.
With the move from Java/Groovy to Go, a Docker image preloaded with Maven and third party libraries (previously known as the full image) is no longer provided. For those who still want to execute unit testing with Maven, this can still be achieved as a separate step in the pipeline using a Docker image other than FlashPipe. Examples can be found in the azure-pipelines and github-actions branches of the flashpipe-demo repository.
In previous versions, when uploading multiple copies of the same integration flow to the same tenant, it was possible to use normalisation to change the MANIFEST.MF file to rename the artifacts. Java provided native handling of the Java-centric MANIFEST.MF file but Go does not, therefore the normalisation capabilities have been dropped.
Summary
Overall, it was a great and beneficial experience to learn a new programming language in the context of developing a real solution. Go is simple and yet powerful, and I can see why it is popular and much loved amongst the developer community.
For those of you who are already using FlashPipe, I encourage you to make the minor adjustments to upgrade to this latest and greatest version. I am confident that you will like it and reap the immediate benefits of it. For the rest, this is a great time to give FlashPipe a go (pun intended 😉) – try it for yourselves and see how it can improve your development workflow for SAP Integration Suite.
This reimplementation sets a very good foundation for FlashPipe, so watch this space for further improvements and updates to your CI/CD Companion for SAP Integration Suite.
Go, Go, FlashPipe!
Comments
Feel free to provide your comment, feedback and/or opinion here.
