26. April 2019
In this article, I would like to put a spotlight on one of our open source tools that makes deployment automation much easier for us: JLineup.
At otto.de, we use Continuous Deployment to release frequently and in small iterations. This is only possible through comprehensive unit, integration, and acceptance testing in our automated build pipelines. Our pipelines are setup in such a way that developers are confident that nothing will break whilst pushing new code – so nobody fears pushing frequently. During working hours, every commit goes live a few minutes after pushing. We covered continuous delivery and automation at otto.de in detail in a former article.
In our team, we are responsible for various microservices that serve website content for otto.de. Through functional Selenium webdriver or Cypress tests, that work on the document object model (DOM) in a real browser, we can check if the page that we deliver is semantically correct, has the content we look for, and can be interacted with as wanted. This is a good base, but those automated tests have a blind spot: The visual aspect. Does the page look the same before and after a deployment? Can a human spot subtle changes that may be unintended by just looking at a web page before and after a deployment? This is very hard without a photographic memory. Besides that, it would be a manual process, making automated builds impossible. Why not let the computer do the visual comparison?
The idea is supported by the fact that the majority of our deployments don’t affect the visual design of our front-end, because they contain back-end features, optimizations and refactorings. Additionally, we use the concept of feature toggles, so big changes are toggled on by a human after the deployment. As they shouldn’t have any impact when not being active, the front-end also should rarely change during automated deployment.
To have automated visual regression tests, we had the idea to simply take screenshots of the website before and after the deployment. After this, the screenshots can be compared to check if there are differences. When we were searching for a tool to do this back in 2015, we didn’t find anything fitting into our setup, so Lineup was born, which later evolved into JLineup. JLineup helps us spotting front-end bugs before they go live since several years now. Although it has a very simple concept and is easy to configure, it is a mighty little tool which we don’t want to miss.
JLineup can be used as a simple command line tool or it can run as a web server which takes your screenshot jobs as REST calls. The only additional requirement is a Chrome or a Firefox web browser. Once you have downloaded JLineup, literally all you have to do is call it before a deployment, deploy, and then run it again. We integrated this into our build pipeline, which looks like this (a little simplified):
Whatever your integration would look like, it’s good to not let JLineup fail your build if it detects changes, because if changes are intended, you want the possibility to go on with your deployment. We solve this through using gates or transitions, which can be opened or closed. Whenever JLineup detects differences, the transition to the Live deployment is closed. Of course we have integrated it into our monitoring systems so that when a Live deployment is closed the team checks the JLineup HTML report which was generated:
If we have a false positive, which means the detected changes are OK, we open the transition manually and the build goes on. Otherwise, the bug has to be fixed and a new build is started thereafter. Depending on your CI system, there may be other ways to integrate JLineup into the flow. If you work with feature toggles, the pipeline should rarely stop because of visual differences.
So, you would like to give it a try? It’s easy and it’s free! JLineup is Open Source, licensed under the Apache-2.0 license. This tutorial is running on Linux, which is the platform we support. It should also work under MacOS or Windows, but we (currently) don’t test this.
You can simply download the CLI version by getting the jlineup-cli.jar from Maven Central:
wget https://repo1.maven.org/maven2/de/otto/jlineup-cli/3.0.2/jlineup-cli-3.0.2.jar -O jlineup.jar
Now you have a jlineup.jar in your current directory.
For best results, you need a „real“ web browser installed on your system. JLineup supports Google Chrome (Chromium also works) or Firefox at the moment. As JLineup is written in Java, you need a Java Runtime Environment version 8 or higher. OpenJDK is fine! JLineup can use PhantomJS internally, if no other browser is available. This is not recommended, as PhantomJS is not maintained any more. More and more modern web features don’t work in PhantomJS, so it’s only an emergency fallback for very simple websites. If these prerequisites are fulfilled, you can make a test run:
java -jar jlineup.jar --help
If you see an error message now, something is wrong. You should see a usage explanation with all possible command line arguments for JLineup. JLineup doesn’t have any other dependencies, just make sure you have the jlineup.jar in your current directory or somewhere in $PATH to use it.
For a first test, you don’t need any configuration:
java -jar jlineup.jar --url https://time.gov/ --step before
Notice the step declaration ‚before‘. JLineup is printing the generated config to standard out and it starts to browse to time.gov and to make screenshots of it’s current state. Wait some seconds and make the ‚after‘ run:
java -jar jlineup.jar --url https://time.gov/ --step after
Below your current directory, there should be a ‚report‘ folder now:
➜ ~ ls report -al
drwxrwxr-x 3 marco marco 4096 Jan 11 15:02 .
drwxr-xr-x 84 marco marco 4096 Jan 11 15:09 ..
-rw-rw-r-- 1 marco marco 1817 Jan 11 15:02 jlineup.log
-rw-rw-r-- 1 marco marco 4548 Jan 11 15:02 report.html
-rw-rw-r-- 1 marco marco 1755 Jan 11 15:02 report.json
drwxrwxr-x 2 marco marco 4096 Jan 11 15:02 screenshots
Open the report.html file with a browser and you can see your first JLineup comparison report! For programmatic analysis of the JLineup job outcome, you can also find the report.json file with detailed insights about the job’s outcome. If the ‚before‘ and the ‚after‘ run of a job show no differences, the ‚after‘ run of JLineup exits with return code 0. If there are differences, it exits with 1. For ‚real‘ usage, you shouldn’t use the –url parameter, but you should create a configuration file.
Here’s a small example config:
Copy the config and save it as file with name lineup.json. If lineup.json is in the current working directory when you run JLineup, it’s used automatically. You can also specify a different config name with the –config parameter. Now you’re ready to run JLineup again. A „real“ Firefox window should open and you can watch JLineup at work. If you want Chrome, change browser to Chrome in the config above.
java -jar jlineup.jar --config lineup.json --step before
Wait a bit and run ‚after‘
java -jar jlineup.jar --config lineup.json --step after
A new report was written, it now should have the time.gov page in three widths, as defined in the configuration. There should be differences, because time always changes. 🙂
If you want to see a ‚green‘ report without changes, use a static page like http://example.com/