Mein Profil. Hier einloggen oder registrieren.

26. April 2019

Automated Visual Regression Tests with JLineup

Automated Visual Regression Tests with JLineup

In this article, I would like to put a spotlight on one of our open source tools that makes deployment automation much easier for us: JLineup.

At otto.de, we use Continuous Deployment to release frequently and in small iterations. This is only possible through comprehensive unit, integration, and acceptance testing in our automated build pipelines. Our pipelines are setup in such a way that developers are confident that nothing will break whilst pushing new code – so nobody fears pushing frequently. During working hours, every commit goes live a few minutes after pushing. We covered continuous delivery and automation at otto.de in detail in a former article.

In our team, we are responsible for various microservices that serve website content for otto.de. Through functional Selenium webdriver or Cypress tests, that work on the document object model (DOM) in a real browser, we can check if the page that we deliver is semantically correct, has the content we look for, and can be interacted with as wanted. This is a good base, but those automated tests have a blind spot: The visual aspect. Does the page look the same before and after a deployment? Can a human spot subtle changes that may be unintended by just looking at a web page before and after a deployment? This is very hard without a photographic memory. Besides that, it would be a manual process, making automated builds impossible. Why not let the computer do the visual comparison?

The idea is supported by the fact that the majority of our deployments don’t affect the visual design of our front-end, because they contain back-end features, optimizations and refactorings. Additionally, we use the concept of feature toggles, so big changes are toggled on by a human after the deployment. As they shouldn’t have any impact when not being active, the front-end also should rarely change during automated deployment.

A tool was born

To have automated visual regression tests, we had the idea to simply take screenshots of the website before and after the deployment. After this, the screenshots can be compared to check if there are differences. When we were searching for a tool to do this back in 2015, we didn’t find anything fitting into our setup, so Lineup was born, which later evolved into JLineup. JLineup helps us spotting front-end bugs before they go live since several years now. Although it has a very simple concept and is easy to configure, it is a mighty little tool which we don’t want to miss.


JLineup can be used as a simple command line tool or it can run as a web server which takes your screenshot jobs as REST calls. The only additional requirement is a Chrome or a Firefox web browser. Once you have downloaded JLineup, literally all you have to do is call it before a deployment, deploy, and then run it again. We integrated this into our build pipeline, which looks like this (a little simplified):

Simplified pipeline with JLineup integration
Simplified pipeline with JLineup integration

Whatever your integration would look like, it’s good to not let JLineup fail your build if it detects changes, because if changes are intended, you want the possibility to go on with your deployment. We solve this through using gates or transitions, which can be opened or closed. Whenever JLineup detects differences, the transition to the Live deployment is closed. Of course we have integrated it into our monitoring systems so that when a Live deployment is closed the team checks the JLineup HTML report which was generated:

Excerpt from a JLineup Comparison Report
Excerpt from a JLineup Comparison Report

If we have a false positive, which means the detected changes are OK, we open the transition manually and the build goes on. Otherwise, the bug has to be fixed and a new build is started thereafter. Depending on your CI system, there may be other ways to integrate JLineup into the flow. If you work with feature toggles, the pipeline should rarely stop because of visual differences.

Quick Tutorial

So, you would like to give it a try? It’s easy and it’s free! JLineup is Open Source, licensed under the Apache-2.0 license. This tutorial is running on Linux, which is the platform we support. It should also work under MacOS or Windows, but we (currently) don’t test this.


You can simply download the CLI version by getting the jlineup-cli.jar from Maven Central:

wget https://repo1.maven.org/maven2/de/otto/jlineup-cli/3.0.2/jlineup-cli-3.0.2.jar -O jlineup.jar

Now you have a jlineup.jar in your current directory.

For best results, you need a „real“ web browser installed on your system. JLineup supports Google Chrome (Chromium also works) or Firefox at the moment. As JLineup is written in Java, you need a Java Runtime Environment version 8 or higher. OpenJDK is fine! JLineup can use PhantomJS internally, if no other browser is available. This is not recommended, as PhantomJS is not maintained any more. More and more modern web features don’t work in PhantomJS, so it’s only an emergency fallback for very simple websites. If these prerequisites are fulfilled, you can make a test run:

java -jar jlineup.jar --help

If you see an error message now, something is wrong. You should see a usage explanation with all possible command line arguments for JLineup. JLineup doesn’t have any other dependencies, just make sure you have the jlineup.jar in your current directory or somewhere in $PATH to use it.

Basic usage

For a first test, you don’t need any configuration:

java -jar jlineup.jar --url https://time.gov/ --step before

Notice the step declaration ‚before‘. JLineup is printing the generated config to standard out and it starts to browse to time.gov and to make screenshots of it’s current state. Wait some seconds and make the ‚after‘ run:

java -jar jlineup.jar --url https://time.gov/ --step after

Below your current directory, there should be a ‚report‘ folder now:

➜ ~ ls report -al
total 28
drwxrwxr-x 3 marco marco 4096 Jan 11 15:02 .
drwxr-xr-x 84 marco marco 4096 Jan 11 15:09 ..
-rw-rw-r-- 1 marco marco 1817 Jan 11 15:02 jlineup.log
-rw-rw-r-- 1 marco marco 4548 Jan 11 15:02 report.html
-rw-rw-r-- 1 marco marco 1755 Jan 11 15:02 report.json
drwxrwxr-x 2 marco marco 4096 Jan 11 15:02 screenshots

Open the report.html file with a browser and you can see your first JLineup comparison report! For programmatic analysis of the JLineup job outcome, you can also find the report.json file with detailed insights about the job’s outcome. If the ‚before‘ and the ‚after‘ run of a job show no differences, the ‚after‘ run of JLineup exits with return code 0. If there are differences, it exits with 1. For ‚real‘ usage, you shouldn’t use the –url parameter, but you should create a configuration file.

Here’s a small example config:


Copy the config and save it as file with name lineup.json. If lineup.json is in the current working directory when you run JLineup, it’s used automatically. You can also specify a different config name with the –config parameter. Now you’re ready to run JLineup again. A „real“ Firefox window should open and you can watch JLineup at work. If you want Chrome, change browser to Chrome in the config above.

java -jar jlineup.jar --config lineup.json --step before

Wait a bit and run ‚after‘

java -jar jlineup.jar --config lineup.json --step after

A new report was written, it now should have the time.gov page in three widths, as defined in the configuration. There should be differences, because time always changes. 🙂

If you want to see a ‚green‘ report without changes, use a static page like http://example.com/

Further Documentation

If you want to go on with JLineup, you can find the full documentation and the source code on GitHub: https://github.com/otto-de/jlineup

Here are some entry points:

Running JLineup as CLI tool
Running JLineup as web service
JLineup Job Configuration
We hope JLineup can also improve your pipelines. Feedback is always welcome! Use the comments section or open a GitHub issue for feature requests: https://github.com/otto-de/jlineup/issues


If JLineup doesn’t fit your needs, you should take a look at some other similar tools for visual regression testing. There are some alternatives with different scopes and capabilities like Galen, BBC Wraith, Gemini, BackstopJS, Shoov, or Spectre to mention some popular ones.

0Noch keine Kommentare

Dein Kommentar
Antwort auf:  Direkt auf das Thema antworten
3410 - 2
Geschrieben von
Marco Geweke

Diese Webseite verwendet Cookies. Durch die Nutzung der Webseite stimmen Sie der Verwendung von Cookies zu. Weitere Informationen: Datenschutzinformationen