Docker and managing Integration tests — A better approach.

Its good to talk about summary in an intro:

As a summary of requirements, there shouldn’t be any more separate docker related orchestrations. Everything that you do with docker are first class citizens in the project that can run along with other test cases in your project. We can see bits and pieces of code everywhere trying to achieve this, but here we strive for the best possible way of doing it.

Let’s list down our requirements first.

Requirement 1

Requirement 2

Although sbt it can solve this to some extent, we need a better name-spacing management. Integration tests are in general refer to a wider context, while we prefer specific management of docker tests.

Requirement 3

Requirement 4

I am motivated to consider this as a requirement because I have had tough times managing the dockerFiles and compose yamls as separate hardcoded files. This will also avoid the complexities of overloaded context being passed to docker daemons and thereby accidentally slowing down the image creations. I initially solved this using a shell-script like a good pragmatic developer, but not anymore!

My philosophy on Integration Tests


Philisophy behind the design where images have built-in sample data, and test environment set up being part of build settings:
-----------------------------------------------------------------
1) The "real" environment for "real" testing shouldn't be mutated anywhere else other than the application itself. This leads to better testing of the lifecycle of an application.
2) Multiple integration tests sharing the same environment is a tragic situation. Stop reusing. Don't be lazy.3) If you want to run concurrency tests, run multiple instances of the same application with in the same test case, with obviously a test environment completely new and not being used/mutated anywhere else.4) To trouble shoot test environment, you should be able to spin up just the test environment in isolation. A build task could be a perfect solution to do this. You are building an environment that your app is supposed to work on.5) If your integration test fails, it means you don't know about the real test environment. So come back to build configs to know more about it, and change your test cases.

A side note:

I would like to add a controversial note along with the philosophical reasons to not put docker as part of the spec.

Docker integration tests are completely indeterministic. We can’t disagree with the fact, we are always in a “fingers-crossed” situation when our boss tries to run our integration test in his machine. At the mercy of God!

This happens due to a variety of reasons.

  • We might have done port mappings in docker-compose files
  • There is a host machine dependency for your docker environment. An example is putting a README of what to change in /etc/hosts on running integration tests, may be to run from an IDE.

There is a whole bunch of issues similar to the above bullet points, but I am stopping it here. Running docker test from outside of the containers from a different network, through port mappings is not really what is happening in a real environment most of the time. Example; Your kafka stream job is part of the same docker-compose file as that of broker, zookeeper, and kafka which you might be deploying through kubes or rancher, and the stream job will successfully talk to the topic and get the data. The same operation of reading the topic may fail in your integration test because you are reading it from outside of the kafka world. Trying to fix it through port mappings is cheating, and in fact it’s a waste of time. This will also involve dependency mismatch — the kafka client that you used in integration test will be different from the image versions. I am going to post a different blog solving these type of problems. But I would definitely want you to get the gist of what I am trying to say in this side note.

Nevertheless, all the solutions that I am going to propose in future, will require the same set up which I am going to explain in this blog. It will be unfortunately become too big, if I try to solve all of these issues as part of the same blog. So, thanks in advance for your patience. I also recommend trying this pattern which will uncover lot many things that you could now do with your test set up especially in the CI-CD side of things. Again I am going off the topic a bit here. Let’s get into the business.

Let’s start the set up now

Don’t worry, we will go step by step.

Step 1

addSbtPlugin("com.tapad" % "sbt-docker-compose" % "1.0.34")
addSbtPlugin("se.marcuslonnberg" % "sbt-docker" % "1.5.0")

Step 2

Ok, now we intend to run only those tests that depends on Dockerin this context. To avoid running other tests in this context, we use tags .

Tests.Argument(
TestFrameworks.Specs2, “include”, “DockerComposeTag”
)`

This is a snippet that I used in build.sbt that says, let me include only those tests with the tag “DockerComposeTag”. You can name the tag as you wish. Don’t worry too much on this now. We will get to know more on tags later.

Note: This work with any test frameworks: ScalaCheck, ScalaTest, Specs, Specs2, JUnit.

In the example above, the argument required for Spec2 is include (or exclude ) to include (or exclude) only those tests with the tag DockerComposeTag , and that in ScalaTest is n(or l) to include (or exclude) tests with the DockerComposeTag .

This ensures that sbt docker-int:test will run only those tests that are tagged with “DockerComposeTag”.

We will see how to tag a test case later on. Before that we need to ensure we are not running these long running test when we call simple sbt test . To do that, we add 1 more level of setting as given below.

This ensures that sbt test avoid long running tests. For those who are wondering, “oh well I could use it:test coming inbuilt as part of sbt without much of a drama”, I am not a fan of separate it folder in my tests, and not being able to reuse the test support functionalities that my other unit tests make use of.

Step 3

Step 4

Step 5

Result:

sbt test will run all tests except IntegrationTest and there won’t be any docker-compose up or any sort of operations related to docker.

If you don’t want to run tests but just compose-up, do sbt dockerComposeUp, and that will provide you with the environment the integration tests are supposed to run.

This is super helpful in troubleshooting the integration tests.
And once done, sbt dockerComposeDown. Yes, it is one of the strong requirement that I had forcing me to put docker deployments as part of sbt and not composing it up within functional tests.

No more bash orchestration and separate compose files, or dockerFiles in the project.

Fantastic CI

sbt dockerComposeDown dockerComposeUp docker-int:test dockerComposeDown

I did a dockerComposeDown as the first step, because who knows if any CI agent has already got these containers running.

Thanks, and have fun !

A software engineer and a functional programming enthusiast at Simple-machines, Sydney, and a hardcore hiking fan. https://twitter.com/afsalt2