A little-known but very useful feature slipped into JUnit 4 and recent versions of the Maven Surefire Plugin: support for parallel testing. This feature has been around for a while in TestNG, but has been missing in JUnit. And now, if you are a JUnit user, you too can run your tests in parallel!

Running your tests in parallel is a powerful way to speed up your tests. Good automated tests should be independent, isolated and reproducible, making them ideal candidates for being run concurrently. However in practice not all test classes are designed with concurrency in mind, and aspects such as shared mutable instance variables, shared file or database access, or the use of embedded web servers may make running the tests in parallel trickier. Nevertheless, running your tests in parallel is decidedly a very neat trick!

Let's see how it works. As of JUnit 4.7, you can configure Maven to run your tests in parallel, just like in TestNG. The following configuration will run your test methods in parallel:

<plugin>
   <groupId>org.apache.maven.plugins</groupId>
   <artifactId>maven-surefire-plugin</artifactId>
   <version>2.5</version>
   <configuration>
     <parallel>methods</parallel>
   </configuration>
</plugin>

Another useful trick is to run your test classes in parallel, rather than the methods. You can do this by setting the configuration item to 'classes' instead of 'methods'. This may or faster depending on the number of test classes you have, but it might also be safer for some test cases not designed with concurrency in mind:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.5</version>
  <configuration>
    <parallel>classes</parallel>
  </configuration>
</plugin>

For finer control over the number of threads, you should check out the the configurable-parallel-computer library (see also this blog entry from the author of the library). (This actually made little difference on my dual-core laptop, but I would expect more significant differences on a large build server). Once you add this dependency to your dependencies (it's not in the standard Maven repositories, so you have to install it yourself), you could run your tests in parallel using 4 threads as shown here:

<plugin>
   <groupId>org.apache.maven.plugins</groupId>
   <artifactId>maven-surefire-plugin</artifactId>
   <version>2.5</version>
   <configuration>
     <parallel>methods</parallel>
     <threadCount>4</threadCount>
   </configuration>
</plugin>

You can also configure the maximum number of threads per CPU core, which allows for a bit more flexibility when the tests are run on different machines:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.5</version>
  <configuration>
    <parallel>methods</parallel>
    <threadCount>4</threadCount>
    <perCoreThreadCount>true</perCoreThreadCount>
  </configuration>
</plugin>

The 'useUnlimitedThreads' will create as many threads as there are classes or methods, and attempt to run them all concurrently. This works fine for unit tests, but when I used this option for web tests, it quickly saturated the server.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.5</version>
  <configuration>
    <parallel>classes</parallel>
    <useUnlimitedThreads>true</useUnlimitedThreads>
  </configuration>
</plugin>

Mileage will vary, depending on your machine capacity and on the nature of your tests, so you should experiment with different configurations and see what works best for you. You will generally get more significant gains from slower tests involving I/O, or web/database access, such as functional or web tests, than from large numbers of small, fast unit tests. Here are some examples I got for a medium-sized web application (running on my Macbook pro, which has 2 cores):

Using nothing:

$ mvn verify:
...
Tests run: 150, Failures: 0, Errors: 0, Skipped: 0

[INFO] [jetty:stop {execution: stop-jetty}]
[INFO] Stopping server 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1 minute 2 seconds

Using 'methods':

$ mvn verify:
...
Tests run: 150, Failures: 0, Errors: 0, Skipped: 0

[INFO] [jetty:stop {execution: stop-jetty}]
[INFO] Stopping server 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 35 seconds

Using 'classes':

$ mvn verify:
...
Tests run: 150, Failures: 0, Errors: 0, Skipped: 0

[INFO] [jetty:stop {execution: stop-jetty}]
[INFO] Stopping server 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 32 seconds

Using 'both':

$ mvn verify:
...
Tests run: 150, Failures: 0, Errors: 0, Skipped: 0

[INFO] [jetty:stop {execution: stop-jetty}]
[INFO] Stopping server 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1 minutes 1 seconds

Whether 'classes' or 'methods' works best depends largely on the number of test classes you have, and the number of methods in each class. For these tests, 'both' had no notable effect.

As mentioned above, in these tests, and on my machine, increasing the threadCount had little effect. But this, again, will depend on the nature of your tests.

TrackBack URI for this entry

Comments (8)

Subscribe to this comment's feed

...
written by Peter Niederwieser, July 06, 2010

http://bit.ly/buqkia argues that

1. "threadCount" has no effect unless org.jdogma.junit:configurable-parallel-computer is on the classpath
2. "perCoreThreadCount" is a boolean property
3. "classes" is usually faster than "methods"

Who's right?

• 
• +0
• 
• 

See updated blog entry
written by John Smart, July 06, 2010

Thanks for your comments, Peter - I've updated the blog entry with some extra details about the tests I ran, so I think you'll find the answers to your questions there.

• 
• +0
• 
• 

Logging ?
written by Frédéric Camblor, July 06, 2010

And what if, in our tests, we were logging things in maven console ?
Will this logging completely anarchistic in the generated output ?
Or will the maven log be "pushed" at the end of a test (method or class) execution ?

• 
• +0
• 
• 

Parallel logging
written by John Smart, July 07, 2010

If you log to stdout, Maven will write the output to the console as it happens, so in an unpredictable order - no magic here, I'm afraid. The JUnit XML reports, on the other hand, are fine.

• 
• +0
• 
• 

coverage
written by Vaidotas, July 12, 2010

I haven't found why, but my coverage report doped down, because coverage reporting don't find surfire-report folder in target for part of submodules. Any recommendations?

• 
• +0
• 
• 

Cobertura doesn't work well with maven modules
written by John Smart, July 12, 2010

Are you using Cobertura or Emma? You will have trouble getting aggregate coverage data out of these in Maven, as the tools don't really support it.
The Hudson Cobertura code coverage plugin handles multiple modules just fine, as does Clover.

• 
• +0
• 
• 

...
written by Vaidotas, July 12, 2010

I use Cobertura and Sonar. As I said, adding parallel to project I get that in some sub-modules surefire-reports are missing.

• 
• +0
• 
• 

...
written by Vaidotas, July 12, 2010

Maybe I'll try this: http://github.com/krosenvold/c...l-computer

• 
• +0
• 
• 

Name

Email

Title

Comment

smaller | bigger

I have read and agree to the Terms of Usage.

Add Comment Preview