Good Tests

Unit Testing – Good Tests

Learn what makes good unit tests

There are a few things to keep in mind as you design and implement your unit tests. Familiarize yourself with the characteristics of good unit tests in the table below and then practice applying these principles in the exercises.

IndependentUnit tests should not depend on each other. There should not be a sequence in which the unit tests must be executed. One unit test should not depend on another unit test to arrange its state. A unit test should arrange its own state for its particular act of what it is testing. A unit test should be able to run independently of other unit tests.
FastA unit test should run fast. It’s setup, execution, and tear down should be fast. The point of unit tests is that we can run them often, many times a day, every day, as we develop our software application. This gives us quick immediate feedback when we break things, so we can fix them right away while the code is still fresh in our mind.
RepeatableUnit tests need to be predictable and have deterministic outcomes. It should not matter what environment they run in, or what time of day they run, or what other threads are doing while they run. A unit test that produces different results every time it runs on the same application code that didn’t change is not repeatable. Running a unit test at 2am compared to 2pm should not affect the results. A threads race condition should not affect the results of a unit test.
One clear goalA unit test should focus on testing one clear condition. One aspect, one scenario. If it tests more than one thing then when it fails we wouldn’t know if the failure in the application code was for the one condition or the other, slowing us down. The name of the unit test method should clearly identify the application method it is testing, which condition of that method it is testing, and what the expected result is.
Non-interactiveA unit test should not rely on user interaction. We want unit tests to run automatically without a user having to interact with the application for them to work.
ValuableA unit test should test something valuable. No need to write unit tests for getters and setters or for third party code or for low risk code.

Exercises

Below is an implementation of two unit tests, but they do not meet the criteria of good unit tests. Can you see why not? At first glance they may seem just fine, but look closer.

public class LoginServiceTest {

  LoginService loginService = new LoginService();
  UserAccount userAccount = new UserAccount("user", "password");

  @Test
  public void changePasswordSucceedsIfPasswordMeetsAllRules () {
    // arrange
    String newPassword = "newPassword1";
    // act
    loginService.changePassword(userAccount, newPassword);
    // assert
    Assertions.assertEquals(newPassword, userAccount.getPassword());
  }
  
  @Test
  public void changePasswordFailsIfOldAndNewPasswordsSame () {
    // arrange
    String newPassword = "newPassword1";
    Date dateOfLastPasswordChangeBeforeAct = userAccount.getDateOfLastPasswordChange();
    // act
    loginService.changePassword(userAccount, newPassword);
    // assert
    Assertions.assertEquals(
      dateOfLastPasswordChangeBeforeAct,
      userAccount.getDateOfLastPasswordChange());
  }

}

First let’s make sure we understand what the bottom unit test is trying to do. The bottom unit test changePasswordFailsIfOldAndNewPasswordsSame() ensures the date when the password was last changed is not updated during an attempt to set the password to the same thing as it was before.

Look at the highlighted arrange section of the bottom unit test. It sets the new password for the test to “newPassword1” and assumes the old password is also “newPassword1”, but where is the old password initialized for this test?

public class LoginServiceTest {

  LoginService loginService = new LoginService();
  UserAccount userAccount = new UserAccount("user", "password");

  @Test
  public void changePasswordSucceedsIfPasswordMeetsAllRules () {
    // arrange
    String newPassword = "newPassword1";
    // act
    loginService.changePassword(userAccount, newPassword);
    // assert
    Assertions.assertEquals(newPassword, userAccount.getPassword());
  }
  
  @Test
  public void changePasswordFailsIfOldAndNewPasswordsSame () {
    // arrange
    String newPassword = "newPassword1";
    Date dateOfLastPasswordChangeBeforeAct = userAccount.getDateOfLastPasswordChange();
    // act
    loginService.changePassword(userAccount, newPassword);
    // assert
    Assertions.assertEquals(
      dateOfLastPasswordChangeBeforeAct,
      userAccount.getDateOfLastPasswordChange());
  }

}

In the implementation of the two unit tests, the bottom unit test depends on the top unit test to run first. The bottom unit test cannot be executed independently of the top one. What if we want to run just the bottom unit test? Or what if they run out of order? In this example the bottom unit test depends on the top unit test to change the password to “testPassword1”. If it doesn’t then our bottom unit test breaks and is of no use to us.

Below is a better implementation that keeps the two unit tests independent of each other. Either one can be run without depending on the other to run first, or they can run in parallel by different threads.

public class LoginServiceTest {

  @Test
  public void changePasswordSucceedsIfPasswordMeetsAllRules () {
    // arrange
    LoginService loginService = new LoginService();
    UserAccount userAccount = new UserAccount("user", "password");
    String newPassword = "newPassword1";
    // act
    loginService.changePassword(userAccount, newPassword);
    // assert
    Assertions.assertEquals(newPassword, userAccount.getPassword());
  }
  
  @Test
  public void changePasswordFailsIfOldAndNewPasswordsSame () {
    // arrange
    LoginService loginService = new LoginService();
    UserAccount userAccount = new UserAccount("user", "newPassword1");
    String newPassword = "newPassword1";
    Date dateOfLastPasswordChangeBeforeAct = userAccount.getDateOfLastPasswordChange();
    // act
    loginService.changePassword(userAccount, newPassword);
    // assert
    Assertions.assertEquals(
      dateOfLastPasswordChangeBeforeAct,
      userAccount.getDateOfLastPasswordChange());
  }

}

Below is an implementation of some unit tests, but they do not meet the criteria of good unit tests. Can you see why not?

public class BatchJobTest {

  @Test
  public void batchJobShouldSucceedWithLargeData () {
    // arrange
    BatchJob batchJob = new BatchJob();
    Data data = new Data("/data/huge_data_file.txt");
    
    // act
    batchJob.runAndReturnWithoutWaitingForItToFinish(data);
    Thread.sleep(10000);
    
    // assert
    Assertions.assertTrue(batchJob.isSuccess());
  }

  @Test
  public void batchJobShouldFailBetween1amAnd6am_v1 () {
    // arrange
    BatchJob batchJob = new BatchJob();
    
    // act
    BatchResult result = batchJob.run();
    
    // assert
    Assertions.assertEquals(BatchResult.BLOCKED_DUE_TO_TIME_OF_DAY, result);
  }
  
  @Test
  public void batchJobShouldFailBetween1amAnd6am_v2 () {
    // arrange
    BatchJob batchJob = new BatchJob();
    DateTimeFormatter formatter = DateTimeFormatter.ofPattern("HH:mm");
    LocalTime now = LocalDateTime.now().toLocalTime();
    LocalTime start = LocalTime.parse("01:00", formatter);
    LocalTime end = LocalTime.parse("06:00", formatter);
    
    // act
    BatchResult result = batchJob.run();
    
    // assert
    if (now.isAfter(start) && now.isBefore(end)) {
      Assertions.assertEquals(BatchResult.BLOCKED_DUE_TO_TIME_OF_DAY, result);
    } else {
      Assertions.assertNotEquals(BatchResult.BLOCKED_DUE_TO_TIME_OF_DAY, result);
    }
  }
}

The unit tests defined in the BatchJobTest class test the behavior of the BatchJob class. Ask yourself, if we didn’t change anything in the code of BatchJob would our unit tests in BatchJobTest always give us the exact same test results no matter what?

The problem with all of the unit tests defined in BatchJobTest is that none of them reliably give the same test results every time they are executed.

  • batchJobShouldSucceedWithLargeData() triggers the method we are testing to run asynchronously and then waits a predetermined time for it to finish, before checking the results of the test. What if we didn’t wait long enough? This unit test is not predictable and not deterministic. It depends on the environment in which it is executed. What if something in the environment is slowing the batch job down past the amount of time we wait. If we wait too long then now our unit test doesn’t run as fast as it can. As long as we don’t change the code we are testing, BatchJob, then the unit tests that test is should predictably and deterministically always return the same test results. The unit tests should not be affected by thread race conditions or how long the thing we are testing takes to run.
  • batchJobShouldFailBetween1amAnd6am_v1() only works correctly when we run it between 1am and 6am. This unit test is not predictable because if we run it at the “wrong” time then it doesn’t work as intended. A good unit test can be executed at any time and give the same test results regardless of the time it was run. A developer may work during the day or late at night and needs the unit tests to work the same anytime and every time. Recall a developer runs unit tests over and over again as they develop the application code.
  • batchJobShouldFailBetween1amAnd6am_v2() works anytime we run it, but it tests something different depending on the time of day we run it. It forces a developer to run it at different times of the day. Who has time for that? A developer needs something reliable that gives the same test results anytime and every time they use it, regardless of when they choose to work. The other problem with this unit test is it tests more than one thing which slows us down when it fails because we have to figure out which condition caused it to fail.

Below is an implementation of some unit tests, but they do not meet the criteria of good unit tests. Can you see why not?

public class FormValidationTest {

  @Test
  public void addressHasValidStreetName () {
    // arrange
    List<String> streetNames = Loader.load("all_street_names_in_USA_(alphabetical_lowercase).txt");
    Random random = new Random();
    String streetName = streetNames.get(random.nextInt(streetNames.size()));
    Address address = new Address("1234 " + streetName + ", boston, ma");
  
    // act
    boolean result = FormValidation.isStreetNameValid(address);
    
    // assert
    Assertions.assertTrue(result);
  }
  
  @Test
  public void addressHasInvalidStreetName () {
    // arrange
    List<String> streetNames = Loader.load("all_street_names_in_USA_(alphabetical_lowercase).txt");
    String streetName = streetNames.get(streetNames.size() - 1) + "abc";
    Address address = new Address("1234 " + streetName + ", boston, ma");
  
    // act
    boolean result = FormValidation.isStreetNameValid(address);
    
    // assert
    Assertions.assertFalse(result);
  }

}

Is the implementation of the unit tests as fast as it can be?

Both unit tests load into memory a very large file of data. The large file is loaded twice, once for each unit test, taking twice the amount of time. Both times it is used for reference only, reading not writing. Recall we want our unit tests to run as fast as possible. Is this implementation of our unit tests the fastest it can be?

Typically we prefer each unit test to arrange its own data, so it is independent of all other unit tests. This is very important. In this case because we are loading reference data into memory that would be used by potentially multiple unit tests, it is ok to load it once and use it in multiple unit tests, as long as we avoid creating a dependency between unit tests. We don’t want one unit test to load it for all other unit tests, since this creates a dependency. What we could do is load it once for all unit tests before they run, as follows. This way the unit tests can run in any order.

public class FormValidationTest {

  List<String> streetNames;

  @BeforeAll
  public void loadBigDataOnce () {
    streetNames = Loader.load("all_street_names_in_USA_(alphabetical_lowercase).txt");
  }

  @Test
  public void addressHasValidStreetName () {
    // arrange
    Random random = new Random();
    String streetName = streetNames.get(random.nextInt(streetNames.size()));
    Address address = new Address("1234 " + streetName + ", boston, ma");
  
    // act
    boolean result = FormValidation.isStreetNameValid(address);
    
    // assert
    Assertions.assertTrue(result);
  }
  
  @Test
  public void addressHasInvalidStreetName () {
    // arrange
    String streetName = streetNames.get(streetNames.size() - 1) + "abc";
    Address address = new Address("1234 " + streetName + ", boston, ma");
  
    // act
    boolean result = FormValidation.isStreetNameValid(address);
    
    // assert
    Assertions.assertFalse(result);
  }

}