Soft rules for testing software

Table of contents

1. Black-Box testing
2. Mock testing
3. Seams between worlds
4. Interfaces
5. Being confident with tests
6. Shift-Left testing
7. Prefer debugability over testability
8. Testing more than code

Black-Box testing

Mock testing

Continuing on from black-box testing is mock testing, mock testing is the antithesis of black-box testing where you assert that methods are invoked with specific parameters in specific orders. I argue that mock tests are always more harmful than any other form of testing, if you imagine your functions and classes as blocks of lego then mock testing is superglue. Any code change requires changing a dozen mock tests and more importantly you have no clue if the code change works. Mock tests verify nothing other than that the code is exactly what you wrote when you wrote the test. Tests are supposed to provide a level of confidence that a change has been made without changing existing behaviour in an undesirable way, mock tests allow no code changes to happen and therefore provide no confidence as the tests immediately fail on any change.

Virtual dependencies over mocks

                
@Component
public interface GlacierClient {
    Vault describeVault(String name);

    Vault createVault(String name);
}
                
                
@Component
@AllArgsConstructor
public class LogArchiver {
    public static final String LOG_ARCHIVE_VAULT = "LogArchiveVault";

    @Autowired
    private final GlacierClient glacier;

    public void createLongTermLogStorage() {
        Vault vault = glacier.describeVault(LOG_ARCHIVE_VAULT);
        if (valut == null) {
            vault = glacier.createVault(LOG_ARCHIVE_VAULT);
        }
    }
}
                
                
public class LogArchiverTest {
    @Mock
    private GlacierClient glacierClient;

    @Autowired
    private LogArchiver logArchiver;

    @Test
    public void testCreateLongTermStorageCreatesVaultIfNeeded() {
        // If the vault doesnt exist
        expect(glacierClient.describeVault(LogArchiver.LOG_ARCHIVE_VAULT)).andReturn(null);
        // then it should be created
        expect(glacierClient.createVault(LogArchiver.LOG_ARCHIVE_VAULT)).andReturn(new Vault(LogArchiver.LOG_ARCHIVE_VAULT));

        logArchiver.createLongTermLogStorage();
        replay(glacierClient);
    }

    @Test
    public void testCreateLongTermStorageDoesNotCreateVaultTwice() {
        // If the vault doesnt exist then the vault should not be created again
        expect(glacierClient.describeVault(LogArchiver.LOG_ARCHIVE_VAULT)).andReturn(new Vault(LogArchiver.LOG_ARCHIVE_VAULT));

        logArchiver.createLongTermLogStorage();
        replay(glacierClient);
    }
}
                
            

                
@Component
public class VirtualGlacierClient implements GlacierClient {
    private Connection jdbcConnection;

    private static class VaultRecord {
        public String name;
    }

    public VirtualGlacierClient() {
        this.jdbcConnection = DriverManager.getConnection("jdbc:sqlite:glacier.db");
        try (Statement statement = connection.createStatement()) {
            statement.execute("CREATE TABLE IF NOT EXISTS vaults (id INTEGER PRIMARY KEY, name TEXT, CONSTRAINT ak_name UNIQUE (name))");
            statement.execute("DELETE FROM vaults;");
        }
    }

    @Override
    public Vault describeVault(String name) {
        try (PreparedStatement statement = connection.prepareStatement("SELECT * FROM vaults WHERE name = ?")) {
            statement.setString(1, name);
            try (ResultSet results = statement.executeQuery()) {
                if (results.next()) {
                    String name = results.getString("name");
                    return new Vault(name);
                }
            }
        }

        return null;
    }

    @Override
    public Vault createVault(String name) {
        try (PreparedStatement statement = connection.prepareStatement("INSERT INTO vaults (name) VALUES (?)")) {
            statement.setString(1, name);

            // Throws a unique key constraint if a duplicate vault is created
            statement.executeUpdate();

            return new Vault(name);
        }
    }
}
                
                
public class LogArchiverTest {
    @Autowired
    private VirtualGlacierClient glacierClient;

    @Autowired
    private LogArchiver logArchiver;

    @Test
    public void testCreateLongTermStorageCreatesVaultIfNeeded() {
        // The vault should not exist beforehand
        Assertions.assertNull(glacierClient.describeVault(LogArchiver.LOG_ARCHIVE_VAULT));

        logArchiver.createLongTermLogStorage();

        // The vault should be created
        Assertions.assertNotNull(glacierClient.describeVault(LogArchiver.LOG_ARCHIVE_VAULT));
    }

    @Test
    public void testCreateLongTermStorageDoesNotCreateVaultTwice() {
        // Create the vault beforehand
        glacierClient.createVault(LogArchiver.LOG_ARCHIVE_VAULT);

        logArchiver.createLongTermLogStorage();

        Assertions.assertNotNull(glacierClient.describeVault(LogArchiver.LOG_ARCHIVE_VAULT));
    }
}
                
            

• Transparent to dependencies: Maybe later LogArchiver is reworked to delegate its vault creation logic to another dependency, this test would continue to function.
• Not coupled to the implementation: A new api could come along to create a vault and supporting it would require only updating the virtual dependency rather than each test case.
• Usable as a system test: theres no peering inside the service to determine what was and wasnt called, so porting this to be a system test is as simple as swapping out the virtual dependency with a real one.

Seams between worlds

Interfaces

Being confident with tests

Prefer debugability over testability

Testing more than code

• The Science of Unit Tests - Dave Steffen - CppCon 2020
• Stopping entropy with ArchUnit by Alexander Schwartz
• Things I Wish I Knew When I Started Testing Spring Boot Applications by Philip Riecks @ Spring I/O
• Mockito javadoc
• Fragile Test