Is "Integrated tests are a scam", a bad title?

Moved learn-c-the-hard-way to public repository

As mentioned last week that since I regularly mention talk about my progress with learn c the hard way that I should make it publicly available. SO here it is https://github.com/MarcMcIntosh/learn-c-the-hard-way currently I’m up to exercise 31, which didn’t involve any programming just reading about undefined behaviour (UB) which as far as my understanding of UB is behaviour that can act differently across different platforms (computers / compilers). There’s also a short list of some common casues of UB, I’ve not been able to find an extensive list yet but did find out that the compiler clang has a -fsanitize=undefined option to help detect sources of UB while compiling. More recent version of GCC also supports the fsanitize option.

Test pyramid

After researching and reading about UB and how to detect UB, I spent a short while reading about the test pyramid trying to figure out if there’s a golden rule that gives the pyramid it’s shape. So with out using pictures I’ll try to describe why the pyramid is a pyramid. The bottom axis (length wise) could be representative of the number of tests, while the side axis could represent the level of integration from totally isolated to totally Integrated or speed as unit tests are faster than UI / cross-browser testing.

Most of the pictures I’ve seen have unit tests on the bottom, as these are faster and isolated. The next level on the pyramid is service or integration tests, some sources have this as a multi-teared level. I like to think of this level as integration tests, because the commonality between all of the different names is that this level should require less tests than the unit tests, (mostly likely because the units should be thoroughly tested) but will take more time to complete the tests. The third and final level is once again domain specific such as end-to-end, GUI tests, acceptance tests, these will take the longest to run and will therefore have the fewest tests. I say fewest, but cross browser visual regression testing would be lots of tests, so maybe saying the number of times the test is run would be a better measure for the bottom axis.

Integrated tests are a scam

Although it does sound contradictory for the next activity to have been watching J. B. Rainsberger’s video Integrated Tests Are A Scam, it’s not. JB does make a distinction between the testing behaviour between the code we control (units) and code we don’t, then further calls the tests we preform between units (collaborators) as contracts.

Contracts are expected behaviours between units/collaborators, such as the interactions between a client application and server. These interactions can be written / tested using mocks and stubs, such as mocking a client request then responding with a hard coded stub and vice-versa for the client receiving a hard code stub to simulate expectations. The overall workflow of using a mock as an interface allows the two collaborators to be isolated and tested to see if they implement the contract required by the mock interface.

A pseudo-simulation of generating an error test case on the client side,

// Mock interface
function mockFetchWithError(request) {

  return new Promise((resolve, reject) => {
    const errorStub = { ok: false, status: 404, message: "Not found", request };

    setTimeout(function() {
      reject(errorStub);
    }, 500);
  }
}

// client handle error
function handleError(error) {
  if(error.status === 404) {
    // redirect to not found page
  }
}

function handleResponse(response) {
  if(!response.ok) { throw response; }
}

// Handle error test
mockFetchWithError('/something-that-does-not-exist')
  .then(handleResponse)
  .catch(handleError);

After a writing a few response stubs we would basically have an explicit interface for our service to implement.

Aside from some discussion of mocking collaborators to create contracts, there only mention of where to traditional integration tests was for code we don’t control such as a third party API. Overall I would recon the video is made to promote the use of unit tests rather than not using integration tests as eventually we’ll have to check that the collaborators work together as expected, but this should be done after testing that each implements the expectations from the mock interface.

Written by Marc McIntosh Find him on github