3 Problems We Encountered Using @xstate/test With TestCafe — and Their Solutions

The Beginning

The @xstate/test Quick Start states that there is the createModel-function which takes any state machine and generates test path plans for it. The created model exposes two functions to generate these plans which can be used to test out your app’s state machine. On one hand there is getSimplePathPlans which will return all simple paths. Simple paths are paths from an initial state to any other state by traversing via any possible other state. On the other hand there is getShortestPathPlans which returns the shortest paths from the initial state to any other state.

For the given case we decided to go with getSimplePathPlans since we like to test every possible state traversal combination. And that's where we encounter our first problem: the guards.

The guards Problem

When using createModel the state machine is executed in the test framework context without invoking any actual services. This leads to a problem, since the services most likely fetch data, which will alter the context and then determines the outcome of a guard. So calling getSimplePathPlans either fails at a specific guard, because no real data is available or the guard returns its default value, which leads to not all paths being generated and in turn not all available paths get tested.

That's why we implemented a guard combination calculator function which takes a guards object and returns an array with all possible combinations of the guards. It actually builds a simple binary table (or a logical or truth table) where each 0 or 1 is a possible result for a given guard function. To make this more comprehensible, let’s take the guards from the people search app:

jsx
const guards = {
  hasQuery: ({ query }) => query !== null,
  hasMore: (_, event) => isPeopleDoneEvent(event) && !!event.data.next,
  hasPeople: (_, event) => isPeopleDoneEvent(event) && event.data.count > 0,
}

Each of the above guard-functions can either return false or true (0 or 1) and therefore lead to a different path within the state chart. But if there is no data since no actual service gets called during the evaluation of the state machine, it is not possible that for instance query or event.data will ever be defined. That's why we implemented a function which takes the guards object and builds an array with all possible combinations:

jsx
const guardCombinations = [
  {
    hasQuery: () => false,
    hasMore: () => false,
    hasPeople: () => false,
  },
  {
    hasQuery: () => true,
    hasMore: () => false,
    hasPeople: () => false,
  },
  {
    hasQuery: () => false,
    hasMore: () => true,
    hasPeople: () => false,
  },
  {
    hasQuery: () => false,
    hasMore: () => false,
    hasPeople: () => true,
  },
  {
    hasQuery: () => true,
    hasMore: () => true,
    hasPeople: () => false,
  },
  // and so forth...
];

Now we can iterate over the array and generate path plans to test for each combination by passing the generated guards into the machine as configs.

jsx
const testPlans = guardCombinations.reduce((plans, guards) => [
  ...plans,
  ...createModel(Machine(statechartToTest, { guards }))
    .withEvents(testEvents)
    .getSimplePathPlans(),
], []);

Finally we have all our path plans. But how does @xstate/test actually tests them?

The meta Problem

From the @xstate/test Quick Start we know that we need to add a meta-property to each state we like to test. So the people search state chart would look like:

jsx
const statechart = {
  // ...
  id: "PeopleMachine",
  initial: "Fetching",
  states: {
    Fetching: {
      // ...
      meta: {
        test: ({ t }) =>
          t.expect(Page.Items.count).eql(0).expect(Page.Loading.exists).ok(),
      },
    },
    Idle: {
      // ...
      initial: "More",
      states: {
        More: {
          // ...
          meta: {
            test: ({ t }) =>
              t
                .expect(Page.Items.count)
                .eql(10)
                .expect(Page.MoreButton.exists)
                .ok(),
          },
        },
        End: {
          // ...
          meta: {
            test: ({ t }) =>
              t
                .expect(Page.Items.count)
                .gt(0)
                .expect(Page.MoreButton.exists)
                .notOk()
                .expect(Page.GoToTopButton.exists)
                .ok(),
          },
        },
        NoResults: {
          // ...
          meta: {
            test: ({ t }) =>
              t
                .expect(Page.Items.count)
                .eql(0)
                .expect(Page.NoResults.exists)
                .ok(),
          },
        },
      },
    },
    // ...
  },
};

But we do not like to have test code in our state charts as it will be delivered to the client’s browser. Therefore we would like to extract the meta-code and programmatically append it to the given state chart, so that the test code can be written separately:

jsx
const tests: StatesTestFunctions<Context, TestContext> = {
  Fetching: ({ t }) =>
    t.expect(Page.Items.count).eql(0).expect(Page.Loading.exists).ok(),
  Idle: {
    More: ({ t }) =>
      t.expect(Page.Items.count).eql(10).expect(Page.MoreButton.exists).ok(),
    End: ({ t }) =>
      t
        .expect(Page.Items.count)
        .gt(0)
        .expect(Page.MoreButton.exists)
        .notOk()
        .expect(Page.GoToTopButton.exists)
        .ok(),
    NoResults: ({ t }) =>
      t.expect(Page.Items.count).eql(0).expect(Page.NoResults.exists).ok(),
  },
  // ... additional state tests
};

Tim Deschryver wrote a very nice article how he did this in combination with Cypress. We took his approach as a basis, added the above guard combination logic and a logging mechanism and released it as a package called @smartive/xstate-test-toolbox.

This package contains the helper createTestPlans which can be used with xstate and xstate/test.

The createTestPlans function from the package helped us to generate all possible test paths for our state machines:

jsx
const tests = {
  Fetching: ({ t }) =>
    t.expect(Page.Items.count).eql(0).expect(Page.Loading.exists).ok(),
  // ...
};

const testEvents = {
  /* Will be discussed in the next section. */
};

const testPlans = createTestPlans({ machine, tests, testEvents });

// Iterate over plans and test paths
testPlans.forEach(({ description: plan, paths }) => {
  fixture(plan).page("<http://localhost:8080>");

  paths.forEach(({ test: run, description: path }) => {
    test(`via ${path} ⬏`, (t) => run({ plan, path, t }));
  });
});

So now we have generated all tests from our state charts and magically everything gets tested — Well, not quite. Since @xstate/test expects us to define what kind of user event or other event does execute a state change, we ran into the next problem.

The async Problem

The idea behind @xstate/test is that we define what should be tested in state X and what kind of action (e.g. a button click) transitions the app into the next state Y. So how do we test invoked services?

Let’s consider the following state chart as an example: