Core Cypress Lessons We Learned That 10x-ed Our E2E Automation

For some time now, I’ve been leading the QA efforts at Stoplight – a USA-based startup that we are the engineering partner. To give you some context, Stoplight builds products in the API domain. They are the creators of the API design Studio, an enterprise API platform for managing APIs at scale and generous contributors to the OpenSource (see Prism or Spectral for example).

Awakening of Cypress

Until about a month ago, our team was happily using Codecept. As it often happens in startups, things change, and we now joyfully code our tests with Cypress :).

Yes – we ditched all tests we wrote, re-thought our approach, and wrote them from the grounds up!

Was it the right choice? What did we learn on that journey? Well, keep on reading to find out!

The Grand Redesign & The Window Of Opportunity

At the time, we were deciding whether to switch the frameworks or not Stoplight was at the end of a significant product redesign. That allowed us to redo most of our automation suite.

As you can imagine switching to a new framework was not a light decision. Our crew was pretty knowledgeable about Codecept and, despite it having many bumps, we weren’t sure if Cypress would help with anything.

The promise was bold, though! Because of how the Cypress is written we were supposed to get:

0 flaky tests,
faster execution,
better onboarding,
and more versatile debugging!

In other words, we were expecting to be able to write lightning-fast and stable tests in less time!

Now, after about a month of working with Cypress, I can share my initial impressions, lessons learned, and top tips that should help you onboard to Cypress and make fewer mistakes than we did!

Ready to learn some essential core concepts and take your Cypress game to the next level?

Here we go! Here is a list of top lessons we learned and a bunch of advice that can help you onboard to Cypress quicker and make fewer mistakes than we did!

The 3 game-changing concepts

If there were only three points I advise you to remember it would be those:

Cypress commands run serially but are asynchronous (they look like Promises, but they aren’t!).
Avoid long command chains because only the last command is repeated.
Change your mental-model from waiting to taking actions.

And that would be it. I could call it a day and finish this article here.

But what good would this article be without examples! Like they say, "code is worth a thousand words."

Just to make it as realistic as possible, the things below are the "real deal". It’s our blood, sweat, and tears since we bumped on each of those issues and learned a ton while sorting them out! I hope you’ll get out of here armed with some handy knowledge.

Everything is asynchronous

Let’s start with the following code. Try to guess what is going to happen when you run it.

// Incorrect: will not work

it('compare input value to a string', () => {
  cy.visit('/project');
  const projectName = Cypress.$('input#project-name').val();
  expect(projectName).to.eq('Your First Project');
});

You may expect this code to visit /project page, grab the value of an input field, and verify that it’s equal to "Your First Project".

However, that is not what is going to happen. If not, then what does happen and how to fix it?

One of the first concepts you’ll need to understand is that all Cypress’s commands appear to be synchronous but are, in fact, asynchronous.

To make the above code pass you’ll need to make the following adjustment:

// Correct

it('compare input value to a string', () => {
  cy.visit('/project');
  cy.get('input#project-name').should(($input) => {
    const projectName = $input.val();
    expect(projectName).to.eq('Your First Project');
  });
});

Great, but why would the above example not work while the latter does? The critical concept to understand is: each Cypress command you invoke adds to a global, serial chain. However, none of those commands run until the function finishes executing!

That explains why const projectName = Cypress.$('input#project-name').val(); fails to find any element. Notice that this line is a plain, synchronous code. It runs exactly when invoked. On the other hand, cy.visit('/project'); does not run immediately but schedules a "job" to run "later".

This means that Cypress.$('input#project-name').val() will actually run before cy.visit!

To solve this and similar problems, you need to make sure you always tap into the Cypress chain (see the "correct" example).

Some of you might wonder how this test is asynchronous if there isn’t any promise returned nor any done function passed to the test.

Cypress does the magic for you. It manages the asynchronous queue in the background so that you can write tests "as if" they were synchronous.

A Promise or not a Promise that is the question

That’s the bit that threw me off initially. I was so used to async/await in the code that when I first saw a PoC of rewriting the code to Cypress using a bunch of then chains, I thought "what the(n) hell"!

It looked something like this (pseudo code):

it('sign up with email & password', () => {
  cy.visit('/auth');
  cy.contains('Username').type('qa');
  cy.contains('Password').type('super-strong!');
  cy.contains('Continue')
    .click()
    // Before we log in, we need to grab a confirmation code 
    // from a database. We use a GraphQL API in order to get it.
    .then(() => graphql.getConfirmationCode())
    .then((code) => {
      cy.contains('Code').type(code);
      cy.url().should('include', '/welcome');
      // etc. some other operations nested in this then
    });
});

My immediate reaction to this code was: rewrite to async/await.

Surprise, surprise! That is not going to work!

The sooner you understand why and get over it the better 😉

The reason this doesn’t work is that Cypress then only reminds a Promise. Internally, though, it’s a "Promise on steroids". Or to be more accurate Promise with retries and timeouts aware of its chain! For example:

it('get and retry', () => {
  // get returns a power-promise that will retry 
  // until the DOM element is found and then passes 
  // the execution to should. If should fails it will 
  // retry to get and should again!
  cy.get('#projects-list').should('have.length', 10);
});

Additionally, when you do things like:

it('test multiple interactions', () => {
  // just grabbing something from the page
  cy.get('[data-test=specs]').should(
    'have.text',
    'Best cats API in the world!'
  );

  cy.window().then((win) => {
    // calling some internal API
    win.__internalAPI.setSpecTo('{ "openapi": "3.0.0"}');
  });

  // typing some code into editor
  cy.contains('[data-test=editor]')
    .find('line:nth-child(1)')
    .type('"info": {}');
});

What you’re doing in the above code is creating a global chain of events that are scheduled and run by Cypress for you (remember the section about asynchronous code?).

Here is one interesting fact. Since everything gets added to the global chain automatically, it’s impossible to ever "lose" a Promise. Everything runs in order even if you don’t explicitly "chain" it.

cy.window().then((win) => doSomething(win));
// Next line will be executed after then 
// even though you have not chained it with an extra then!
cy.contains('Add Model').click();

Do not chain `then` if not necessary

Following the previous example, it is not necessary to do things like:

cy.visit()
  // 1. you can but don't need to chain this one
  .then(makeHttpRequest)
  .then((response) => {
    cy.get('input').type(response);
    // 2. unnecessary nesting
    cy.get('button').click();
    // 3. unnecessary chain
    return cy.visit('/');
  })
  .then(() => {
    cy.get('button').click();
  });

(1) is required but can be done differently.
(2) is simply not necessary. You can step outside the then as soon as you do your thing. Cypress puts everything on a serial queue – even commands nested in then!
(3) is a pure waste of indentation 😉 Don’t do that.

See how you could refactor this code.

cy.visit()
  .then(makeHttpRequest)
  .then(response => cy.get('input').type(response));
cy.get('button').click();
cy.visit('/');
cy.get('button').click();

The only required then here is the one after visit. You need to explicitly resolve your promises (or use should!).

There are obviously trade-offs to this solutions including:

not being able to catch errors
not being able to use async/await

Read more about it on the official docs page: Commands are Not Promises. I really recommend it!

Only last command before an assertion repeats.

That’s the most significant source of flake we encountered. Read carefully.

If you chain multiple commands and put an assertion at the end, then only the last command repeats! It means that your assertion sometimes fails because the last command you executed is working with an outdated DOM. If that happens, you are up to some nasty flakes.

Consider this:

cy.get('[data-test=spec]').find('h2').should('have.text', 'Boom!');

Assume that there can be multiple [data-test=spec] elements on the page. If they get added asynchronously (with some delay) this code may fail in such way:

It will get the all immediatelly available [data-test=spec] elements.
Find all h2 available in the list of [data-test=spec].
Will try to find h2 matching Boom!.
If should fails it will retry find but not get!
In the meantime, new [data-test=spec] is added but get is not retried and your test fails.

A recommended way to do this is:

cy.get('[data-test=spec] h2').should('have.text', 'Boom!');

That may fail at should but retries get until should passes or timeouts!

It’s a must-read, very well documented on the Cypress website: read it!

Prefer `should` over `then`

A somewhat similar to the previous but also cause some flake for us.

Remember: then does not repeat!

For instance, if you do:

cy
  .window()
  // then is not repeated and should 
  // will fail if data is not already set
  .then(window => window.SOME_GLOBAL.data)
  .should('eq', 'expected');

A better way to do this is to replace then with should and use explicit assertions.

cy.window()
  // this way if should fails it will retry
  .should((window) => {
    expect(window.SOME_GLOBAL.data).to.eq('expected');
  });

`should` ignores returned value

In one of our cases, we had to add a particular explicit wait to ensure our backend did its thing.

cy.visit('/')
  // do some stuff with forms
  // ...
  .wait(2000)
  .then(getInvitation)
  .as('link');

We thought we could avoid using wait by replacing then with should and asserting we received the data. The idea was that it should automatically repeat until the server is ready.

cy.visit('/')
  .contains('Confirm Invitation')
  .should(() => {
    return getInvitation().then((link) => {
      expect(link).not.to.be.undefined;
      return link;
    });
  })
  .as('link');

We found out (after trying this and reading the docs more carefully) that:

should ignores explicitly returned value and returns the last "subject"! In this case, it’s the result of contains('Confirm Invitation').

Just bear that in mind when trying tricks like the one above.

Use explicit `wait()` sparingly

You might tend to add a wait if you’re coming from selenium based projects.

Typically, you don’t need to do that in Cypress.

We had only three cases when we were forced to use wait (and to be frank, even those cases have workarounds):

when waiting for a backend job to finish without a clear sign in UI;
when dealing with asynchronous operations that are difficult to predict (one of our components needs about a second to reactivate)
when grabbing async data from the window object (we need to wait for 1s on average for some data to propagate to our react store because of all the crazy stuff it’s doing in the background).

However, normally you should be do well with waiting for UI changes:

check a URL change,
check for an element or text to appear on a page,
or just click the next element you expect to appear, simple as that.

If everything else fails, try bumping an inline timeout.

// waits 10s for the element
cy.get('#some-delayed-element', { timeout: 10000 }).click();

Mind your environment variables

Moving from Codecept, where everything runs on the server, made us bump into a wall at some point.

Longs story short, we use GraphQL API to set the state of the app. GraphQL needs credentials which we set via an environment variable in CI (Continuous Integration).

Here is pseudocode.

// client.ts
function client() {
  return createGraphQlSdk({
    host,
    password: process.env.GRAPHQL_PASSWORD || 'letmepass!',
  });
}

We use that client in test cases as well as tasks. If you’re not sure what a task is I recommend reading about it here;

Pseudo plugin code

import client from './client';

module.exports = (on) => {
  on('task', {
   seed() {
      return client.createUser();
    },
  });
};

Pseudo test case

import client from './client';

beforeEach(() => {
  cy.task('seed');
});

it('create a project', () => {
  cy.wait(0).then(() => client.createProject());
});

The above was a classic case of "runs locally, fails on CI".

Symptoms were that the beforeEach statement ran correctly and seeded our DB, but the test case failed, saying that we didn’t provide the GraphQL password…

Wait. What?!

Both the seed and createProject rely on password being set, so why would only one of them fail?

See if you can spot the problem.

Ready?

Cypress runs in two contexts.

One is the browser. The second one is node.

They run in different processes, and the browser process has no idea about the environment variables you set in your shell if you do something like:

GRAPHQL_PASSWORD=123456789 yarn e2e start

To fix this in the browser, you need to use Cypress.env(). However, Cypress.env is not defined in node… That means you need to do a bit of juggling when sharing environment variables between the two.

// a quick fix for the above issue
function env(varName: string) {
  return (Cypress || process).env[varName];
}

Not a big deal, but don’t let that surprise you!

Conclusion

We’ve had a great experience with Cypress so far (except few bumps described here and having to use puppeteer to test OAuth login flows).

Cypress does prove to be less flake-prone than selenium so far, but it isn’t wholly flake-resistant.

Writing tests in Cypress was easy, and efficient. We’ve decided to stick to it for the time being.

We did an internal meetup to share our thoughts and best practices with other engineers, and I’ll be hosting an internal Cypress workshop soon. If all goes well, the hope is that we can enable engineers to be more involved with e2e tests and QA.

We’re now experimenting with their paid Dashboard and analytics (beta). My initial impressions are that it will be very handy and help us track:

stability
coverage growth over time
e2e performance

Overall, it was eye-opening to discover the capabilities of this framework, and we’ve learned a lot. Go ahead and try it out!

Core Cypress Lessons We Learned That 10x-ed Our E2E Automation

Awakening of Cypress

The Grand Redesign & The Window Of Opportunity

The 3 game-changing concepts

Everything is asynchronous

A Promise or not a Promise that is the question

Do not chain `then` if not necessary

Only last command before an assertion repeats.

Prefer `should` over `then`

`should` ignores returned value

Use explicit `wait()` sparingly

Mind your environment variables

Pseudo plugin code

Pseudo test case

Conclusion

Let's get in touch!

Members of

Company

Business

Get in touch

Awakening of Cypress

The Grand Redesign & The Window Of Opportunity

The 3 game-changing concepts

Everything is asynchronous

A Promise or not a Promise that is the question

Do not chain then if not necessary

Only last command before an assertion repeats.

Prefer should over then

should ignores returned value

Use explicit wait() sparingly

Mind your environment variables

Pseudo plugin code

Pseudo test case

Conclusion

Let's get in touch!

Members of

Company

Business

Get in touch

Do not chain `then` if not necessary

Prefer `should` over `then`

`should` ignores returned value

Use explicit `wait()` sparingly