Nightwatch and BrowserStack

Recently I’ve been using Nightwatch.js and BrowserStack for E2E testing. Nightwatch.js is one of E2E testing frameworks which provides Test Runner, Assertions, Commands and etc. It also allows us to create Custom Assertions and Custom Commands as well. BrowserStack provides cross browser testing environment. For example, it allows us to operate any browsers which are supported by BrowserStack it our own browser. It even enable us to test our localhost using Chrome extension. Furthermore, we can operate mobile devices such as iPhone and Android.

Automate testing

BrowserStack supports Nightwatch.js on their automate testing offering. Following is a link for it.

Selenium with Nightwatch

You need only a few installations and configurations. Basically it’s easy. But there’re some pitfalls to work with browsers on BrowserStack. I’m happy to share them below.

Make it works

Use Nightwatch.js v0.9.*

Current stable version of Nightwatch.js is v0.9. But there’s pre release version v1.0. Pre release version doesn’t work at all with BrowserStack regardless of type of testing such as local or remote. So it appears to me v0.9 is only a option to work with BrowserStack.

Choose remote testing over local testing

Local testing worked on my environment, but it wasn’t stable. So remote testing which specifies public URL must be easy. That’s more stable. Also, there’s a limitation for local testing on Safari (both macOS and iOS). We can’t use localhost.

I face issues while testing localhost URLs or private servers in Safari on macOS/OS X and iOS.

Use “retries” option

Probably this depends on an application, E2E is unstable sometimes. There’re many factors which affects E2E testing like browser, network, 3rd party services and etc. So specifying retries option makes sense.

Use waitForElementPresent over waitForElementVisible

Edge requires elements are actually shown in screen on waitForElementVisible. So it make problems on a tall screen. waitForElementPresent works well even when elements are not shown. This behavior only appears on Edge.

Tips

Specifying session name

If you do nothing special, sessions are named from test directories and files like Home / Timeline. This will make many duplications if you write plural tests in single file. In that case, you should specify session name.

browser.options.desiredCapabilities.name = 'Test A';

If you want session name automatically be specified from its test name, you can do something like this.

before: function (browser) {
  this.browserName = browser.options.desiredCapabilities.browser;
  this.baseName = browser.options.desiredCapabilities.name;
  this.steps = browser.currentTest.results.steps.slice(0);
}

beforeEach: function (browser) {
  let bsTestName = self.baseName;
  let localTestName = '';
  const [ nextTestName ] = browser.currentTest.results.steps;
  if (nextTestName) {
    const idx = this.steps.indexOf(nextTestName) - 1;
    if (idx >= 0) {
      localTestName = this.steps[idx];
      bsTestName = `${self.baseName} / ${localTestName}`;
    }
  } else {
    localTestName = this.steps[ this.steps.length - 1 ];
    bsTestName = `${this.baseName} / ${localTestName}`;
  }
  this.localTestName = localTestName;

  bsTestName = sanitize(bsTestName);
  this.bsTestName = bsTestName;
  browser.options.desiredCapabilities.name = bsTestName;
}

Update session’s result (if it failed)

Most of cases, sessions resulted in passed on BrowserStack even though it actually failed. If you want to know correct results on BrowserStack, you have to update session’s status using BrowserStack API like following.

afterEach: function (browser, done) {
  const API_PREFIX = 'https://api.browserstack.com/automate';
  const auth = {
    username: process.env.BROWSERSTACK_USERNAME,
    password: process.env.BROWSERSTACK_ACCESS_KEY
  };
  const resBuild = await axios.get(`${API_PREFIX}/builds.json`, { auth });
  const [ build ] = resBuild.data;
  const buildId = build.automation_build.hashed_id;

  const resSessions = await axios.get(`${API_PREFIX}/builds/${buildId}/sessions.json`, { auth });
  let session = null;
  for (const sess of resSessions.data) {
    if (sess.automation_session.name === this.testName &&
      sess.automation_session.browser === this.browserName) {
      session = sess;
      break;
    }
  }
  const sessionId = session.automation_session.hashed_id;
  const data = {
    status: 'failed',
    reason: ''
  };
  await axios.put(`${API_PREFIX}/sessions/${sessionId}.json`, data, { auth });
  done();
}