Nightwatch and BrowserStack

Recently I’ve been using Nightwatch.js and BrowserStack for E2E testing. Nightwatch.js is one of E2E testing frameworks which provides Test Runner, Assertions, Commands and etc. It also allows us to create Custom Assertions and Custom Commands as well. BrowserStack provides cross browser testing environment. For example, it allows us to operate any browsers which are supported by BrowserStack it our own browser. It even enable us to test our localhost using Chrome extension. Furthermore, we can operate mobile devices such as iPhone and Android.

Automate testing

BrowserStack supports Nightwatch.js on their automate testing offering. Following is a link for it.

Selenium with Nightwatch

You need only a few installations and configurations. Basically it’s easy. But there’re some pitfalls to work with browsers on BrowserStack. I’m happy to share them below.

Make it works

Use Nightwatch.js v0.9.*

Current stable version of Nightwatch.js is v0.9. But there’s pre release version v1.0. Pre release version doesn’t work at all with BrowserStack regardless of type of testing such as local or remote. So it appears to me v0.9 is only a option to work with BrowserStack.

Choose remote testing over local testing

Local testing worked on my environment, but it wasn’t stable. So remote testing which specifies public URL must be easy. That’s more stable. Also, there’s a limitation for local testing on Safari (both macOS and iOS). We can’t use localhost.

I face issues while testing localhost URLs or private servers in Safari on macOS/OS X and iOS.

Use “retries” option

Probably this depends on an application, E2E is unstable sometimes. There’re many factors which affects E2E testing like browser, network, 3rd party services and etc. So specifying retries option makes sense.

Use waitForElementPresent over waitForElementVisible

Edge requires elements are actually shown in screen on waitForElementVisible. So it make problems on a tall screen. waitForElementPresent works well even when elements are not shown. This behavior only appears on Edge.

Tips

Specifying session name

If you do nothing special, sessions are named from test directories and files like Home / Timeline. This will make many duplications if you write plural tests in single file. In that case, you should specify session name.

browser.options.desiredCapabilities.name = 'Test A';

If you want session name automatically be specified from its test name, you can do something like this.

before: function (browser) {
  this.browserName = browser.options.desiredCapabilities.browser;
  this.baseName = browser.options.desiredCapabilities.name;
  this.steps = browser.currentTest.results.steps.slice(0);
}

beforeEach: function (browser) {
  let bsTestName = self.baseName;
  let localTestName = '';
  const [ nextTestName ] = browser.currentTest.results.steps;
  if (nextTestName) {
    const idx = this.steps.indexOf(nextTestName) - 1;
    if (idx >= 0) {
      localTestName = this.steps[idx];
      bsTestName = `${self.baseName} / ${localTestName}`;
    }
  } else {
    localTestName = this.steps[ this.steps.length - 1 ];
    bsTestName = `${this.baseName} / ${localTestName}`;
  }
  this.localTestName = localTestName;

  bsTestName = sanitize(bsTestName);
  this.bsTestName = bsTestName;
  browser.options.desiredCapabilities.name = bsTestName;
}

Update session’s result (if it failed)

Most of cases, sessions resulted in passed on BrowserStack even though it actually failed. If you want to know correct results on BrowserStack, you have to update session’s status using BrowserStack API like following.

afterEach: function (browser, done) {
  const API_PREFIX = 'https://api.browserstack.com/automate';
  const auth = {
    username: process.env.BROWSERSTACK_USERNAME,
    password: process.env.BROWSERSTACK_ACCESS_KEY
  };
  const resBuild = await axios.get(`${API_PREFIX}/builds.json`, { auth });
  const [ build ] = resBuild.data;
  const buildId = build.automation_build.hashed_id;

  const resSessions = await axios.get(`${API_PREFIX}/builds/${buildId}/sessions.json`, { auth });
  let session = null;
  for (const sess of resSessions.data) {
    if (sess.automation_session.name === this.testName &&
      sess.automation_session.browser === this.browserName) {
      session = sess;
      break;
    }
  }
  const sessionId = session.automation_session.hashed_id;
  const data = {
    status: 'failed',
    reason: ''
  };
  await axios.put(`${API_PREFIX}/sessions/${sessionId}.json`, data, { auth });
  done();
}

Aurora Serverless

Since Aurora Serverless became available from early this month, I started to use it for this blog’s database.

Impression

Though it took several seconds to connect to an Aurora instance for the first time after its launch, it’s quite natural since then. It responds quickly. It appears to me this is already practicable enough as long as I use it simply.

There’re many limitations such as only a few supported cluster parameters, no support for cross region replicas, etc. So it may be still early to use it for complicated scenarios.

Price

As for storage and I/O, it’s same to provisioned Aurora. But computing capacity which is called “Aurora Capacity Unit (ACU)” is kind of confusing. AWS website states “Pay only for the database resources you consume, on a per-second basis”, but it also states “$0.06 per ACU Hour” in a pricing page. I was not sure it will be billed per second or hour.

FAQ page states in detail.

You pay a flat rate per second of ACU usage, with a minimum of 5 minutes of usage each time the database is activated.

That says AWS bills by accumulated seconds per month, but there’s an hidden factor. When will database be deactivated after last request? This minimum of 5 minutes of usage won’t affect much if a database has constant requests, but I’m still interested in this internal behavior.