Behat – part two

Introduction

In the previous part (part one), I covered getting up and running with running acceptance tests sequentially and using a Ramdisk to speed things up. In this second part, we’ll look at running tests in parallel, specifying a custom theme to test on and writing your own tests.

Disclaimers

Firefox® is a registered trademark of the Mozilla Foundation – https://www.mozilla.org/en-US/foundation/trademarks/policy/.

Ubuntu® is a registered trademark of Canonical Ltd – https://ubuntu.com/legal/intellectual-property-policy.

Moodle™ is a registered trademark of ‘Martin Dougiamas’ – https://moodle.com/trademarks.

Other names / logos can be trademarks of their respective owners. Please review their website for details.

I am independent from the organisations listed above and am in no way writing for or endorsed by them.

The information presented in this article is written according to my own understanding, there could be technical inaccuracies, so please do undertake your own research.

References

Starting position

We will start where part one left off, so an installed and configured Moodle installation with Behat all configured for single sequential runs. I’ve configured my virtual machine to have four processors (my CPU has six), so that should be plenty to consider that (and I assume) that each Behat process would run on its own processor as we will be running three runs at the same time.

Altering Behat for parallel running

As we will be running three runs at the same time, so we need to tell Behat what ports each of the Selenium processes will be on:

sudo -u www-data nano /var/www/moodle/config.php

and add:

$CFG->behat_parallel_run = array (
    array ('wd_host' => 'http://127.0.0.1:4444/wd/hub'),
    array ('wd_host' => 'http://127.0.0.1:4445/wd/hub'),
    array ('wd_host' => 'http://127.0.0.1:4446/wd/hub'),
);

just after ‘$CFG→behat_profiles’:

Moodle Behat config

Looking at the fact that the processes are specified with an IP based URL, I did try running them on a separate virtual machine, however this failed and the Chrome driver stated that for security reasons it will only run on local (to the machine) addresses.

Now we can initialise the tests, so in ‘/var/www/moodle’:

sudo -u www-data php admin/tool/behat/cli/init.php --parallel=3 –add-core-features-to-theme="dennis"
Behat initialisation

There are two new parameters here ‘parallel’ where we state the number of concurrent runs and ‘add-core-features-to-theme’ where we want to run the tests against another theme (the default is Boost), here I’ve stated ‘dennis’ which is my own child theme of Boost that I use on my Moodlebites courses.

Once completed then we can login (as before) to Moodle and navigate to ‘Site Administration > Development > Acceptance Testing’, where we should see the step definitions which will help with writing tests:

Acceptance tests in the UI

If they are not listed then you’ll get an informational error message instead and it means that the initialisation has failed.

Running tests

Just like in the first post. we are ready to run again. Last time I ran the Topics format tests, this time though I’ll run the Collapsed Topics format tests instead as I’ve created a few already and this will lead onto the writing tests section of the post (which you can find on the Moodle 3.11 version in the folder ‘/course/format/topcoll/tests/behat’). In the ‘behat’ (as per part one) folder:

printf %d\n {4444..4446} | xvfb-run xargs -n 1 -P 3 java -jar selenium-server-standalone-3.141.59.jar -port
Three Seleniums running

where the output of ‘printf’ is piped to the end of the xvfb-run command to append each port integer to Selenium’s port parameter.

Then in another terminal in the /var/www/moodle folder:

sudo -u www-data php admin/tool/behat/cli/run.php --format="pretty" --out="report_behat.txt" --tags="format_topcoll" --suite="dennis"
Parallel running

When ‘behat’ finishes you can see the results in greater detail with:

tail report_behat.txt
Behat report

Note: I have specified the format as ‘pretty’ here, and that does appear to be passed to behat by ‘run.php’, however the report is has not been made ‘pretty’ as in a single run. I’m currently not sure why. Also, at the start I did find Behat temperamental in finding the tests, i.e. reporting ‘No steps. No scenarios’, so I initialised for a single run, ran, then initialised for a parallel run then it worked, odd.

So is it faster? In this case I don’t think so as I’m only testing against one small set of tests and the purpose of this article is to explain how parallel running can be undertaken. But it did occur to me that I should at least try, the run above is without the ramdisk, but for the runs to follow they will use the ramdisk. Hardware wise, all I’ve done is to change the number of CPU’s available to the virtual server, everything else is the same, i.e. 2GB of ramdisk for the database and 4GB in total ram available. Software wise, the single runs have one Behat and Selenium processes running with the database using the ramdisk via Docker. The parallel runs have three Behat and Selenium processes running with the database using the ramdisk via Docker. The tests are the same Collapsed Topics ones with 11 scenarios over 5 features with a total of 124 steps:

Configuration Time taken
Single run one CPU 2m4.90s
Single run two CPU’s 1m49.06s, 1m21.20s and 1m18.26s
Single run four CPU’s 1m55.19s, 1m16.30s and 1m15.73s
Single run six CPU’s 1m17.88s
Parallel run six CPU’s 1m49.04s
Parallel run four CPU’s 1m51.04s
Parallel run two CPU’s 1m53.28s
Parallel run one CPU 2m20.22s

You will see on two of the tests that I did a further two runs afterwards that were much quicker as I was curious. Therefore, perhaps something is happening on the first run to speed things up a bit? I did contemplate repeating the same with the other configurations, but then thought that when running a big test run its likely to be for the first time, therefore the first reported time would be the one you’re most interested in anyway.

Writing tests

I am very new to this, so I looked at the existing ‘feature’ files and the documentation, copied one and began to figure out what to change. Looking at the ‘Acceptance Testing’ screen, we can discover what a step definition does:

I click on

Thus, I was able through trial and error to come up with (one of many), a test for the toggle:

@format @format_topcoll @javascript
Feature: Toggle
  In order to see and hide the sections
  As a student
  I need to be able to open and close toggled sections.

  Background:
    Given the following "users" exist:
      | username | firstname | lastname | email              |
      | dennis   | Dennis    | Topcoll  | dennis@topcoll.com |
    And the following "courses" exist:
      | fullname | shortname | format  | numsections |
      | CollTop  | CT        | topcoll | 3           |
    And the following "course enrolments" exist:
      | user     | course | role    |
      | dennis   | CT     | student |
    And the following config values are set as admin:
      | config                | value | plugin         |
      | defaultuserpreference | 0     | format_topcoll |
    And I log in as "dennis"
    And I am on "CollTop" course homepage

  Scenario: Open a toggle
    When I click on "Section 1 - Toggle" "text"
    Then "#toggledsection-1" "css_element" should be visible
    And "#toggledsection-2" "css_element" should not be visible

  Scenario: Close a toggle
    When I click on "Open all" "text"
    And I click on "Section 1 - Toggle" "text"
    Then "#toggledsection-1" "css_element" should not be visible
    And "#toggledsection-2" "css_element" should be visible

  Scenario: Open all toggles
    When I click on "Open all" "text"
    Then "#toggledsection-1" "css_element" should be visible
    And "#toggledsection-2" "css_element" should be visible

  Scenario: Close all toggles
    When I click on "Open all" "text"
    And I click on "Close all" "text"
    Then "#toggledsection-1" "css_element" should not be visible
    And "#toggledsection-2" "css_element" should not be visible

  Scenario: Toggle open after reloading the page
    When I click on "Section 1 - Toggle" "text"
    And I click on "CT" "link"
    Then "#toggledsection-1" "css_element" should be visible
    And "#toggledsection-2" "css_element" should not be visible

Where certain elements are for human use only and others are acted upon by the machine. In the first part there are some ‘at’ prefixes that state that the test is a course ‘format’, then the actual Frankenstyle name and lastly that ‘javascript’ is used. Then we have the ‘Feature’ description which is just for us and outlines the intent of the test in a specific syntax. Its a bit like writing a requirements specification where you use a sub-set of English in a very precise manner that is as unambiguous as possible.

Then we have the ‘Background’, this is used by Behat to setup the system to the starting point that you want for each ‘Scenario’:

A Behat step

Now we can have each scenario where we test the actions that the user can take with the user interface that we want to test. This follows a pattern of doing one or more things to produce one or more expected outcomes, such as:

  Scenario: Close a toggle
    When I click on "Open all" "text"
    And I click on "Section 1 - Toggle" "text"
    Then "#toggledsection-1" "css_element" should not be visible
    And "#toggledsection-2" "css_element" should be visible

Where two actions are undertaken to then check two outcomes. The steps can also be found in ‘Acceptance Testing’:

Should be visable step.

In a theme it is possible override existing steps (in PHP) ones if your theme is different and a test fails because the markup has changed sufficiently. This has been done in Adaptable, but not by me. I’m still learning! I also think its possible to create your own steps, but I’ve not tried yet as the existing ones work for me so far.

Conclusion

I hope both posts are informative and you’ve gained an insight into Behat.

I’ve been chatting with a peer of mine and he thinks that performance can be improved by the way the tests are written in terms of set-up to avoid the ‘log on’ step (And I log in as “dennis”) and instead be already logged in from a scenario perspective. Perhaps that’s something for a third post and when I understand more.

What do you think? Please let me know in the comments.

Gareth Barnard
Latest posts by Gareth Barnard (see all)
blank

Gareth Barnard

Gareth is a developer of numerous Moodle Themes including Essential (the most popular Moodle Theme ever), Shoelace, and other modules such as course formats.

One thought on “Behat – part two

  • 16th July 2021 at 10:55 am
    Permalink

    Unfortunately this level of technical work is beyond me.
    However, from reading through it does give me “an idea” of how and why this whole thing works, which is really useful background familiarity useful for when talking with developers 🙂

    Reply

Add a reply or comment...

%d bloggers like this: