A Look At How WooCommerce Tests using Continuous Integration

While going through some tools that would do continuous integration whenever anyone would make a commit to our GitHub repository, some of the tools I came across were Travis CI, Codeship, CircleCI, SideCI, etc. of which Travis CI had caught my attention.

Let us see the steps for setting up Travis CI. If you are already familiar with Travis CI and its setup, you can proceed to the next section of the post.

Installation Guide for Travis CI:

Follow these simple steps to set up Travis CI:

1. You need to sign up for a GitHub account if you do not have one already.
2. Sign in to Travis CI with your GitHub account, accepting the GitHub access permissions confirmation.

   

3. Once you’re signed in, and Travis CI has synchronized your repositories from GitHub, go to your profile page and enable Travis CI builds for your repositories. You can also enable Travis CI for your forked repositories.

   

   Note: Travis CI is 100% free for open source projects. To access the ones with private access, they have a paid plan that you need to subscribe to.

4. And that it is. Now whenever you push a commit to master or make a pull request, Travis CI will build a test.

Perks of Travis CI

There are many perks of using this tool:

• Setting it up takes few seconds
• Supports many languages apart from php
• Test your pull requests before merging them
• Deploy directly to your staging or your production site once the test passes
• and many more.

As they have mentioned on their website, Travis CI lets you Test and Deploy with Confidence, indeed its true. It easily syncs your GitHub projects with Travis CI and you’ll be testing your code in minutes!

After setting up Travis CI, let’s go ahead and have a look at how WooCommerce builds test automatically using Travis CI.

How WooCommerce Runs Automated Tests:

The perks with Travis CI made me very inquisitive of how other companies are using this tool. I came to know that WooCommerce uses this tool to run their unit tests. When I went through their .travis.yml file (the file responsible for building tests), I got to know the workflow of their test builds that are run for every commit pushed to their master branch.

There are a few files that need to be gone through to understand how WooCommerce builds test on every commit pushed.

As mentioned above, WooCommerce builds the tests in parts. They are performing different jobs in each section. The sections are before_script, script and after_script. Each section has a dedicated job to perform.

before_script:

So when anyone pushes a commit, Github will trigger Travis CI which will look for the .travis.yml file which reaches this part of the code after going through all the pre-requisites.

When it comes to before_script part, WooCommerce does these things:

1. It sets composer in the environment variable.
2. Runs a bash file which creates a test database called ‘woocommerce_test’ using the commands added in the woocommerce/tests/bin/install.sh file.
3. And it runs the ‘before’ condition that is declared in the travis.sh file which sets all the dependencies for testing (like checks for composer and then updates it).

 script:

This is the part where actually the unit tests are run. WooCommerce has added 2 tasks in this section that will do the following things:

1. Runs the phpunit.sh file.
   This file calls ‘phpunit -c phpunit.xml.dist’ or ‘phpunit -c phpunit.xml.dist –coverage-clover ./tmp/clover.xml’ command depending upon which condition is fulfilled.
   The condition met with ‘phpunit -c phpunit.xml.dist –coverage-clover ./tmp/clover.xml’ will create an xml file which will display the test result along with code coverage in a better view in a browser. Whereas, when ”phpunit -c phpunit.xml.dist’ is called, the test build result will be displayed in a terminal.You can find the phpunit.xml.dist file in the core plugin’s folder.
   The phpunit.xml.dist file has the ruleset for running phpunit. WooCommerce uses PHPUnit to test its code, so for running test for WooCommerce, they have created unit tests. And to run these unit tests phpunit command is used. You can see the content of the phpunit.xml.dist here:
   Apart from setting variables, this file declares a testsuite called ‘WooCommerce Test Suite’ under testsuites, which is its parent class.
   <testsuitename=“WooCommerce Test Suite“>
   Under this testsuite, it declares a class called directory with a suffix ‘.php’ which says it will execute all the files under ./tests/unit-tests directory with an extension of .php.
   <directory suffix=“.php“>./tests/unit-tests</directory>
   So when a commit is pushed, Travis CI will build a test for all the files under the unit-tests directory. Any failures in the test in this part are considered to be crucial.
   Adding to it, the phpunit.xml.dist file also filters the files of the plugin for showing the status of the code in the files that are getting executed. This is done by setting this:
   <whitelist addUncoveredFilesFromWhitelist=“true“>
   This is performed on all the files and folders in the core folder.Under the <exclude> tag, all the files with an extension of .php in the declared directories, will not be executed while running the test.
2. Runs the ‘during’ condition declared in the travis.sh file.Here are the contents of the travis.sh file:The above code checks the WordPress Coding Standards. It will check the code of all the files to make sure the coding standards are followed. If there is an error in the code, then it will point it out in the result and will notify that your test has been failed. The test failures in this part are not considered to be crucial.

after_script

This above code runs the ‘after’ condition declared in the travis.sh. Here it runs the test only when the commit is made to the master branch and on a latest stable PHP version. It will not build a test for any pull requests (defined in .travis.yml).

You can see the status of your test build by Travis CI on https://travis-ci.org/ once you are logged in.

And whenever your test fails in any build jobs, it will show that particular job in red and on opening the job, you will be able to see the exact part of the code where the failure has occurred.

Another perk that comes with this tool is, whenever a test fails, the user will get an email about the build fail.

So this is how WooCommerce’s builds tests using Travis CI tool. Now, let me shed some light on my experience and learnings while trying a hand at Travis CI for WooCommerce and a few other things I got to learn.

My Learnings:

With Travis CI:

When I came across Travis CI, I wanted to first know what it does so I went on to their website https://travis-ci.com/ where they have very well explained what it does, how it does and how to configure it. And following their guidelines I was able to setup, run the test in less than 60 mins. As a non-developer, I love how easy to use this tool is. The documentation is simple to understand and quick to work with.

While I was going through the documentation of Travis CI, they gave a sample code for testing purpose. I quickly forked it and followed the steps they asked to do, and bam I was able to run the test via Travis CI. Though it was a minuscule test, which just checked and assert that 1+1 is 2, but running it gave a big smile on my face. It was an achievement for me!

Building automated test for WooCommerce:

When I got to know WooCommerce is using this tool to run their unit tests, I forked WooCommerce‘s repository and went on to check the flow of the test in their plugin. I got to know the complete flow of the test, which I have tried to explain above. While at it I also got to know that they are building the test only when a commit is pushed to master and not when it is a pull request.

I also managed to build and fail a test for WooCommerce by making changes in the core plugin. What I did was simply changed the ‘Add to cart’ text to ‘Add to bucket’ that is set while creating a dummy product for test. In the file product-simple.php which is under the unit-test folder, it checked if the button has the text ‘Add to cart’ or not. Eventually, it failed when I build the test with the changed text.

The failed test result will be highlighted in red and there will also be a message displaying the expected and the actual output of your test. It will also show the line number of the file where the test has failed.

You can check the failed test below:

I also managed to fix the bug by easily changing the text back to ‘Add to cart’ and on re-building the test, the test passed.

While at it, there was an error in the build for PHP v7.1. I came to know about the issue, that each line in the array declaration should end with a comma, which occurred while it was checking for WordPress coding standards. So I went ahead and added the issue on WooCommerce’s GitHub repository. Just after 15 mins, the issue was fixed and the comma was added in the master branch. Another achievement :)!

A small guide that indicates your progress of the build:

For each test run, the PHPUnit command-line tool prints one character to indicate progress:

. - Printed when the test succeeds.

F – Printed when an assertion fails while running the test method.

E – Printed when an error occurs while running the test method.

R – Printed when the test has been marked as risky.

S – Printed when the test has been skipped.

I – Printed when the test is marked as being incomplete or not yet implemented.

PHPUnit distinguishes between failures and errors. A failure is a violated PHPUnit assertion such as a failing asserEquals() call. An error is an unexpected exception or a PHP error. Sometimes this distinction proves useful since errors tend to be easier to fix than failures. If you have a big list of problems, it is best to tackle the errors first and see if you have any failures left when they are all fixed.

Few more things I learned:

1. hhvm: HHVM is an open-source virtual machine designed for executing programs written in Hack and PHP.
2. Using Travis CI, I would be able to test the code on different versions of PHP in a single build.
3. While writing this post, I wanted to show the code snippets how GitHub has the code. Googled a bit and bam GitHubGist came to rescue.

Our Future Plans:

In order to make use of Travis CI along with PHPUnit, your project should be unit testable.

1. So we firstly we need to make all of our projects unit testable.
2. After which we need to write unit test code for all our plugins. It is going to take a lot of time and effort of both our developers to refactor the code and myself to write the unit tests for each plugin. I know this is going to take time but eventually, it is going to make life easier for all.
3. Create a file structure and all the necessary files like .travis.yml,phpunit.sh, phpunit.xml.dist, composer.json, phpunit.xml and many more for triggering Travis CI every time a commit is pushed to the repository.
4. Hence, when any of our developers will commit code to our master branch, the unit test should automatically run, as WooCommerce is currently running.

I hope this post would guide and encourage you to work on building your plugin/theme/project unit testable and make it Continuous Integrated using Travis CI or any other tool that you think meets your needs.

If you are already using any such tool, we would love to know your experience in our comment section.