Let’s assume that we need to create a new Milestone in TestRail that contains two test runs — the one with Acceptance tests, and another one with Regression tests. The step in your .gitlab-ci.yml Gitlab configuration file would look like this:
This step is reading CI_COMMIT_REF_NAME environment variable which is the tag name for which the project is built. /^v\d+\.\d+\.\d+$/ regular expression in the only field allows this step to be executed only for the tags which have this particular format, e.g. v15.1.0. Thus our TestRail milestone will be named My Application v15.1.0. We are also setting two more environment variables for Acceptance and Manual Regression test run names: My Application v15.1.0 — Acceptance Testing and My Application v15.1.0 — Manual Regression Testing. The script prepare-tr-milestone.js is where the magic happens, let’s review it in detail.
This script begins with initialization of TestRail and Gitlab API clients. Valid access credentials have to be used. You can see the implementation of createNewMilestone method below, along with all the helper methods. It only creates a new TestRail milestone if it doesn’t exist yet, and skips creation of a milestone otherwise. getMilestones method also uses pagination because TestRail API limits the maximum number of results which are returned for /api/v2/get_milestones endpoint.
The next part of the script creates a test run with Regression test cases (this is a Test Case Type) and assigning this test run to the milestone that we’ve created. You can additionally filter test cases by references, but this is only needed for Acceptance test runs that will be explained in the next paragraph.
Generic methods of the TestRail API client which get the list of runs, test cases and test case types for your project. Note the use of pagination for runs and cases, the responses could contain thousands of entries. Case Types are account-specific, not project-specific, this is why the number of case types is limited and pagination is not needed.
These are the generic POST methods which create and update TestRail runs. There’s no need to delete and create a run again if it can be updated with a new set of tests.
In order to add test cases to Acceptance test run we would need to additionally filter test cases by references to avoid including test cases which had been created for older releases. This filter is already included into the createTestRun method, but it’s only used when the list of references is provided as the last parameter.
So where do we get those references from? References are Issue IDs, and we can get them from a tag in Gitlab that is created for every product release. This is where we need to use our Gitlab API client. Gitlab API has a special /repository/tags/ endpoint for tags, we just need to parse the response result, converting it into the list of Issue IDs.
Note that we use a human friendly GOT HTTP request library for Node.js for TestRail and Gitlab API clients above. We also retry failed requests when we receive 409, 429 or 500 codes. You could also use any other open-source libraries to wrap the API, but they might not be so easy to update in case of breaking API changes.
The initialization of Gitlab API client (you have to provide valid Project ID and Gitlab private token):
The initialization of TestRail API client (you have to provide valid username and API key):
The demonstrated integration of Gitlab and TestRail helps us to make our testing efforts more efficient, bringing together the best features of both tools. Feel free to use these tips to create a test run for your automated tests, but this would also require the implementation of a TestRail reporter. Depending on your test automation framework, this reporter could be implemented in many different ways.