WI-05 Test Execution

Work Instruction for Test Execution configuration items

1. Introduction

1.1. Purpose

This Work Instruction provides the set of tasks required to be performed as part of the Test Execution configuration item lifecycle.

1.2. Scope

This Work Instruction covers the complete Test Execution lifecycle.

1.3. Records and evidence

Records for each Test Execution will be held based on the records and retention policy. Test Executions are used to generate the following artifacts:

• Change request verification report

• Risk control matrix (risk control measures only)

• Risk management file

• Release notes

• Testing report file

• Traceability matrix file

1.4. Responsibilities

As listed in the procedure description, each task in the Test Execution item’s lifecycle will be completed by a member that is part of one of the following approval groups. When any of these approval group members can perform the task, Anyone is listed.

• Item Assignee: The assignee is typically a Quality Control Engineer and the person responsible for filling and carrying out the Test Execution. They execute the assigned Test Case and documents its results as well as file any resulting anomalies in case of failure.

• Quality Manager: The person accountable for the Test Execution. The Quality Manager verifies that all the necessary fields (specified in the Procedure description) are correctly filled out and vouches for their correctness and completeness by approving it.

2. Procedure description

2.1. Step 1: Log into Jira

Anyone

Log into your Jira organization, e.g., your-company.atlassian.net.

2.2. Step 2: Create a Test Execution

Anyone

2.2.1. Recommended procedure using the Test management page in Ketryx

1. Open the Test management page in Ketryx.

2. Select the version being tested.
3. Select the Test Cases to execute.

4. Press Create executions for selection and confirm.

2.2.2. Using the Traceability widget in Jira

1. Open a Test Case for which you would like to create a Test Execution in Jira.

2. In the Traceability widget, press New test execution.
3. By default, Ketryx will copy all the necessary fields from the Test Case.

4. (Optionally) Define an appropriate Assignee, i.e., Item Assignee.

5. Press Create.

2.2.3. Manual procedure in Jira

1. Press the Create button in the top-level navigation of Jira to open the form to create a new Jira issue.
2. Select Test Execution as the issue type.
3. Select the Test Case that the Test Execution is testing in the Test being executed field. Furthermore, be sure to copy over the Steps and Expected behavior fields from the Test Case.
4. Give the Test Execution a title. It is common practice to prepend the word “Execution: ” before repeating the title of its Test Case parent.

5. (Optionally) Define an appropriate Assignee, i.e., Item Assignee.

6. Press Create.

2.3. Step 3: Navigate to the issue page

Anyone

Using the popover shown by Jira or through other means, navigate to the Jira page of the Test Execution.

2.4. Step 4: Change status to In Progress

Item Assignee

1. If you are not yet the Assignee of the ticket, assign the ticket to yourself by pressing Assign to me.
2. Select the status button and move the ticket to In Progress by clicking Start Progress.

2.5. Step 5: Perform the Test Execution

Item Assignee

1. Carry out the steps of the Test Execution and document the observed behavior in the Test Execution.

2. Attach any files resulting from the test procedure, a screen recording of the test procedure itself, and/or a screenshot of the observed outcome as proof of behavior.

3. If the observed behavior matches the expected behavior, record the Test Execution as Pass. If not, record it as Fail.

2.6. Step 6: Filing an Anomaly

Item Assignee

1. If a Test Execution deviates from the expected behavior and is marked as Fail during a formal Validation & Verification cycle, the Assignee should report this deviation in a new Anomaly (refer to WI-06 Anomaly on how to report an anomaly).

2. Link the newly created Anomaly in the Found anomalies field.

2.7. Step 7: Set the version

Item Assignee

Set the appropriate product version in Introduced in version for which the Test Execution is carried out.

If the Test Execution was created by Ketryx from the Test management page, the version will already be set automatically.

2.8. Step 8: Change status to Resolved

Item Assignee

Select the status button and move the ticket to Resolved by clicking Ready for Review.

2.9. Step 9: Review as Owner

Item Assignee

Read through the Test Execution and ensure the following fields are correctly filled out:

• Test being executed: Is it linked to the correct Test Case?

• Introduced in version: Is it linked to the correct version?

• Steps & Expected behavior: Do they match the Test Case?

• Observed behavior: Is the answer complete?

• Test result: Is it filled out?

• Found anomalies: In case of failure, has an anomaly been filed and linked to the Test Execution?

If the review fails and the content is deemed incorrect or incomplete, reopen the ticket by clicking on Reopen. Restart the procedure from Step 4 and skip steps as deemed necessary to correct the review failure.

2.10. Step 10: Review as Quality Manager

Quality Manager

Read through the Test Execution and ensure the following fields are correctly filled out:

• Test being executed: Is it linked to the correct Test Case?

• Introduced in version: Is it linked to the correct version?

• Steps & Expected behavior: Do they match the Test Case?

• Observed behavior: Is the answer complete?

• Test result: Is it filled out?

• Found anomalies: In case of failure, has an anomaly been filed and linked to the Test Execution?

If the review fails and the content is deemed incorrect or incomplete, reopen the ticket by clicking on Reopen. Restart the procedure from Step 4 and skip steps as deemed necessary to correct the review failure.

2.11. Step 11: Transition to a controlled state

Ketryx

Only Ketryx can move a Test Execution to a controlled and effective state by transitioning its status to Closed. Ketryx moves the Test Execution to a Closed state after all approval rules have been passed, i.e., all required groups have approved the Test Execution.

Once closed, a permanent, immutable record is created in Ketryx, which is not affected by any changes in Jira (e.g., reopening a ticket or removing files). Said immutable record includes any attached files (i.e., that Jira supports uploading) at the time of closure.

3. Procedure flow diagram

4. Item schema

• Description (rich text): The description of this execution. The text is typically a copy of its Test Case at the time of creation.

• Steps (rich text): The steps needed to execute this test. The text shall be a copy of its Test Case at the time of creation.

• Expected behavior (rich text): The expected behavior of this execution. The text shall be a copy of its Test Case at the time of creation.

• Observed behavior (rich text): The observed behavior of this execution.

• Test result (dropdown): The result of this execution derived from the observed behavior.

  • Fail

  • Pass

  • Pass with exception

• Test being executed (-> Test Case): The Test Case this execution is carrying out.

• Found anomalies (-> Anomaly): One or multiple anomalies discovered in this execution, if any.

• Introduced in version (version reference): The version this item is effective in. If empty, this execution is ignored, and the result of the latest Test Execution for the executed Test Case and version is used.

5. Traceability to other configuration items

The following relations can be defined from a Test Execution to other configuration items:

• Test Execution executes Test Case

• Test Execution finds Anomaly