background
Core system restructuring project for a major travel agency
Various inconveniences have occurred due to the aging of core systems, both for customers and for internal use.
■ For Clients
No longer able to meet diverse customer needs.
(e.g.) Unable to provide multilingual support for the increasing number of inbound travelers.
(e.g.) Unable to customize various options for facilities and stores.
(e.g.) Legacy UI makes it difficult for customers to use.
■ for internal use
We have become inferior to our competitors in terms of operational efficiency.
(e.g.) There is no function to automatically generate combinations of patterns when creating travel products. Therefore, travel products must be created one by one according to the patterns.
In order to solve these problems, the core system needed to be reconstructed.
This was a large restructuring project worth 1 billion yen and lasting about 2 years.
During this project, the expansion of testing hours became an issue.
issue
Inflated development cost/time due to huge amount of testing.
The scale of development for this project was large, so the amount of functional testing was enormous in proportion, and there was concern that delays would occur due to the expansion of the development period/time.
However, there is a tradeoff between test volume and quality, so it is difficult to make decision on whether to cut back or not.
If the amount of testing is reduced, there is a risk of system failure due to insufficient testing. As a result, there are many cases where a huge number of testers are deployed to complete testing.
In order to resolve this issue, we decided to develop a test plan.
[reference]
■ Development Scale
約100万SLOC (1,000KSLOC)
■ Estimate number of functional test cases
5 standard value by IPA /KSLOC(*1)×1,000KSLOC＝ 50,500
*1) https://www.ipa.go.jp/files/000069381.pdf
chart 7-5-20 ● Basic statistics on number of test cases and bugs detected per SLOC size by testing process
activity
Test plan development
The following four initiatives were implemented as means to prevent delivery delays and to ensure that test quality is not compromised.
(1) Definition of quality goals
It is important to define the goal to what extent quality is to be assured by testing.
The reason is that it is impossible to test for all patterns.
In other words, it is impossible to achieve no bugs.
Specifically, the goal is defined as the extent to which bugs are tolerated.
[Examples in this case]
We defined the targets as shown in Figure 1 with the policy of deterring bugs that cause significant damage.
(Figure 1)
(2) Quality targets are determined for each function.
It is important to weight the extent to which tests are to be covered in terms of quality goals.
The reason is that it is a step toward reducing testing while ensuring quality.
Specifically, the idea is to focus testing on functions that are directly related to quality goals and reduce testing on functions that are not.
.
[Examples in this case]
Quality targets were set for each function of the core system using a high/low quality target.
Although not all of them can be described due to confidential information, a rough excerpt is shown in Figure 2.
(Figure 2)
(3) Set test targets for each quality goal
The All-pairs Method is the key to test efficiency.
For those with low quality goals, quality goals can be achieved without having to cover so much testing.
In other words, a testing policy like the one shown in (Figure 3) can reduce the number of testing hours.
(Figure 3)
[All-pairs method]
Based on the idea that most defects are caused by interaction of at most two factors, this is a testing method that covers all combinations between two items.
http://www.pairwise.org/
About 70% or more of the bugs are caused by a combination of two factors.
https://www.atmarkit.co.jp/ait/articles/1505/29/news015.html
.
[Approximate number of test cases that can be reduced]
For example, if there are five factors (all 4 patterns for each factor), the number of cases can be reduced by 864.
# For all combinations
4x4x4x4x4 = 1,024 patterns
# For all-pair method
5C2 x 16 = 160 patterns
(4) Creation of specific test cases
Test cases are created using either the decision table or the all-pairs method according to the quality goals. There is no special knowhow for decision tables.
[Best Practices for all-pair methods]
The all-pair method is best generated automatically using a tool. The reason for this is that omissions are more likely to occur when all-pair test cases are created by hand.
[Tools that can automatically generate test cases for the all-pair method]
The two tools are used in combination (Figure 4).
Instructions on how to use are provided below (Figures 5 and 6).
(Figure 4)
(Figure 5)
(Figure 6)
consequence
Succeeded in reducing the amount of testing while ensuring quality.
The amount of functional test were decreased.
And testing was completed within the scheduled development period.
In addition, bugs could be extracted at the planned bug density.
■Good points
(1) It was good that we were able to come up with ideas for improving efficiency during the test planning phase and realize them. As a result, the concept of all-pair method was shared with stakeholders in advance, which enabled smooth implementation.
(2) The introduction of the tool for creating test cases for the all-pair method was a good point. As a result, the efficiency of test case creation was improved. In addition, test case omissions were prevented.
