Little Secrets of Testing a Large LMS
Today, we want to tell you about one of our most interesting projects. We have been working productively with this customer for over 10 years now. During this time, we found a way to enact optimal process organisation, decided to use monthly releases and successfully adapted our process to the quarantine conditions this year.
This particular project is a learning management system, which is used by more than seven million users from different countries. The system contains 1000+ web pages and about 10,000 test cases.
Fifteen development teams are working on the project right now. One part is working from the customer side in Norway, and another part is working from the Arcadia side in Saint Peterburg.
The meaning of optimal process in our project
Our main purpose is to meet the end users’ needs. This includes the implementation of new functionality and the maintenance of the current system state. We pay additional attention to the performance of the system and its stability in the case of a high load.
First of all, the development process should allow for the possibility of reaching business objectives such as timely release with good quality. In addition, the process should be comfortable for development teams so that all members involved in the process are satisfied. We established optimal processes in our teams, so let’s talk about the tools we used to accomplish this.
Development process in general:
a) The implementation approach meets the team’s needs.
We use the scrum approach and three-week sprints. Sprint goals are presented, and a set of requirements is collected during the presentation session before each sprint. After the presentation, we have a planning session where we estimate all tasks and define a set that will be included in the next sprint. After each sprint, we have a review session where we demonstrate all that has been done in sprint tasks and announce the sprint goals that have been reached. Such an approach is optimal for us because during sprints, we have time to implement enough new functionality and also fix and test a certain count of support bugs (10% of sprint time is used for fixing and testing such bugs).
Basic team members include the team lead, developers, and QAs. The ratio of developers to QAs in our teams is 3:1. Such a ratio allows us to reach sprint goals with the necessary capacity, so there are no periods of inactivity for some team members: while developers are implementing functionality, QAs are creating/updating test cases; when implementation is finished, QAs test the changes, and developers move on to the next tasks or help the QAs with testing (in the case of huge changes requiring wide testing).
The product owner defines the goals and requirements at the beginning of each sprint and accepts them at the end of the sprint. In addition, each team has a scrum master who helps to solve different team problems.
Time spent on sprint tasks is equal to 80% of teamwork time. Time is divided in this way in consideration of different risks and to allow the team to engage in different activities that are not directly related to sprint tasks. Thus, two hours per day is reserved for communication, meetings, seminars and bugs found in sprints.
b) Accurate requirements and good planning are essential.
We try to take to sprint only those changes with clear and accurate enough requirements so that planning will not take long, and the sprint will not fail due to unknown details or additional difficult changes. If the team isn't familiar with the project area related to changes, or the product owner cannot answer a lot of questions during planning, the team can take on the task of becoming familiar with the current area or investigating to implement the prototype of functionality as a result.
Retrospectives and error analysis for sprints and releases are important. Sometimes we do not have enough time to finish some tasks before the deadline, and we may have to work overtime to finish the work until a specific date. For example, it may be necessary to finish release testing when we are behind schedule, and the release may need to be postponed. However, overtime is an additional expense for the company. Therefore, we administer retrospectives for sprints or releases to discuss what was done well or wrong and to create action points to avoid such issues for future sprints/releases.
d) Support is provided by management.
From time to time, we face problems that cannot be solved on a team level. In such cases, it is necessary to change the process or involve company management in solving the problem. It is also important to understand that everyone can find help and support. Our entire company, from the Arcadia side to the customer side, is at its best in such cases; all questions are answered, and problems are resolved in ways that satisfy all sides.
e) The foundation of comfortable work is good communication, kindness, and the willingness to compromise.
This point relates to everyone, including each team member, product owners, scrum masters, company management, and other participants in the process. Everyone is ready for questions and discussions; developers consult with QAs on requirements, and together they decide on how to enact specific changes (from implementation through to testing). The product owner is always in touch with the team, answers questions as soon as possible, and tries to compromise in reaching sprint goals. The scrum master is always ready to help by finding resources (QAs/developers for a sprint or release) or advising on how a sprint can be organised within the available timeline.
a) QA guidelines for writing test cases.
Test cases are the main detailed documentation of system functionality in our project, so we pay a lot of attention to them. Each team’s change requires the creation of new test cases or the updating of existing ones.
Previously, when the system was not as huge as it is now, test cases were sufficiently detailed with different specific steps. Now, this is not appropriate, since updating such test cases requires a lot of time. As a result, QA leads changed the approach to writing test cases: test cases are now written as scenarios (each step is a separate case) instead of as detailed specific steps.
b) QA guidelines for testing in sprints.
QA guidelines for sprint testing were developed so that the team could provide good quality changes. The foundation of such guidelines is maximum testing coverage, which includes:
- testing of team changes (functionality and UI if necessary)
- testing of the system after making the changes using a checklist, which consists of mandatory checks including accessibility testing, testing of translations, testing of existing data, testing using special characters, security testing and testing of changes on the mobile app.
c) QA guidelines for release testing.
The release process and QA guidelines used in the release are described separately below.
d) Using automated testing for regression.
Creating automated tests for regression testing made the team’s work easier. Now, if it is necessary to check some area after the team’s changes have been made, it is possible to run regression auto tests for an appropriate area (if such autotests have been created). If some part of the system is broken by the team’s changes, it will be found by failed automation tests.
e) Assistance from QAs and developers.
We do not have a lot of QAs (about one QA for every three developers). Moreover, each QA spends time on release testing and other tasks not related to sprint tasks from time to time. Thus, some QAs may lack time. In such cases, another QA with a lower workload could help with testing. QA leads or scrum masters can help to find such a QA.
In addition, developers can help with the testing of changes in sprints if test cases are ready for these changes.
f) Communication between QAs.
Microsoft Teams is used for communication. In Teams specific QA chat, it is possible to ask questions and get answers quickly, and all QAs can learn actual information about discussed theme from this QA chat. We also have monthly QA meetings where QAs share their teams’ current tasks and discuss different issues, such as the testing approach, location of test cases related to unknown areas for some QAs, questions about a release such as the establishment of a release team and the schedule of release testing, or new mandatory checks that have been added to the common QA checklist after some customer bugs.
The above-mentioned approaches allow us to provide a good quality system in a no-stress working process.
Our path to monthly releases: organisation of testing process during release
Previously, we had quarterly releases. Using this approach, many changes made by teams were collected for each release. Such a huge release package required regression testing before each release, so we had a separate period (usually about two weeks) for regression testing, and all teams participated in this testing, including both QAs and developers.
Later, automation tests were used in the release testing process. Not all tests for regression were covered by the autotests; some things were still tested manually. In general, automation testing reduced testing time, but due to the huge size of the system, full regression testing still took a lot of time.
The implementation cycle looked like this:
It was obvious that this approach to releases was expensive, not optimal, and that it could not allow for end users’ needs to be quickly and flexibly met. The release process approach required changes, and it was decided that releases needed to happen more often—in this case, once per month.
When we moved to monthly releases, it became clear that we did not have enough time for regression testing before a release, even when using automated regression tests. Now, the full period of preparation for a release takes two weeks. Therefore, regression testing is only done in a sprint if necessary (if developers consider that regression testing of a certain area of a system is required after changes of this area have been made).
However, for each release, we still need to test not only the new functionality but also the common condition of the system. Therefore, we use stabilisation testing instead of regression testing during release testing.
a) testing of new functionalities added to the current release by each team;
b) testing of critical areas, including a basic functionality check of the main areas of a system, which obviously requires less time than a full regression cycle;
c) testing of bugs found in the current release package.
Now, the entire implementation cycle looks like this:
Let’s talk about what else is necessary to prepare for a release.
When stabilisation is obtained and the quality of the release package is good, configuration testing is done in a pre-production environment. Operations are used to practice and update the environment, and QAs test the configuration with the current release package before real release on production.
It may seem that two weeks of preparation for release is a lot, and that not much time is available for sprint testing. In general, for each team, QA preparation for a release takes 4–6 days depending on the difficulty and the number of functions that the team is going to release :
A common timeline of release testing looks like this:
|Monday–Friday, week 1||Monday/Tuesday, week 2||Saturday, week 2|
|Stabilisation testing||Configuration testing on a pre-production||Configuration testing on a production|
We partially automated the test cases for critical areas and the configuration testing because this part of the process is permanent and repeated for each release. Automation significantly reduced the time of testing during preparation for release, so we plan to extend the use of autotests in the future.
A QA coordinator (one of the QA leads) is chosen for each release for organisation and control purposes. More time is needed by the QA coordinator to prepare for the release than the teams’ QAs because of the additional responsibilities that include:
- definition of the release team and checking the availability of all release team participants during the release;
- preparing test plans for the release;
- running autotests during stabilisation and configuration testing;
- coordination of QAs’ work during release testing;
- helping QAs to resolve any questions and problems related to the release;
- control of testing execution and the reallocation of resources if some team QAs are overloaded with team change testing (other teams’ QAs, developers of teams or the QA coordinator can help with this).
Of course, no release is without its problems. Usually, we solve these problems and develop plans to avoid them in the future. Some problems that we have faced recently are as follows:
Problem 1: Sometimes teams merged their changes to release packages in the last minutes before stabilisation, so stabilisation was initiated later than scheduled. Moreover, such last-minute changes usually contained bugs (due to the lack of deep testing in sprint).
Solution to problem 1: We try to avoid adding such risky changes to the release package.
Problem 2: Bugs have been found during configuration testing in the pre-production environment. This requires fixing and recreating the release package, and so there is a delay in finishing testing.
Solution to problem 2: After resolving such bugs, we add additional cases (from the bugs) to critical area test cases so we can find them upon stabilisation next time.
Problem 3: Increasing the time of release testing can be a reason for postponing a release.
The following factors can lead to increased testing time:
a) moving release day for some reasons (e.g., operations and/or the pre-production environment are busy with work, updates, etc.) which leads to an increased number of team changes in the release
b) unstable state of environments for stabilisation and/or configuration testing
c) configuration bugs in environments for stabilisation/configuration that block testing of the release
d) a large count of bugs found during stabilisation (usually integration bugs from changes enacted by different teams in one area).
Solution to problem 3: In this case, we try to test all release phases without postponing the release day by using all available resources for testing or working overtime; this is because release is a high-priority task.
There can be other problems, but we can solve all of them by releasing teams or using experts in the problem area. This is possible due to the well-organised, united and effective work in our project.
Work during quarantine: organisation of testing
For our project, it is mandatory to work in the office. This allows one to easily find other members of the project during work hours (colleagues from another office can ping a person who does not respond in a chat). This is important because some of the teams are located in Norway, and the others are in Saint Petersburg.
During the pandemic, when Norway and Russia enacted self-isolation measures, we had to start working from home. Nevertheless, we have continued working in our regular mode: teams finished sprints with good sprint acceptance as usual, and releases were made according to the planned schedule.
Communication was maintained at a good level. Microsoft Teams covered all needs: active discussions were held in chats, and meetings happened without problems. All emergent questions were discussed by calling other project members.
From the testing organisation perspective, there were also no problems: all QAs responded in chats and participated actively in release testing during work hours. We prepared a list of phone numbers in case we could not reach some QAs, but we did not need to use it.
The only problem that we faced while working from home was that we could not test our system in a team environment using phones/tablets due to VPN specifications. However, we solved this problem by using a proxy server on our home network connections. This allowed us to conduct tests on mobile devices from home.
Now, we are partly back to working in the office, but part of the company continues working from home. Even while located in different parts of the city, we remain united in solidarity as a company committed to our work during such a difficult period of time, with issues arising such as children and pets requiring attention, internet connections being slow or disappearing or lights being turned off from time to time. Only one question remains in mind through all of this: “How could I work effectively today?”.
We will be able to endure these difficult times, and we will finally return to normal office life, where we will smile more widely at each other than before.
In summary of the above:
- In a huge long-term project, it is possible to organise an optimal process that can meet end users’ and all development process members’ needs. Our project is a good example of this.
- It is obvious that the optimal process will have several iterations until the necessary tools and approaches are found.
- Based on our project experience, the basis of comfortable work is good communication regardless of the chosen tools and approach. In such circumstances, the project can be successful, and each member of the development process can engage in their own tasks with pleasure and do their best to ensure the success of the project.