Geethu K Nambiar

How Agile Methodologies Challenge Testing


Every software development lifecycle model, from sequential to spiral to incremental to Agile, has testing implications. Some of these implications ease the testing process. We don’t need to worry about these implications here. Some of these testing implications challenge testing. Here we discuss only the challenges created by the Scrum methodology, and distinguish those from other types of testing issues. ​​

1) Dealing with the Volume and Speed of Change

One of the principles of Agile development is that project teams should “welcome changing requirements, even late in development”. Many testing strategies, especially analytical requirements-based testing, become quite inefficient in such situations. However, risk-based testing accommodates change, since we can always add risks, remove risks, change risks, and adjust the level of risk. If test execution is underway, we can adjust our plan for the remaining period based on this new view of quality risk. Smart automated testing also accommodates on-going change. With care, automation at the graphical user interface can be maintainable. Automation at stable command-line interfaces does not tend to suffer severe maintainability problems. Many of these challenges arise from change in the definition of the product and its correct behavior . When the test team is not kept informed of these changes, or when the rate of change is very high, this can impose inefficiencies on the development, execution, and maintenance of tests. ​​

2) Remaining Effective during Very Short Iterations

In sequential lifecycles, test teams can have a long period of time in which to develop and maintain their, in parallel with the development of the system, prior to the start of system test execution. Some more formal iterative lifecycle models, such as Rapid Application Development and the Rational Unified Process, often allow substantial periods of time between test execution periods for each iteration. These intervals allow teams develop and maintain their test systems. Agile methodologies like Scrum are less formal and faster moving. Consistent with the evocative name, sprints, these methodologies use short, fast-paced iterations. This will further squeeze the test team’s ability to develop and maintain test systems, compounding the effects of change noted earlier. Testing strategies that include an automation element have proven particularly sensitive to this challenge. The risk-based element of the recommended strategy can help. Risk-based testing focuses on the important areas of test coverage, and de-emphasizes or even cuts less important areas, relieving some of the pressure created by the short iterations. This ability to focus proves especially helpful for test teams also under tight resources constraints. Test teams in an Agile world should develop, maintain, and execute tests in risk priority order.

3) Receiving Code after Inconsistent and Often Inadequate Unit Testing

Agile methodologies stress good, automated unit testing. Open-source test harness such as J-Unit and Cpp-Unit minimize the tool costs of doing so, overcoming one keyobstacle to automation. Such automated unittests allow what Agile proponents call refactoring. Refactoring is the redesign of major chunks of code or even entire objects. The automated unit tests provide for quick regression testing of refactored code. Some Agilists recommend substituting automated unit tests for design, as in Test Driven Development. While this sounds good in theory, there are two problems that we tend to observe with this approach in practice. First, unit testing has limited bug-finding utility So, while unit testing helps, the main filter to prevent excessive field failures remains system testing Second, under the guise of excuses both valid and not-so-valid, many programmers do not create automated unit tests and in some cases don’t do any unit testing at all. This creates a great deal of trouble for the test team, which, as mentioned above, remains a critical bug-removing filter. The short test execution periods on Agile sprints, compared to sequential projects, means that the degree of damage caused by one or two days’ of test progress blockage due to highly buggy code is higher than in a sequential project. Delivery of unstable, buggy code will undermine one of the key benefits of the risk-based testing portion of the strategy, which is the discovery of the most important defects early in the test execution period. It also inevitably leads to a large degree of code churn during testing, since so much must change to fix the bugs. amount of change can ultimately outstrip even the ability of the best automated regression test system to keep up, which would then lead to lower defect detection effectiveness for the test team.

4) Managing the Increased Regression Risk

In iterative lifecycles like Scrum, though, code that worked in previous sprints gets churned by new features in each subsequent sprint. This increases the risk of regression. Agile methodology advocates emphasize good automated unit testing in part to manage the regression risk inherent in such churn. However, good unit testing has limited defect removal effectiveness, as cited earlier. So, automated regression testing via unit tests will likely miss most of the regression bugs. Therefore, we need effective regression testing at the system test level (which has a higher level of defect detection effectiveness). By combining risk-based testing with the automated regression testing, test teams can effectively manage the increased regression risk.

5) From Detailed Documentation to Many Meetings

Agile advocates promote “faceto-face conversations.” This, of course, is another name for a meeting. From a marketing perspective, it was smart of the Agile advocates not to use the word meeting in the Agile Manifesto, but the reality remains. The flip side of this problem, in some organizations, is that everyone gets invited to ever meeting, the meetings balloon in number and duration, managers and leads are not available to manage and lead their team because they are in meetings much of the day, and effectiveness and efficiency drop. Teams using Agile methodologies must achieve the right balance between documentation and meetings. Teams using Agile methodologies must have crisp, efficient meetings that move the project forward and course-correct, not meetings that grind the team down and go around in circles.

6) Holding to Arbitrary Sprint Durations

In some cases, in spite of reducing the scope of testing, the test team still can’t execute all the tests in the available time at the end of a sprint. If so, rather than ruining their weekend to run every test, the test team can select the most important tests using the risk priority number. Less important tests can slip into the next sprint. (You’ll notice that Scrum and other Agile methodologies allow user stories to slip from one sprint to the next in the same way.) Of course, if the test team must consistently triage its tests in this fashion, they should again adjust the test extent mapping downward for future sprints.

7) Dealing with Blind Spots in the Sprint Silos

Most of our clients adopting Agile methodologies have retained the independent test team, or at least the independent tester, to some extent. For those retaining the independent team, most have chosen to partition the team across the sprints in some way, often making each tester answerable on a day-to-day task basis to the Scrummaster or other sprint leader, rather than to the test manager. In the case of having independent testers but not independent test team, there is no test manager. This provides some advantages, especially to the person leading the sprint.

  • Each tester focuses entirely on sprint related tasks, with minimal outside distractions.
  • Each tester allocates time entirely to the benefit of the sprint and its exigent goals.
  • The sprint leader can re-direct any tester to focus on what is most important to the sprint team, often without having to consult the test manager.
  • The sprint leader can – and, at the end of a sprint, often will – call on the tester to put in extra efforts to hit sprint targets.
  • The amount of test effort allocated to the sprint does not vary once the number of testers for the sprint is determined, and the test manager cannot surprise the sprint leader with a sudden reduction in test effort to serve other test team priorities.
    As you can see, some of the advantages have zero-sum-game elements that carry within them the seeds – if not the actual fruit – of various potential problems. The challenges of this approach – some of which are obvious corollaries of the advantages– include the following:
  • The tester thinks of himself as – and conducts himself as – less of a tester and more of a specialized developer. This can include a loss of interest in growing test specific skills in favor of growing technical skills.
  • The tester has less contact with people outside the sprint team, reducing the wider, system-level perspective provided by an independent test team.
  • In the absence of coordinative guidance from the test manager, the tester starts to make mistakes related to gaps and overlaps. The tester fails to perform some test tasks that make sense, especially longer term investments that don’t necessarily benefit the current sprint. The tester redundantly performs test tasks done by other testers on other sprints, because she wasn’t aware of the other testers’ work.
  • The test manager, having lost the ability to manage the workload of their resources, finds the morale suffers and turnover increases as unsustainable workloads at the end of each sprint take their tool.
  • The ability of the test team to grow a consistent, powerful, maintainable test system – the test scripts, the test tools, the test data, and other stuff needed for effective and efficient testing in the long-term– goes down because of the exigent focus on the sprint’s immediate needs.
  • Testers display a tendency to “go native” and lose some of the objectivity that an independent test team normally provides, suffering a loss of defect detection effectiveness.

None of these challenges (or advantages, for that matter) arises from Agile methodologies per se; it just happens that Agile methodologies as typically practiced tend to accentuate them. The challenges are not insurmountable, if an independent test team exists. The test team, lead by a good test manager, can introduce centripetal forces that will bind the team together and makes its actions consistent and congruent. These forces then balance the sprint-specific centrifugal forces that tend to push the test team members into sub-optimizing for their sprint as well as isolating the test team members from broader testing considerations

One approach to managing these challenges is to have a separate test period that follows the development sprint. (To keep the terminology clean, some like to call this approach an alternation of development sprints following by test sprints.) Some of our clients have pursued this approach. It does seem to work for them, when the test team remains engaged in the development sprint. In other words, if the test team disengages from the development team during the development sprint, you are just exchanging one form of siloing for another.