What You May Have Got Wrong About Agile Software Development and How You Can Make it Right
Software technology has boosted the productivity of every industry. Thanks to the techies, all industries across the world are more productive than they have ever been. Ironically, this is not true for the software industry itself. By the way, this is a very controversial topic. My friends in the software field bristle at the usage of the word ’productivity. They associate terms like productivity and output with factory-like setups or with workstations that churn out products and services that are more tangible. Virtual products, they argue, cannot be spoken of in the same manner.
Does this suggest that productivity is not important in the software industry? It does not. The success rate of the software industry can be measured. It is less than 30%. This means that over 70% of the projects fail. As per the annual chaos report, software projects fail either in terms of budget overrun, scope creep or time overrun. This is not a great situation for an industry with the enormous potential to impact all other industries. The saviour seems to be needing to be saved first!
Numerous studies have been done to investigate why projects fail in the ways they do. Compared to physical products, development of software products has its own share of advantages as well as challenges. For example, the software industry faces the peculiar challenge of ‘visual complexity’- it is difficult to clearly verbalize and communicate various aspects of a virtual product that doesn’t have a physical form.
Over the years, many new product development philosophies have tried to offer solution to improve the success rate of software projects. Despite these efforts, the industry has not seen any significant improvement. To understand why, let’s delve deeper into the typical stages of product development.
The stages flow in this order. One of best management practices is to ensure that requirement detailing is thorough and complete before designing stage begins. Designs have to be completed and closed before development (implementation) begins.
This advice makes a lot of sense for teams creating conventional or physical products. If they move back and forth, the teams might have to undo or redo work completed. Going back to design after beginning development could mean making structural changes to physical structures. The ‘damage’ is visible. For example, in a building construction project, design changes would not be permitted after civil construction is completed. Allowing such changes would mean tampering with columns and beams that hold up the building. The amount of rework would be enormous. It would also mean loss/wastage of resources (material, time, capacity). Similarly, entertaining design changes late in the development stage can create havoc.
Therefore, the traditional product development model has always insisted on strict adherence to the sequence. Here are a few examples of rules the traditional model prescribes.
· Do not start development (C) before freezing the design (B). This helps to reduce avoidable rework at a later stage.
· Do not start work on the design (B) before freezing the requirement (A). This is to ensure clear accountability between the requirement provider (user/customer) and the product developer
In the world of software development, this model is referred to as ‘Waterfall methodology’. Despite its popularity in other industries, this approach was widely seen as unsuitable for use in developing software products. Many thought leaders in the software community began to question the applicability of waterfall methodology in the domain of software development. They put forth their arguments.
· Why not allow design/requirement changes after the development stage? Yes, it would open the door for rework. But it would be a simple matter of erasing and rewriting code. The damage is not as visible or catastrophic as it is in traditional environments. Software development is more forgiving. So why not take advantage of this aspect and follow a flexible, iterative approach?
· Why does scope have to be frozen before the project begins? Why not allow it to evolve as the development progresses? This would be easier on customers who are unable to specify clearly their requirements upfront.
Many new approaches were proposed as alternatives to Waterfall methodology. One of the most popular amongst them was Agile philosophy introduced in 2001.
There are many versions of Agile. Following are the most commonly used solution-elements.
Fluid requirements and design: Iterative development instead of sequential approach (as in waterfall). This allows for work to begin with good enough preparation and eliminates the need to freeze requirement and design upfront., Developers elicit feedback in the early stages, utilizing it to stay on track as they move forward. Welcome changing requirements, even late in Development (1*).
Self-organizing teams: The self-sufficient team works in a cohesive manner; concurrently designs, develops, tests, and delivers workable features.
Agile solutions are considered path-breaking. The success rates of projects, however, do not match up to the stellar reviews Agile receives.
If you are one of those who has already implemented Agile but aren’t reaping the expected results, and continue to suffer low productivity issues such as increasing technical debt, unexpected refactoring, growing backlog of bugs, difficulty in justifying the budget or launch dates and high stress, you might want to blame the team for bad implementation of Agile or weak compliance to Agile philosophy.
Before you look for gaps in Agile implementation, you must examine the inherent assumptions behind Agile philosophy. You must check the validity of these assumptions for your environment. That way you will understand the causes of low productivity or wastage in your system. Following guidelines will help you to do the same.
Assumption 1: Self-Organizing team
The Agile frame work comes from the premise that the best architectures, requirements, and designs emerge from self-organizing teams (2*). For a team to self-organize, it needs to be self-sufficient. It cannot self-organize without adequate skill sets of all types required, especially high-end skills like designing, architecture and requirement assessment. While generic skill resources like developers and testers are aplenty, subject matter experts (SME) are scarce. When Agile intervention begins, these experts, who are mostly groomed within the company, are further shared among many small self-organizing teams.
Even though, on the face of it, each agile team may seem adequate in terms of number of resources, they may not be self-sufficient in the true sense, lacking key skills (like designing, architecture and requirement assessment). Therefore, the label ‘self- organizing team’ is often incorrect and misleading.
Assumption 2: Fluid requirements and design
When teams adopt Agile, they do away with formal processes of handover and completion of requirement and design phase. They also dispense with traditional documentation, processes, and tools that are used for confirming proper requirement and design. The underlying assumption is requirements and design cannot be frozen; these are left to the judgement of self-organizing Agile teams (3*).
On the one hand, Agile teams are short of required skill level (assumption 1), and on the other hand, formal process checks and tools are discarded (assumption 2). Undoubtedly, this leads to deterioration of design and requirement assessment. Consequently, testing becomes the primary source of feedback on design and development gaps — continuous flow-back from testing causes frequent interruptions. Leaving out comprehensive design analysis to save time proves costlier in the long run. Teams end up spending more time (than they saved) on bugs, missed use cases, regression, and reopened cases. All the unplanned rework silently guzzles up teams’ capacity (particularly that of experienced SMEs) and, at the same time lengthens development time.
Assumption 3: Sprint cycle
One of the key differentiators proposed by Agile was iterative software development instead of sequential development (as in waterfall). Iterative development involves eliciting feedback and making corrections. Iterative development by definition is the opposite of pre-planning (4*).
If Agile is to be followed in its purest form, then teams cannot plan all the sprints in advance. If that is so, how can they plan an overall project/version delivery while they acknowledge that this overview is required for budgets and commitment? To address this, large enterprises have modified Agile to suit their needs. Here are a few key features of recent versions.
- Every sprint becomes a deadline with pre-planned and committed scope.
- Feedback on current sprint is not required to plan subsequent sprint.
Each self-organizing team is expected to independently deliver the committed scope within the sprint deadline. SMEs, however, face a severe capacity crunch close to sprint deadline as they support developers for current sprint and prepare for subsequent sprint.
Consequently, tasks of current sprint get priority over preparatory tasks of future sprint (design and requirement detailing). Thus, development kicks off with suboptimal design. At times, work packets are split to fit the sprint. Resultant rework (bugs, regressions, reopened cases, missed use cases/implicit use cases) create a snowball effect of task spillovers from one sprint to another. Known errors move to subsequent sprints. This generates a growing list of bugs and leads to massive effort-spiking during the months close to “final release”. At times, the product is released with known defects.
When deadline of sprints are imposed in an environment of resource constraints and fluid requirements, the resultant effect can be highly damaging in terms of system productivity.
In-depth understanding of this cause and effect, will help teams identify the sources of wastages in the system. Plugging the losses will boost productivity and improve the rate of success.
For more insights you may read this article : https://www.vectorconsulting.in/blog/pull-solution-for-projects/not-so-agile/
*Principles values of agile manifesto:
1* — Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage
2* — The best architectures, requirements, and designs emerge from self-organizing teams
3* — Individuals and interactions over processes and tools, Working software over comprehensive documentation.
4* — Responding to change over following a plan
