The waterfall model originated in manufacturing and construction where changes are costly and investment in design of the production line is often much less than potential loss if the actual production fails. It is based on idea that planning and design costs are much lower than those used in the actual production.
Software development life-cycle (SDLC)
The software waterfall model often uses some variation of following phases:
- Requirements specification (Requirements Analysis) resulting in Requirements Document
- Software design resulting in Software Design (SD) document
- Implementation resulting in the actual software
- Testing (or Validation)
- Deployment (or Installation)
This model expects that each phase is finished before the next one starts. Between each of those phases there are often "gates" that consist of reviews of produced material. In order to pass those "gates", review of produced material needs to confirm that the previous phase is indeed complete. It usually ends with some form of a sign off. For example, requirements specification can be reviewed by customers. They are expected to sign off the document thus confirming that the result of the requirements phase (for example requirements specification document) is satisfactory and the team can proceed to the design.
Waterfall model discourages revisiting and revising any completed phase. For that reason there are "gates" that are acting as barriers towards the next phase. Each needs to be "perfect" since there is no going back.
McConnell shows that a bug found in the early stages (such as requirements specification or design) is cheaper in money, effort, and time to fix than the same bug found later on in the process. Waterfall responds to this claiming that time spent early on making sure requirements and design are correct saves much time and effort later. Thus, the thinking of those who follow the waterfall process goes, make sure each phase is 100% complete and absolutely correct before proceeding to the next phase. Program requirements should be set in stone before design begins (otherwise work put into a design based on incorrect requirements is wasted). Same can be said for the rest of phases. Implementation should not start until the design is 100% complete. Integration needs implementation, testing needs integration... Each phase should be "perfect" before people begin to implement the next one.
Waterfall model was a good start in the software industry at the beginning. There was no other model to compare with. In the mean time, problems started to become evident.
In almost all but non-trivial projects it is impossible to finish a phase perfectly. Customers often cannot know all the details of the end product up front, design is prone to fail to be 100% accurate without doing part of the implementation, testing will find defects that might require the team to change the design.
"Many of the details only become known to us as we progress in the implementation. Some of the things that we learn invalidate our design and we must backtrack." - David Parnas
One of the side effects of the waterfall model is creation of "silos". Bigger companies, while adopting the model and growing bigger tend to create departments for each of the phases. As a result, requirements specification phase is done by business analysts, design by architects, implementation by developers, etc. Each of those departments tends to work in partial (if not full) isolation from others. End result of one phase is often "thrown over the wall" to the team in charge of the next phase. That same department often moves on to the next project and has limited availability to provide support to the team working on the next phase.
The "silos" effect often prevents the team to establish necessary cohesion required for employment of techniques like Continuous Integration (CI) and Behaviour Driven Development (BDD).
Common result of having phases and "silos" is increase in management and documentation effort. It is not uncommon that medium to big projects spend one third or even half of the budget in documentation and management. Since team communication is reduced as the result of phases and silos, it needs to be compensated with the increase in documentation.
Each phase with its own team suffers from different problems that are getting accumulated with time. Those listed below are not applicable to all companies practising waterfall model. However, due to the nature of the model, its fostering of silos and types of organizations that still practice this type of delivery, they are more common than not.
One of arguments for the usage of the waterfall model is that it has more accurate estimations and budget and that pleases the customers. However, since it is based on the idea that it will start with the "perfect" plan this often tends not to be true due to increases in costs produced by inevitable changes that are requested later. Even when estimations are accurate that is done at the expense of big increase in costs that extensive documentation and planning requires. Due to pressures to do the project within the budget, estimations tend to be inflated so that everyone is safe from possible repercussions.
Requirements Specification (Requirements Analysis)
Requirements specification is often done by business analysts. Their primary role is to establish communication with the client and produce a document that is signed off and will be used as a "bible" of what should be designed by architects and coded by developers. BAs often lack skills to make actual decision about the final solution and act as proxies between client on one side and the rest of the team on the other. Due to the nature of phases, that communication is limited and tends to be based mostly on revision of already produced documents.
Another common side effect of a specialization to work in one specific phase is conversion of best developers and testers into "MS Office users". Senior developer would be promoted to, for example, architect. Since implementation is the job of developers, architects dedicate a lot (if not all) of their time writing the design document that is passed to developers. Ex "star developer" gets converted into a person that is more and more disconnected from the code resulting in someone who knows how things were done before but is not up to date with current tendencies nor actual implementation of previous projects.
Developers are supposed to "blindly" implement the Requirements Document as it is, often without having the direct contact with the customers. They often don't know their motivations and reasons that resulted in a request for that product. Common phrase when it is discovered that the solution is not what the customer wants is "I did it as it is written in the Requirements Document". This problem is increased even more with further splitting development team into the above mentioned product or component teams. Not only that in that case developers might not have enough information, but they sometimes receive or are interested only in requirements for their product or component.
Developers tend to rely on the next phases to validate their work and do not have incentives to switch towards Test Driven Development (TDD).
If development team is fractured into product and component teams, integration of these products and components is often left for the phase after the implementation, resulting in the "integration hell". This is the phase when everyone crosses fingers that components and products done individually by separate teams are working well together. Continuous Integration is made to solve this problem and make sure that everything is integrated most of the time. However, teams working with waterfall model tend to not use the CI due to separations of teams and tendency to do their part. In an extreme (but not uncommon) case integration is done by a separate team that puts it all together, does smoke testing and passes problems back to developers.
Testing (or Validation)
Since testing is a separate phase that is performed after the integration, work is mostly based on quality checking (QC). It is too late to assure that the implementation is done with quality built-in. The effect is often that testers are acting as "police" trying to prevent that the implementation passes their "gate" and is deployed unless all checking passed successfully.
This phase suffers from the similar problem as development. Since there is often no direct contact with the customer and all the work in this phase is based on the Requirements document, validation can check whether the software conforms to the document, not whether it fulfills customer expectations.
Since testing is left for late stages of the project, actual discovery of bugs is postponed. McConnell's claim that a bug found in the early stages is cheaper in money, effort, and time to fix than the same bug found later on in the process tends to work against the waterfall model even though that is the main argument used by those practicing it.
Test automation is often limited or non-existent since testers often don't have enough coding skills to build the automated solution. On the other hand, developers are working on implementation and are concerned mostly to do their phase and fulfill their schedules. Due to the lack of cooperation between teams, testers are left on their own and often forced to do most of the work manually. As a result, testing phase can last as long as the implementation if not longer.
In extreme but very common cases, maintenance is handled by a dedicated team. Those that did the implementation and testing phase often move to the next project leaving the maintenance team to fix bugs that will occur in production. In such a situation knowledge of what was done during the project is based on documents instead actual experience and the team needs more time to debug and solve problems.
Change requests are probably the most painful part of the process. Since the model assumes that each phase is complete before the next one starts and that the project has the "perfect" plan, any change to the scope tends to break the whole process, moves the team to the beginning and costs a lot of effort. For all but the most trivial projects, there is no perfect plan nor there is the perfect scope and design defined in advance. Change will happen and when it does it must be rejected or it will produce a major setback.
Waterfall model can be useful for static projects where there will not be many changes. For most others it is outdated, too inflexible and too expensive.
Often overlooked aspect of the waterfall model is its indirect effect on the company structure and the organization of teams that tends to create silos of highly specialized people and the expense of decreased collaboration. There is increase in management costs and creation of rigid environment with little space left for improvement.