The insurance industry is in the midst of a complete technological transformation, and fierce market competition is increasingly driving the importance of getting IT initiatives to market as fast as possible. The methodology used—whether for an externally facing digital portal, internal data analytics or full core systems replacement—can have substantial implications on both the speed-to-market as well as the overall success of the initiative.

In recent years, the traditional waterfall methodology has given way to a variety of agile models, with Scrum being the most popular. Unfortunately, while Scrum has led to many positive outcomes that have garnered a lot of press, there are an equal number of failures where Scrum has been labeled as the culprit.

What is the secret sauce that makes some agile projects succeed while others fail? Is it the nature of the project itself, poor execution or some other nuanced factor like “culture”?

To help answer these questions, insurers must identify when agile should be used and what guidelines should be followed.

Agile vs. Waterfall: Key Differences

Before addressing these questions, let’s review some of the notable differences between agile and waterfall. Each stage of a waterfall project—requirements, design, development, testing, UAT—is performed in sequence, with formal signoff at the completion of each. Once a stage is complete, changes are not allowed until a post-production maintenance release or second phase of the project.

When properly executed, waterfall offers reliable and predictable costs, timeline and scope, making it particularly attractive for organizations with a low tolerance for cost or timeline variance. The drawback is that rigid and early scope definition can result in a delivery that doesn’t meet users’ needs because of changes since the project started.

Conversely, agile instead calls for short, iterative cycles called sprints, which tackle only a few specific and narrow use cases. Each sprint has loose requirements, written as user stories, which are firmed up before the solution is designed, coded, tested and documented. With Scrum, end users can review a working system after each sprint to provide feedback that can be implemented in subsequent ones. Sprint stories normally entail only minimal planning prior to the start, allowing for frequent changes in priority and scope throughout the project.

When properly executed, agile offers quick delivery of a prototype or minimally viable product that can deliver real value, driving a steady and transparent progression toward the desired goal with regular incremental deliveries. The drawback of agile is the lack of detailed solution specifications and a project plan, making it difficult to predict the delivery timeframe and cost, as the project can extend forward while more features and capabilities are added.

When Does Agile Make Sense?

Projects that align well to agile are those where the scope is not easily defined and priorities and features are expected to change regularly based on user feedback. Some examples include a digital portal, mobile app, analytics dashboard or platform driving the processing for a new product.

In contrast, a legacy core system migration onto a new platform or the implementation of an enterprise data warehouse would likely be better suited to a waterfall approach as the objective and requirements can be clearly defined up front. Those types of deployments require careful architectural design, particularly for the numerous interfaces involved, and the need for frequent changes is unlikely.

Beyond the nature of the project, IT leaders also must assess their organizational mindset and culture. Shifting a traditional waterfall mentality to agile is more difficult than expected, as are the potential implications that can arise if mishandled. The tendency to continue building overly detailed specifications can stifle developer innovation and creativity. For companies that lack experience in agile, it’s advisable to create a formal training program and engage external consultants to provide counsel on key roles of Product Owner and Scrummaster.

It’s also important to educate all stakeholders about the tradeoffs listed above, especially regarding the scope and timeline variability. Furthermore, the need for close involvement from business subject matter experts throughout the project can’t be discounted and must be present during resource planning. For organizations that have successfully implemented Scrum on small projects and now want to tackle a large one involving multiple Scrum teams, moving to Scaled Agile Framework (SAFe) also requires additional training from experts. For best results, the recruitment of an experienced agile practice manager to lead the organization should be considered.

A Hybrid Methodology

A compromise hybrid methodology has also emerged to realize the predictability of waterfall with the flexibility of agile. This approach is sometimes labeled “iterative waterfall” and calls for waterfall’s detailed requirements analysis, architectural analysis and in-depth design for all complex areas, leading to a formal scope and solution document and signoff. Then agile takes over with iterative sprints to build out a minimally viable product and then prioritize changes and new features based on user feedback.

This approach has served us well in providing clear scope definition while offering our insurer clients the flexibility to re-prioritize and make changes during the project. Naturally, strong change control and communications processes are needed to ensure that transparency and alignment persist through the project.

The insurance industry is changing at a faster pace than ever before, and an agile structure tends to adapt to that change rather than collapse under it. But if insurers do not have the organizational framework to support the methodology, then selecting a waterfall or hybrid approach can be incredibly valuable as long as they have the right team or partner to drive the strategy.