From Waterfall to EVO in a Medium Size Norwegian Software House

FIRM was established in 1996, and has 70 employees in 4 offices (Oslo, London, New York and San Francisco). FIRM delivers one software product: Confirmit . Confirmit is a web-based application which enables organizations to gather, analyze and report key business information across a broad range of commercial applications.

The Path and Experiences

Background

FIRM was established in 1996, and has 70 employees in 4 offices (Oslo, London, New York and San Francisco). FIRM delivers one software product: Confirmit . Confirmit is a web-based application which enables organizations to gather, analyze and report key business information across a broad range of commercial applications. Confirmit can be applied to any information-gathering scenario. Its three main data sources are: Customer Feedback, Market Feedback and Employee Feedback.

The FIRM R&D department consists of about 20 people, including a Quality Assurance department of 3 people where I work. We are mainly involved in product development of Confirmit, but we also do custom development for clients who fund new modules of the software.

In the very beginning, when FIRM only had a couple of clients, our development was very ad-hoc and customer driven. The software was updated on an almost daily basis based on client feedback. As our client base grew, we formalised the development process according to a waterfall model. We were unhappy with several aspects of the model: risk mitigation postponed until late stages, document-based verification postponed until late stages, attempting to stipulate unstable requirements too early, operational problems discovered too late, lengthy modification cycles and much rework. The requirements were focused on functionality, not on quality attributes.

FIRM CTO Peter Myklebust and I heard Tom Gilb speak about Evolutionary Project Management [EVO] at a software conference ( ITPro 2003). We found the ideas very interesting, and Tom and Kai Gilb offered to give a more detailed introduction to the concept. They spent one day in our offices teaching and preaching EVO. We decided to use EVO as best as we could for the next release, with a development phase of 3 months.

FIRM's Interpretation of EVO: Basis for the 3 Month Trial Period

EVO is in short: Quickly evolving towards stakeholder values & product qualities, whilst learning through early feedback.

After the one day crash course with Tom and Kai Gilb and a literature study ("Competitive Engineering" by Tom Gilb and other material on the subject), our overall understanding of EVO was this:

• Find stakeholders (End users, super-users, support, sales, IT Operations etc)

• Define the stakeholders' real needs and the related Product Qualities

• Identify past/status of product qualities and your goal (how much you want to improve)

• Identify possible solutions for meeting your goals

• Develop a step-by-step plan for delivering improvements with respect to Stakeholder Values & Product Quality goals:

  • Deliveries every week

  • Measure: are we moving towards our goals?

Requirements

With EVO, our requirements process changed. Previously we focused mostly on function requirements, and not on quality requirements. It is the quality requirements that really separate us from our competitors. There is an analogy with the spell checker in MS Word: why was this a killer application? There was no new functionality; authors of documents have been able to spell check with paper dictionaries for ages. The real difference was a superior product quality: speed of spell checking and usability. [ Surely MS-Word wasn't the first WP with automated spell checking? Ed. ]

We tried to define our requirements according to a basic standard:

• Clear & Unambiguous

• Testable

• Measurable

• No Solutions (Designs)

• Stakeholder Focus

Example:

Usability.Productivity

Scale:

Time in minutes to set up a typical specified Market Research Report (MR)

Past:

65 min, Tolerable: 35 min, Goal: 25 min (end result was 20 min)

Meter:

Candidates with knowledge of MR-specific reporting features performed a set of predefined steps to produce a standard MR Report. (The standard MR report was designed by Mark Phillips, an MR specialist at our London office)

The focus is here on the day-to-day operations of our MR users, not a list of features that they might or might not like. We know that increased efficiency, which leads to more profit, will please them.

After one week we had defined more or less all the requirements for the next version of Confirmit.

Solutions/Designs

For every quality requirement we looked for possible solutions (Design Ideas)

E.g. for Quality Requirement: Usability.Productivity we identified the following design ideas:

• DesignIdea.Recoding (See IET below)

• DesignIdea.MRTotals

• DesignIdea.Categorizations

• DesignIdea.TripleS

• ...and many more

We evaluated all these, and specified in more detail those we believed would add the most value (take us closer to the goal).

EVO

We collected the most promising solutions/design ideas and included them in an Impact Estimation Table (IET) - see Figure 1.

The IET is our tool for controlling the qualities and deliver improvements to real stakeholders, or as close as we can get to them. (E.g. ProS/Support department acting as clients)

FIRM EVO Week

We decided that one EVO step should last one week (see Table 1) because of practical reasons, even though we violate the rule of not spending more than 2% of project schedule in each step.

Table 1. An EVO Step

At the Project Management meetings on Fridays each project leader presented the results from the previous step (IET), as well as the content of next EVO step (one week). Possible new Solutions are discussed and weighed against each other.

We launched our first major release based on EVO in May 2004 and we have already received feedback from users on some of the leaps in product qualities. E.g. the time for the system to generate a complex survey has gone from 2 hours (of waiting for the system to do work) to 20 seconds!

Figure 1. Impact Estimation Table

Internal Feedback on EVO After the Trial Period

Project leaders:

1. Defining good requirements can be hard.

2. It can be hard to find meters which were practical to use, and at the same time measured real product qualities.

3. Sometimes you think it's necessary to spend more than a day on designs, but this was not right according to our understanding of EVO - the concept of backroom ^{[ 1 ]} activity was new to us.

4. Sometimes it takes more than a week to deliver something of value to the client - again, the concept of backroom activity was new to us.

Team members (developers):

• Sometimes it felt like we're rushing to the next weekly step before we had finished the current step.

• Testing was sometimes postponed in order to start next step, and some of these mistakes were not picked up in later testing.

Overall, the whole organization has embraced EVO. We all think it has great potential, and we will work hard to utilize it to the full.

^{[ 1 ]} A backroom activity is programming activity not visible for the end user. This is not essential information, we seldom use this activity. We always try to produce some value to some stakeholders every week.