Performitis (Part 3)

Last time we discussed how to overcome Performitis, and learned about technical approaches that basically resemble what I'd call sound software engineering practices. If the teams had followed them, Performitis would never have had a real chance to emerge.

However, slips in sound practice do happen - and not necessarily by novice developers. Seasoned developers abandon solid practices because of their convictions, prejudices, and assumptions.

Process and attitude

While an extensive process cannot compensate for sloppy engineering, there are process elements that can help to keep the technical issues from becoming quality issues that endanger a release.

• A Test-Oriented Process supports early feedback and fosters a quality driven attitude. While it is not necessarily directed towards performance or against Performitis, it helps to soothe most of the symptoms. When the tests include performance criteria, both the overall design quality can be improved and the performance goals met.
• Time Boxed Releases discourages spending time on activities with low return - such as too early, too detailed performance tuning measures that can lead to Performitis.

Both process elements reduce the overall project risk and are advocated by agile development methods [ Agile01 ]. While it is always a good idea to reduce risk, these two process elements are particularly helpful in compensating for Performitis.

When Performitis occurs, how the architect interprets his role is essential, as well as how he is perceived by the other project participants. In many teams the architect is as tangled in prejudices and assumptions as any developer. A new interpretation of the architect's responsibilities can help: if the architect is not just a technical decision maker but a coach for all developers, he needs to stand back and reflect about the project and what to coach. This Architect Also Coaches stance is much less determined to result in Performitis, both due to the reflection and to the coaching.

During this coaching or in separate workshop settings, the team can explore the Synergies Between Qualities that the project offers.

Change

The technical symptoms of Performitis can be attacked by sound practice or suppressed by extensive process. The causes of Performitis can be taught and the team led to avoid and master it.

However, sometimes sound practices do not happen, the process does not help, and all costly initiatives create more friction than effect. Let's face it, it's not the technology and not the process, it's the people. A change is needed. In this situation any organizational change is good per se - as there is no point in continuing the old way.

Projects endangered by Performitis have a size that typically results in an explicit team structure. This follows either the technical domains involved, or the feature aspects of the application domain. Projects need to cover both dimensions and fill the entire matrix that is spanned, but the organization necessarily focuses on one aspect. To Reorganize the Project Team along another dimension changes the focus of the developers and can lead to remission of Performitis.

When the team is still inclined towards Performitis, Staff Exchange is the last resort of management. Remember that Performitis thrives best in teams that close themselves against external influence. When the people cannot cure the project, the project has to cure itself.

Chance removes known paths and rites, and induces further change at a smaller scope as reaction. Bringing in experience from other cultures causes friction; the project's progress will degrade before something actually improves. Combine the therapies of change with other process and technical therapies to ensure a positive learning curve with respect to performance and intrinsic qualities.

Pattern form and approach

Both diagnoses and therapies follow their own forms, including sections that contribute to the medical metaphor. With each diagnosis, symptoms and examination are discussed and concluded by a checklist. A description of possible pathogens and the etiology closes the diagnosis. Each diagnosis comes with a brief explanation of applicable therapies. This includes possible therapy combinations and treatment schemes that combine several therapies. These are suggested starting points for a successful treatment of the actual situation. Therapies are measures, processes or other medications applicable to one or several diagnoses. Their description includes problem, forces, solution, implementation hints and an example or project report. Their initial context is kept rather broad since most can be applied for different diagnoses. In addition to the common pattern elements, therapeutic measures contain additional, optional sections of pharmaceutical information. These are introduced by symbols: the mechanisms of a therapy and how it works; the involved roles and related costs; counter indications, side and overdose effects; cross effects when combined with other therapies.

Test-Oriented Process

Applies to projects whose major implementation decisions are derived from the developers' gut feelings. No effort is made to find out whether these decisions are appropriate.

In a development team that is overly concerned about specific system properties, expensive measures are taken without evidence of necessity or possible prove of effectiveness. Important aspects of the project goals are hidden behind the self adopted blinkers.

• Experienced developers have gone through extensive learning processes,...but every new project comes with problems you have not yet experienced.
• It is unlikely that somebody learns from theoretical lessons without a match in the daily practice,...but each project has stakeholders whose practice differs from the developers' world and promote their particular foci.
• Changing the development methodology can be much harder than changing some technology,...but tackling a process issue with process actions causes the least friction.

Therefore , focus the development on testable achievements and create a test for every development step. Consider no functionality completed until it has been tested. Explicitly include things that are hard to test, like the architecture, and the system performance.

Most blinkers are based on previous experience, much like a Pavlov reflex. Typically the developers can well describe what the actual problem back then was, how it was discovered, and what they would do differently to avoid similar problems in future projects. From here, defining some kind of test is usually a small step. Messed up dependencies can be tested by evaluating generated dependency graphs; insufficient or unfavourable architectures can be tested by reviews according to self defined criteria; failed performance goals can be tested by load tests and frequent profiling against estimated thresholds. If a blinder cannot be expressed in terms of some test, it is probably just a prejudice instead of an experience.

When changing the project's way of working, make sure that the people with the greatest level of concern feel like the winners. If possible, push them into suggesting tests themselves. Never insist that the idea comes from you, and never fully expose why you were so eager to introduce tests.

	Test-Oriented Process is effective against all diagnoses that stem from process related blinkers. Its mechanism is to relate all experiences to a common currency called 'test' - similar to business decisions that require all known influences to be converted into 'money' for comparison. This offers a global view on a lot of different aspects of development, including missing ones as well as blinding ones.
	Changing the development process requires buy-in from all project stakeholders. Like all changes to the way of working, the key costs are proportional to the resistance they create. Agree on compromises that help to reduce resistance. Suggest the changes early while the team is still small, and the blinkers have not had a severe effect.
	The only real counter indication against Test-Oriented Process would be to introduce it near the end of the project, and it is not definitely sure that it will fail without drastic measures. The side effects are the reason you introduce it in the first place: attention to previously blind spots, and time and effort spend there. The overdose effect would be to test to an extent that is no longer cost effective - but you are not very likely to experience it.
	Test-Oriented Process goes well together with all other therapies that do not change the development process. If you need to introduce other methodology changes as well, like a focus on architecture, it might be appropriate to find the common ground among the changes first, and then change the process according to which risks are the most important ones in the current situation.

This appears to be the obvious thing for every software developer who has seen some kind of development methodology. Testing is the broadest intersection between waterfall-like and agile ways of thinking.

Management and quality control will appreciate your initiative. The project development progress becomes visible and easily traceable, while the releases inherently bring a minimal level of quality that makes formal testing much easier.

We started to introduce integration tests on a unit test granularity, for each completed function that was available on multiple platforms. These helped to limit the negative effect of changes to a common code base that were impossible to test on all systems employing the code with reasonable effort. Performance was an issue, so we expanded the test framework with time measurement. A function that had at least once caused a performance problem became accompanied by an additional unit test with no functional test criterion, but with an execution time criterion. The code had to satisfy the timing requirements of the most performance limited client, otherwise the test was not considered passed.

Time Boxed Releases

Applies to projects that spend their time preparing for some future event that might never come, meanwhile neglecting sound software engineering practices.

In a development team that has lost focus of the project's purpose or the key architectural solution issues, you need to reinforce the main objectives of the project.

• Former experience leads to anticipation of problems to come,...but when the problems do not come, you lose the effort spent on anticipation.
• Preparing for future possibilities requires time,...but the time is best spent on the problems you have at hand, and on increased customer value.
• Time pressure reduces the willingness to spend effort in quality work,...but adequately balanced internal qualities enable the team to proceed faster, and cope with yet unforeseen situations.

Therefore , schedule the releases of your software in frequent, small intervals. Convince your management that sticking to the release dates is of major importance, just as important as keeping the internal Visible Qualities .

When the team remains unchanged, the only variable you can negotiate is the scope, the expected functionality contained in each delivery. However, all project stakeholders will strive to have as much usable functionality with each release. This leads to a strongly perceived lack of time to care for tiny details early in the project, including early tuning measures except when well motivated as Measurement-Based Tuning .

	The mechanism of Time Boxed Releases is to focus the development team, and spend only effort in those tasks that pay off within the next weeks. Implicitly, everything that has only a vague chance to pay off will not be started unless all alternatives are worse.
	Time Boxed Releases require management decisions and affects the entire team as well as the project's other stakeholders. Its costs are comparable to other substantial process change costs, and are not caused by the mere measures themselves. Due to reduced project risk and less effort spend in vain on irrelevant topics, it probably pays off quickly.
	Counter indications to Time Boxed Releases are violations to the entry conditions. Internal qualities may be hard to achieve when under time pressure, and the temptation to ignore them is high. Do not start with Time Boxed Releases unless you have established some taboo on the internal qualities. Side effects are the desired emphasis changes of the projects. Some developers might feel uncomfortable with the increased time pressure, with their personal ways of reaction blocked, so prepare for some demotivation and help to establish a fearless environment. Overdose effects are reached when the time boxes are so small that your development environment does not allow for significant achievements, or when you fail to mitigate the side effect risks and induce stress and fear to individual team members.
	Accompany with quality oriented process measures. It is necessary to ensure that the Visible Qualities are established and taken seriously. A Test-Oriented Process helps with the necessary frequent integration, to measure progress, and to achieve internal qualities. Introduce Measurement-Based Tuning as the standard way to motivate tuning measures.

are a two-edged sword. I have been at a team that was forced to implement features, and ignore performance for a long time. When performance had degraded to a degree that the customer could no longer reasonably execute his acceptance and usability tests, the next iteration became dedicated to performance tuning alone.

Architect Also Coaches

(Also known as: Emphasize ~Ilities)

Applies to project teams that focus their work and thoughts to a few essential ideas, but ignore all other issues that might also be or become important to the project's success.

In a development team that has an informal design and architecture process without a dedicated role assignment, and that focuses on a particular issue of development, you need to address further important system qualities that are essential to adequately manage the system architecture.

• People's experience is valuable to the project,...but the experience needs to fit the current project's size and criticality; building a large system requires attention to issues that hardly matter in smaller systems.
• Education opportunities for developers are readily available in courses and classes,...but learning as you do your job is more efficient, and has the potential to teach lessons that you never forget.
• An architect's experience and view on the software world is limited, ...but the architect has the broadest view of the developers.
• Neglecting internal qualities can cause a large system to break under its own weight,...but the system can not become any better than the architect and the team know how to build it.

Therefore , the architect becomes responsible for coaching the development team about the internal qualities (the '~ilities') that are essential to crafting large software systems. This is an efficient way to succeed in his core duties - maintain the Big Picture Architecture [ Marquardt02b ], support management and development, balance the different forces on the architecture, ensure consistency, and broaden the architectural view - because it involves all developers and avoids resistance.

This sounds simple, but its implementation requires serious effort. While most project situations can live with a developer informally taking that role, Architect Also Coaches requires significant time and effort. You need to have the architect's role defined and assigned, as in Dedicated Architect .

The internal qualities the architect needs to emphasize correspond to the size and complexity of the solution under construction. Testability is a favourite one that pays off quickly. Ensuring testability is closely related to a design with a clear distribution of responsibilities and strict adherence to dependency rules. These two are also needed to allow parallel development of several tasks and potentially several teams. Maintainability is another key quality for a large project, as during initial development the first of its components are already being maintained.

	The mechanism behind Architect Also Coaches is respect and trust, and to spread knowledge. The team will respect an architect that knows the system, has a stake in it, and solves the day-to-day problems. Trust is necessary to learn and to change the own behavior. Spreading the knowledge helps convincing developers and avoiding resistance.
	Architect Also Coaches involves the architect and potentially all team members. The costs can become significant because you need to dedicate a lot of time to it. However, the costs for education and consistency will reduce the project risk and likely pay off over the project's course.
	If individual developers send signals that they would not accept coaching, this might be a counter indication. Architect Also Coaches has side effects on the work load that the team can manage. It will decrease in the short term, but eventually increase in the mid to long term.
	The acceptance of the architect coaching is likely fostered when you apply Architect Also Implements [ Coplien04 ]. Pair programming [ Beck99 ] or Joint Design [ Marquardt02 ] are good opportunities to start coaching. When some team members do not accept any coaching, consider Staff Exchange .

After the business case was established, the project had to change. The effort and schedule estimations demanded that the team of initially ten developers, located in two sites in different countries, had to grow to sixty within one year. While one department started growing the way management expected, the other team just grew to sixteen developers within thirty months. Most developers came straight from university, and the local architect and his manager took significant time to coach them. Years later, the project had missed the initial expectations. However, that smaller team was still working with a high quality and at a good pace. The other department had collapsed due to the mismatch between expectations and fulfilment, and most of the developers had been fired.

Synergies Between Qualities

Applies to projects in domains that require high system responsiveness.

In a development team that is blinded by a particular experience and tends to ignore or even deny all issues that do not fall into the selected category, you need to address architectural needs that are critical to the success of the project.

• People tend to concentrate on a single issue, neglecting everything else,...but a broad overview uncovers connections among different system qualities.
• Some external qualities are more relevant to reach than other external or internal ones,...but in most cases achieving some quality does not necessarily prevent other qualities.

Therefore , identify the relation between different qualities, and separate actual contradiction from developers' superstition. Outline which technical methods and means would foster which internal and external qualities, and which would prevent you from achieving them. Initiate those actions that support multiple quality aspects of your system and that complement each other. Teach your peers about the mutual amplification of qualities, and show that many assumed or perceived contradictions are not existent in reality.

	Synergies Between Qualities works by focusing attention at the self-sustaining quality gains, and reflecting on unspoken assumptions that are blinkers to developers and managers.
	All developers need to be involved, and it is helpful to include technical management as well to avoid contradicting your efforts by restrictive management policies. The costs come from the two phases of the therapy. Identifying measures and effects to qualities require some preparation that depends on the architect. The ongoing teaching is a mentoring effort that needs to last for some time; its effort is similar to other mentoring or coaching techniques.
	There are no counter indications to Synergies Between Qualities . Possible side effects or overdose effects are not attributable to Synergies Between Qualities directly.
	Combine Synergies Between Qualities with Visible Qualities and with Architect Also Coaches . Performance-Critical Components gives some concrete examples of frequently perceived contradictions.

The therapy patterns Performance-Critical Components and Architecture Tuning exhibit one of the most popular virtual contradictions. If you choose a clear, concise structure, you are (a) more likely to quickly locate performance relevant issues, and (b) be able to fix them with much less effort. In the end, you get a system that runs faster and shows a better internal structure, increasing testability and maintainability.

Furthermore, performance and the qualities fostered by separation of concerns can go nicely together. However, you need to separate along the lines that help increase responsiveness - which is probably not the way you would initially decompose a system. Scalability immediate relates to system performance. Testability can also increase performance and responsiveness, both directly and indirectly by enabling layers that can be mocked for testing and later replaced with an optimized implementation. Even portability does not need to impose a performance penalty depending on the techniques the team chooses.

Some years ago I worked for a business unit that was supposed to build both a domain specific framework, and the first product based on that framework. My role was product family architect, so I was in close contact to the management and to future users of the framework. Sensing tough decisions, I asked the management for their priorities. Of the choices I offered to them, they selected two things being both on top of the list: quality, and time to market. At first, I was frustrated from not getting a clear priority. After some time I learned that this priority combination strengthened the position of the architecture a lot, and was a perfect motivation to emphasize a healthy, consistent, concise and thus respected system architecture - the best thing one could do to reach both goals.

After project failure (due to non-technical reasons), the intellectual property and most of the framework team became absorbed by other projects. While not initially intended, the architecture and the team building started to pay off. More projects became more successful than expected. Currently, its results are implicitly reused in several products and form the basic of a now successful framework.

Reorganize the Project Team

Consider a project that is implemented by a team of more than a dozen developers.

A software project team that is structured into several sub-teams, the distribution of team responsibilities can only follow one possible view on the system decomposition. Each project must satisfy a number of different aspects and cover a multi dimensional decomposition.

• The organization into sub-teams enables a focused work,...but each project needs to have different foci, and priorities change over time.
• Different foci could be supported by having a multi dimensional team structure,...but reporting to different leads obtains more overhead than even most large projects can afford.
• Changes cause friction in reestablishing working teams,...but important goals need to be reflected in the organization to get significant attention.

Therefore , change your project organization occasionally during the project's course in a way that it reflects the highest project risk.

Dividing the project team into sub-teams according to functional components or layers is a very natural thing for architects to suggest, and can be highly effective in technical domains. Dividing the project team according to user visible function and workflow enables the team to deliver quickly what the user expects. All significant systems need to cover both views, but the organization cannot reflect both at the same time (Conway's Law, see [ Coplien04 ]).

A deliberate change in the organization forces all project participants to think in multiple dimensions, those of the new and the former organization. The implementation and architecture follows this change with a delay, a phase shift in time. The possibility of repeated organization reversion gets the participants used to multilateral thinking and minimizes the influence of Conway's Law.

	The main mechanism is change, resulting in reactions and further changes. Different aspects are addressed in the most effective way, by changing the organization.
	Reorganize the Project Team requires management decision and can only be suggested by subordinates. Its costs are similar to other costs caused by change and should be seen as an insurance fee.
	Reorganize the Project Team has a number of side effects including communication changes, irritation, and tighter integration. If the risks associated are higher than the chances it is counter indicated. Overdose effects occur when you reorganise too often: hidden communication due to fear, ineffectiveness due to uncertainty, and an increase in staff turnover.
	An alternative team structure would be Team per Task [ Coplien04 ] that avoids a breakup into sub-teams and forms teams for each small task. The tasks may both be technical or application bound.

Reorganize the Project Team when other therapies have failed to change the prevalent mindset. The maximum dosage is twice during the project's course. Do not apply it in the first quarter of the estimated project time.

The new team organization needs to support the area of the highest project risk. This will likely compensate for the costs and friction of the restructuring. However, be aware that initiated changes cause further changes in possibly unexpected areas, and that none of the symptoms of the actual disease is directly addressed.

The initial prototype phase ended with a team of five that did not need further substructure. When more developers joined the project team, the tasks and later the teams were split into different areas: database, GUI, and network. Further teams were established for quality and for connection to particular devices. When field tests began, the workflows slightly beyond the trivial standards failed or were unstable. To overcome this deficiency, the team focus was shifted towards making the workflow operable, and the team structure was reorganized according to the workflows. The workflow teams were composed so that each technical competence was represented.

Staff Exchange

Applies to projects stuck with old ideas that work to some extend, but lead to unsatisfying results.

In a development team that is stuck, caught within their own ideas, and blinded by their own limited experience, you need to bring in new ideas.

• People's experience is valuable to the project,...but repeated similar experiences can blind you and let you ignore new possibilities.
• Changing the staff of your project is risky both socially and by means of the technical and organizational learning curve,...but new people bring new ideas and different blinkers.

Therefore , suggest exchanging some members of the development team for developers new to the domain or the company. Make sure that management replaces at least one of the key team members, and looks for replacements that are personally able to become key players within a short time. Look for developers that bring experience, a strong personality and good communication skills, so that the project really profits from their knowledge.

Ensure that you have a stake in the job interviews. Be clear about your goals and the difficult situation during the job interview, to avoid later disappointment that could counter your intentions. Management could make the first raise depending on the influence the newcomer gains during his first months.

	The mechanism of Staff Exchange is to influence the development team by a factor they cannot ignore: new colleagues. These bring new knowledge and different experience and inject this into the developers'minds while the entire team is going through all team building phases.
	Staff Exchange is beyond the scope of an architect and can only be suggested to management. There is no one-fits-all answer on the related costs, but the required learning curve and the intended friction make it expensive.
	A strong counter indication to Staff Exchange is when at least some of the key developers are willing to learn and accept offered opportunities. A side effect is that you run into more discussions than you really desire. Besides all irrationalities of the newly started team forming process, the arising discussions will cover development practice and methods, coding and quality standards, architectural ideas, just to mention a few. An overdose can cause the team to get lost in discussions, break its motivation, and eventually loss of the project and valuable employees. Another overdose effect could be that the company loses the expertise it once had, without being able to adequately replace it.
	Staff Exchange is kind of an entropic therapy causing undirected activities. You need to complement it by problem and goal oriented therapies to focus the direction of its effect.

Before you consider exchanging the entire team, think of exchanging just the architect. An architect in the wrong place can do more harm than good. For indications of such a step, see the diagnoses in [ Marquardt03 ].

Another important variant is to expel the consultants. Having consultants in the role of an architect is particularly dangerous as significant competency, the reasoning behind decisions, and key knowledge will be gone at a time you cannot predict. Consultants that follow their own agenda are also an obstacle.

A division of a consulting company specialized in technical projects was particularly good at taking the entire technical leadership of their customers projects. While about half the staff in the development team was new to that kind of project, a large number of experienced developers were distributed over the teams. They also came to join particular projects as senior consultants when problems occurred. The internal turn-over made sure that the knowledge was spread, and the new developers became familiar with different projects quickly while maintaining a common understanding and corporate identity with the company.

A contractor had managed to become the mind monopole at one of my customers. He motivated his queer data model with reasons about local performance gains. When the system went productive, it was a factor of 100 slower than comparable systems, which would have caused annual operation costs of several 100,000 €. Confronted with radical ideas to save a factor of 1000, the contractor reacted with denial, without being able to give reasons. Due to cost saving measures, the contractor finally had to leave the project, and system responsibility was passed to an internal team. For political reasons, only the simplest of the suggested changes became implemented, and these confirmed the initial performance gain estimation.

Suggested treatment schemes

Now that the individual therapies are known in detail, it is time to think about which therapy to apply when.

While some therapies are known to be risky, there is good news about therapy combinations: no combination of the suggested therapies can be harmful to the project. They all have mutually increasing effect. However, too many therapeutic changes at the same time might break more than they heal. Take your time to introduce one after another, and anticipate which one will have the most effect in the current situation.

Figure 1 is one treatment scheme that might be useful for you if the project has been on its way for a significant time. In this case, you need to focus on therapies that can be used any time during the project. Which of these you can trigger depends on your role and your ability to convince other project participants.

Figure 1

First, establish Measurement-Based Tuning - this one imposes small costs, does not require major project changes, and is the basis for further tuning measures. In case it is not accepted, you may need to wait until the problems of the project are more severe. Then you can offer a Dedicated Architect role as an approach with a clear responsibility to ensure adequate system performance.

With the acquired data, start Architecture Tuning on the basis of the observations and their interpretation. With this initiative you can prove that performance is taken seriously and covers most concerns.

The following steps depend strongly on whether you succeed, or at least perceive a growing acceptance. If you fail to get that acceptance, you should consider addressing a missing Dedicated Architect very soon. In case some key developers remain ignorant, consult with management and suggest process changes that help to reduce the overall project risk, like Time Boxed Releases . Process changes possibly happen at the expense of the developers' motivation. So alternatively you could try to influence QA to demand a Test-Oriented Process . You need to consider that such a therapeutic schema against the team's adopted habits might lead to an unintended Staff Exchange , and possible drawbacks will be attributed to some scapegoat.

With growing acceptance, you can in parallel introduce Visible Qualities to start a team learning process. Over time, those developers eager to learn will notice the Synergies Between Qualities and will have learned for their professional careers. Depending on the reactions, be relaxed on the strict timing related to Visible Qualities , it might be OK to check them with every release only.

A very short treatment scheme (Figure 2) can be recommended whenever the team composition changes. Take your time to coach new members for a while and Emphasize the ~Ilities of system architecture.

Figure 2

Annual health check

As with all diseases, the success of the healing depends on the patient, not the doctor. While the doctor applies due diligence, offers support and adequate medication, therapeutic success is the key interest of the patient and ultimately his responsibility. The healing depends not only on the quality of medication, but on compliance to the advice and on the willingness to change personal habits and attitude.

The doctor is now satisfied with your situation. Please come back next year for your annual health check!

Acknowledgements

Many thanks to Ric Parkin and the Overload team for their feedback and engagement.

A word amongst colleagues: Dear fellow medical architects, if you could report on your results with different therapies and patients, I'd be interested to add this to the body of knowledge. Please contact me at pattern@kmarquardt.de. Many thanks in advance!

References

[ Agile01] http://www.agilemanifesto.org 'The best architectures [...] emerge from self-organizing teams'

[ Beck99] Kent Beck: E xtreme Programming Explained: Embrace Change . Addison-Wesley 1999

[ Coplien04] James Coplien, Neil Harrison: Organizational Patterns of Agile Software Development . Prentice-Hall 2004

[ Marquardt02] Klaus Marquardt: 'Supporting the Software Architect: Selected Patterns Covering Different Perspectives'. In: Proceedings of EuroPLoP 2002

[ Marquardt02b] Klaus Marquardt: 'Patterns for the Practicing Software Architect'. In: Proceedings of VikingPLoP 2002

[ Marquardt03] Klaus Marquardt: 'Neglected Architecture'. In: Proceedings of VikingPLoP 2003