Managing Complexity in the Software Industry

[Draft Proposal]

Background Ideas
Types of Complexity
Consequences of Complexity
Organizational Responses
Empircal Method



Most projects completed for CSE 584 this term probably delve into a particular topic in software engineering.   This paper, by contrast, adopts a higher-level perspective on software development and attempts to develop an interesting and possibly useful characterization of the phenomena observed from this perspective.  "Interesting and useful to whom?" is the key question.  The answer: to business types.

Building on earlier studies of the private PBX and semiconductor industries, this paper proposes a study of the management of complexity in the software industry.  The key questions addressed are the following.  How can the complexity of the software industry be usefully characterized?  How do firms manage complexity?  Which organizational strategies have been most successful, and which are likely to be successful in the future?  The remainder of the document is a draft proposal for a research project to answer these questions.


Background Ideas

The perspective on software development outlined in this paper draws on three ideas: bounded rationality in organizations, technological trajectories, and the management of complexity.

Bounded Rationality.  Among the central concerns of business administration is the simple question: What determines the success or failure of a business firm?  Academics research, theorize about, and teach answers to this question.  Students and industry decision makers are keenly interested in understanding and executing the right answers.  Though any single answer is necessarily a partial one, this paper draws on a school of thought within business administration originated by computer science's own Herb Simon [1969, 1976].  One determinant of firm success, Simon argues, is an organization's ability to make rational decisions in the face of limited and uncertain information and limited resources to process information.  This is the problem of bounded rationality.  Organizational decision makers necessarily must act on imperfect information.  Those firms are most likely to succeed that most effectively gather the needed information about the business, formulate appropriate strategies, and implement appropriate policies and organizational structures.  Appropriate means that the firm is able to achieve solid results under expected scenarios, but can adapt to unexpected problems, and capitalize on unexpected opportunities.

Technological Trajectories.  The second idea is based on the work of Joseph Schumpeter [1976].  Schumpeter's main concern was the political and economic evolution of capitalism.  He believed that technology in general and technological innovation in particular were important determinants of the political and economic evolution of societies.  His key idea in the present context, however, is that the adoption of technologies by organizations are commitments that start an organization down one path, or trajectory, rather than another.  Once an organization commits to a technological trajectory, it becomes difficult and costly to abandon the commitment and adopt alternative technologies.  This idea is intuitively clear and familiar to software engineers: Once a commitment has been made to a particular platform, development tool, and code base, it is difficult and costly to change platforms or tools, or abandon the code base.  The thing to note, however, is not only the current costs of making a change, but the way in which past commitments constrain future choices.  Economists employ the familiar concepts of entry and exit costs to help explain the differential rates at which new entrants enter and incumbents exit industries in different sectors of the economy.  A technological trajectory is a finer-grained concept that helps explain the ease or difficulty wiith which firms can reposition themselves in industries in which they are incumbents.  If the technological commitments are significant and  -- as developed later -- complex, firms are locked into their existing market positions and cannot easily change them.

For example, if a firm markets a line of products that run on the Windows platform and were developed with C++, most rational business decisions in the future will involve incremental innovations closely related to this line of business: enhancements to existing products, additions to the line of Windows products, complementary lines of products that can be sold through the same channels to the same customer base  or into the same target market, support for the same products on different platforms.  Any number of theoretically plausible business opportunities -- for example, a highly profitable and untapped market for unrelated Unix products -- are unlikely ever to make good business sense for this particular firm because of its past and continuing commitments -- commitments of capital, personnel, organizational expertise, distribution channels, customer base.  The firm is an incumbent in the industry, but its technological trajectory limits its business options and makes it difficult and costly for the firm to reposition itself, even when this is an otherwise reasonable business strategy.

Complexity.  There is a secular trend in modern industry toward increasing complexity.  The term complexity is used here in the common software engineering sense, but applied more broadly.  The concept is not quantified, but it can be.  Any system can be thought of as a set of atomic components.  The components of a system can be partitioned into exhaustive and mutually exclusive subsets or subsystems.  (Subsystems can be nested hierarchically.)  The complexity of the system depends on the number of components, the number of subsystems, and the number of relationships among components or subsystems.  A component or subsystem is coupled to another component or subsystem if a relationship exists between the two.  A system or subsystem is cohesive if a high percentage of participating subsystems (in a system) or components (in a subsystem) have relationships with a high percentage of the other subsystems or components.  A system is relatively complex when it is highly cohesive, and relatively simple when it is loosely coupled.  Simple systems are generally preferred over more complex ones because they are easier to grasp, easier to monitor, and easier to maintain and modify.  The utilitariian preference for simplicity has led many disciplines, including software engineering, to strive for a system design goal of relatively few loosely coupled subsystems, each of which is highly cohesive.  That is, a modular design in which the overall system is relatively simple and the complexity is contained in smaller and more manageable subsystems.

Types of Complexity in the Software Industry

Software firms must manage three distinct but related types of complexity: environmental, organizational, and technological.

Environmental Complexity.  The business environment of the software industry is exceptionally complex.  Most software firms are acutely tuned to this complexity.  Much of the time and attention of senior executives is devoted to strategic decision making about it.   Among the contributing factors are:

In the current software industry environment, when management gets it wrong -- emerging technological standards, strategic alliances and partnerships, platforms supported, quality, customer service, marketing  -- the capital markets are swift and severe in their punishment.  So, understanding environmental complexity and effective decision making based on this understanding are crucial goals of software firms.
Technological Complexity.  Technological complexity has a natural tendency to increase in the software industry.  At the firm level, it is driven by the enhancement of the feature sets of existing products and the introduction of more powerful products.  Every increase in the size of the code base, feature enhancement, and functional differentiation of products increases the technological complexity of the firm.  At the industry level, the synergy between hardware and software innovation and the rapid build-out of institutional and technological infrastructure opens up markets that require new, functionally more ambitious products.  The effect is that firms in the industry collectively support an ever larger and more complex set of products.  (Theoretically, product evolution at the firm or industry level can be executed while holding technological complexity steady, or even decreasing it.  But holding the line on complexity has a cost -- in time and resources.  In practice, it is typical of the industry that product changes are incremental and evolutionary and driven by market exigencies that preclude attention to controling the growth of technological complexity.  That is, controling technological complexity is not a priority.  It is allowed to increase in a manner dictated by other, more pressing, considerations.)

One useful way to think about the so-called software crisis that has been with the industry almost since its inception is in terms of this increasing technological complexity.  At any point in time, the technological complexity of a single firm or the industry as a whole is the result of a dynamic balance between two trends.  One is the demand-driven imperative to deliver to market ever more complex software products as quickly as possible.  The other is the ability of software engineers and organizations to manage the complexity of the software systems they work with.  The market demands ever-more-complex software; software engineering advances permit more ever-more-complex systems.  Under present industry conditions, the level of complexity at which the equilibrium is achieved is supply constrained.  The market demands more complexity than software engineering can deliver.  So the equilibrium is right up against what is technically possible with the given supply of manpower, expertise, tools, and design strategies available to the firm or the industry.  The software development community is never quite overwhelmed -- as evidenced by the stream of increasingly complex products successfully delivered to market.  But neither does it gain much breathing room -- as evidenced by symptoms such as the stream of schedule slippages and cost overruns, high defect rates, defective and unmaintainable systems, canceled projects, generalized concern about software development practices, and worker burnout.  (See, for example, the  Chaos Report of the Standish group which presents survey data from 365 IT executives representing 8.380 IT-application projects.  The study finds that 31.1% of projects will be canceled before they ever get completed and that 52.7% of projects will cost 189% of their original estimates.)  As long as industry economics dictate rapid growth in system complexity, firms will continue to push development efforts right to the edge of the feasible development envelope.

Organizational Complexity.  Organizational complexity in the software industry does not receive a great deal of attention, partly because environmental and technological complexity are so prominent and partly because organization charts and personnel policies are inherently less interesting than strategic alliances, mergers and acquisitions, and cool technology.  Nevertheless, it is especially true in complex industries like software development that the successful firms are those that execute most effectively.  Organizational execution is the glue that binds strategic vision and technological product.

Organizational execution in the software industry is interesting on two levels: the software development team, and the higher levels of coordination within the firm.  As organizational tasks, software development projects are not particularly complex activities.  Typically only modest personnel and interdepartment coordination is involved.  Nevertheless, organizational aspects of development projects are important because they are a significant determinant of the level at which technological complexity balances.  In CSE 584, we have dealt predominantly -- and appropriately enough -- with the management of software complexity from a fairly narrow, engineering perspective.  But there is a significant software engineering literature that argues that both the problems with software projects and their solutions are primarily organizational in nature, not technological.   McConnell [1996: 12] finds that personnel issues are the single most important factor effecting the success or failure of software projects:

McConnell also found that the organization of the development process ran a close second to personnel issues in effect on project success [McConnell, 1996: 14].  Organizations such as Hughes Aircraft, Lockheed, Motorola, NASA, Raytheon, and Xerox that have explicitly focused on improving their development processes have, over several years, cut their times-to-market by about one-half and have reduced cost and defects by factors of 3 to 10 [Pietrasanta, 1991; Myers, 1992; Putnam and Myers, 1992; Gibbs, 1994; Putnam, 1994; Basili et al., 1995; Raytheon, 1995; Saiedian and Hamilton, 1995 -- cited in McConnell, 1996: 14].  Raytheon was able to reduce their rework costs from 41% to less than 10% and simultaneously triple their productivity [Raytheon, 1995].

Greater complexity arises at higher levels of aggregation within the organization.  The overall need of the organization is to ensure that information flows upward to inform decision making, downward to communicate goals, instill corporate culture, and execute decisions, and laterally, to facilitate collaboration and coordination as needed.   Among the coordination and communication tasks that software firms must handle are:

These are mundane organizational tasks.  But all successful organizations must master them.  The well-established organization theory literature describes the pressures and problems predictably encountered as small firms grow into large enterprises.  Problems also can arise when organizational or environmental changes lead to a mismatch between internal mechanisms and the changed needs at the organizational boundary [see, for example, Perrow, 1986 and Scott, 1998].  The types of adaptations generally required are to replace informal communication and coordination mechanisms with formal ones, to formalize, differentiate, and specialize responsibilities within the organization, and to rework existing policies, reporting mechanisms, and structures to fit changed internal and external conditions.

Consequences of Complexity in the Software Industry

The central thesis advanced in this paper is that organizational management of complexity provides a useful filter through which to view the problems that successful software firms must master.  All firms make strategic and operational decision under conditions of bounded rationality.   All firms manage organizational complexity to optimize information flows and efficient execution of decisions.  And, all firms follow technological trajectories represented by capital expenditures on plant and equipment.  What is unusual about and characteristic of the software industry is the high levels of environmental and technological complexity.  This complexity has consequences both for the individual firm and for the structure and dynamics of the industry.  The points that follow summarize consequences expected to follow from this complexity.  These points are best understood as hypotheses to be investigated empirically (see Empirical Method).

Consequences for the firm

Consequences for the industry


Organizational Responses to Complexity

Though environmental complexity is very important to firm success, firms have little ability to control it.  In general, they are limited to two strategies.  The first is to work to limit technological complexity at the industry level by supporting industrywide standards.  The second is to organize internally to achieve -- and maintain -- the optimal posture toward environmental complexity.  Optimal postures vary, but generally include: strong environmental monitoring and information gathering; strong information synthesizing and strategic planning functions; rapid responsiveness to change; maximum organizational flexibility to facilitate responsiveness.

Firms have significantly more latitude to deal with technological complexity.  Available strategies fall into two broad categories: reducing or displacing complexity, and coping with it.

Firms can reduce (or displace) technological complexity

Firms can cope with technological complexity


Empirical Method

The main goal of this paper has been to present the perspective on complexity in the software industry detailed in the preceding sections.  It is, however, appropriate to conclude by sketching how the issues raised will be translated into an empirical study.

The basic method to be employed is statistical validation of hypotheses generated from quantitative data, rather than purely descriptive case study methods.  Data will be collected through both informal and structured interviews and written surveys with individuals in a number of software development firms -- the number of firms and individuals to be governed, broadly, by statistical needs.


Basili, Victor R. and Frank McGarry.  1995.  "The Experience Factory: How to Build and Run One."  Tutorial M1, 17th International Conference on Software Engineering, Seattle, Washington, April 24.

Boehm, Barry W.  1981.  Software Engineering Economics.  Englewood Cliffs.

Boehm, Barry W., T.E. Gray, and T. Seewaldt.  1984.  "Prototyping Versus Specifying: A Multiproject Experiment."  IEEE Transactions on Software Engineering SE-10 (May): 290-303.

Card, David N.  1987.  "A Software Technology Evaluation Program."  Information and Software Technology.  29 6 (July/August): 291-300.

Curtis, Bill.  1981.   "Substantiating Programmer Variability."  Proceedings of the IEEE.  69 7 (July): 846.

Curtis et al.  1986.  "Software Psychology: The Need for an Interdisciplinary Program."  Proceedings of the IEEE.  74 8 (August): 1092-1106.

DeMarco, Tom and Timothy Lister.  1985.  "Programmer Performance and the Effects of the Workplace."  Proceedings of the 8th International Conference on Software Engineering.  Washington, D.C.  IEEE Computer Society Press, 268-272.

Gibbs, W. Wayt.  1994.  "Software's Chronic Crisis."  Scientific American.  September: 86-95.

Klooster, Marnix, Sjaak Brinkkemper, Frank Harmsen, Gerard Wijers.  1997.  "Intranet Facilitated Knowledge Management: A Theory and Tool for Defining Situational Methods."  Olive and Pastor, 1997:303-317.

McConnell, Steve.  1996.  Rapid Development: Taming Wild Software Schedules.  Microsoft Press.

Mills, Harlan D.  1983.  Software Productivity.  Little, Brown.

Myers, Ware.  1992.  "Good Software Practices Pay Off -- Or Do They?"  IEEE Software.  March: 96-97.

Olive, Antoni and Joan Antoni Pastor eds.  1997.   Advanced Information Systems Engineering.

Perrow, Charles.  1986.  Complex Organizations: A Critical Essay.  McGraw-Hill.
Pietrasanta, Alfred M.  1991.  "A Strategy for Software Process Improvement."  Ninth Annual Pacific Northwest Software Quality Conference.  October 7-8.  Portland, Oregon.

Putnam, Lawrence H.  1994.  "The Economic Value of Moving Up the SEI Scale."  Managing System Development.  July: 1-6.

Putnam, Lawrence H. and Ware Myers.  1992.  Measures for Excellence: Reliable Software On Time, Within Budget.  Yourdon Press.

Raytheon Electronic Systems.  1995.  Advertisement, IEEE Software.  September: back cover.

Sackman, H. W.J. Erikson, and E.E. Grant.  1968.  "Exploratory Experimental Studies Comparing Online and Offline Programming Performance."  Communications of the ACM.  11 1 (January): 3-11.

Saiedian, Hossein and Scott Hamilton.  1995.  "Case Studies of Hughes and Raytheon's CMM Efforts."  IEEE Computer.  January: 20-21.

Schumpeter, Joseph A.  1976.  Capitalism, Socialism and Democracy.   5th edition.

Scott, W. Richard.  1998.  Organizations: Rational, Natural, and Open Systems.  4th edition.  Prentice Hall.

Simon, Herbert A.  1976.  Administrative Behavior: A Study of Decision-Making Processes in Administrative Organizations.  3rd edition.

Simon, Herbert A.  1976.  The Sciences of the Artificial.  MIT Press.

Standish Group.  The Chaos Report of the Standish group: at
Valett, J. and F.E. McGarry.  1989.  "A Summary of Software Measurement Experiences in the Software Engineering Laboratory."  Journal of Systems and Software.  9 (2): 137-148.

Weinberg, Gerald M. and Edward L. Schulman.  1974.  "Goals and Performance in Computer Programming."  Human Factors.  16 1 (February): 70-77.