Managing Complexity in the Software Industry

[Draft Proposal]

Introduction
Background Ideas
Types of Complexity
Consequences of Complexity
Organizational Responses
Empircal Method

Bibliography

Introduction

Most projects completed for CSE 584 this term probably delve into a particular topic in software engineering. This paper, by contrast, adopts a higher-level perspective on software development and attempts to develop an interesting and possibly useful characterization of the phenomena observed from this perspective. "Interesting and useful to whom?" is the key question. The answer: to business types.

Building on earlier studies of the private PBX and semiconductor industries, this paper proposes a study of the management of complexity in the software industry. The key questions addressed are the following. How can the complexity of the software industry be usefully characterized? How do firms manage complexity? Which organizational strategies have been most successful, and which are likely to be successful in the future? The remainder of the document is a draft proposal for a research project to answer these questions.

Background Ideas

The perspective on software development outlined in this paper draws on three ideas: bounded rationality in organizations, technological trajectories, and the management of complexity.

Bounded Rationality. Among the central concerns of business administration is the simple question: What determines the success or failure of a business firm? Academics research, theorize about, and teach answers to this question. Students and industry decision makers are keenly interested in understanding and executing the right answers. Though any single answer is necessarily a partial one, this paper draws on a school of thought within business administration originated by computer science's own Herb Simon [1969, 1976]. One determinant of firm success, Simon argues, is an organization's ability to make rational decisions in the face of limited and uncertain information and limited resources to process information. This is the problem of bounded rationality. Organizational decision makers necessarily must act on imperfect information. Those firms are most likely to succeed that most effectively gather the needed information about the business, formulate appropriate strategies, and implement appropriate policies and organizational structures. Appropriate means that the firm is able to achieve solid results under expected scenarios, but can adapt to unexpected problems, and capitalize on unexpected opportunities.

Technological Trajectories. The second idea is based on the work of Joseph Schumpeter [1976]. Schumpeter's main concern was the political and economic evolution of capitalism. He believed that technology in general and technological innovation in particular were important determinants of the political and economic evolution of societies. His key idea in the present context, however, is that the adoption of technologies by organizations are commitments that start an organization down one path, or trajectory, rather than another. Once an organization commits to a technological trajectory, it becomes difficult and costly to abandon the commitment and adopt alternative technologies. This idea is intuitively clear and familiar to software engineers: Once a commitment has been made to a particular platform, development tool, and code base, it is difficult and costly to change platforms or tools, or abandon the code base. The thing to note, however, is not only the current costs of making a change, but the way in which past commitments constrain future choices. Economists employ the familiar concepts of entry and exit costs to help explain the differential rates at which new entrants enter and incumbents exit industries in different sectors of the economy. A technological trajectory is a finer-grained concept that helps explain the ease or difficulty wiith which firms can reposition themselves in industries in which they are incumbents. If the technological commitments are significant and -- as developed later -- complex, firms are locked into their existing market positions and cannot easily change them.

For example, if a firm markets a line of products that run on the Windows platform and were developed with C++, most rational business decisions in the future will involve incremental innovations closely related to this line of business: enhancements to existing products, additions to the line of Windows products, complementary lines of products that can be sold through the same channels to the same customer base or into the same target market, support for the same products on different platforms. Any number of theoretically plausible business opportunities -- for example, a highly profitable and untapped market for unrelated Unix products -- are unlikely ever to make good business sense for this particular firm because of its past and continuing commitments -- commitments of capital, personnel, organizational expertise, distribution channels, customer base. The firm is an incumbent in the industry, but its technological trajectory limits its business options and makes it difficult and costly for the firm to reposition itself, even when this is an otherwise reasonable business strategy.

Complexity. There is a secular trend in modern industry toward increasing complexity. The term complexity is used here in the common software engineering sense, but applied more broadly. The concept is not quantified, but it can be. Any system can be thought of as a set of atomic components. The components of a system can be partitioned into exhaustive and mutually exclusive subsets or subsystems. (Subsystems can be nested hierarchically.) The complexity of the system depends on the number of components, the number of subsystems, and the number of relationships among components or subsystems. A component or subsystem is coupled to another component or subsystem if a relationship exists between the two. A system or subsystem is cohesive if a high percentage of participating subsystems (in a system) or components (in a subsystem) have relationships with a high percentage of the other subsystems or components. A system is relatively complex when it is highly cohesive, and relatively simple when it is loosely coupled. Simple systems are generally preferred over more complex ones because they are easier to grasp, easier to monitor, and easier to maintain and modify. The utilitariian preference for simplicity has led many disciplines, including software engineering, to strive for a system design goal of relatively few loosely coupled subsystems, each of which is highly cohesive. That is, a modular design in which the overall system is relatively simple and the complexity is contained in smaller and more manageable subsystems.

Types of Complexity in the Software Industry

Software firms must manage three distinct but related types of complexity: environmental, organizational, and technological.

Environmental Complexity. The business environment of the software industry is exceptionally complex. Most software firms are acutely tuned to this complexity. Much of the time and attention of senior executives is devoted to strategic decision making about it. Among the contributing factors are:

rapid emergence and growth of new markets and distribution channels
high interdependence among the products of different firms -- hardware platforms, software platforms, industry standards for product interoperability
head-to-head competition among firms with aggressive feature extension and product upgrade cycles driving the market
technically complex and highly political negotiations over industry standards. Multiple standards frequently compete for de jure industry acceptance. Firms often must choose among several competing de facto standards without waiting for industry consensus to emerge.
multiple interlocking alliances and strategic partnerships -- by the firm itself, and among competitors and potential competitors
multifaceted relationships with other firms: direct competition in one market, partnership in a second, shared support for a common standard in a third

In the current software industry environment, when management gets it wrong -- emerging technological standards, strategic alliances and partnerships, platforms supported, quality, customer service, marketing -- the capital markets are swift and severe in their punishment. So, understanding environmental complexity and effective decision making based on this understanding are crucial goals of software firms.

Technological Complexity. Technological complexity has a natural tendency to increase in the software industry. At the firm level, it is driven by the enhancement of the feature sets of existing products and the introduction of more powerful products. Every increase in the size of the code base, feature enhancement, and functional differentiation of products increases the technological complexity of the firm. At the industry level, the synergy between hardware and software innovation and the rapid build-out of institutional and technological infrastructure opens up markets that require new, functionally more ambitious products. The effect is that firms in the industry collectively support an ever larger and more complex set of products. (Theoretically, product evolution at the firm or industry level can be executed while holding technological complexity steady, or even decreasing it. But holding the line on complexity has a cost -- in time and resources. In practice, it is typical of the industry that product changes are incremental and evolutionary and driven by market exigencies that preclude attention to controling the growth of technological complexity. That is, controling technological complexity is not a priority. It is allowed to increase in a manner dictated by other, more pressing, considerations.)

One useful way to think about the so-called software crisis that has been with the industry almost since its inception is in terms of this increasing technological complexity. At any point in time, the technological complexity of a single firm or the industry as a whole is the result of a dynamic balance between two trends. One is the demand-driven imperative to deliver to market ever more complex software products as quickly as possible. The other is the ability of software engineers and organizations to manage the complexity of the software systems they work with. The market demands ever-more-complex software; software engineering advances permit more ever-more-complex systems. Under present industry conditions, the level of complexity at which the equilibrium is achieved is supply constrained. The market demands more complexity than software engineering can deliver. So the equilibrium is right up against what is technically possible with the given supply of manpower, expertise, tools, and design strategies available to the firm or the industry. The software development community is never quite overwhelmed -- as evidenced by the stream of increasingly complex products successfully delivered to market. But neither does it gain much breathing room -- as evidenced by symptoms such as the stream of schedule slippages and cost overruns, high defect rates, defective and unmaintainable systems, canceled projects, generalized concern about software development practices, and worker burnout. (See, for example, the Chaos Report of the Standish group which presents survey data from 365 IT executives representing 8.380 IT-application projects. The study finds that 31.1% of projects will be canceled before they ever get completed and that 52.7% of projects will cost 189% of their original estimates.) As long as industry economics dictate rapid growth in system complexity, firms will continue to push development efforts right to the edge of the feasible development envelope.

Organizational Complexity. Organizational complexity in the software industry does not receive a great deal of attention, partly because environmental and technological complexity are so prominent and partly because organization charts and personnel policies are inherently less interesting than strategic alliances, mergers and acquisitions, and cool technology. Nevertheless, it is especially true in complex industries like software development that the successful firms are those that execute most effectively. Organizational execution is the glue that binds strategic vision and technological product.

Organizational execution in the software industry is interesting on two levels: the software development team, and the higher levels of coordination within the firm. As organizational tasks, software development projects are not particularly complex activities. Typically only modest personnel and interdepartment coordination is involved. Nevertheless, organizational aspects of development projects are important because they are a significant determinant of the level at which technological complexity balances. In CSE 584, we have dealt predominantly -- and appropriately enough -- with the management of software complexity from a fairly narrow, engineering perspective. But there is a significant software engineering literature that argues that both the problems with software projects and their solutions are primarily organizational in nature, not technological. McConnell [1996: 12] finds that personnel issues are the single most important factor effecting the success or failure of software projects:

the productivity of individual programmers with similar experience can vary by up to a factor of 10 to 1 [Sackman, Erikson, and Grant, 1968; Curtis, 1981; Mills, 1983; DeMarco and Lister, 1985; Curtis et al., 1986; Card, 1987; Valett and McGarry, 1989];
team productivity can vary by factors of 3 to 1 or even 5 to 1 [Weinberg and Schulman, 1974; Boehm, 1981; Mills, 1983; Boehm, Gray, and Seewaldt, 1984];
citing Basili et al. [1995], McConnell notes that NASA SEL found after 20 years of research that the most effective development practices are not those that exploit technology, but those that effectively utilize developer capabilities.

McConnell also found that the organization of the development process ran a close second to personnel issues in effect on project success [McConnell, 1996: 14]. Organizations such as Hughes Aircraft, Lockheed, Motorola, NASA, Raytheon, and Xerox that have explicitly focused on improving their development processes have, over several years, cut their times-to-market by about one-half and have reduced cost and defects by factors of 3 to 10 [Pietrasanta, 1991; Myers, 1992; Putnam and Myers, 1992; Gibbs, 1994; Putnam, 1994; Basili et al., 1995; Raytheon, 1995; Saiedian and Hamilton, 1995 -- cited in McConnell, 1996: 14]. Raytheon was able to reduce their rework costs from 41% to less than 10% and simultaneously triple their productivity [Raytheon, 1995].

Greater complexity arises at higher levels of aggregation within the organization. The overall need of the organization is to ensure that information flows upward to inform decision making, downward to communicate goals, instill corporate culture, and execute decisions, and laterally, to facilitate collaboration and coordination as needed. Among the coordination and communication tasks that software firms must handle are:

ensure consistency, backward compatibility, and interoperability of lines of software products,
strategic orchestration of distinct lines of products and business centers,
allocation of corporate resources, coordination of business functions -- such as sales and marketing, production, finance, personnel, and of course product development
collection of information crucial to strategic and operational decisions
communication of decisions and follow-through on execution

These are mundane organizational tasks. But all successful organizations must master them. The well-established organization theory literature describes the pressures and problems predictably encountered as small firms grow into large enterprises. Problems also can arise when organizational or environmental changes lead to a mismatch between internal mechanisms and the changed needs at the organizational boundary [see, for example, Perrow, 1986 and Scott, 1998]. The types of adaptations generally required are to replace informal communication and coordination mechanisms with formal ones, to formalize, differentiate, and specialize responsibilities within the organization, and to rework existing policies, reporting mechanisms, and structures to fit changed internal and external conditions.

Consequences of Complexity in the Software Industry

The central thesis advanced in this paper is that organizational management of complexity provides a useful filter through which to view the problems that successful software firms must master. All firms make strategic and operational decision under conditions of bounded rationality. All firms manage organizational complexity to optimize information flows and efficient execution of decisions. And, all firms follow technological trajectories represented by capital expenditures on plant and equipment. What is unusual about and characteristic of the software industry is the high levels of environmental and technological complexity. This complexity has consequences both for the individual firm and for the structure and dynamics of the industry. The points that follow summarize consequences expected to follow from this complexity. These points are best understood as hypotheses to be investigated empirically (see Empirical Method).

Consequences for the firm

Environmental and technological complexity increase the information processing burden on firms, slow the decision-making process, and effect the quality of decisions. Organizational systems, procedures, and tools must be developed to

process and make trade-offs about existing resources, environmental conditions, and firm technology .
identify and evaluate preliminarily the increasing range of options generated by environmental and technological complexity. Which markets to target? Where to commit resources?
Sift through the options and select a manageable subset to
evaluate in detail.

Technological complexity binds firms more tightly to their technological trajectories. Other things being equal, more complex technologies are more difficult and costly to modify in fundamental ways or to abandon
Technological complexity has a detrimental effect on development efforts

increased costs
slowed development efforts
decreased number of manageable projects for given resource commitments
increased resource requirements per project
increased testing costs
increased uncertainty about feasibility, timing, success, final product introduces uncertainty and risk into coordinated efforts -- marketing and sales, revenue projections, strategic planning

Technological complexity increases organizational complexity

higher management and coordination burdens
differentiation and specialization of personnel and expertise
more complex staffing trade-offs: retention and utilization of specialists; cross-project coordination of less specialized personnel; staffing levels and timing in dependent functions such as testing, sales, and marketing

Technological complexity raises the costs and stakes of strategic and operational decisions
Environmental complexity induces organizational complexity. Internal organization of sales, marketing, customer support, and development must be effectively aligned with diverse geographic and product market segments. Effective interfaces with industry bodies, partners, and suppliers are crucial. Different target markets must be supported by partially overlapping development efforts requiring multiple development efforts (in parallel or sequentially), coordination, and evolutionary consistency
Environmental complexity induces technological complexity. Different target markets lead to multiple versions of similar products. Competing industry standards may lead firms to hedge their bets and support multiple standards
Organizational complexity induces technological complexity. Parallel or sequential development efforts of different product versions can lead to unintended and unnecessary variations.

Consequences for the industry

Technological complexity promotes diversity of technical trajectory. Bounded rationality constrains firms to search in and pursue only part of the possible solution space. As firms make choices they embark upon often irreversible path-dependent trajectories, from which it is difficult to backtrack.
This leads to industry competition often based on firm and customer commitments to incompatible solutions, rather than exclusively on price, quality, or service.
Technological complexity can be a source of non-imitability. Sometimes it applies, sometimes it does not. To reproduce a particular solution, entrants or imitators may need to reproduce specific history and assets required to reach an incumbent’s end point. This can create non-imitability, a potential source of competitive advantage. Other things being equal, technological complexity makes reverse engineering more difficult. On the other hand, entrants and imitators may be able to successfully reverse engineering a solution. If the incumbent has unnecessary technological complexity as a residual of earlier product evolution (or poor design), imitators may leapfrog the incumbent and gain competitive advantage with similar functionality and less technological complexity.
Industry-level technological complexity has a strong effect on returns to R&D. R&D costs fluctuate over time as the industry passes through periods with large design spaces (high uncertainty about successful designs, few standards, high complexity implies low returns) and smaller design spaces as paradigms emerge and are accepted by industry (lower uncertainty, consolidating, standardizing, and encapsulating the complexity implies quicker progress and higher returns). Over the long haul and on average, the industry will take on just about as much complexity as the firms can handle -- this is the dynamic balance described earlier. Whether a particular firm can negotiate these fluctuations and handle the proprietary complexity required to compete is an important factor in its long-term viability.
Technological complexity opens the possibility of industry inflexion points. The idea here is that innovation slows, R&D becomes less profitable, capital leaves rather than enters the industry, and there is a net destruction of value. At the limit, new entry becomes increasingly difficult and incumbents locked into low-potential trajectories exit. Certainly, this industry scenario is far from a pressing concern of current industry leaders. But, the same can be said about other industries at other times, such as railroads, textile manufacture, steel, and upstream oil and gas, to name a few. The hypothesis here, however, is that increasing technological complexity rather than traditional industry maturation, may lead to an inflexion point. Again, the key consideration is the balance between the demand for technological complexity and the ability to deliver it.

Organizational Responses to Complexity

Though environmental complexity is very important to firm success, firms have little ability to control it. In general, they are limited to two strategies. The first is to work to limit technological complexity at the industry level by supporting industrywide standards. The second is to organize internally to achieve -- and maintain -- the optimal posture toward environmental complexity. Optimal postures vary, but generally include: strong environmental monitoring and information gathering; strong information synthesizing and strategic planning functions; rapid responsiveness to change; maximum organizational flexibility to facilitate responsiveness.

Firms have significantly more latitude to deal with technological complexity. Available strategies fall into two broad categories: reducing or displacing complexity, and coping with it.

Firms can reduce (or displace) technological complexity

Direct efforts to control the growth of complexity during product development and evolution. As noted earlier, controling the growth of technological complexity is seldom an explicit goal of either the development team or higher management during product evolution (an hypothesis). (It may be an instrumental goal during the execution of a given project.) The perspective presented here suggests that controling its growth may be strategically important to the firm. Firms can adopt policies and procedures from the top down to impose this strategic goal on development efforts.
Direct efforts to reduce it by re-engineering efforts. Same rationale as the previous point. The good news for both this point and the previous one is that if the existing code base was not optimized to control complexity, significant reductions in complexity may be achievable with modest efforts.
Technological complexity can be displaced to organizational complexity. This strategy does not necessarily reduce the total technological complexity of the firm but it effectively partitions it -- organizationally -- into smaller, less complex modules that are completely independent. Such partition decisions are not easily reversed. And they are risky. By allowing completely independent evolution of the modules with no attempt to enforce coordination or consistency, the firm is allowing the separate business units to evolve along separate technological trajectories.

Firms can cope with technological complexity

Essentially the entire discipline of software engineering properly falls under this heading. In the language used in this paper, this heading corresponds to the various tools and techniques to manage increasing technological complexity.
Firms can also attempt to cope by throwing ever more corporate resources at the complexity -- money, time, personnel. This paper argues that this strategy is unlikely to work for a firm over the long term. On the other hand, there is considerable anecdotal evidence that this strategy has worked -- so far -- for many successful software firms. This makes it an extremely interesting empirical question.

Empirical Method

The main goal of this paper has been to present the perspective on complexity in the software industry detailed in the preceding sections. It is, however, appropriate to conclude by sketching how the issues raised will be translated into an empirical study.

The basic method to be employed is statistical validation of hypotheses generated from quantitative data, rather than purely descriptive case study methods. Data will be collected through both informal and structured interviews and written surveys with individuals in a number of software development firms -- the number of firms and individuals to be governed, broadly, by statistical needs.

Bibliography

Basili, Victor R. and Frank McGarry. 1995. "The Experience Factory: How to Build and Run One." Tutorial M1, 17th International Conference on Software Engineering, Seattle, Washington, April 24.

Boehm, Barry W. 1981. Software Engineering Economics. Englewood Cliffs.

Boehm, Barry W., T.E. Gray, and T. Seewaldt. 1984. "Prototyping Versus Specifying: A Multiproject Experiment." IEEE Transactions on Software Engineering SE-10 (May): 290-303.

Card, David N. 1987. "A Software Technology Evaluation Program." Information and Software Technology. 29 6 (July/August): 291-300.

Curtis, Bill. 1981. "Substantiating Programmer Variability." Proceedings of the IEEE. 69 7 (July): 846.

Curtis et al. 1986. "Software Psychology: The Need for an Interdisciplinary Program." Proceedings of the IEEE. 74 8 (August): 1092-1106.

DeMarco, Tom and Timothy Lister. 1985. "Programmer Performance and the Effects of the Workplace." Proceedings of the 8th International Conference on Software Engineering. Washington, D.C. IEEE Computer Society Press, 268-272.

Gibbs, W. Wayt. 1994. "Software's Chronic Crisis." Scientific American. September: 86-95.

Klooster, Marnix, Sjaak Brinkkemper, Frank Harmsen, Gerard Wijers. 1997. "Intranet Facilitated Knowledge Management: A Theory and Tool for Defining Situational Methods." Olive and Pastor, 1997:303-317.

McConnell, Steve. 1996. Rapid Development: Taming Wild Software Schedules. Microsoft Press.

Mills, Harlan D. 1983. Software Productivity. Little, Brown.

Myers, Ware. 1992. "Good Software Practices Pay Off -- Or Do They?" IEEE Software. March: 96-97.

Olive, Antoni and Joan Antoni Pastor eds. 1997. Advanced Information Systems Engineering.

Perrow, Charles. 1986. Complex Organizations: A Critical Essay. McGraw-Hill.

Pietrasanta, Alfred M. 1991. "A Strategy for Software Process Improvement." Ninth Annual Pacific Northwest Software Quality Conference. October 7-8. Portland, Oregon.

Putnam, Lawrence H. 1994. "The Economic Value of Moving Up the SEI Scale." Managing System Development. July: 1-6.

Putnam, Lawrence H. and Ware Myers. 1992. Measures for Excellence: Reliable Software On Time, Within Budget. Yourdon Press.

Raytheon Electronic Systems. 1995. Advertisement, IEEE Software. September: back cover.

Sackman, H. W.J. Erikson, and E.E. Grant. 1968. "Exploratory Experimental Studies Comparing Online and Offline Programming Performance." Communications of the ACM. 11 1 (January): 3-11.

Saiedian, Hossein and Scott Hamilton. 1995. "Case Studies of Hughes and Raytheon's CMM Efforts." IEEE Computer. January: 20-21.

Schumpeter, Joseph A. 1976. Capitalism, Socialism and Democracy. 5th edition.

Scott, W. Richard. 1998. Organizations: Rational, Natural, and Open Systems. 4th edition. Prentice Hall.

Simon, Herbert A. 1976. Administrative Behavior: A Study of Decision-Making Processes in Administrative Organizations. 3rd edition.

Simon, Herbert A. 1976. The Sciences of the Artificial. MIT Press.

Standish Group. The Chaos Report of the Standish group: at http://www.standishgroup.com/chaos.html

Valett, J. and F.E. McGarry. 1989. "A Summary of Software Measurement Experiences in the Software Engineering Laboratory." Journal of Systems and Software. 9 (2): 137-148.

Weinberg, Gerald M. and Edward L. Schulman. 1974. "Goals and Performance in Computer Programming." Human Factors. 16 1 (February): 70-77.