Types of Complexity
Consequences of Complexity
Building on earlier studies of the private PBX and semiconductor industries, this paper proposes a study of the management of complexity in the software industry. The key questions addressed are the following. How can the complexity of the software industry be usefully characterized? How do firms manage complexity? Which organizational strategies have been most successful, and which are likely to be successful in the future? The remainder of the document is a draft proposal for a research project to answer these questions.
Bounded Rationality. Among the central concerns of business administration is the simple question: What determines the success or failure of a business firm? Academics research, theorize about, and teach answers to this question. Students and industry decision makers are keenly interested in understanding and executing the right answers. Though any single answer is necessarily a partial one, this paper draws on a school of thought within business administration originated by computer science's own Herb Simon [1969, 1976]. One determinant of firm success, Simon argues, is an organization's ability to make rational decisions in the face of limited and uncertain information and limited resources to process information. This is the problem of bounded rationality. Organizational decision makers necessarily must act on imperfect information. Those firms are most likely to succeed that most effectively gather the needed information about the business, formulate appropriate strategies, and implement appropriate policies and organizational structures. Appropriate means that the firm is able to achieve solid results under expected scenarios, but can adapt to unexpected problems, and capitalize on unexpected opportunities.
Technological Trajectories. The second idea is based on the work of Joseph Schumpeter . Schumpeter's main concern was the political and economic evolution of capitalism. He believed that technology in general and technological innovation in particular were important determinants of the political and economic evolution of societies. His key idea in the present context, however, is that the adoption of technologies by organizations are commitments that start an organization down one path, or trajectory, rather than another. Once an organization commits to a technological trajectory, it becomes difficult and costly to abandon the commitment and adopt alternative technologies. This idea is intuitively clear and familiar to software engineers: Once a commitment has been made to a particular platform, development tool, and code base, it is difficult and costly to change platforms or tools, or abandon the code base. The thing to note, however, is not only the current costs of making a change, but the way in which past commitments constrain future choices. Economists employ the familiar concepts of entry and exit costs to help explain the differential rates at which new entrants enter and incumbents exit industries in different sectors of the economy. A technological trajectory is a finer-grained concept that helps explain the ease or difficulty wiith which firms can reposition themselves in industries in which they are incumbents. If the technological commitments are significant and -- as developed later -- complex, firms are locked into their existing market positions and cannot easily change them.
For example, if a firm markets a line of products that run on the Windows platform and were developed with C++, most rational business decisions in the future will involve incremental innovations closely related to this line of business: enhancements to existing products, additions to the line of Windows products, complementary lines of products that can be sold through the same channels to the same customer base or into the same target market, support for the same products on different platforms. Any number of theoretically plausible business opportunities -- for example, a highly profitable and untapped market for unrelated Unix products -- are unlikely ever to make good business sense for this particular firm because of its past and continuing commitments -- commitments of capital, personnel, organizational expertise, distribution channels, customer base. The firm is an incumbent in the industry, but its technological trajectory limits its business options and makes it difficult and costly for the firm to reposition itself, even when this is an otherwise reasonable business strategy.
Complexity. There is a secular trend in modern industry
toward increasing complexity. The term complexity is used here in
the common software engineering sense, but applied more broadly.
The concept is not quantified, but it can be. Any system can be thought
of as a set of atomic components. The components of a system can
be partitioned into exhaustive and mutually exclusive subsets or subsystems.
(Subsystems can be nested hierarchically.) The complexity of the
system depends on the number of components, the number of subsystems, and
the number of relationships among components or subsystems. A component
or subsystem is coupled to another component or subsystem if a relationship
exists between the two. A system or subsystem is cohesive if a high
percentage of participating subsystems (in a system) or components (in
a subsystem) have relationships with a high percentage of the other subsystems
or components. A system is relatively complex when it is highly cohesive,
and relatively simple when it is loosely coupled. Simple systems
are generally preferred over more complex ones because they are easier
to grasp, easier to monitor, and easier to maintain and modify. The
utilitariian preference for simplicity has led many disciplines, including
software engineering, to strive for a system design goal of relatively
few loosely coupled subsystems, each of which is highly cohesive.
That is, a modular design in which the overall system is relatively simple
and the complexity is contained in smaller and more manageable subsystems.
Environmental Complexity. The business environment of the software industry is exceptionally complex. Most software firms are acutely tuned to this complexity. Much of the time and attention of senior executives is devoted to strategic decision making about it. Among the contributing factors are:
One useful way to think about the so-called software crisis that has been with the industry almost since its inception is in terms of this increasing technological complexity. At any point in time, the technological complexity of a single firm or the industry as a whole is the result of a dynamic balance between two trends. One is the demand-driven imperative to deliver to market ever more complex software products as quickly as possible. The other is the ability of software engineers and organizations to manage the complexity of the software systems they work with. The market demands ever-more-complex software; software engineering advances permit more ever-more-complex systems. Under present industry conditions, the level of complexity at which the equilibrium is achieved is supply constrained. The market demands more complexity than software engineering can deliver. So the equilibrium is right up against what is technically possible with the given supply of manpower, expertise, tools, and design strategies available to the firm or the industry. The software development community is never quite overwhelmed -- as evidenced by the stream of increasingly complex products successfully delivered to market. But neither does it gain much breathing room -- as evidenced by symptoms such as the stream of schedule slippages and cost overruns, high defect rates, defective and unmaintainable systems, canceled projects, generalized concern about software development practices, and worker burnout. (See, for example, the Chaos Report of the Standish group which presents survey data from 365 IT executives representing 8.380 IT-application projects. The study finds that 31.1% of projects will be canceled before they ever get completed and that 52.7% of projects will cost 189% of their original estimates.) As long as industry economics dictate rapid growth in system complexity, firms will continue to push development efforts right to the edge of the feasible development envelope.
Organizational Complexity. Organizational complexity in the software industry does not receive a great deal of attention, partly because environmental and technological complexity are so prominent and partly because organization charts and personnel policies are inherently less interesting than strategic alliances, mergers and acquisitions, and cool technology. Nevertheless, it is especially true in complex industries like software development that the successful firms are those that execute most effectively. Organizational execution is the glue that binds strategic vision and technological product.
Organizational execution in the software industry is interesting on two levels: the software development team, and the higher levels of coordination within the firm. As organizational tasks, software development projects are not particularly complex activities. Typically only modest personnel and interdepartment coordination is involved. Nevertheless, organizational aspects of development projects are important because they are a significant determinant of the level at which technological complexity balances. In CSE 584, we have dealt predominantly -- and appropriately enough -- with the management of software complexity from a fairly narrow, engineering perspective. But there is a significant software engineering literature that argues that both the problems with software projects and their solutions are primarily organizational in nature, not technological. McConnell [1996: 12] finds that personnel issues are the single most important factor effecting the success or failure of software projects:
Greater complexity arises at higher levels of aggregation within the organization. The overall need of the organization is to ensure that information flows upward to inform decision making, downward to communicate goals, instill corporate culture, and execute decisions, and laterally, to facilitate collaboration and coordination as needed. Among the coordination and communication tasks that software firms must handle are:
Consequences of Complexity in the Software Industry
The central thesis advanced in this paper is that organizational management of complexity provides a useful filter through which to view the problems that successful software firms must master. All firms make strategic and operational decision under conditions of bounded rationality. All firms manage organizational complexity to optimize information flows and efficient execution of decisions. And, all firms follow technological trajectories represented by capital expenditures on plant and equipment. What is unusual about and characteristic of the software industry is the high levels of environmental and technological complexity. This complexity has consequences both for the individual firm and for the structure and dynamics of the industry. The points that follow summarize consequences expected to follow from this complexity. These points are best understood as hypotheses to be investigated empirically (see Empirical Method).
Organizational Responses to Complexity
Though environmental complexity is very important to firm success, firms have little ability to control it. In general, they are limited to two strategies. The first is to work to limit technological complexity at the industry level by supporting industrywide standards. The second is to organize internally to achieve -- and maintain -- the optimal posture toward environmental complexity. Optimal postures vary, but generally include: strong environmental monitoring and information gathering; strong information synthesizing and strategic planning functions; rapid responsiveness to change; maximum organizational flexibility to facilitate responsiveness.
Firms have significantly more latitude to deal with technological complexity. Available strategies fall into two broad categories: reducing or displacing complexity, and coping with it.
The basic method to be employed is statistical validation of hypotheses
generated from quantitative data, rather than purely descriptive case study
methods. Data will be collected through both informal and structured
interviews and written surveys with individuals in a number of software
development firms -- the number of firms and individuals to be governed,
broadly, by statistical needs.
Basili, Victor R. and Frank McGarry. 1995. "The Experience Factory: How to Build and Run One." Tutorial M1, 17th International Conference on Software Engineering, Seattle, Washington, April 24.
Boehm, Barry W. 1981. Software Engineering Economics. Englewood Cliffs.
Boehm, Barry W., T.E. Gray, and T. Seewaldt. 1984. "Prototyping Versus Specifying: A Multiproject Experiment." IEEE Transactions on Software Engineering SE-10 (May): 290-303.
Card, David N. 1987. "A Software Technology Evaluation Program." Information and Software Technology. 29 6 (July/August): 291-300.
Curtis, Bill. 1981. "Substantiating Programmer Variability." Proceedings of the IEEE. 69 7 (July): 846.
Curtis et al. 1986. "Software Psychology: The Need for an Interdisciplinary Program." Proceedings of the IEEE. 74 8 (August): 1092-1106.
DeMarco, Tom and Timothy Lister. 1985. "Programmer Performance and the Effects of the Workplace." Proceedings of the 8th International Conference on Software Engineering. Washington, D.C. IEEE Computer Society Press, 268-272.
Gibbs, W. Wayt. 1994. "Software's Chronic Crisis." Scientific American. September: 86-95.
Klooster, Marnix, Sjaak Brinkkemper, Frank Harmsen, Gerard Wijers. 1997. "Intranet Facilitated Knowledge Management: A Theory and Tool for Defining Situational Methods." Olive and Pastor, 1997:303-317.
McConnell, Steve. 1996. Rapid Development: Taming Wild Software Schedules. Microsoft Press.
Mills, Harlan D. 1983. Software Productivity. Little, Brown.
Myers, Ware. 1992. "Good Software Practices Pay Off -- Or Do They?" IEEE Software. March: 96-97.
Olive, Antoni and Joan Antoni Pastor eds. 1997. Advanced Information Systems Engineering.
Perrow, Charles. 1986. Complex Organizations: A Critical
Pietrasanta, Alfred M. 1991. "A Strategy for Software Process Improvement." Ninth Annual Pacific Northwest Software Quality Conference. October 7-8. Portland, Oregon.
Putnam, Lawrence H. 1994. "The Economic Value of Moving Up the SEI Scale." Managing System Development. July: 1-6.
Putnam, Lawrence H. and Ware Myers. 1992. Measures for Excellence: Reliable Software On Time, Within Budget. Yourdon Press.
Raytheon Electronic Systems. 1995. Advertisement, IEEE Software. September: back cover.
Sackman, H. W.J. Erikson, and E.E. Grant. 1968. "Exploratory Experimental Studies Comparing Online and Offline Programming Performance." Communications of the ACM. 11 1 (January): 3-11.
Saiedian, Hossein and Scott Hamilton. 1995. "Case Studies of Hughes and Raytheon's CMM Efforts." IEEE Computer. January: 20-21.
Schumpeter, Joseph A. 1976. Capitalism, Socialism and Democracy. 5th edition.
Scott, W. Richard. 1998. Organizations: Rational, Natural, and Open Systems. 4th edition. Prentice Hall.
Simon, Herbert A. 1976. Administrative Behavior: A Study of Decision-Making Processes in Administrative Organizations. 3rd edition.
Simon, Herbert A. 1976. The Sciences of the Artificial. MIT Press.
Standish Group. The Chaos Report of the Standish group: at http://www.standishgroup.com/chaos.html
Valett, J. and F.E. McGarry. 1989. "A Summary of Software Measurement Experiences in the Software Engineering Laboratory." Journal of Systems and Software. 9 (2): 137-148.
Weinberg, Gerald M. and Edward L. Schulman. 1974. "Goals
and Performance in Computer Programming." Human Factors.
16 1 (February): 70-77.