The Logic of Diversity The Complexity of a Controversial Concept By +Cosma Shalizi … The division of labor is, in part, an adaptation for handling complex problems, but only those which are complex in the straightforward sense of being very large. It relies on finding a way of decomposing the large problem into nearly-separate parts, so that it can be attacked through a strategy of divide-and-conquer, with different people specializing in conquering the various divisions. (This topic, and its relation to hierarchical structure, was explored by Herbert Simon in his classic Sciences of the Artificial.) Diversity, in the sense Page is talking about, is another way of adapting to complexity, and specifically to complex problems which are not decomposable into neat hierarchies. Put strategically, the idea is like this: Agents have only a limited capacity to represent, learn about, and predict their world, and so solve their problems. When the problem or environment is too complex for any one agent, then you should have many weak agents make partial, incomplete, overlapping representations. You’ll be better off by doing this, and then learning a way to combine them, than by trying to find a single, globally accurate representation, such as a single super-genius agent which can handle the problem all by itself. Collectively, the combined representations of the group of agents are equivalent to a single high-capacity representation. But nobody, individually, has anything like the complete picture; in fact, everybody’s individual picture is pretty much wrong, or at best drastically incomplete. Powerful, high-level capacities which emerge from the interplay of low-level components are a common feature of complex systems, but here as elsewhere, just having the components and letting them interact is not enough. The organization of the interactions is crucial. In the brain, for instance, this is the difference between […]