THE LOGIC OF DIVERSITY THE COMPLEXITY OF…

The Logic of Diversity
The Complexity of a Controversial Concept
By +Cosma Shalizi

…
The division of labor is, in part, an adaptation for handling complex problems, but only those which are complex in the straightforward sense of being very large. It relies on finding a way of decomposing the large problem into nearly-separate parts, so that it can be attacked through a strategy of divide-and-conquer, with different people specializing in conquering the various divisions. (This topic, and its relation to hierarchical structure, was explored by Herbert Simon in his classic Sciences of the Artificial.) Diversity, in the sense Page is talking about, is another way of adapting to complexity, and specifically to complex problems which are not decomposable into neat hierarchies.

Put strategically, the idea is like this: Agents have only a limited capacity to represent, learn about, and predict their world, and so solve their problems. When the problem or environment is too complex for any one agent, then you should have many weak agents make partial, incomplete, overlapping representations. You’ll be better off by doing this, and then learning a way to combine them, than by trying to find a single, globally accurate representation, such as a single super-genius agent which can handle the problem all by itself. Collectively, the combined representations of the group of agents are equivalent to a single high-capacity representation. But nobody, individually, has anything like the complete picture; in fact, everybody’s individual picture is pretty much wrong, or at best drastically incomplete.

Powerful, high-level capacities which emerge from the interplay of low-level components are a common feature of complex systems, but here as elsewhere, just having the components and letting them interact is not enough. The organization of the interactions is crucial. In the brain, for instance, this is the difference between coherent thought and delirium, or even epilepsy. In distributed problem solving, social organization is the key to realizing the potential benefits of diversity, and avoiding mutual incomprehension or socially-amplified folly. Improving organization raises performance in diverse groups by making it easier for the agents to utilize each others’ abilities and efforts, which can be more important, as we’ve seen, than improving those individual abilities. Page and Hong’s model shows, in a sense, how well the group could do with the right organization, but not how to find that structure.

When political scientists, say, come up with dozens of different models for predicting elections, each backed up by their own data set, the thing to do might not be to try to find the One Right Model, but instead to find good ways to combine these partial, overlapping models. The collective picture could turn out to be highly accurate, even if the component models are bad, and their combination is too complicated for individual social scientists to grasp.
___________

This is older work by Shalizi, the author of the recent (and excellent) piece of #cognitivedemocracy , written in response to some of the difficulties with the Wisdom of Crowds approach to complex systems. +Gideon Rosenblatt, Shalizi is a professional academic statistician at Carnegie-Mellon, and he raises some of the same concerns I had with the beans in a jar experiment. However, he phrases those criticisms in a very constructive way that might interest you, because it has serious implications for understanding the dynamics of organized crowds and how the differ from behavior of independent agents.

In any case, Shalizi is clearly a social media user but doesn’t seem to have made much of an appearance on G+. Maybe we can bring him out by gushing over his work some more 🙂