A point I want to make more clearly is that one major reason that accurately predicting events that people are particularly interested in is so hard is because many of those events are the result of some kind of tournament.
We are fascinated by tournaments. (Just look at all the complaints that tonight`s college football championship game only represents a quasi-tournament rather than an explicit tournament like the NCAA basketball championships).
So many of the things we most want experts to predict for us are explicit tournaments (e.g., the Super Bowl playoffs) that have been carefully designed to create maximum uncertainty in the later, more climactic rounds by matching the best contestants against each other.
For example, in about 90 or 100 tries, a #16 seeded team in the men`s NCAA basketball team has never upset a #1 seeded team in the opening round, so basketball games are actually quite predictable when there is a fair-sized difference in quality between teams as determined by their seasonal performance. But subsequent round games become less predictable as the quality gap narrows, so public interest builds.
Or, the things we are interested in can be semi-explicit tournaments (e.g., the Presidential primary/general election process).
Or, unplanned events take on some of the nature of tournaments.
For example, people in the 19th Century were utterly fascinated by the Battle of Waterloo (June 18, 1815), which determined the basic political arrangements of Europe up through 1914. It was often remarked that the next century of European dominance was determined by the events of a few minutes in the crisis of the battle in which Napoleon`s hitherto-undefeated Imperial Guard nearly broke through the British lines, but were stopped just short. Then, they faltered, broke, and ran.
Waterloo — which Wellington called "a damn nice thing — the nearest run thing you ever saw" — was seen as evidence against large-scale deterministic theories of history, since so much depended upon something so close.
Contributing to Waterloo`s fame was its numerous tournament-like aspects. For example, Bonaparte was the old champion making a stunning comeback. Wellington was the challenger who had never faced Napoleon before, but had worked his way up to the top by defeating his best marshals.
Finally, much that interests us are forged by vaguely tournament-like processes. For example, stock prices are the result of, in effect, competitions between those who think the price is too low and those who think it is too high.
On the other hand, the kind of phenomena that the social sciences (and much of public policy) are concerned with — crime rates, test scores, and the like — tend not to be very tournament-like at all, and thus tend to be fairly predictable.