Musings Report #40 10-5-13 False Positives & the Limits of Predictive Analysis
You are receiving this email because you are one of the 400+ subscribers/major contributors to www.oftwominds.com.
For those who are new to the Musings reports: they are basically a glimpse into my notebook, the unfiltered swamp where I organize future themes, sort through the dozens of stories and links submitted by readers, refine my own research and start connecting dots which appear later in the blog or in my books. As always, I hope the Musings spark new appraisals and insights. Thank you for supporting the site and for inviting me into your circle of correspondents.
A Note on the Rest of the Year
I am setting off on a family visit that will occupy my October, and I wanted those of you who might email me this month to know that I will be reading email at least once a week but will be unable to respond. I learn a great deal from the links, critiques, commentaries and polite disagreements you send me, and I am grateful for your understanding that a lack of response reflects a severe lack of time/energy rather than a lack of interest.
I will be devoting more time to family and fitness for the rest of the year due to burnout and the recognition that if the average American with a full-time job works about 1,800 hours a year, I completed my 1,800 hours some time in early August. We all know work/life balances are tough to modulate; my basic strategy is to push myself to finish a book a year and then spend the rest of the year restoring the balance lost to that madness.
False Positives & the Limits of Predictive Analysis
Correspondent Lew G. recently sent me this thought-provoking commentary on the limits of "total information awareness" in terms of any information system's intrinsic rate of generating false positives.
In essence, the rate of false positives limits the effectiveness of any predictive system. The process of attempting to eliminate false positives is inherently one of diminishing return: even with no expense spared, the effort to eliminate false positives runs into boundaries of signal noise and generation of false positives. Here are Lew's comments:
Resources to deal with reality are inherently limited by that reality. Information, to the contrary, is inherently infinite, because of the fractal nature of reality.
A property of that information reality is that 'meaning' is relative to other items of info, and that any single item can change the interpretation of a big set of facts. E.g., < Muslim, bought pipes, bought gun powder, constantly visits jihadi sites, watches Syrian battles, attends the Mosque weekly, tithes ...> can be completely changed in meaning by a fact such as 'belongs to the Libertarian Party', even <is a plumber, 'is a target shooting enthusiast'>.
This will continue to be true no matter how much info NSA gathers: it will be a small subset of the information needed to answer the question 'possible terrorist?'.
Thus NSA's tradeoff of privacy vs security is inconsistent with reality: no matter how much info they gather, no matter how sophisticated their filters, they can never detect terrorists without a false positive rate so high that there will be insufficient resources to follow up on them.
In other words, if the system's lower boundary is one false positive per million, no additional amount of information gathering or predictive analysis will lower that rate of false positive generation to zero.
Why does this matter? It matters because it reveals that large-scale analytic systems are limited by their very nature. It isn't a matter of a lack of political will or funding; there are limits to the practical effectiveness of information gathering and predictive analysis.
Though Lew applied this to the NSA's "total information awareness" program, couldn't it also be applied to other large-scale information gathering and analysis projects such as analyzing financial markets?
This was the conclusion drawn by the father of fractals, Benoit Mandelbrot, in his book The (Mis)Behavior of Markets. As Mandelbrot observed: "When the weather changes, nobody believes the laws of physics have changed. Similarly, I don't believe that when the stock market goes into terrible gyrations its rules have changed."
All this should arouse a sense of humility about our ability to predict events, risks and crashes of one kind or another. In other words, risk cannot be entirely eliminated. Beyond a certain point, we're sacrificing treasure, civil liberties and energy for not just zero gain but negative return, as the treasure squandered on the quixotic quest for zero risk carries a steep opportunity cost: what else could we have accomplished with that treasure, effort and energy?
Market Musings
Sometimes it's better not to force a trade, and now seems to be one of those times. As I look at charts of gold, oil (WTIC) and the S&P 500, I don't see much in the way of tradeable short-term trends or extremes that suggest tradeable highs or lows.
Take a look at this chart of the NYMO, the McClellan Oscillator. When NYMO spikes up or down to extremes, it is a pretty good indicator of a near-term top or bottom in the stock market and hence a tradeable extreme. Right now it's noodling around the middle of its range, just like a lot of other things.
It's possible to trade directionless, volatile markets, but there is a definite risk of any profits being ground up by the daily chop.
There may be a tradeable bottom forming in the US dollar (ETF UUP), as it appears to have bounced off key support. There is always a danger, however, that UUP could enter another trendless meat grinder period of volatile swings up and down that would only reward nimble daytraders. If the yen and the euro establish clear trends, that would (by definition) give UUP a trend as well.
The ideal chart is one a 7-year old can read: the trend is up or down. The downtrend was easy to spot; now let's see if the trend will reverse.
Best Thing That Happened to Me This Week
We hosted a BBQ potluck for 24 friends, neighbors and clients. It's easy not to make these sort of events happen, as they require huge amounts of planning, cooking, effort and energy, but then we would miss out on bringing a memorable event to life.
From Left Field
The Writer Automaton, Switzerland (video) (via Bart D.) A 240 year old doll that can write, a clockwork creation by Pierre Jaquet-Droz, a Swiss watchmaker
The Intriguing Health Benefits of Qigong: The ancient Chinese practice shows promise in helping ease hypertension and depression
American Schools Are Failing Nonconformist Kids. Here’s How -- a mess of an essay, part critique and part parental anguish, but it nails a critical failing in our society and educational Factory: if you don't fit in, we have two solutions: meds or tutoring.
"The jaws of power are always open to devour, and her arm is always stretched out, if possible, to destroy the freedom of thinking, speaking, and writing." John Adams
Thanks for reading--
charles
