The Perils Of Data-Fitting

We are delighted to see the Felix Salmon piece (highlighted by Abnormal Returns) reviewing the supposed relationship between Japan in the 90’s and the current US market.  Check out the article to see the “great fit.”

Here at “A Dash” we have frequently exposed such  bogus relationships, even when the loud voices took the other side.  There is a simple process for generating a misleading relationship:

1. Examine a bunch of charts.  Look for variables that are all correlated to the economy in some way.
2. Find a couple of prominent features to the charts.  It is especially good if one of them (the leader) shows a move not yet reflected in the other.
3. Adjust the scales to make the prominent features line up.  Ignore any other activity.
4. Disseminate the predicted next move as widely as possible.

We exposed this process in the “Batman” chart allegedly showing a relationship between bank lending criteria and personal consumption expenditures.  This widely publicized chart was never defended by any of the original big-time proponents.  The collapse in PCE did not occur in the predicted time frame, as we showed here.  No one was keeping score.

It was an obvious case of data-fitting, although our analysis did not get the buzz that we see for Felix Salmon’s piece.  Perhaps it will do better in the book version.

What You Do Not Understand

Let us assume that everyone now sees the peril of twisting old chart data to fit a pattern.  This is a lesson that can be applied to something you do not already know.

When someone reaches into history for an analogy – -it is the same thing.  It is back-fitting of data, using the human mind, the most powerful computer.

When someone presents a trading system — based upon many runs of old data, it is back-fitting.  The spectacular returns are not real.

Our Take

It is a shame that people are drawn to bad charts, bad analogies, and bad systems.

Too bad for them.  For the rest of us, it is a time of opportunity.

Subscribe by RSS Subscribe by Email