The most intriguing thing about the new peer-reviewed paper of the same name as today’s post in Nature: Scientific Reports by Abbas Golestani and Robin Gras is that it is longer than the one word it takes to authoritatively answer the question.
No. We cannot predict the unpredictable.
Nor do we do that good a job at predicting the predictable, as anybody who has ever spent time with models of complex systems like the stock market and global climate system realize (or should realize).
Those who have long been at this game have seen every form of hype and promise come down the pike, each new method touted to fix the shortcomings of the previous wonder. Yet the glory always fades.
Anyway, our authors have devised a “novel” algorithm which they christen GenericPred, which fixes the shortcomings of models such as ARMIA, GARCH, and MLP. Before describing it, at the top of this post is a picture of one of the predictions of the Dow Jones Industrial Average.
We don’t need to understand how forecast “goodness” was defined to be able to see that GenericPred handily beats three other standard methods. Two other pictures show similar performance for other periods of the DJIA. The GenericPred also seems to do well predicting (one version of) the global average temperature anomaly (it predicts an increase of a whopping ~2.5C over the next century).
Looking only at these pictures, it appears GenericPred is hot stuff. Obviously, the old-fashioned methods bit into the seeming increase of the DJIA (pictured above) and were burnt. But not GenericPred. No, sir. It—somehow—sensed that changes were coming and, more or less, nailed the decrease.
I don’t buy it. I may be wrong—yes, dear reader, it is possible that I am mistaken—but I don’t think so. My misgivings flow from the nature of their algorithm and its relation to chaotic signals. But, hey, if I’m wrong and these guys really can predict the stock market two years in advance, they’ll be billionaires in short order and I’ll still be running this penny-ante blog.
GenericPred is based on a good idea, which I’ll roughly summarize. Take a series y1, y2, …, yT and compute some measure of chaos, like a Lyapunov exponent. Now posit new values of the series, yT+1, yT+2, …, yT+m, and then compute the (say) Lyapunov exponent for the augmented series y1, …, yT+m. Pick those values of yT+1, …, yT+m that minimize the distance between the (say) Lyapunov exponent of the original and the augmented series.
This makes some sense because, if the series is indeed chaotic, and no external changes in the causes of this series are expected, then the nature of that series, as measured by various indexes, should remain constant. That there are or can be external changes in causes is obvious, and is why the authors look before and after the last financial “crisis.”
Still with me?
Now if you have had any experience with chaotic time series, you know that, like the DJIA, they are apt to fly off hither and yon. Chaotic signals are those that are caused (as is every time series) but which are sensitive to initial conditions. The weest perturbation at the beginning (or really anywhere) could and does result in wildly different values now. This is why they are so difficult to predict.
Take another look at the prediction picture above. The data before (on or about 1 July 2007) were used to fit the various models, and the predictions came afterward. What would this plot look like if the authors had used date up to, say, 29 June 2007? The old-school models would probably still stink. But it’s not at all clear GenericPred would do as well—because the series is chaotic. Chaotic signals are just as likely to rise as fall. Very curious they got it so startlingly right.
And could something as humble as the Lyapunov exponent (or some other univariate measure) really hold the secret to all possible future values of the stock market?
Every scientist wants to think the best of his creation. Could these researchers have played (in honest earnestness) with the dates to show us the most dramatic discrepancies between their model and the old-school methods? Since these series are chaotic it would be truly remarkable if the method weren’t sensitive to the dates chosen to model and forecast.
The pictures look a little too neat, the points of departure of the old-school models and GenericPred a little too sharp. I do not suggest any nefariousness. But if this new method is to follow the history of all other methods, and prove to be not as exciting as initially promised, it will be because the authors fooled themselves by tinkering one step too many to present their best case.
Anyway, I predict at least a brief GenericPred boom among prediction clients. If the model works as well as promised by these figures, then the authors ought to get rich off the stock market.
Incidentally, all predictions methods should be accompanied by measures of uncertainty. The old-school methods automatically give these, but GenericPred does not. That should be the authors’ next step.
Update One of the authors (Gras) responded in comments below.
Thanks to reader Rich Kyllo for bringing this paper to our attention.