British readers of a certain age may remember a groundbreaking TV series from the autumn of 1979 – The Mighty Micro
In many ways the programmes – from what I can remember (and I was an avid viewer) – rather underestimated the impact of what was to follow. But the last programme did – and still does – stick in the memory because of what seemed, and seems, like a hyper-optimistic prediction: that microchips could save us from war.
Essentially Evans’s view was that by hugely increasing computational power, micro-powered computing would allow us to accurately predict the outcome of military conflict and so prevent it (why start a war when you know you are bound to lose or if you do win you, and your domestic critics, know it will devastate your society).
There are a lot of flaws in this argument. One only has to think of the jihadist claim to “love death” to recognise that the certainty of defeat might not be deterent enough and the 2008 financial crisis also demonstrates that increased computing power might just create new ways to mess things up, not to solve them.
But, but, but… maybe there is something to it after all. This week’s New Scientist reports on the release of the “Global Data on Events, Location and Tone” (GDELT) data set and the way it has been successfully used by Jay Yonamine, then a PhD student at Penn State, to model the spread of conflict in Afghanistan.
Yonamine was able to successfully model how the conflict would spread through Afgahnistan using GDELT, which geolocates major news stories and uses natural language processing to store a very short summary of them.
Modelling how the conflict spread is not the same as predicting where the next jihadist inspired conflict will take place though, of course, but it may be the first step on being able to draw out undercurrents of news stories and issue early warnings. The key question is whether it can be an effective leading indicator.
Maybe the idea has promise. At the very beginning of my memories of the world are the events of August 1969 – when the British Army was drafted on to Northern Ireland’s streets to avoid a bloodbath. Just six months before no one would have predicted that would have happened – even if the tempo of civil disputation had been increasing and certainly no one expected them to stay on the streets, as they did, for the next 30 years. And more importantly, perhaps, nobody – beyond some zealots on either side – would have wanted either outcome.
Again, think of the 2007 – 2008 financial crisis. Could it have been foreseen as early as 2004? Certainly some politicians claim that it could – but how could you tell whether they were any good at prediction: Mitt Romney,a pretty serious person after all, really believed that would be president even on the night of election day – does that mean everything he says is nonsense or just some things?
Big data might help sort some of that out too. Another piece of research highlighted by New Scientist, and undertaken by Tobias Preis at the University of Warwick, Helen Susannah Moat and H. Eugene Stanley of Boston University suggests that an investment strategy based on analysis of Google Trends could have made substantial sums over the 2004 – 2011 period (see graph).
Their abstract states:
Crises in financial markets affect humans worldwide. Detailed market data on trading decisions reflect some of the complex human behavior that has led to these crises. We suggest that massive new data sources resulting from human interaction with the Internet may offer a new perspective on the behavior of market participants in periods of large market movements. By analyzing changes in Google query volumes for search terms related to finance, we find patterns that may be interpreted as “early warning signs” of stock market moves. Our results illustrate the potential that combining extensive behavioral data sets offers for a better understanding of collective human behavior.
So, risking the wrath of John Rentoul, this could be A Question To Which The Answer Might Not Be No.
Related articles
- GDELT: a big data history of life, the universe and everything (guardian.co.uk)
- World’s largest events database could predict conflict (newscientist.com)
- What can we learn from the last 200 million things that happened in the world? (beowulfjournal.wordpress.com)
- Google Trends Big Data For Predicting the Market: Deep Dive and Current Predictions (forbes.com)
- Mapping the GDELT data (and some Russian protests, too) (r-bloggers.com)
- Predictive Application Tracks Global Events (arnoldit.com)
- Google big data sets forecast stock market movements, researchers claim (itpro.co.uk)
- Can Wikipedia Predict The Stock Market? (shinesquad.me)
- Using Big Data to Save the Planet (slashdot.org)
- When will computers out think You? (workingthewebtowin.blogspot.com)
cartesian product ... stuff about computing mostly 
3 responses to “Big data to save the planet? A Question To Which The Answer Might Not Be No”
At least on this side of the pond, a have been treated repeatedly of late to ideologues who are undeterred by substantial bodies of evidence contradicting their views. Get them fired up about going to war and then hand them a prediction of defeat, and my guess is that they will blithely dismiss the prediction. There are the things you know to be true, and then there are those pointy-headed intellectuals who keep insisting the Earth revolves around the sun.
A fair comment, as always! Though more evidence that a policy choice is the wrong one is surely still a good thing?
Interesting question. A decision theorist would say yes, because the expected value of information is never negative. A psychologist might wonder if additional evidence would just make the decision maker more stubborn, particularly if the decision maker has a gift for being counterfactual. A pessimist might look in his back yard to see if there is room for a bunker.