Sunday, May 19, 2013

Living by numbers

When someone starts talking about data most of us switch off. If we think about it at all we imagine boring numbers. But data is power. Stick with me and I'll share the story of my introduction to the power of data.

Most of us will have  heard the saying that information is power, but we don't usually pay it much heed.

Even when it comes to sharing information, on social media for example, if we think about anything it will probably be about privacy. We pay some regard to who we share what with. That's not necessarily a bad thing. After all sometimes knowing something about you can give another person power. You will likely do better haggling if the seller doesn't know how much you are willing to spend.

But data does not need to be personal in order to be revealing.

My Introduction to the power of data

My first job out of college was working with transportation engineers. Back then, in pre-PC days, they used huge computers to make sense of road use surveys, and  build computer models to decide where to built new roads. That's power, but not the power I'm talking about.

The engineers had a problem, the models they built using the survey data just did not work. First they asked all the brilliant computer people to check everything, then their best engineers, nobody came up with an answer. As a last resort they piled big bunches of raw data on the desk of anyone who was not too busy. That's how I ended up looking at books and books of almost incomprehensible numbers. With no skill or training in the area I decided to see if I could find any patterns in the numbers.

That turned out to be surprisingly easy. For example in one spot the model veered off after ten pm.  I tried to figure out why. As a then recent ex student the only bell ten pm rang for me was 'time for one beer before the bar closes'. Now near the spot there was a big steel mill, and it changed shifts at ten pm.   The pattern in the numbers showed that the survey staff headed off early and missed the shift change. That helped make the model work and I was promoted.

The survey staff had not realised that the numbers could tell much more than how many cars drove along that road.

Revealing our secrets

Now the world is full of data gatherers, from smart phones to the  supercomputers. We leave a data trail everywhere we go. And that data gets used.

One US retailer can accurately predict when a customer is pregnant with such accuracy that they have to include irrelevant items in the offers they send out so as not to make customers feel like they are being spied on.

That data trail we leave can be used to create surprisingly detailed pictures about us.

The lessons 

There are two important lessons about the power of information.

First we need to remember why people collect data.   The answer is of course power.  That retailers wants to know when customers are pregnant so they can use insights from psychology to form lifelong habits in those customers. Or to put it another way apes who run companies want to use their super powers to be dominant: to have more resources, to get a  better choice of mate, to ensure their offspring are more likely to survive and prosper.

We should, I believe, be worried about how much power the data trail we create puts in the hands of the companies we deal with.

It's almost impossible to predict how information about us can be used. The survey staff could not predict that numbers of cars along a road could prove they left work early. A woman buying unscented soap can't tell if she is letting the store know she is pregnant.

In a world where companies change the rules about what information they store and who they share it with, understanding who knows what about you can be next to impossible.

What we do know is why the do it.

A criminal example

Sudhir Venkatesh studied the economic activity of people involved in criminality in a poor neighborhood in Chicago, drug dealers making less than minimum wage for example. Why did the leaders of a gang of drug dealers allow a researcher to interview the gang? To get more power.

Here's how they used that power. Venkatesh found that many in the gang supplemented their income by doing, often legitimate, jobs in their spare time. The gang leaders used that information to take a bigger cut of these peoples money. That is cutting the income of people who could barely afford to live independently.

Criminal; but we should not be surprised. We are not that many generations past a time when millions of people had no freedom. Their lives were totally controlled by a few powerful individuals. Those apes with power sexually preyed on and exploited their serfs and slaves.

Today, some estimate, that less than two thousand corporations control most of the wealth of the world. It is the same species of ape running those corporations. They are gathering data at an unprecedented rate. They have access to more information than the wildest dreams of a spymaster a century ago. That information is about billions of ordinary people.

Now imagine that the next time you negotiate your pay your boss has knows what you and all your family spend and how you spend it, all your secrets; matched to a psychological profile.

Do you think you would get a good deal?

Oh and the other lesson. Sometimes, as in my introduction to data, the information is wrong, but it still gets used.

DUHIGG, CHARLES. 2012. How Companies Learn Your Secrets. [ONLINE] Available at: [Accessed 16 May 13]. 2012. How Target Figured Out A Teen Girl Was Pregnant Before Her Father Did. [ONLINE] Available at: [Accessed 16 May 13].

LAMB , BRIAN. 2008. ”Q&A”: Sudhir Venkatesh. [ONLINE] Available at: [Accessed 16 May 2013]. 2012. Other ethical problems in Venkatesh's research. [ONLINE] Available at: [Accessed 16 May 13].