{"id":41970,"date":"2018-06-05T04:45:14","date_gmt":"2018-06-05T09:45:14","guid":{"rendered":"https:\/\/michaelhyatt.com\/?p=41970"},"modified":"2018-06-05T04:45:14","modified_gmt":"2018-06-05T09:45:14","slug":"big-data-is-overrated","status":"publish","type":"post","link":"https:\/\/fullfocus.co\/big-data-is-overrated\/","title":{"rendered":"Big Data is Overrated"},"content":{"rendered":"

If thou gaze long into an algorithm, the algorithm will also gaze into\u2026 well, not exactly thee, right? More like a patchy portrait of your likeness churned out by a mimeograph low on ink\u2014sharply delineated in a few areas, sure, but hazy and obscure in many others.<\/p>\n

Yet close enough in the broad strokes to serve as a makeshift Rorschach test for those who believe a selective data set can provide the imprimatur of scientific detachment to a conclusion in search of confirming evidence.<\/p>\n

Back in 2013 an \u201cIndustry Insights\u201d report published by IBM estimated there were 2.5 quintillion bytes of data created every single day. That\u2019s the number one followed by eighteen zeroes and drawn from \u201csensors used to gather shopper information, posts to social media sites, digital pictures and videos, purchase transaction, and cell phone GPS signals to name a few.\u201d<\/p>\n

Now, five short years later, the ante has been upped considerably. In 2017, Forbes reports, \u201cmore personal data was harvested than in the previous 5,000 years of human history\u201d\u2014an impressive figure considering the Sumerians of Mesopotamia only invented the written word 5,500 years ago.<\/p>\n

Go back any further and we\u2019d have to start arguing over whether the European Union\u2019s new General Data Protection Act requires us to erase Paleolithic cave paintings in France.<\/p>\n

Big Data's boosters<\/h3>\n

This, friends, is the grist that feeds the much-ballyhooed mill we call Big Data\u2014and its boosters are legion. In fact, a writer for App Developer Magazine recently fretted that \u201cless than 0.5 percent of that data is actually being analyzed for operational decision making.\u201d<\/p>\n

\"\"<\/p>\n

But is it really a positive development to find our data points tossed to and fro in oceans of petabytes, exabytes, zettabytes, and bytes likely so vast they can not yet be named, occasionally to be fished out by some technocratic administrator or another to determine whether we\u2019re worthy for a loan, an education, a job, an insurance policy, an early release from jail?<\/p>\n

\u201cWe risk falling victim to a dictatorship of data, whereby we fetishize the information, the output of our analyses, and end up misusing it,\u201d Viktor Mayer-Sch\u00f6nberger and Kenneth Cukier warn in Big Data: A Revolution that Will Transform How We Live, Work, and Think<\/em>. \u201cHandled responsibly, Big Data is a useful tool of rational decision-making. Wielded unwisely, it can become a tool of the powerful, who may turn it into a source of repression, either by simply frustrating customers and employees, or, worse, by harming citizens. The stakes are higher than is typically acknowledged.\u201d<\/p>\n

Hmm. Seems like perhaps something of an understatement?<\/p>\n

It also raises the question, what\u2019s the next new thing? It\u2019s one thing to blithely click \u201cagree\u201d on a social media user agreement; it\u2019s quite another to have the information gleaned from there married to location data from your cell phone and facial recognition systems embedded in advertising.<\/p>\n

Despite the velocity of change and the abstruseness of this digital architecture, we have largely placed our trust in the benevolence and good intentions of those who either constructed or administer these contrivances\u2014without realizing how much flesh and blood creator has ceded to a machine.<\/p>\n

Once gone will we ever get it back?<\/p>\n

The tyranny of the collective<\/h3>\n

Towards the end of his expansive, mind-bending tome Antifragile: Things That Gain From Disorder<\/em>, acclaimed statistician\/scholar\/essayist Nassim Nicholas Taleb goes on an extended and convincing riff about the many ways \u201cthe abundance of data\u201d can be \u201cextremely harmful to knowledge.\u201d<\/p>\n

\u201cMore data means more information, perhaps,\u201d he writes, \u201cbut it also means more false information.\u201d Taleb illustrates his warning by asking us to imagine \u201cdistinction between real life and libraries.\u201d<\/p>\n

\u201cSomeone looking at history from the vantage point of a library will necessarily find many more spurious relationships than one who sees matters in the making, in the usual sequences once observes in real life,\u201d he explains. \u201cHe will be duped by more epiphenomena, one of which is the direct result of the excess data as compared to real signals.\u201d<\/p>\n

Now hold on a second, you say, libraries have done alright by me!<\/p>\n

Yes, of course. But we\u2019re not talking about your local or university library in which the curation process is driven by compounding human consensus and diversity. This is more akin to being ushered into a theoretical library full of books chosen by an entity possessing a singular presupposition and devoted to a very specific agenda or agendas. Any volume that challenges or offers contrary evidence is ignored or discarded.<\/p>\n

\"\"<\/p>\n

If you\u2019d like to personalize this thought experiment even further, imagine all of these books are volumes detailing how your life should be run, what you should be allowed to do, the degree of trust you should be granted.<\/p>\n

\u201cThe researcher gets the upside, truth gets the downside,\u201d Taleb posits. \u201cThe researcher\u2019s free option is in his ability to pick whatever statistics can confirm his belief\u2014or show a good result\u2014and ditch the rest. He has the option to stop once he has the right result.<\/p>\n

\u201cThe spurious,\u201d he adds, \u201crises to the surface.\u201d<\/p>\n

Big Data raises this \u201ccherry-picking to an industrial level,\u201d Taleb argues, adding: \u201cModernity provides too many variables (but too little data per variable), and the spurious relationships grow much faster than real information, as noise is convex and information is concave.\u201d We find ourselves, he fears, at the mercy of a \u201ctyranny of the collective,\u201d made all the more formidable by those daily quintillions added to its mass.<\/p>\n

Beware mathematicians bearing magic formula<\/h3>\n

\u201cWhen I was a little girl, I used to gaze at the traffic out the car window and study the numbers on the license plates,\u201d Cathy O\u2019Neil writes in the opening chapter of her always elucidating, frequently harrowing 2016 book Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy<\/em><\/a>. \u201cI would reduce each one to its basic elements\u2014the prime numbers that made it up. 45 = 3 X 3 X 5. That\u2019s called factoring, and it was my favorite investigative pastime.\u201d<\/p>\n

Not exactly the sort of material that\u2019ll send folks into wistful exclamations of “Hey, kids will be kids!,” but for O\u2019Neil math \u201cprovided a neat refuge from the messiness of the real world.\u201d She went to math camp, followed numbers into college, eventually got her Ph.D. \u201cMy thesis was on algebraic number theory,\u201d she writes, \u201ca field with roots in all that factoring I did as a child.\u201d She got a tenure-track gig at Barnard then made the jump to the hedge fund D.E. Shaw to \u201cput abstract theory into practice.\u201d<\/p>\n

\u201cAt first I was excited and amazed by working in this new laboratory, the global economy,\u201d she writes. Then, a little more than a year later, came the 2008 crash which \u201cmade it all too clear that mathematics, once my refuge, was not only deeply entangled in the world\u2019s problems but also fueling many of them.\u201d<\/p>\n

\u201cThe housing crisis, the collapse of major institutions, the rise of unemployment\u2014all had been aided and abetted by mathematicians wielding magic formulas,\u201d she continues. \u201cWhat\u2019s more, thanks to the extraordinary powers that I loved so much, math was able to combine with technology to multiply the chaos and misfortune, adding efficiency and scale to systems that I now recognized as flawed.\u201d<\/p>\n

Yet in what should have been a humbling moment for her field, she notes in the aftermath, \u201cnew mathematical techniques were hotter than ever and expanding into still more domains.\u201d<\/p>\n

Without any real discussion and a tacit approval seemingly based purely on awe, humanity had surrendered to techno-deities and \u201clike gods, these mathematical models were opaque, invisible to all but the highest priests in their domain: mathematicians and computer scientists.\u201d<\/p>\n

\"\"<\/p>\n

If this seems like hyperbole before you read Weapons of Math Destruction<\/em>, it certainly won\u2019t by the time you finish it.<\/p>\n

From chasing excellent teachers out of failing public schools where they\u2019re desperately needed to keeping qualified applicants out of good jobs and educational institutions to calculating the potential recidivism of inmates in dubious ways that keep the reformed as wards of the prison-industrial complex to creating false feedback loops that reinforce and amplify inequality and limit social mobility to preventing us from advocating for ourselves in the very political system that frequently appears determined to foist all of this upon us, O\u2019Neil makes a convincing case that we should all begin and end our days acknowledging, There but for the grace of the algorithmic gods go I\u2026<\/p>\n

\u201c[Y]ou cannot appeal to a WMD,\u201d she writes. \u201cThat\u2019s part of their fearsome power. They do not listen. Nor do they bend. They\u2019re deaf to charm, threats, and cajoling but also to logic\u2014even when there is good reason to question the data that feeds their conclusions.\u201d<\/p>\n

From her perch at the Mathbabe blog, through her books, and in a bracing TED Talk entitled \u201cThe Era of Blind Faith In Big Data Must End,\u201d O\u2019Neil has established herself as an eloquent advocate for humanity in the face of technocracy.<\/p>\n

\u201cModels,\u201d she warns we laypeople, \u201care opinions embedded in mathematics.\u201d Ah, but therein lies the rub. What incentive does any person or institution that benefits from a given algorithmic outcome to acknowledge to themselves or anyone else that the dispassionate automation that got them to it is a fiction, polite or otherwise?<\/p>\n

What\u2019s at stake<\/h3>\n

Of course, none of this is new. When in Ecclesiastes it is said that, \u201cGod made men upright, but they have sought out many devices,\u201d it\u2019s not a reference to iPhones and supercomputers\u2026though it easily could be updated as such. Nor is the state of affairs hopeless. It simply requires more honest grappling and pushback than what has hitherto occurred. For that to happen we must first set aside a bit of convenience and have a frank discussion of the stakes.<\/p>\n

\u201cThe essential point about big data is that a change of scale leads to a change of state,\u201d Mayer-Sch\u00f6nberger and Cukier write. \u201c[T]his transformation not only makes protecting privacy much harder but also presents an entirely new menace: penalties based on propensities. That is the possibility of using big data predictions about people to judge and punish them even before they\u2019ve acted. Doing this negates the idea of fairness, justice, and free will.\u201d<\/p>\n

One hopes those remain universal ideals we can all come together to defend.<\/p>\n","protected":false},"excerpt":{"rendered":"

If thou gaze long into an algorithm, the algorithm will also gaze into\u2026 well, not exactly thee, right? More like a patchy portrait of your likeness churned out by a mimeograph low on ink\u2014sharply delineated in a few areas, sure, but hazy and obscure in many others. Yet close enough in the broad strokes to […]<\/p>\n","protected":false},"author":48,"featured_media":41984,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[34,16],"tags":[44,417,33,38,232,373,39,469],"acf":[],"yoast_head":"\nBig Data is Overrated | Full Focus<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/fullfocus.co\/big-data-is-overrated\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Big Data is Overrated | Full Focus\" \/>\n<meta property=\"og:description\" content=\"If thou gaze long into an algorithm, the algorithm will also gaze into\u2026 well, not exactly thee, right? More like a patchy portrait of your likeness churned out by a mimeograph low on ink\u2014sharply delineated in a few areas, sure, but hazy and obscure in many others. Yet close enough in the broad strokes to […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/fullfocus.co\/big-data-is-overrated\/\" \/>\n<meta property=\"og:site_name\" content=\"Full Focus\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/facebook.com\/michaelhyatt\" \/>\n<meta property=\"article:published_time\" content=\"2018-06-05T09:45:14+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/fullfocus.co\/wp-content\/uploads\/2018\/06\/data.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1920\" \/>\n\t<meta property=\"og:image:height\" content=\"1080\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"shawnmacomber\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/fullfocus.co\/big-data-is-overrated\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/fullfocus.co\/big-data-is-overrated\/\"},\"author\":{\"name\":\"shawnmacomber\",\"@id\":\"https:\/\/fullfocus.co\/#\/schema\/person\/f6b26f230785c4266338934e4a30ce42\"},\"headline\":\"Big Data is Overrated\",\"datePublished\":\"2018-06-05T09:45:14+00:00\",\"dateModified\":\"2018-06-05T09:45:14+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/fullfocus.co\/big-data-is-overrated\/\"},\"wordCount\":1748,\"publisher\":{\"@id\":\"https:\/\/fullfocus.co\/#organization\"},\"keywords\":[\"Decision-Making\",\"Decisions\",\"Leadership\",\"leading teams\",\"Making decisions\",\"relationships\",\"Success\",\"Teams\"],\"articleSection\":[\"Leadership\",\"Personal Development\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/fullfocus.co\/big-data-is-overrated\/\",\"url\":\"https:\/\/fullfocus.co\/big-data-is-overrated\/\",\"name\":\"Big Data is Overrated | Full Focus\",\"isPartOf\":{\"@id\":\"https:\/\/fullfocus.co\/#website\"},\"datePublished\":\"2018-06-05T09:45:14+00:00\",\"dateModified\":\"2018-06-05T09:45:14+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/fullfocus.co\/big-data-is-overrated\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/fullfocus.co\/big-data-is-overrated\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/fullfocus.co\/big-data-is-overrated\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/fullfocus.co\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Big Data is Overrated\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/fullfocus.co\/#website\",\"url\":\"https:\/\/fullfocus.co\/\",\"name\":\"Full Focus\",\"description\":\"Win at Work and Succeed at Life\",\"publisher\":{\"@id\":\"https:\/\/fullfocus.co\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/fullfocus.co\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/fullfocus.co\/#organization\",\"name\":\"Full Focus\",\"url\":\"https:\/\/fullfocus.co\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/fullfocus.co\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/fullfocus.co\/wp-content\/uploads\/2022\/02\/Full-Focus-Logo-Tan.svg\",\"contentUrl\":\"https:\/\/fullfocus.co\/wp-content\/uploads\/2022\/02\/Full-Focus-Logo-Tan.svg\",\"width\":805,\"height\":178,\"caption\":\"Full Focus\"},\"image\":{\"@id\":\"https:\/\/fullfocus.co\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/facebook.com\/michaelhyatt\",\"https:\/\/www.instagram.com\/fullfocusplanner\/\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/fullfocus.co\/#\/schema\/person\/f6b26f230785c4266338934e4a30ce42\",\"name\":\"shawnmacomber\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/fullfocus.co\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g\",\"caption\":\"shawnmacomber\"},\"url\":\"https:\/\/fullfocus.co\/author\/shawnmacomber\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Big Data is Overrated | Full Focus","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/fullfocus.co\/big-data-is-overrated\/","og_locale":"en_US","og_type":"article","og_title":"Big Data is Overrated | Full Focus","og_description":"If thou gaze long into an algorithm, the algorithm will also gaze into\u2026 well, not exactly thee, right? More like a patchy portrait of your likeness churned out by a mimeograph low on ink\u2014sharply delineated in a few areas, sure, but hazy and obscure in many others. Yet close enough in the broad strokes to […]","og_url":"https:\/\/fullfocus.co\/big-data-is-overrated\/","og_site_name":"Full Focus","article_publisher":"https:\/\/facebook.com\/michaelhyatt","article_published_time":"2018-06-05T09:45:14+00:00","og_image":[{"width":1920,"height":1080,"url":"https:\/\/fullfocus.co\/wp-content\/uploads\/2018\/06\/data.jpg","type":"image\/jpeg"}],"author":"shawnmacomber","twitter_card":"summary_large_image","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/fullfocus.co\/big-data-is-overrated\/#article","isPartOf":{"@id":"https:\/\/fullfocus.co\/big-data-is-overrated\/"},"author":{"name":"shawnmacomber","@id":"https:\/\/fullfocus.co\/#\/schema\/person\/f6b26f230785c4266338934e4a30ce42"},"headline":"Big Data is Overrated","datePublished":"2018-06-05T09:45:14+00:00","dateModified":"2018-06-05T09:45:14+00:00","mainEntityOfPage":{"@id":"https:\/\/fullfocus.co\/big-data-is-overrated\/"},"wordCount":1748,"publisher":{"@id":"https:\/\/fullfocus.co\/#organization"},"keywords":["Decision-Making","Decisions","Leadership","leading teams","Making decisions","relationships","Success","Teams"],"articleSection":["Leadership","Personal Development"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/fullfocus.co\/big-data-is-overrated\/","url":"https:\/\/fullfocus.co\/big-data-is-overrated\/","name":"Big Data is Overrated | Full Focus","isPartOf":{"@id":"https:\/\/fullfocus.co\/#website"},"datePublished":"2018-06-05T09:45:14+00:00","dateModified":"2018-06-05T09:45:14+00:00","breadcrumb":{"@id":"https:\/\/fullfocus.co\/big-data-is-overrated\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/fullfocus.co\/big-data-is-overrated\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/fullfocus.co\/big-data-is-overrated\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/fullfocus.co\/"},{"@type":"ListItem","position":2,"name":"Big Data is Overrated"}]},{"@type":"WebSite","@id":"https:\/\/fullfocus.co\/#website","url":"https:\/\/fullfocus.co\/","name":"Full Focus","description":"Win at Work and Succeed at Life","publisher":{"@id":"https:\/\/fullfocus.co\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/fullfocus.co\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/fullfocus.co\/#organization","name":"Full Focus","url":"https:\/\/fullfocus.co\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/fullfocus.co\/#\/schema\/logo\/image\/","url":"https:\/\/fullfocus.co\/wp-content\/uploads\/2022\/02\/Full-Focus-Logo-Tan.svg","contentUrl":"https:\/\/fullfocus.co\/wp-content\/uploads\/2022\/02\/Full-Focus-Logo-Tan.svg","width":805,"height":178,"caption":"Full Focus"},"image":{"@id":"https:\/\/fullfocus.co\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/facebook.com\/michaelhyatt","https:\/\/www.instagram.com\/fullfocusplanner\/"]},{"@type":"Person","@id":"https:\/\/fullfocus.co\/#\/schema\/person\/f6b26f230785c4266338934e4a30ce42","name":"shawnmacomber","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/fullfocus.co\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","caption":"shawnmacomber"},"url":"https:\/\/fullfocus.co\/author\/shawnmacomber\/"}]}},"_links":{"self":[{"href":"https:\/\/fullfocus.co\/wp-json\/wp\/v2\/posts\/41970"}],"collection":[{"href":"https:\/\/fullfocus.co\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fullfocus.co\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/fullfocus.co\/wp-json\/wp\/v2\/users\/48"}],"replies":[{"embeddable":true,"href":"https:\/\/fullfocus.co\/wp-json\/wp\/v2\/comments?post=41970"}],"version-history":[{"count":0,"href":"https:\/\/fullfocus.co\/wp-json\/wp\/v2\/posts\/41970\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/fullfocus.co\/wp-json\/wp\/v2\/media\/41984"}],"wp:attachment":[{"href":"https:\/\/fullfocus.co\/wp-json\/wp\/v2\/media?parent=41970"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fullfocus.co\/wp-json\/wp\/v2\/categories?post=41970"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fullfocus.co\/wp-json\/wp\/v2\/tags?post=41970"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}