Featured Post

Well, Wadda Ya Know?

I can still remember taking my first dive into data mining.  My voice had just recovered from incessantly reminding people that there was no year zero, so the millennium doesn’t begin until 2001.  In the information science world, we were all abuzz about this new era of database technology and...

Read More

A Survey of the Emergent Media Delivery Economy

Posted by Chad | Posted in Data Science, Economics, Network Science | Posted on 06-08-2015


Below is a paper I wrote in April, so it may seem a bit anachronistic, but some of the things outlined here are starting to take shape. It was also my first foray into network theories of economics, so the whole topic was quite an eye opener for me.


On February 26, 2015, the Federal Communications Commission (FCC) decided that they would implement rules to protect the openness of the Internet. Its report, published on March 12, 2015 established what it calls, “clear, bright-line rules (F.C.C., p6.)” banning three practices that pose a threat to the openness of the Internet. In turn, the report cites bans on blocking of lawful destinations, throttling and degradation of traffic to and from lawful destinations, and paid prioritization of lawful traffic on the Internet (F.C.C., p6.).

This decision came on the heels of a lengthy public debate on the economic, sociological, and political effect of government regulation of Internet service providers (ISPs). Some arguments against the concept of Net Neutrality pointed to the economics of the Internet backbone and how removing the profit motive from network investment, the growth of the Internet backbone in the United States will stagnate (Hahn & Wallsten, 2006, p6) (Misra, n.d.) (Yglesias, n.d.).

There exists an economic reality that was always there, lurking beneath the crust of the conversation, but has come into full view and is having an immediate effect on the landscape of broadcast media. So-called over-the-top media providers are springing up from all corners of the broadcasting world, causing a fundamental shift in the way consumers buy content, how providers deliver it, and how advertisers are trying to worm their way in (Poggi, 2015).

Over-the-Top Services as Alternatives to Cable and FiOS

With broadband download speed capabilities reaching 35.7 megabits per second (mbps) in the United States and 23 mbps globally on average, and a global mobile platform download speed average of 12.1mbps (“Download Speed in United States | Net Index from Ookla,” n.d.), high definition video and audio streaming services have found a new market of consumers and delivery architecture for the lawful distribution of licensed media content.

Because of this access to bandwidth and content, coupled with a regulatory platform that ensures unfettered delivery of those services across backbone networks and into the home and mobile networks of broadband Internet customers, consumers are “cutting the cable,” canceling their cable service and relying entirely on streaming services for their content (Captain & Scharr, n.d.). This is also creating an economic climate for traditional television providers to offer a la carte services to their customers (Kang, 2015) much to the dismay of content providers who rely on fees collected from cable providers. Sports broadcasting company, ESPN, a subsidiary of media giant, Disney, is currently suing Verizon over their new bundling package, citing a breach of contract, but ostensibly rooted in the fear that they won’t collect fees for users who don’t watch their channels (Swanner, n.d.). One can understand ESPN’s frustration as it has positioned itself as a top price brand for cable companies, regardless of actual viewership, year on year. It sustains its popularity among sports enthusiasts by paying premium rates to carry sports events, but this comes at a cost. That cost is apparently $5.54 per month per subscriber (Ross & Maglio, n.d.).

Another driver for the growth of over-the-top services is a growing number of people who have never had cable and are instead skipping the cable rates and terrible service of the worst company in America (Matyszczyk, 2014) and going straight to the on-demand online media market. As many as one in four Millennials have never had cable and are using over-the-top services to access entertainment at home (Newman, n.d.). It is not a great stretch to see how those that never had cable are proving that it can be done and are cannibalizing the cable industry customer base, at least in part.

Poor service and availability of alternatives aren’t the only drivers of change. According to the U.S. Bureau of Labor Statistics, US consumers spent an average of $2,482 in entertainment categories in 2013 (B.L.S., 2015). This includes television services, with averages ranging from $360 per year for basic services to as much as $2,160 per year when bundled with other services such as home security and phone (Fottrell, n.d.).

Therein lies the rub. Analysts have long warned the cable industry that its model is inherently flawed (Diallo, n.d.) and that their pricing practices were just simply not in line with available discretionary spend (Eggerton, n.d.), so cable has had to turn to other services that are more important to consumers such as telephone service and security systems, but even this approach has a market cap that is constrained by price and non-price determinants of demand.

Convergence has rendered the home phone a relic of a bygone era, relegated to a device used for those who need a number on which they need to call the baby sitter and to those who have been sold on a discount for providing the phone service. A longer than expected recession has lowered the amount of household discretionary income, and prevalence of wireless telephone service has diminished the need for home telephone. This is a dangerous prospect for any company hinging their future on cross selling telephone service.

We are now at a stage of market development where the services provided by cable that aren’t television are necessary to justify the existence of television. In fact, the pricing model set up by Time Warner Cable has plans with television included equal to or less than plans without television (“Cable Deals and Internet Offers | Time Warner Cable,” n.d.) as if to say “Please just let us install television service in your home. We need you to say you want television for our shareholders’ sake.” It smacks of the desperation of a ponderous, lumbering behemoth gasping for air in an acrid economic climate.

The Real Cost of Cutting the Cable

The new regulatory environment has positioned the online streaming media market to grow very quickly. On its face, this relatively nascent market is an exciting place to be, and it is tempting for many to try to break in by providing their own licensed and original content over the Internet, on-demand, for a price. Any who would jump into these waters must do so with the knowledge that they are jumping into a space whose novelty will expire. The abundance of choices for the consumer are creating severe disequilibrium in the market, recklessly burdening the market with supply with little or no understanding of the demand.

As we’ve seen before, American consumers have a moderate budget for television services at just 4.45% of annual income, on average, and that this money has a nonlinear inversely proportional relationship at the consumer unit level (B.L.S., 2015). Even with consumers opting to shift their entertainment spend away from cable television services and to over-the-top services, there is still a dearth of spending relative to the ever-increasing supply of services. The problem with the model is if a consumer wants to take advantage of all the content provided by the prevailing leaders in the space, one could spend over $700 just to get content on par with a moderately priced (for some relative definition of “moderate”) cable service with much of the same content and on-demand capabilities.

Americans have proved resilient, if not apathetic, to content glut, tuning only 17 channels on average, despite an increase in available options (“Nielsen: Changing Channels: Americans View Just 17 Channels Despite Record Number to Choose From,” n.d.) (yet another failure of the cable marketing model by not understanding its customers wants). Even as more services emerge in the over-the-top media delivery space, consumers will settle into viewing patterns, and services that may provide quality content are likely to die on the vine or be subsumed by emerging leaders in the industry.

Over the Top Services and Emergence Economics

What we are going to see in the coming years is the concept of Emergence Economics in action. Whitt and Schultze define Emergence Economics as “an umbrella phrase” to describe the interaction between complexity economics and other competition theory and net-effect economic models (2009, p223). This model deviates from neoclassical economic models by acknowledging that there is far more than just an invisible hand driving the market toward equilibrium. Emergence economics, at their core, are a formula representing interactions between agents and networks and their evolution through network growth (Whitt & Schultze, 2009, p231).

Whitt & Schultze are purposefully vague on the concept of an agent, saying agents in emergence economics “consist of the full spectrum of economic actors (2009, p231)” In the case of our new model of over-the-top services, the agents of change in the network are many, but principally include consumers, the FCC, content creators, ISPs, the cable carriers, and the over-the-top services themselves.

Consumers are agents of the change exercising their options in media consumption expenditure, creating a demand, however amorphous, that the new services are trying to capture. The consumer role in the viability of the new market will have direct effect on the macro and micro aspects of the industry. As consumers react to changes in the network and from other agents in the sphere of influence, the network will fluctuate.

The FCC has emerged as the most visible single agent by establishing that the network infrastructure on which these services will work is free from artificial and purposeful hindrances.  Now that the federal agency has dipped its gargantuan toe into the murky waters, it must be prepared to further smooth the regulatory and economic landscape they have helped shape.

Content creators will find their role in the plot to come as an important one by establishing the economy of licensing in the new delivery mechanism. The prevailing service providers are facing a problem with their licensing model by having to ask studios to produce or license exclusive content, a gamble in such an uneasy market being requested of an already uneasy industry (Rayburn, n.d.).

Despite being somewhat neutered by the FCC ruling, ISPs will play a significant role in the future of streaming media by changing their own consumer pricing models. At risk are the unlimited data streams in the current plans. It is likely that ISPs will evaluate, if not adopt, a pay-per-use model that has been the de facto model for mobile data carriers for some time. Such a model shifts the burden of cost to the consumer directly, rather than allowing the media providers to absorb the variable cost (or fixed, depending on how savvy their negotiations) into their own pricing structures.

The cable carriers do still have a role to play, as they are reacting already and can have a direct effect on the network by changing their model. Verizon FiOS is already marching down that road with their bundled services and, endangered though it may be, is but the first in many steps toward ensuring the survival of the industry and long-term viability of the service as a content delivery platform.

The over-the-top providers will certainly be the foci of the network as it evolves. The advantage these services have is that they are currently en vogue, the beneficiaries of the “endogenous novelty (Whitt & Schultze, 2009, p241)” in the model. They are also currently new and small enough to adapt to rapid change, which benefits them as they are the first in what looks to be the future of media consumption.

The Evolution

Given the regulatory, consumer, and organizational landscape spread out before us, we can make an educated guess to the track of the evolution of the market. Agents will find their role in the long-term model by differentiating themselves from other agents (in and out of their own classifications and networks), developing a core competency in which they will play. Those that do not differentiate will disappear or be subsumed by those that do.

We are already seeing evidence of this in the way Time Warner is offering its premium content network, HBO, via the online service, HBO Now (“HBO NOW. It’s HBO. All you need is the Internet.,” n.d.). Superficial views of this move would indicate that Time Warner is cannibalizing their cable television consumer base, and assuming HBO Now customers will retain their cable television accounts, but a closer look reveals a company shrewdly putting a physical distance between its media interests and its delivery infrastructure, decoupling a once tightly integrated vertical, and evolving to compete in a new economic landscape.

Over-the-top providers such as Netflix and Hulu are differentiating by providing exclusive content, often funded by the services at great expense in the assumption it will bring subscribers. If someone wants to watch House of Cards, they will have to carry a Netflix subscription. If that same person wants to watch episodes of popular cable or broadcast television shows such as The Walking Dead and not wait for Netflix to get the content, they will need an $8 Hulu subscription (“Watch TV and movies on Xbox, PS3, Apple TV, and more | Hulu Plus,” n.d.).

Consumers have been evolving for some time and have driven a lot of the change that led to the current landscape. Individuals will make their voices heard through their buying decisions. As they allocate their sparse discretionary income in a climate of increased options, they will define the leaders and will arbitrate at the scorers’ table for who adapts in the most effective ways.

ISPs will evolve through a tenuous process of managing pricing models, tempered by consumer fickleness and antitrust concerns. If they try to pass along too much cost, they will arouse the suspicions of the Department of Justice and could exacerbate already tense relationships with their customers who will clamor for alternatives.

The Shape of Things to Come

Given the stakeholders involved, we can posit many outcomes, but there are some factors on which we can rely to determine what shape the market will take. We will see a number of over-the-top streaming services emerge. As the market saturates, a few clear winners will emerge and will either push out their under performing competitors by leveraging customer bases and media licensing options, or acquire them and add their assets to their arsenals. These organizations will grow to become leaders in the industry and will have increasingly larger influence over the network. In two to five years, it will not be surprising to see three solid performers in the industry with a majority of the market shared among them, with niche players operating at much smaller volumes delivering premium content to narrower consumer markets.

ISPs will dance with their customers and ultimately find a compromise between controlling revenue growth while keeping prices within customer expectations. Customers that use data at rates at or beyond the 97.5th quantile will pay more than they do today, but net price to consumer will remain relatively flat for most users. In the next half-decade, ISPs will be charging for use, but customers will still be getting value for the spend.

Cable companies will continue to distance their media services from their delivery infrastructure. We will see a continued decoupling of once-vertical industries and these companies will start to look more like over-the-top services or ISPs. They may operate under the same shingle, but they will be subject to new shareholder expectations, and will overcome their identity crisis by acknowledging their role in the space.

The FCC and other regulatory agencies will spend the next five years monitoring the economic landscape, ensuring that everyone is playing fairly in the field. It is not unreasonable to assume that some other issue will come to the fore, and the FCC will have to determine in what capacity they will participate. Indeed, the FCC has struck a note in the symphony, and now it must be harmonious in its participation.

Consumers will have a direct and influential effect on how the next five years of this emergent market will progress. Through their spending choices, they will shape how the network evolves. In five years, broadband-connected households will stream their media from their favorite service directly to their televisions and computers (if one can still make such a distinction), adding premium content from smaller providers in an a la carte fashion, commensurate with their discretionary expenditures and television-watching proclivities.

In time, the market will evolve enough to expose new vertical industries, and larger conglomerations will emerge to dominate the landscape. As fewer and fewer companies control the market from content creation to delivery, the market will achieve its critical mass. A new agent, source or form yet unknown, will appear and create a new conversation about a new market and the cycle will begin anew.

The emerging market will be exciting to watch. In fact, with a story containing as much drama, as many diverse, multidimensional characters, and as unpredictable a plot as the new streaming media economy…

Who needs television?




B.L.S. (2015). Consumer Expenditures in 2013 (No. 1053). US Bureau of Labor Statistics. Retrieved from http://www.bls.gov/cex/csxann13.pdf

Cable Deals and Internet Offers | Time Warner Cable. (n.d.). Retrieved April 29, 2015, from http://www.timewarnercable.com/en/plans-packages/cable-internet.html

Captain, S., & Scharr, J. (n.d.). Cable TV Alternatives – A Guide to Cutting the Cord. Retrieved April 29, 2015, from http://www.tomsguide.com/us/cord-cutting-guide,news-17928.html

Diallo, A. (n.d.). Cable TV Model Not Just Unpopular But Unsustainable. Retrieved April 29, 2015, from http://www.forbes.com/sites/amadoudiallo/2013/10/14/cable-tv-price-hikes-unsustainable/

Download Speed in United States | Net Index from Ookla. (n.d.). Retrieved April 29, 2015, from http://www.netindex.com/download/2,1/United-States/

Eggerton, J. (n.d.). Moffett: Where Will Money Come From For Pay TV Price Increases? | Multichannel. Retrieved April 29, 2015, from http://www.multichannel.com/news/cable-operators/moffett-where-will-money-come-pay-tv-price-increases/328425

Ehrenfreund, M. (n.d.). This hilarious graph of Netflix speeds shows the importance of net neutrality. Retrieved April 29, 2015, from http://knowmore.washingtonpost.com/2014/04/25/this-hilarious-graph-of-netflix-speeds-shows-the-importance-of-net-neutrality/

F.C.C. (n.d.). Report and Order On Remand, Declaratory Ruling, and Order, GN Docket No. 14-28 (No. FCC 15 – 24).

Fottrell, Q. (n.d.). Why cable TV bills are only going up – MarketWatch. Retrieved April 29, 2015, from http://www.marketwatch.com/story/why-cable-tv-bills-are-only-going-up-2013-02-28

Hahn, R., & Wallsten, S. (2006, June). The Economics of Net Neutrality. Economists’ Voice, 1–7.

HBO NOW. It’s HBO. All you need is the Internet. (n.d.). Retrieved April 30, 2015, from https://order.hbonow.com/

Kang, C. (2015, April 17). Verizon breaks up its bundle to give consumers more choice – The Washington Post. Retrieved April 29, 2015, from http://www.washingtonpost.com/blogs/the-switch/wp/2015/04/17/verizon-breaks-up-its-bundle-to-give-consumers-more-choice/

Matyszczyk, C. (2014, April 10). Comcast wins! (Worst company in America) – CNET. Retrieved April 29, 2015, from http://www.cnet.com/news/comcast-wins-worst-company-in-america/

Misra, V. (n.d.). Net neutrality is all good and fine; the real problem is elsewhere | Dept. of Computer Science, Columbia University. Retrieved April 29, 2015, from http://www.cs.columbia.edu/2014/net-neutrality/

Newman, J. (n.d.). Nearly 1 in 4 millennials have cut the cord or never had cable | TechHive. Retrieved April 29, 2015, from http://www.techhive.com/article/2833829/nearly-1-in-4-millennials-have-cut-the-cord-or-never-had-cable.html

Nielsen: Changing Channels: Americans View Just 17 Channels Despite Record Number to Choose From. (n.d.). Retrieved April 29, 2015, from http://www.nielsen.com/us/en/insights/news/2014/changing-channels-americans-view-just-17-channels-despite-record-number-to-choose-from.html

Poggi, J. (2015, March 23). Apple, Sony, Dish Push OTT Viewing Over The Top | Media – Advertising Age. Retrieved April 29, 2015, from http://adage.com/article/media/apple-sony-dish-push-ott-viewing-top/297712/

Rayburn, D. (n.d.). Netflix’s Long-Term Business Model Flawed, Competitors Willing To Give Content Away – Dan Rayburn – StreamingMediaBlog.com. Retrieved April 30, 2015, from http://blog.streamingmedia.com/2011/11/netlfixs-long-term-business-model-flawed-competitors-willing-to-give-content-away.html

Ross, L. A., & Maglio, T. (n.d.). Your Unfair Cable Bill: Most Expensive Channels Aren’t the Most Watched – TheWrap. Retrieved April 29, 2015, from http://www.thewrap.com/cable-bill-battle-subscribers-providers-carriage-fees/

Swanner, N. (n.d.). ESPN sues Verizon over new FiOS bundling – SlashGear. Retrieved April 29, 2015, from http://www.slashgear.com/espn-sues-verizon-over-new-fios-bundling-27380933/

Watch TV and movies on Xbox, PS3, Apple TV, and more | Hulu Plus. (n.d.). Retrieved April 30, 2015, from http://www.hulu.com/

Whitt, R. S., & Schultze, S. J. (2009). The New “Emergence Economics” of Innovation and Growth, and What It Means for Communications Policy. Journal on Telecommunications & High Technology Law, 7(2), 217–315.

Yglesias, M. (n.d.). Net neutrality economics: The economic case against the broadband providers. Retrieved April 29, 2015, from http://www.slate.com/blogs/moneybox/2014/01/16/net_neutrality_economics_the_economic_case_against_the_broadband_providers.html


The Velvet Smog

Posted by Chad | Posted in Life | Posted on 02-08-2015


Those who know me know that I despise talking on the phone. I just communicate better in writing and in person than I do on the phone. When my phone rang around 10am on August 2, 2014 and my mom’s number appeared on the screen, my heart was sinking before I even hit the accept button.

A year ago today, the world lost Bob Brewster, a dad, husband, friend, and my step-dad. The music world lost one of its most musically talented untapped treasures (and one of the best bass players I’ve ever heard – who, by all accounts, never picked up a bass guitar). I had the honor of speaking at his funeral, and for the last year, I have wanted to post this to preserve his memory and to honor him as best I could. The following is what I said as we laid him to rest, slightly edited to account for some things I missed that I wish I would have said (and for grammar… cut me some slack, it was a tough time).


The Phantoms. Clockwise from Left: Mike Berman, Bob “the Velvet Smog” Brewster, Richard Chaffee, Frank Filardo

I never addressed Bob as “dad.” I also never officially dropped the “step” from his title when referring to him with others. If you pay attention to the popular culture or Grimm’s tales, it may seem like I never regarded him as fit for the role of dad, but I can assure you this could not be farther from the truth. You see, in my early years, there was a man in my life whom I called “dad” but, for reasons that will remain forever elusive, that man found other things in life to be more interesting than a gangly nerd who always got in the way.

And so, he left.

To call Bob “dad” would demean Bob and belittle his role in my life. To drop the “step” from his title would, in a way, negate the fact that he plowed through the reckless hubris of a preteen boy still reeling from abandonment, and work to establish a relationship that would transcend the role of dad. In short, he chose me. I suppose that if I had one regret it’s that I didn’t elaborate this to him. I think, in hindsight, he wanted me to just call him dad. We had many erudite conversations about the nature of our relationship and the fabric of our funny little (big) family, but we never crossed that threshold.

In the contemporary imagination, dads teach their sons how to throw a fastball, how to keep their elbows tight when shooting a jumpshot, or when to close the choke on a two-stroke motor. The teaching Bob had for me went far beyond such banal trivialities. Sure, he did offer advice on relationships, career, and many of those traditional lessons imparted from father to son, all sagacious in its own right, but as with all of life’s best lessons, the greatest thing I learned from him came from what he did.

We all know Bob was a musician practically from birth. Nobody here can doubt he was among the best in the world at the art of music. It was how he played that taught the greatest lessons. Bob could play a song you had heard a thousand or more times before and you would know that tune from the start. His true genius, however, was in how he would simply shift his pinky 1/4″ to the right or left and the song was instantly his own. He could play a song you had never heard before, but the way he played it made you pensive and nostalgic, as if it was always your favorite song. Today, if I hear keyboards in a song, I can tell in three notes that it’s Bob. The man spoke through the keys.

They were his voice.

Among many other things, mom gave me the tools, support, and teaching to understand and love the purity of machines. The ability to see Leibnitz’ ones and zeroes as beautiful and powerful forces capable of doing amazing things came naturally to me. Bob taught me to see that there is an amazing world somewhere between 0 and 1. That life, like music, cannot be truly mine without knowing that area between true and false. Bob saw and taught me to see that the true beauty of the universe lies somewhere between the notes, a place we all know is there but cannot see with our eyes or hear with our ears. A place whose existence we must feel. This has become the model by which I approach all endeavors. To make this world my own, I have to find my own voice and use it to shape everything that I do. To be truly individual, I need to move my finger slightly to the left or right.

Though Bob heard music in everything in his life, there was no greater music for his ears than his grandchildren. If he spoke of all of his children with love, caring, and admiration, he spoke doubly or triply so about his grandkids. If “In the Lounge with Mel Torme” was an album he could listen to for hours, the sound of his grandchildren near him was a symphony he could listen to for a lifetime. They were the source of all his pride and elevated his spirits in fulfillment of what he truly held as his duty. [ed. note: If any of you are reading this, I mean that. He spoke so highly of each and every one of you and your parents. He was almost insufferable about it 🙂 ]

Now, I will never be the musician he was, I doubt anyone will for a long time, but if I remember this and apply it to whatever I do, I can certainly make my own mark on the world. These are things that make him more than just “dad” to me, and so he will always be Bob. And that, to me, is far more important than syntax.

We have lost a father, a friend, a husband, and a friend. Our music is a bit less interesting now.


If we always remember to find and use our own voice, to throw the best of ourselves into everything we do and find that beauty that lives between the notes, we will never lose Bob.

RIP, Robert Leo Brewster. March 24, 1942 – August 2, 2014. Thank you for the precious memories that will live forever in our hearts.

Follow Chad on Twitter or friend him on Facebook.  Or don’t.  Or whatever.

Data Science and The Librarian

Posted by Chad | Posted in Data Science, Librarianship | Posted on 29-03-2014


Last week I was talking with a few fellow librarians and library students. I mentioned that I was pursuing my CAS in Data Science in addition to my MSLIS and their comment was, “so you’re thinking of going into business?”

I don’t mean to dissuade any would be librarians from exploring businesses as potential employers. In fact, I am a staunch advocate for businesses large and small looking to librarians to help manage the mountains of information the day to day operations in an organization create. Information literacy should be of paramount concern to any organization, and businesses would improve their position by leveraging the skills librarians bring to the table.

The comments got me thinking about the perception of data science in the library community. Since the beginning of what we know of as the library, we, as librarians have been charged with connecting communities to the information the world creates. It is in this way that we facilitate knowledge creation. For so long, this was mostly limited to access to print resources, and recently web-delivered content and databases. Now our communities are faced with a deluge of data and information they don’t know exists or don’t know how to manage.

More and more, we have a need to connect communities not only to the data and information generated elsewhere in the world, but also to that data which they themselves create. Wherever we go, we are creating data. When we swipe our rewards card, we are creating marketing data. When we check out a book in a library, we are creating data about our tastes in reading. If you have a cell phone in your pocket right now – you are creating a continual data stream to your wireless provider. Merely by accessing this page, you’ve created data (which I will mine for information on my readers… all 2 of you).

Data are everywhere in our lives – and all that data are being mined for insights into our trends and tastes. Using data wisely, both as consumer and producer, is at the core of information literacy. Who are better equipped than librarians to connect our communities to data and derive meaning and personal responsibility for the data?

Let us also not forget that we librarians are very much a part of that community. ILS software is generating data at an increasing rate, and we need to know how to use that data to gain understanding of our communities’ needs. Wouldn’t you love to gain some insight and build empirical assessments of library services – and NOT write yet another survey?

Lastly, after we’ve ingested, chewed upon, and become nauseated by our data, what do we do with it? In order for the fruit of our analytical efforts to persist, we need to make it available for future use. Curation and preservation of born-digital information is a complex and intriguing field and who better than librarians and archivists to inform the discussion?

This is data science.



Librarians, Missionaries, and Improving Society

Posted by Chad | Posted in IST511, Librarianship | Posted on 09-10-2013


Whenever I think of activism as a societal improvement, I am almost always excited for the opportunity.  I look around and see so many people engaged in arguments involving things about which they have no knowledge (government shutdown, I’m looking in YOUR direction).  That lack of knowledge infuriates them and takes any teeth out of their discourse.  Instead, those discussions decay into personal attacks and, voila, the YouTube comments section is born.

By the way, here’s a short list of phrases that will immediately turn me off to whatever you are saying:

  • A whole nother (there’s no such word as nother)
  • irregardless (drop the ir)
  • Nookyoolar (when you mean nuclear)
  • Subliminable (remind me who actually voted for you?)

Sounds arrogant, right?  Especially if one of those phrases above are in your lexicon.  This is my bias.

I’m reminded of a feeling I got when I watched the movie, Black Robe.  Black Robe is Bruce Beresford’s film, fictionalizing a Jesuit mission into Huron territory in the 17th century.  If you haven’t seen it, I highly recommend you take the time to do so.  If you take me up on that, I recommend you hide, or at least secure beyond use, all rope, firearms, and antifreeze (SPOILER ALERT: it’s not exactly a rom-com).

In the film, Father Laforgue, played by the ever intense Lothaire Bluteau (yes, that is the toughest sounding French name there ever was), makes his way through the wilderness of Nouvelle France (later Quebec, for those keeping score).  The trek is to go in search of a Jesuit mission.  Oh, by the way, it’s freakin winter in northern damn Quebec in 1635.

Allons-y pour une promenade!

Cela ressemble à une bonne idée!”

If you survive your own overwhelming despair after sitting through the film, you should note that the thing the movie does well is show the situation into which the missionaries thrust themselves and the resultant damage it caused the indigenous people.  I can very easily sit here in my underwear (you’re welcome) and shoot flame out of my fingers at them for thrusting themselves upon and bringing about the destruction of that culture.  If you want the evidence for how they were just pawns in the game of colonial imperialism, check out this book by Takao Abé which spends 234 pages explaining why it was all a bad idea. (Skip to page 61 to see what I mean).

But let’s hold up a minute.  Getting angry at things that happened 4 centuries ago solves nothing, except to prove that I’m not much better than a YouTube commenter.

So what’s the point then?  (Hang in there, the payoff is coming.)

The point is these guys weren’t trekking into the wilds because they wanted to destroy a culture.  They were bringing their God to these people.  They were improving a society.  They were bringing enlightenment to a heathen population who didn’t know what they were missing.  Armed with their faith, they would show the indigenous people of this new world happiness and peace through Godly work. The Church had centuries of missionary work under its belt. This would be easy.

For all the well-meaning they had, they were wrong.

It was not a problem of intention.  They truly wanted to do good by the native people.  It was a problem of bias.  They believed that their religion and pedagogy was what was needed to lift the Huron and Algonquin from their dark place and into paradise.  As it was, the Huron and Algonquin were perfectly happy just the way they were.

This is the challenge we face in the new librarianship.  We can have all the good intentions and well meaning in the world, but if it does not truly jibe with what the society needs (wants), it can actually damage the community. Everything we do is biased by our perspective and is driven by our worldview. Libraries have evolved from large buildings holding books to become a secular temple where knowledge is enlightenment, and librarians are illuminating the paths.

We must not temper our dedication to the mission, but we must ensure that what we do is for the good of our community and for the true betterment of society.

Well, Wadda Ya Know?

Posted by Chad | Posted in IST511, Knowledge, Librarianship | Posted on 16-09-2013


I can still remember taking my first dive into data mining.  My voice had just recovered from incessantly reminding people that there was no year zero, so the millennium doesn’t begin until 2001.  In the information science world, we were all abuzz about this new era of database technology and information management. Business Intelligence wasn’t quite the buzzword it is now, though it wasn’t quite as far away as the horizon. Information was the pathway to knowledge, and knowledge was the currency in an information buzzword knowledge empowered hyperbole-laden buzzword world.  We were discovering knowledge in data.

Or so we thought.

Knowledge, as it turns out, isn’t San Salvador and cannot be “found” or claimed for Spain.  We have entangled the idea of knowledge and it’s creation with the idea of a Eureka moment.  Eureka, for those of you who aren’t fluent in ancient Hellenistic dialects (I’m looking in your direction, Scott), derives from the ancient Greek meaning “I have found it.”  Found what?  Was it in the couch cushions?  Did you happen to see the remote for the surround sound in there?  Bigger question: What were you doing rooting around in my house?

It seems that we have adopted the word from the story of Archimedes of Syracuse having himself a bath after a long day and noticing that as he sank into the tub and gave his rubber duck a squeeze, he noticed the water rise and calculated it as the volume of his mass.  As the synapses fired and linked up in his brain, he leaped from the tub and ran, naked, probably to the horror of many of his fellow Syracusans, shouting “Eureka!”  And that, kids, is how the Mountain Goat Race was born[citation needed].

What the Archimedes myth is telling us is that he found a link between concepts that, when put together, help explain something.  In his case, it was the effect of putting something in water to measure the volume of irregular objects and the question of determining the purity of gold in a crown.  What I find refreshing is the humility shown by regarding himself as an irregular object.




What he didn’t find was knowledge.  The knowledge gained here was created from the assembly of different concepts through an internal conversation.  One cannot find knowledge.  No matter how many coupons you clip, you can’t just run down to Piggly Wiggly and grab a box of knowledge real quick.  Whatever you stepped in out on the lawn was not knowledge (speaking of which, please take off your shoes on the porch, kthx).

Knowledge simply doesn’t exist until its created.

Next time, I’m going to rant good and long about that terrible DIKW pyramid.

Follow Chad on Twitter or friend him on Facebook.  Or don’t.  Or whatever.

Your Mission, Should You Choose to Accept It

Posted by Chad | Posted in IST511, Librarianship | Posted on 09-09-2013


I make it a personal rule to never write or say the phrase “Merriam-Webster defines…” Excepting that particular statement, I have done pretty well so far. Let’s see if we can define a mission statement without resorting to Webster.

A mission statement is intended to be the prime operating directive for an organization, an individual, or a profession.  Simply said, anything that entity does or any task or endeavor it undertakes should be held against the stencil of the mission statement to see if it fits the pattern. The mission statement serves as a barometer against which all of its adherent’s actions are measured.

This is in contrast to a slogan.  A slogan is a clever marketing tool designed to stick in the mind of customers and help create a brand identity.  Often these two concepts are confused and mission statements get bogged down with vapid, ephemeral marketing buzzwords.

I think of it this way: Your slogan is designed to attract and keep customers through brand recognition.  Your mission statement reminds you of what’s important to you, your organization, or your profession.

In the context of librarianship, specifically new librarianship, the mission statement as offered by R. David Lankes states that

The mission of librarians is to improve society through facilitating knowledge creation in their communities


There is a lot of specificity in the vague language there.  As a librarian subscribing to the new librarianship mission statement, when you embark on an endeavor, you have to look at your task through the lens of the mission statement.

Am I improving society?

Am I facilitating knowledge creation?

Do I have a clear understanding of my community?

If the answer to any of those questions is “nay,” I would first say a simple “no” would suffice there, Josiah Bartlett, but I would also ask you to revise the task to address the mission statement.  If the task is absolutely necessary, but still doesn’t mesh with the mission statement, maybe it’s time to rethink the mission statement.

In any event, I’m not entirely sure if this gets us closer to a concrete definition of a mission statement, but I will end with an exercise.

What is Google’s mission statement? (Hint) Do you feel what they do (or seem to do) adheres to that statement? How does it differ from their slogan?

What is Microsoft’s mission statement? (Hint) Do you feel what they do (or seem to do) adheres to that statement?

What is YOUR mission statement?

You're Welcome. Source

Dammit. So close.


Re-purposing Chad

Posted by Chad | Posted in IST511, Knowledge, Librarianship | Posted on 01-09-2013


I had myself a little laugh when giving some thought to what to write. I had to make a choice between creating a new blog for my reflections on librarianship, or re-purposing this site for those posts. I got to the end of that thought and just about slapped myself.  “You idiot,” said the voice in my head. “You have one post.  How can you re-purpose anything that didn’t even have a purpose to begin with?”

It seems that, in addition to a penchant for ending sentences with prepositions, my internal monologue is both honest and right. What I was really thinking was I needed to re-purpose myself. I was in the midst of a change in perspective and was finding a new drive. The last ten to eleven months have been marked by an incredible transition.

And it all boils down to my personal worldview.

What do you think when you think of a librarian?  What about librarianship?  We all have some definition, even if we can’t exactly couch it in concise language, of what is a librarian, what they do, or how they add value to our lives and communities. In your mind, is that all the value they can add?  What are your expectations of librarians and, by extension, a library?  For years I had very specific answers to these questions.  The last ten days have completely unwound that.

My worldview is changing on a daily basis.  I’m quickly learning that I’ve been a librarian for many years. No, I didn’t leer at whisperers from behind stacks of dusty books (well, I did, but not in a library. That was because I’m kind of a creeper.), but rather I managed a place to which people came for information.  I provided an information and content delivery service.

Hey, Chad, what was the material margin on [insert product line here] for [insert time frame here]?

Hey, Chad, what was the scrap percentage for the [insert work center here] for each month?

Hey, Chad, what’s on the cafeteria menu today?

I’m now learning that there were essential professional skills missing from my toolbox that contributed greatly to my frustration and eventual withdrawal from corporate life. I could very easily conjure up “42%, 8.5%, grilled cheese and tomato soup,” and I could complain the next time they asked, citing the fact that, not only did I already provide this information, but I provided the tool to do so. What I did not provide was conversation.  I didn’t offer them context or facilitate the creation of knowledge.  Instead, I relied on the data to be the knowledge as opposed to understanding knowledge as a transient and personal concept. I inadvertently fostered the environment that said “if you need it, have Chad build it.”

It’s time to rearrange the tool chest and make some room.  While I’m at it, I should reshape some of the wrenches to fit different bolts.

Follow Chad on Twitter or friend him on Facebook.  Or don’t.  Or whatever.


Posted by Chad | Posted in Gaming probabilities | Posted on 12-11-2012

Tags: , , , ,


“Dammit,” I muttered under my breath.  That was my fifth game and my fifth consecutive experience with not winning.  The game was QuickDraw, a game which to most people looks like something drunken people do while they wait for their appetizer sampler.  To me, it had become something more.  It was a challenge.

I stood up from my bar side perch and meandered over to the lotto scanner.  The machine was devised as a way for people to check their tickets for winners before handing them over to the bartenders to scan into the exact same machine on the other side of the bar.  The only difference between the two machines was the bartender’s machine was attached to a cash register, so that makes hers infinitely cooler than the one to which I had access.  Apparently people who wear tin foil hats are being scammed by bartenders on a basis consistent enough to merit an alternate infrastructure.

I wasn’t using the machine to avert any potential malfeasance.  Nor was I using the machine to possibly catch a big winner that I may have missed.  I knew beyond a doubt that I had not won.  Instead, I was after absolution.  The dialogue must be two-way.  “Sorry, this ticket is not a winner,” the message read loud and clear on the brilliant bluish green LED.  Such is the life of a gambler.

I crumpled my ticket and returned to my seat.  As I was making the fifteen foot walk-of-lotto-shame back to my pint of Great Lakes Nosferatu, I noticed the Silver Fox was accepting a small stack of cash from the bartender.  I quietly mumbled an expletive that may or may not have accused the Silver Fox of fornicating with a maternal unit and plopped my large frame on the seat at my ale.

I was not at all surprised.  I don’t think I’ve seen the Fox pay for a meal at a bar that has QuickDraw with money he actually brought with him.  What was his system?  I had to figure it out.  Papa needed some mozzarella triangles and it was high time New York Lottery ponied up for some fried food for me.  While my new games were playing, I paid particular attention to the patterns that emerged and came to a pretty interesting realization.

This is the worst game ever.

Let’s look at the numbers.

For those who don’t know, QuickDraw is merely a revised version of Keno.  For each game, you select between one and ten numbers from the numbers one to eighty.  Then twenty numbers are drawn at random from the eighty.  However many numbers you have in common with the drawn numbers determines your payout.  If you played one number, and that number came in, you win $2 for every dollar you wagered.  There’s also the potential for a bonus multiplier, but like the police, overtime pay, and two-ply toilet paper, it’s not guaranteed to be there when you need it, so let’s leave it out to keep things simple.

Quick point to make:  I’m not the first to post about this by any stretch, but this is my approach.  Google “keno odds” to get a list of folks who’ve done a better job of explaining it.

The least picks, or “spots” to the QuickDraw elite, you can make is one, and the smallest wager per game is $1 if you don’t play the bonus.  Couple that with the twenty out of eighty numbers selected and you can easily draw the conclusion that you have a 1 in 4 chance of winning $2.  But what happens if you play a two-spot game?

Now it’s getting interesting.  Might want to put in your order for the wings now.

The numbers are “drawn” one at a time, so it’s easy to see a pattern emerging in the probabilities.  The first number chosen has a 2/80, or 2.50% chance of being one of your numbers.  If that number isn’t one of yours, your odds improve to 2/79, or 2.53% chance.  Not a huge improvement, but it’s better than a turd sandwich. Let’s assume for the sake of argument that 18 numbers have been picked, and neither of your two spots have been hit yet.  That makes the odds of the next number being one of yours 2/62, or 3.23%.  Wait, this is starting to look not so awesome.

So, ball 19 drops and it’s one of your numbers.  There is hope.  Maybe.  Now the odds of that next ball being yours are 1/61, or 1.64%.  Now you have a less than 2% chance of winning $10.  According to the New York Lottery web site, your chances of winning on a 2 spot game are 1/16.63, or a whopping 6.01%

So what does this look like for 3 and 4 spot games?  What about 10?  Percentages and probabilities are interesting creatures in that the rules my very precocious six-year old step daughter knows about math don’t apply and you can’t just willy-nilly add things up.  We have to math a little harder than that to get to the answer.

The pattern we see here is called a hyper-geometric distribution.  This is a common statistical model in gambling because a lot of games of chance deal with predicting outcomes of selection from increasingly diminishing population sets.  This is how Texas Hold’em players can predict how good their hand is relative to others on the table they can’t see.  It’s also how those MIT cats beat the system by counting cards in blackjack.  Most applicably, it’s how to figure out the odds of winning in QuickDraw’s cooler older cousin, Keno.  The basic premise is, if I know what has been drawn, and I know what is left, I can establish the likelihood that the next item selected will buy me a plate of loaded nachos.

And it works like this.

We need a population size.  In the case of our QuickDraw drama, there are 80 numbers on the board from which to choose.  Our population is 80.

We need a sample size.  This means, “how many selections will be made during the game?”  In QuickDraw, the game is played by drawing 20 numbers, so our sample size is 20.

We need a count of success states.  How many numbers are we trying to hit?  This is the number of “spots” you have chosen.  Let’s take 2 spots and try to figure out how they got to that 6.01% chance of getting the bartender to hand you cash.

Then, let our friend Fortuna take over with a little help from a probability mass function, or PMF:


Probability mass function

Not pictured: your soul

How to build a PMF is best left to people smarter than me, but suffice it to say we can take all of the variables we gathered earlier, and plug them into this model and get back our odds of winning like this:

N is the population size, or 80.

m is the number of spots we want to hit, so 2.

n is the number of selections made from the population.  QuickDraw draws 20 numbers, so n = 20.

k is the number of successes, or how many spots we actually hit. Because we have to get both spots to win money, this number is 2.

Some quick replacement and we have this:

PMF with numbers

Still with me? Hello?

Let’s get rid of those polynomials.  My trick for this is following this formula:

The proper way to read this is to shout it at the top of your lungs.

If you run the above numbers through a calculator with a lot of room for numbers, you’ll get this:

OK, math, go home. You’re drunk.

No, actually.  Do the division and you get 0.06012blah blah blah, or as I like to call it, 6.01%.

Neat, huh?

Ok, so now that we’ve barfed out all the math we need to solve the problem, let’s see what it looks like when we chart it for all of the winning combos!


So what’s the takeaway here?  The way I read that is that it is statistically impossible to get rich playing QuickDraw.

But wait a second.  The Silver Fox always seems to win, so what’s the system there.  Well, we have a couple options to explore.  Maybe he just gets really lucky when he’s in the same room as me (though I’ve confirmed with other people that he always wins with them).  Maybe he has beaten the gambler’s fallacy and found numbers that seem to come in.

My next step in the quest is to utilize Python to analyze some QuickDraw data.  Perhaps then we can see if there is a way to get a free burger.

Until then, take it deep, QuickDraw.


Follow Chad on Twitter or friend him on Facebook.  Or don’t.  Or whatever.