Featured Story

Contrary to what you heard at the top of the show an algorithm has nothing to do with former Vice President Al Gore or his ability to keep a beat. Algorithms predate the internet, Al Gore, and even computers.

Rather than convolute my story with an etymology lesson that starts in Persia in the year 825, let’s just say an algorithm is a set of rules that precisely defines a sequence of operations. Wikipedia

Let’s further put that in context for today: an algorithm is how a computer processes data. Come with me back in time to the late 1990’s. There were young entrepreneurs who saw great potential in the newly commercialized internet. These youthful visionaries knew the world could benefit by this linking of computers and the digitizing of all the world’s knowledge, but how could you make such a large catalog of information useful?

Companies like Yahoo! had people that scoured the web and built directories, like the yellow pages, to help people find what they were looking for. It had real live humans deciding what content others saw, which meant moderation was built in, but so was censorship. Leaving the decision up to human beings wasn’t really the best way to rank and present the internet, especially as it expanded exponentially. Besides that, it was expensive and time consuming. Wayback Machine – Yahoo | Wayback Machine – The Mining Company

Then along came a startup called Google. Google used what’s called a spider to crawl through the web automatically, creating an index of all the sites it could find. If all Google did was read every page on the internet and build an index that contained every word on every page, it wouldn’t make for a very useful search engine. If you searched for the word President, Google would return every webpage that had the word president on it. So, Google created a set of rules that would identify keywords on a page. This is an algorithm.

Now, in our example here, we see how Google identifies keywords to figure out where in the index a webpage would go, but now we encounter the problem of how to rank those pages. So, Google then identifies the number of other websites that link to that page with that keyword. The more sites and pages that point to a page on president, the higher that page is in the search results.

Cool, now we have a basic understanding of how Google works. All the same basic principles are applied to recommendation algorithms as well. Amazon keeps a shopping history for every user that visits its site. Let’s say you’re looking to invest in a new, red, Swingline-brand stapler. Amazon looks in its vast data stores and sees that out of 150,000 people who bought this item, 1,000 of them also bought a 10 pack of sticky notes. Now you see a link to that pack of sticky notes because you want a new stapler.

This all seems fine, right? So, why do algorithms get such a bad name? Well, once you understand how they work it’s easier to manipulate them.

Just from this brief description you know that by loading a page with keywords for Google to index and by getting lots of other pages to link to that page, you can quickly move up in Google’s search rankings. Using this knowledge, we pick keywords we know are being searched regularly and add those to our page, even if our page has nothing to do with those keywords.

Then we place links from regularly indexed sites, like social media, to the content we want to promote, and the search and recommendation algorithms do their magic promoting our content to the masses.

Now, we have a simple method for promoting whatever we want, globally. This is how word of the New Zealand massacre spread so quickly across the internet. A few malicious people creating content in the right places at the right times, and we have a virtual wildfire of hatred.

But it didn’t start with the massacre of innocent Muslims in Christchurch. There are countless stories of these algorithms being used for evil. YouTube’s recommendation algorithm has been called into question for helping conspiracy theorists and child abusers promote their content. Twitter and Facebook pushed video of a reporter shooting his former colleagues on live television in 2015. Pinterest’s algorithm is accused of driving a 14-year-old girl over the edge to commit suicide by emailing her images of self-harm.

So, how can it be fixed?

The answer is complicated and not one-size-fits-all. Swing too far into human moderation or keyword bans and the critics cry censorship. Change nothing and our existing problems are exacerbated by copycats.

The algorithm and its brawnier, brainier cousin Artificial Intelligence are creeping into more places in the world to make decisions for humans. Creating ethical automated decision-making systems are critical to our future.

I’ve included a bit more reading in the show notes if you’d like to get an idea of some approaches to solving this issue.

Alright, Let’s wrap up this episode with the good news.

Google’s Clever Plan to Stop Aspiring ISIS Recruits

Up Next: A Better Recommendation System

The Mass Shooting in New Zealand was Designed to Spread on Social Media

The People Trying to Make Internet Recommendations Less Toxic

New Zealand ISPs are Blocking Sites that Do Not Remove Christchurch Shooting Video

Researchers Built an ‘Online Lie Detector.’ Honestly, That Could Be a Problem

People Want to Know About Algorithms – but Not Too Much