Chapter 15. Making the World a Little Smaller

Data Compression and You

So, we’ve finally reached the end of this book, which details algorithms that began in the 1960s and have had a clear and pronounced influence on computing and technology to the present time. But where are things going from here? Many engineers will happily throw up their hands and say that compression is a solved problem, or that it is not significant to their skill set. The truth is that over the next few decades, data compression will remain as important as it was back then. So, it’s perhaps worthwhile to take a look at how interconnected data compression is with you, your company, and the future of technology.

Data Compression and the Bottom Line

When it comes down to you and your company, it’s all about the money. Hard, cold, fast cash. Data compression is so interwoven to your bottom line that companies who can get this part of their tech stack right save when it comes to the following things:

  • User acquisition and retention

  • Running costs

  • Planning ahead

Let’s dig in a bit.

User Acquisition and Retention

There is a direct relationship between how fast web pages load and conversion rates. If pages don’t load fast enough, users will abandon whatever they are doing, including buying your products. On the other hand, the slimmer the data you pack into your product page, the faster it will load, and the more likely users will buy and return in the future.

The average statistics show that one in four users will abandon a mobile page load that takes longer than four seconds. It’s worth testing your pages, because this could mean massive impact on the bottom line, just from page loads. If you need convincing, here are some real-life stories:

  • Amazon has shown that for every 100 milliseconds of slowdown, it experienced a 1% drop in revenue. Or, to take a different turn on stats, a giant, such as Amazon, would suffer an annual loss of $1.6 billion if its pages loaded just 1 second slower. On the other hand, Amazon increased revenue by 1% for every 100 milliseconds of improved page load speed.

  • Walmart’s latest reports show that for every 1 second of page load improvement, it experienced up to a 2% increase in conversions. For every 100 milliseconds of improvement, it grew incremental revenue by up to 1%.

  • Shopzilla sped up its average page load time from 6 seconds to 1.2 seconds and increased revenue by 12%, as well as page views by 25%.

  • Small sites, such as AutoAnything, cut their load time in half and saw revenue grow by 13%.

  • President Obama’s 2012 presidential campaign based its entire fundraising success on making its website load instantly.

In addition, better site reviews, more word-of-mouth downloads, and better retention can probably save your marketing departments lots of money.

Running Costs

The content of your websites has to be stored somewhere, and that somewhere is usually a bunch of hard drives in a big fat cloud. Hard drives cost money. Shipping data to and from the cloud costs money. Renting (or building and running) data center space and bandwidth costs money. And even though cloud technology is normalizing and costs are coming down significantly, bandwidth and storage remain challenging financial problems for big companies.

In 2015, Netflix announced that it was going to begin modifying the compression technology for its video streams; adjusting which algorithm was used on what video depending on the noisiness of the content itself. This move was projected to save the company massive amounts in terms of bandwidth costs, and allowed it to accommodate to the performance of specific devices.

Also in 2015, Facebook published details on how it serves preview images, which are only 200 bytes each. Considering how many images the social media network sends around on a daily basis, this was a huge win for them, especially for users on 2G devices.

You don’t need an advanced degree in mathematics to understand this one: smaller data results in smaller outbound costs, less inbound costs, and fewer storage costs.

Planning Ahead

Web pages are getting bigger. An independent website, HTTPArchive, has been downloading, cataloging, and gathering statistics on the top 1,000 web pages since about 2011. One of the interesting stats gathered during analysis is the average size of these websites. This statistic sums the bytes required to display the page information, including JavaScript, HTML, CSS, JPG, and video. This page size reportedly grew by 24% in 2013, landing at about 1.5 MB per page, and in 2015, it hit 2 MB in size.

One reason why websites are growing larger is because they contain more images, and those images are increasing in size. In addition, there is more code as websites are becoming more complex overall. This problem is becoming so bad for users on 2G, that big companies like Google launched a new framework called Accelerated Mobile Pages, which specifically targets sites to reduce their dependency graph, image sizes, and basically tries to offer leaner, faster content to users on bad bandwidth.

Making Your Users’ Lives a Little More Magical and Less Expensive

Mobile devices make up a surprisingly important part of modern life. It’s madness to think that the user experience of that hinges so critically on how fast pictures of cats can be streamed from a server to a phone.

And while you’re trying to cut costs for outbound data, users are trying to do the same for inbound data. Let’s be clear here: users pay for everything. And most of them pay for data, by the megabyte, and at outrageous rates.

mobiForge did a nice little analysis in 2013 that showed the costs for connecting to data on high-tariff data plans. At that time, AT&T’s roaming rates charged $12 per megabyte. Users who browsed to microsoft.com ended up paying AT&T $17.50 each time.

But it’s not just the money cost. Consider battery overhead as well. Users who are on slower connections take longer to download content, which means that their battery is on longer for the same piece of content than on a faster connection. The result is that users on slower connections drain batteries faster.

Data compression is directly related to all these issues. Smaller assets means less time to download on worse connections, and less battery drain. The end result? Faster cat pictures to your users.

Thinking About What’s Next in Technology

If you’re lucky enough to live in a country with amazingly well-adopted connectivity, congrats to you! Chances are you’re paying pretty sane rates for a pretty good mobile connection. Worldwide, however, this isn’t the case. More humans on the planet have bad connections than good ones. Looking forward, the future of mobile computing is going to be defined by the next five billion humans who are about to jump online for the first time, and the quality of their mobile networks.

The Next Five Billion Users

There are 7.4 billion humans on the planet right now. About 2 billion of them are currently connected to the Internet. Most of the rest live in countries that are rapidly expanding their connectivity. This means that your largest growth potential is in the emerging markets of Asia and Africa. The advancement in mobile computing technology over the last decade means that the next billion humans will be coming online, for the first time, from a mobile phone rather than a desktop or laptop device.

In their book, New Digital Age (Penguin Random House), authors Eric Schmidt and Jared Cohen lay out the topic well:

There are already more than 650 million mobile phone users in Africa, and close to 3 billion across Asia. The majority of these people are using basic-feature phones—voice calls and text messages only—because the cost of data service in their countries is often prohibitively expensive, so that even those who can buy web-enabled phones or smartphones cannot use them affordably. This will change, and when it does, the smartphone revolution will profoundly benefit these populations.

Mobile Networks

Building networks for these high-saturation-potential countries will not be cheap. If you consider that Verizon had to shell out $50 billion to simply upgrade its network to 4G, the cost of building an entirely new network for such a massive population must be astronomical. Such costs will often be inflated due to government entanglement (as is common for government and telecom companies), and this generally will result in passing off all costs to the end customer.

The world has been seeing great improvement in network speeds over the past few years. However, it’s important to see how this improvement is not uniform in terms of numbers, or geolocation. Google Analytics has a set of fantastic charts showing the trends in connectivity, worldwide. It’s easy to see that the idea of improvement is not homogeneous. For instance, China saw an 8% increase in median page load time for desktop computers (things got slower), whereas their mobile performance time decreased 33% (things got faster), but still landing at >3.5 seconds load time. This is a pretty big number, considering 42% of their 1.53 billion population is online.

The short form is this: mobile networks will continue to grind their way to increase speed, slowly, unevenly, and at great expense. If you’re waiting for the mobile web to suddenly get faster, you might need to find a more comfortable chair to wait around in.

For example, a 2G network has ~0.021 MB/sec transfer speed, whereas GZIP can compress 61 MB/sec. Even a reduction of 10 times in GZIP speed would still access a single megabyte faster than the network could transfer it. Colt’s analysis of these data points suggests that it would be cheaper to invest in a better compressing-slower decompression codec than it would be to invest millions of dollars upgrading the network hardware.

...Starting Now

The picture painted here should be immediately clear. The next great computing revolution will come from areas of massive populations that tend to be skewed toward the entry-level economic ladder, meaning that their choices in mobile hardware and cell provider will be skewed toward slower hardware and slower networks.

But there will be the same demand for fast data, and there will be the same competition between developers for the attention of those users. The trends in mobile computing continue, and average app data costs continue to skyrocket; however, these individuals are playing a catch-up game in which they are already far behind. The user cost to send down 25 thumbnails or load a page of news headlines with images might be too much and too slow for these users, causing them to abandon slow experiences for much faster ones.

This problem is so important in 2015 that even large developers like Facebook have rolled out slimmed down, 2G-friendly versions of their experience, just to reduce barriers for acquiring users in this budding mobile market.

And as a developer, you can’t really control the networks, and you can’t control the hardware. But you can control the data, and with that, you can do a great amount to ensure that it is compressed aggressively so that it arrives to users with a speed and quality that lets them have a valid computing experience and remain faithful to your application over time.

What are you waiting for?