From the course: Big Data in the Age of AI

The volume, velocity, and variety of big data

From the course: Big Data in the Age of AI

Start my 1-month free trial

The volume, velocity, and variety of big data

- [Narrator] The one thing we know absolutely for sure about big data is that it's really big. There's a lot of data, it's big. But you know, what counts as big data changes with the times. Once upon a time, a bunch of punch cards might've been big data or back in 1969, this was a massive amount of programming. This is Margaret Hamilton with the code for the Apollo Guidance Computer for which she eventually won the Presidential Medal of Freedom. And then what might be big at one time becomes normal or even small at another. So for example, back in 1992, I got my first computer, an Apple Macintosh Classic II with the optional larger 80 megabyte hard drive. I got all the way through grad school. I wrote my PhD thesis on that computer and now I have a relatively modest MacBook Pro and when I home it's connected to ten terabytes of external storage. That's 125,000 times as much storage as my first computer and truthfully, I've bumped up against the limits. So it's massive by comparison to my 1992 self but it's middling for consumers and really puny by any commercial standards. And so, if we want to think about really, what do we mean by big data given this relative shifting frame of reference, well, one thing that's pretty consistent is the definition that relies on what are called the three Vs of big data. And so, the first V, the first characteristic of big data, is volume, simply meaning there's a lot of it. And the second is velocity, having to do with the speed with which the data arrives. And the third one is the variety or the nature and the format of the data. Take those together and you have what most people would consider big data and I want to talk about each of these a little bit separately. So volume is the most obvious one. This is when you have more data than fits in a computer's RAM, its memory, or maybe you have more data than fits on a single hard drive and you have to use servers and distributed storage. I mean think, for example, about the data on Facebook's 2+ billion users. You can't put that on a single computer. Or the information on Amazon's 120+ million items that they sell online. Keeping track of all that is obviously going to overwhelm any one computer, or even a collection of computers. Next is velocity. You know, a gentle breeze through the trees is nice but a hurricane is a whole other situation. The velocity refers to data that comes in rapidly and changes frequently. So think about, for instance, it's been estimated that nearly 200 million emails are sent each minute of each day. Or that five billion videos are watched on YouTube every day. If you're trying to keep track of this stuff as it happens, it's going to be a completely overwhelming job. And then the third V is variety. Data comes in a lot of different formats and you can have video, photos, audio, you can have GPS coordinates with time and location, you can have social network connections between people, and all of these represent distinct kinds of data from the regular rows and columns of numbers and letters that you would expect to find in, like, a spreadsheet. And all of these require special ways of storing, managing, manipulating, and analyzing the data. Taken together, those usually constitute big data. Another way to think about it is big data is data that is hard to manage. It's the idea like some animals are a little more challenging to deal with and have not been domesticated, like the zebra. Big data's a little like the zebra of the data world. It's simply not easy to work with, not through conventional standards, and you're going to have to be very adaptable to get the value out of that data. Now, I do want to say something historical about the term. This is Google Trends data on the search popularity for the term "big data" on Google, and we have data from 2011 through 2019. And what you can see is there's an obvious peak right there in the middle at October of 2014. That is when the Google searches for "big data" were the most common overall. Now, this doesn't mean that people don't care about big data anymore. You see how it's gone down maybe a third, maybe even 50% since then. Well, there's a saying that a fish, or in this case, a seahorse, would be the last to discover water. That's because it's everywhere around them. It's literally the medium in which they live and move. The same thing is true for big data. While the searches for big data may have declined a little over the few years, not because nobody cares about big data anymore, it is because big data has become the air that we now breathe, the water that we move in. Big data has become the new normal data for use in data science and machine learning and artificial intelligence, and because of that, understanding what big data means, the special challenges that it creates, and how to work with it is as relevant now as it ever has been.

Contents