A friend of mine at work was joking that if he was given $5 for every time he hears the phrase “big data” he could retire before the end of the year. It has become a term well over used. I also hear it a lot and often times it is not being applied correctly.
Big data often means you have a lot of data. We define that as having more than 250 GB of data to process. It may not sound like much and it isn’t, especially when you take into account all the data a large Fortune 500 has access to. But big is so subjective that I think evolutionary data might be a more accurate phrase to use. It is evolutionary because the data itself is dynamic.
If you really think about it, data never stops. We are constantly collecting data all the time. So why would I look at it and think, hey, let’s just call it big. 15 years ago, 1GB was huge! If you have that much stuff, you have a huge system. Most smartphones today have more than that. So big is always changing. Moreover, the kinds of data we have coming in is always changing. Now, mobile phone data and tablet data are new sources for many companies.
One of the areas we have to think about when developing our big data programs is, what will data look like in a few years? Will our system be able to handle the new sources of data we can think up? We have to think about this when we design because data does evolve and only system that are able to adapt, will survive.