The Data Age

Big Data and Living in an Age of Data

The Data Age - Big Data and Living in an Age of Data

The Wall Street Big Data Conversation

The other day I had a great conversation about big data and retail. But in the conversation things went to Wall Street and big data. Being as this person I was talking with has worked in the NYC financial services industry, he knew a thing or two about why things went south.

I always assumed that the people on Wall Street probably hired a lot of math PHd to write these great algorithms and that they probably hired more than just math or computer science majors. Turns out, probably not. Also, I would have hoped they do like I do and at the very least, learn the technology, yeah, that didn’t seem to have happened either.

What ended up happening is, the developers were left to do their own thing with little oversight or understanding by the business side and then things went south. When it comes to big data and predictive analytics, for it to work right and be of the most benefit for the customer base and the business, you need to have the business engaged in the process to the level of really understanding what is going on. I don’t expect the business to get in there and start coding, but the business side should know what the tools are and what are the real capabilities of those tools and if the tools are the best for the job. The business needs to deliver on this and not leave it to IT.

If the business leaves these kinds of decisions to IT then the business owner of the product or project will one day wake up and realize, they have something other than what they thought. They also won’t know how to use the tools anymore and that’s where real problems arise. I myself do my best to learn what my team is using for tools. Now I don’t know how to work varnish, but I know what it does, same with flume. I know what it can do and what it can’t do, so when I work on my next iteration of the product strategy, I know what I really can do and where the gaps are. It is what allows me to deliver faster than other teams and to deliver a more accurate solution to the customer or business problem I am solving.

The social implications of machine learning and big data

As much as I see the benefits for big data and machine learning to help corporations, these technologies, like any technology, have a dark side to them if used incorrectly.  Often times the movie, Minority Report comes up as a model of the darker side.  People being accused of precrimes, tried and convicted before they ever do anything.  In essence, taking a probability and proclaiming it a fact.  When applied to society in reality, this has some bad implications.  However, there is a solution.


I really do think that machine learning and big data are tools that can help us.  We can manage street traffic better, help understand crime and staff police accordingly, or maybe even use it to track the flu better.  There are all great ideas for helping society.  But what makes big data and machine learning something that can go dark is who controls it.  Data in the public sector is often controlled by the government for the government.  We the people, who task the government, often do not get such access to the data.  But with open source tools in machine learning and big data, government actually would be better served if it did hand over the data to people, not companies but make it available to anyone who is a part of that country, state, county or town.


This would help people understand what their data looks like and how it is being used, a big part of the issue people have with data systems is that they have no idea what their information is being used for.  This might also help raise awareness as to what data people give up and this might lead to better personal data management.


When it comes to companies, transparency is always a good policy.  What you use big data for, will help you position yourself.  A company that creates products for consumers that helps consumers is far better positioned than a company that just uses the data for their own profit.  One is using data to benefit consumers; the other is taking data (as is often the case) and uses it for self-interest gains.  Most people will have a problem with the latter.


Data freedom is something that will help everyone, not just companies.  The more people have access to the tools around machine learning and big data, the better we all will be.  People will get a better understanding of the data they have and what it means.  I hope that we can keep our collective heads level and use data for benefit, not self-interested gains only.

Big Data Gets Personal

It is not surprise that social media sites have been playing around with big data.  What gets a lot of people is, just how personal this information can get.


Imagine joining a dating site that uses big data to determine what you are probably lying about, along with everyone else and then using the data to find you a better match.  Or social media can determine if you actually have real friends that meet offline or just chat about sports online.


I am a huge fan of big data, when used properly.  But as anyone, there are some things that I am not interested in seeing big data be used for.  As long as people have the right to manage their data, then it is ok.  But when data is basically taken, that can be a problem.

Why I am a Fan of Big Data

Why am I a fan of big data, because it works.  Many large consumer goods and B2B companies are jumping on the big data bandwagon because of what it can do.  I am not just talking about analysis but the ability to make real work changes on the fly for your organization.  The ability to take information internally and externally and use it in new ways to outsmart competitors.


Imagine the ability to spot a hot trending product before your competitors and then being able to order and promote it long before the wave hit.  You would be in the perfect spot to capitalize and gain a higher margin on that product, before your competitors got into the space and started competing on price.  Your ability to profit would go up.  That’s what big data can do for just about any company.  The more data you have, the better position you are in to capitalize on it to make money and win customers.


What is also great about big data is, it gives you the ability to test and experiment your ideas out.  Allowing for your organization to be more agile in how it approaches the market.  It also helps with personalization of services and creating dynamic demographics that actually mean something.


Big data can even augment management, giving leadership insight that they never thought possible.  And this is where things get interesting.  Because right now, most people in the big data space are working from a technical point of view.  I work from the business side.  Looking at how to apply the data in new ways.  This is where the competitive advantage comes into play, looking at the market and consuming base and understanding how to apply these new concepts to those markets and bases.


Big Data Taking Off

If you haven’t heard of Hadoop, you really should go out and learn about it.  I am always talking about hadoop and what it can do.  This is a great tool for analyzing data and since it is open source, it doesn’t cost an arm and a leg.  But the problem is that, big data has been the domain of IT, the rest of the organization tends to not notice it all that much and this is a big shame.


Big data and the ability to analyze it can help in marketing, sales, finance, logistics, you name it.  It is why I really like hadoop, it is one of those core tools I think everyone should get to know so they can understand, how can they use it in their area of business.