I have not written for a while as I have been launching and scaling a cluster. One of the interesting things about big data is the difference between what I read about these start ups and how it is done in a large established company like I am in. There are some definite differences between the two.
For starters, I work with some people who do want to pretend we are in a start up. Having worked in a start up, I know the difference all too well. It sometimes can take weeks to get on the calendar of someone in a large company, in a start up, I hardly ever had real meetings. We just got up and went and talked to someone and working it out right then and there. Currently, I have days that are books solid with meetings from 9 till 5.
Another big difference is the utter complexity of the job. In a start up, setting up systems was easy; you didn’t have to worry about legacy system, there were none! Today, I am not only dealing with legacy systems but also other teams spinning up new projects that are not fully fleshed out. How I approach, the two is very different. The legacy teams want analysis, reports from consulting companies and change management analysis. The newer groups just want to know if there is an API call and when and how can they tap into it and that’s normally all they want to know.
There is a whole side to building a big data system in a large company that I don’t think you often hear about when reading articles these days. There are many articles on big data right now, I try to read as many as I can and often they talk about the technology. However, there is the entire business aspect that is not discussed and especially how do you set up big data in a way that is smart for a company that is already established. That is actually a rarer skill set than a big data architect skill set and those are fairly rare skills in and of themselves!
The truth is, on the technology side, you can count with your fingers the number of people in most company who know the area. But on the business side, normally it is just one or many two people. If you really want to make big data work for you, you are going to have to poach one of those people just as much as you will need the technology people. Otherwise, you may end up with a great system that has little business direction given to it.