One thing I often run into when talking about big data or evolutionary data, is the fact that most people get hung up on the IT side of it and believe that big data should be in the hands of IT. Now I some what agree but there is a part of it that I am very much in disagreement with. The data analysis of big data is not the domain of IT.
If you work in a company as I do, IT plays a big role in many things. We set up a hadoop cluster in a day. If you have the equipment and a person who knows how to put on the hbase or hive aspect onto hadoop, it is not that difficult to set up. This is IT’s domain. Putting those systems together. Where I run into issues is that IT also wants to control the data analysis part, claiming that is their domain. It really is not.
One of the things I have learned in talking to people who have set up big data projects is that you do not want IT doing the analysis. What does a system admin really know about economic trends, customer trends or marketing trends? You should leave that in the hands of the data scientists and analyst who specialize in those areas. It is not IT’s domain, unless you truly are analyze IT systems with big data, then it makes sense. However, if you are like me and work in a company where you are using big data to identify consumer trends, then IT is not the place to have the analysis done.
I am very adamant about the need for data scientist who is not in IT to be doing the analysis. I am not saying IT cannot do the job but at least in my company, IT is focused on systems. When it comes to data analysis, the business and customer need is the main focus and being able to spot the trends in the data does take a skill set that requires experience in having seen those trends before.
It is important that when setting up a big data system that you do take this into account, IT can own the hardware but think long and hard about who is doing the actual analysis.