I attended two interesting technology conferences last fortnight, and listened to two energetic speakers. At the SAP Forum Mumbai, we had an invigorated Steve Lucas, Executive Vice President and General Manager, SAP Database & Technology telling the audience why SAP was a better choice for enterprise databases. And at Teradata Universe 2012, the audience listened attentively to Stephen Brobst, Chief Technology Officer, Teradata Corporation, who gave a highly lucid explanation of Big Data. Both spoke about the role of the Data Scientist, and why every organization should have one on its employee roster.
So what does a data scientist do, what academic qualifications do you need to qualify as one, and why is the data scientist going to be much sought after tomorrow? For that matter, is this just a replacement term for ‘data analyst’?
Going by one definition, a data scientist is a person who excels at analyzing large amounts of data, to help a business gain competitive edge.
A example is the call detail record or CDR generated by a telecom company. A data analyst would scan the CDR and then advise the business what services to introduce based on usage trends and the movement of users in the network.
The data scientist is obviously associated with Big Data and analytics. Everyone knows that big data or high volume data was always around; the difficult part is identifying the most useful parts of this data, and establishing patterns and trends, so that one could make smart business decisions.
That’s what a data scientist will do.
“A data scientist need not be a programmer, but he should be able to extract value from the data,” said Teradata’s Stephen Brobst. “But he should be able to use tools to discover new insights from this data.”
A data scientist would have a combination of analytical and statistical skills, and would be adept at machine learning and data mining. And that gifted person would also have experience with algorithms and coding. But the data scientist would also need to have a special ability of explaining business trends to others in the organization.
SAP’s Steve Lucas said a data scientists must have a degree in economics, statistics, or mathematics (or all three preferably).
“They should also have spent a good part of their career with practical applications like analytics, predictive technologies, data management, data modeling, statistics, forecasting etc. I also see that Wall Street seems to be a good source for data scientists to hire from,” said Lucas.
Lucas opines that companies are becoming data hoarders — not knowing what to do with so much data. “It’s a lot of noise and not much signal,” he says. “We need to look for new signals to understand our business.”
Well, it looks like we’ll soon have a new position in corporations, designated chief data officer (CDO) or equivalent. But rather than having a single person responsible for this task, it should really be a team of people. For instance, many organizations have in-house or external social media teams to manage their social media pages; these teams scan posts and send a customer sentiment report back to marketing (customer sentiment analysis).
This team could evolve into a data analysis team, and be led by data scientists. The output from this team would be intelligence reports that advise the business on what directions it needs to take, based on market and customer dynamics.
Well, financial analysts and consultants are doing this on a larger scale today. But I’m suggesting an in-house team, working full time. The task is also becoming easier with the availability of data visualization tools.