Teradata released the third State of Business Intelligence Survey today, which found that interest in big data analytics careers is on the rise. However, there is a significant gap between the number of students interested in these opportunities and the amount of hirable talent in the pipeline.

Working as a data scientist (or related position) requires taking a large data set, modeling it, and gleaning insights. As the hype surrounding big data continues, the demand for people with these skill sets is increasing, along with salaries. But just like top tier athletics, not everyone has the requisite talent or ability to follow a career in this field and there is a major, global shortage of data scientists. And despite the fact that two out of three students surveyed agree or strongly agree that there are job opportunities for them in data-savvy careers, only 16 percent are actually considering careers in this area.

Accordingly, employers still struggle to find potential hires. Analytics hiring managers reported that their largest needs for recent graduates are for IT or systems analytics (35%), program developers (32%), data managers (30%), and business analytics (22%). In the survey, one-third of employers identified a lack of experience as their greatest challenge, followed by insufficient business skills, insufficient technical skills, and a general lack of candidates. A lack of communication skills is also an issue.

Preparing students for real-world employment is partly the responsibility of the teachers. Forty-one percent of professors reported an increase in the number of BI or analytics courses at their university, but they also identified six areas where businesses can help them meet the challenges of creating qualified employees: providing large data sets, suitable cases, staying current with practices, technical support and training, realistic and meaningful experiences, and access to contemporary enterprise software.

The tech industry is a competitive job market and companies are always looking for qualified, experienced data scientists who combine technical expertise with creative thinking and communication skills. The responsibility does not lay only on the students, but on the universities, teachers, and companies that turn them from students into successful employees. A degree alone is not enough.

The survey was conducted by Barbara Wixom, an associate professor of commerce at University of Virginia’s McIntire School for Commerce and a research affiliate at MIT’s Sloan School of management. The study was sponsored by the Teradata University Network, which includes 3,400 faculty members, 1,600 universities, and thousands of students. TUN is a free web portal for students and teachers working in this field.

Photo Credit: Shutterstock

Everybody and their mother is on the hunt for a data scientist, dubbed the “sexiest job of the 21st century,” by the Harvard Business Review.

To train the next generation, New York University (NYU) has launched a data science and statistics initiative, which is intended to drive breakthrough research in other programs, including medicine and technology. The new website for the program cites research from McKinsey that projects the U.S. alone will need 140,000 to 190,000 people with deep analytic skills by 2018 as well as 1.5 million managers and analysts for “big data” jobs.

NYU has created a Center for Data Science composed of faculty from across the university and will offer a variety of graduate-degree programs to its students. A two-year master’s program will begin accepting applications this month and will commence this fall. In addition, the university plans to establish a doctoral program in the field.

“Data Science is becoming a necessary tool to answer some of the big scientific questions and technological challenges of our times,” said Gerard Ben Arous, director of NYU’s Courant Institute of Mathematical Sciences, which will house the Center for Data Science.

Machine learning expert Yann LeCun will lead the initiative. He said in a statement that we can use a variety of data sources to answer questions like, “How does the brain work?” The new program will focus on managing a vast volume of data, but also on exploiting it in ways that can yield myriad “social and economic benefits,” according to the statement.

As we reported when eBay opened its New York office, the city’s startups and tech companies have a keen interest in analytics and “big data.”

Bloomberg Ventures’ Matt Turck, who also runs the Data Business meetup in New York, said with initiatives like these, the city is in a position to become a “big data mecca.” “[But] let’s just hope that all the graduates don’t get scooped up by hedge funds or the Facebooks of the world,” he warned. 

Jacob Brody, a New York based partner at seed fund MESA+, said this is a first step in addressing the talent shortage for data scientists in New York. “So many problems currently being solved by companies in New York require expertise in Big Data,”  he said.

“The program will help startups somewhat in the short term, but it will help longer term when alumni leave big jobs in a few years to join startups or start their own,” said Brody, who is also a former tech reporter for VentureBeat. “It’s about creating an ecosystem, and this is a start,” he explained. 

NYU isn’t the first university to offer training to budding data scientists: Syracuse University has a certificate in advanced study in data science, Columbia University recently launched its own Data Science Institute, and Stanford University has classes in data mining. Check out this fairly comprehensive list.

Graduation hats image via Shutterstock 

Sept. 9 - 10, 2013
San Francisco, CA

Early Bird Tickets on Sale

The appeal of “big data” has made data scientists a hot item, commanding respect and income worthy of supermodels or football stars.

But perhaps a better term would be “data artists,” befitting the artfulness that goes into interpreting and illustrating big datasets. Perhaps these scientists are not the Einsteins and Edisons but the Van Goghs and Picassos of the big data revolution.

The current revolution in big data is inherent in the name: datasets trending to petabyte size (that’s 1 million GBs) or more. And it’s all about complexity: unstructured data, multiple datasets, and structured data, all jumbled up. (Even Facebook is looking to hire a data scientist.) So big data requires big brains and massive computing brawn.

But it requires more, too, much more, according to author and data scientist Bill Franks.

He’s the author of Taming the Big Data Tidal Wave and the chief analytics officer for Teradata, the data warehousing and analytics company.

According to Franks, even more than size and complexity, big data is increasingly about creativity: finding interesting patterns and following them down the rabbit hole.

Sometimes, nothing turns up. Other times, data artists discover wonderland. A very profitable wonderland.

What is big data?

“You will not find an agreed-upon definition for ‘big data,’” Franks says. “I’ve had vigorous debates about that.”

The general guidelines he offers are a combination of factors: volume, size, variety, and complexity. One of the most interesting and relevant is complexity. Big data doesn’t always fit into neat columns. Some data works in standard traditional SQL (structured query language) databases, and some does not.

For example, web analytics data from server logs is fairly structured, but tweets or Instagram photos are not. Sensor data reporting customer traffic in a retail store may be neat and clean, but data artists will probably want to match it up against an entirely different dataset: sales.

And then factor in weather data, and time of year, and the price of gasoline. Probably not the position of Pluto, but in big data you never really know what bit of information might prove to be critical.

‘Data artists’ … really?

“There have always been individuals who take a company’s data and find interesting patterns using data-mining and predictive analytics,” says Franks. “The technical abilities to do so are now table stakes to be successful in a business environment.”

But now in the era of big data, these technical abilities need to be married with softer skills: commitment, and especially creativity.

So the key to being a good data artist, Franks suggests, is creativity.

“There’s no need to be creative if you have the exact data for the exact problem … but the reality is that you have to make assumptions, deal with inconsistencies, and then choose a model that may not be perfect.”

That’s what takes creativity, and the best data scientists are, in fact, artists.

The tools of their trade, of course, are not palette, brush, and canvas. And increasingly they are not traditional databases and SQL, but tools like Hadoop and MapReduce, tools for processing huge amounts of data across large number of machines. Or hand-coding spur-of-the-moment solutions in Java or other languages.

And the results?

The business results are clear. Better products, better customer service, better asset utilization: All results of using big data.

Walmart’s rise over the last few decades was largely a technology-powered revolution in retail. Google’s capability to be Google is entirely dependent on the company’s success to acquire and manage massive sets of data. The same is true of Facebook’s capability to keep almost a billion people connected.

But it’s not just about the big names in the technology world. Airlines scheduling flights and setting prices use big data to maximize profitability. Banks stocking ATMs with cash manage datasets on seasons and events in addition to their own customer data. And Teradata now has 36 members of its “Petabyte Club” — clients who have at least one but often dozens or scores of petabytes of data — up from just five in 2007.

“Big data is so much in the news lately that some might argue it’s overhyped,” Franks says. “But so many people are getting excited … they can’t all be wrong. And the new value that can be driven is just really cool.”

Franks sums it the job of a data artist this way:

“You need to use imperfect data and imperfect methods in an insufficient timeframe to get enough data to make a good business decision.”

The end result? Green on the balance sheet.

Image credit: Atypeek Design/ShutterStock, Bruce Rolf/ShutterStock, Val Lawless/ShutterStock, AnatolyM/ShutterStock, medul.la via photo pin cc

Finally, a clear description of what it means to be a data scientist! Facebook is seeking a “data scientist” to join the team of a dozen researchers in the Menlo Park HQ tasked with uncovering insights from the most extensive data set on human relationships ever assembled. The job is lumped into the general category of “software engineering careers,” but is a far more product-driven role than you might expect.

According to the job description, the ideal candidate is a strong communicator with an appreciation for products, someone with expertise in the full gamut of technical skills. To qualify, you’ll need:

• Fluency with at least one scripting language (Python or PHP)
• Experience working with large data sets. Bonus points if you can work with distributed computing tools, such as Map/Reduce, Hadoop, Hive, etc.
• Familiarity with relational databases and SQL
• A strong passion for empirical research and for answering hard questions with data
• Ability to communicate findings to product managers and engineers
• Ability to answer and identify product questions using appropriate statistical techniques on available data

Who is this wunderkind? This data hero — or data artist? It’s certainly not your typical low-level developer or analyst, many of whom have taken to passing themselves off as data scientists. You can’t blame them — it’s the tech industry’s sexy new profession. From LinkedIn to Netflix, every tech company worth its salt has developed a data-dedicated team. According to a rather brilliant feature in Technology Review, Facebook is set to double its data team in the next 12 months.

The right candidate for this job will be willing and able to find Facebook new ways to make money from personal information, whether it’s our family photos, conversations, life achievements, or other tidbits of data. Under the company’s top data scientist Cameron Marlow, the team’s goal is to find patterns among the data that the product team can make use of. For instance, data scientists helped Facebook identify users you may know but haven’t “friended.” The team is working with terabytes of potentially valuable data that is lodged in the company’s servers.

There still isn’t much consensus on what it actually means to be a data scientist, so I was relieved to stumble upon this opening, which sheds some light. Still, even Facebook’s HR department is going in a bit blind. Even Marlow prefers to go by the academic-sounding term of “chief sociologist.”

The job description says the ideal candidate will have a Ph.D or masters degree. Technology Review reported that several members of the team are trained in sociology or social psychology. In their free time, they can use Facebook’s raw data to uncover patterns in human behavior, and publish the results in academic journals.

The salary isn’t specified, but according to a recent study, it averages at $55,000 for a data analyst to $132,000 for a vice president of analytics. This is Facebook, so expect to earn significantly more than that.

Click here to read more about what it means to be a data scientist.

Often, the clues that lie in big data can be the key to an enterprise’s future success, offering insights to optimize supply chains, uncover consumer behavior patterns, identify traffic and energy patterns, and much more. Rather than be a burden, all of this newfound knowledge can allow businesses to make effective long-term and real-time business decisions.

How can businesses grapple with the task of not only managing this data explosion but more importantly, extracting value from it and turning it into an asset? How do you remove the noise from this high velocity, highly variable data and discover key insights while they are still relevant?

That’s a question I and and other panelists broached at VentureBeat’s recent CloudBeat session on “The New Social Layer.” The session examined how companies are using social media to gain insight into what customers think.

Those of us on the panel reached a surprising agreement: Much of the big data tools and technology needed for businesses to begin exploring this data is already in place. So why are so few companies making use of this data? What they’re missing is the right culture and people.

What struck me about this agreement was that we had executives from a startup, a major media company, an SMB and a Fortune 500 firm all agreeing on this catalyst to succeed with big data.

Organizations will always need individuals with strong traditional data management skills, but today business and IT staff need to collaborate in order for enterprises and SMBs to evolve. The individuals who will be highly sought by organizations will be people who can serve as change agents, pushing organizational collaboration and information innovation. I have been speaking extensively this year about the need for data scientists and big data skills and plan on continuing to do so.

At IBM, part of my job is to help develop new big data products and partner with customers to figure out the cultural shift needed. One company I’ve worked with, Vestas Wind Systems, a Danish energy company, is using our big data analytics technology to determine the best locations for wind turbine placement. Vestas is able to analyze everything from weather reports, tidal phases and satellite images to geospatial and sensor data, and to use that analysis to select locations that will generate optimal return on investment. This type of analysis used to take weeks but can now be done in less than an hour. Once the turbine has been sited and assembled, engineers can use these big data solutions to predict turbine performance and determine the best times to schedule maintenance.

That’s just one example of what is possible today. In the future, advanced and predictive analytics will help business users gain powerful insights into data so they can forecast future scenarios. One form of predictive analytics currently underway is analyzing consumer sentiment, a new phenomenon where companies can mine the countless insights gained from streaming human interactions across social networks and the Internet to establish consumer behavior or corporate best practices.

Dutch entertainment company RTL Nederland, for example, is using predictive analytics to mine viewer sentiment and feedback from blogs and social media sites, allowing the company to make real-time adjustments to improve viewer satisfaction for shows like “The X Factor” and “So You Think You Can Dance.” For “The X Factor” alone, RTL analyzed over 70,000 online conversations, enabling it to address viewer likes and dislikes so it could effectively tweak its programming.

Lastly, whether we were aware of it at the time or not, many of us witnessed big data analytics in action on the TV show Jeopardy! this past February, when IBM’s Watson, faced off against Jeopardy champions Brad Rutter and Ken Jennings. Zipping through category after category (with a few stumbles along the way), Watson demonstrated its ability to search roughly 200 million pages of text and buzz in once it reached a “confidence threshold.” Being able to understand Natural Language – the way humans speak to each other – including puns, double entendres and pop culture references, was a groundbreaking achievement for technology. Being able to do it all in about three seconds blew the human competition away. The reach of this exciting new big data analytics technology will be broad and will certainly play a role in many areas of our future.

Yes, the amount of big data that exists can be intimidating, but it’s exciting to envision the countless possibilities it can achieve for businesses. In order to capitalize on big data, businesses need to recast their views of their data problem. Instead of looking for quick fixes to handle the vast amounts of incoming data, they should look at it as an opportunity to make a significant impact on their business. If they have people on staff dedicated to determining the types of insights they’d like to uncover from their data, the technology exists to help them find it.

Anjul Bhambhri is vice president of Big Data Products at IBM.

[Image via Eric Isselée/Shutterstock]

What does a data scientist do?

DJ Patil led LinkedIn’s data science team and is now the Data Scientist in residence at Greylock Partners. His free ebook “Building Data Science Teams” provides an excellent introduction to the basic areas of data science and how to build a team.

For startups, the most relevant applications of data science are probably decision science and product and marketing analytics. Decision science, as the name implies, allows you to identify and monitor key metrics for your business and answer strategic questions like “Which country should we expand into next?” or “What is the impact on the business if we lose this client?”. Google’s data science team even drives its HR policies.

Product analytics covers anything from how users are reacting to new features to developing standalone data products. LinkedIn’s “People you may know” feature and Amazon’s recommendation system are data-driven features that attempt to keep users on the site longer or drive more sales.

Using data to showcase or market a product is the domain of marketing analytics. One of the best known examples is okCupid’s okTrends blog, which features posts like “The case for an older woman” or “The 4 big myths of profile photos”. The blog drives massive traffic to the site and is regularly covered in the media.

Who are the data scientists?

Since data science is a new area, practitioners often migrate from other fields. You may see maths, statistics, machine learning or computer science on their resumes or a data-intensive field like meteorology. Data scientists want to be of central importance to a business, especially when it’s a startup. The best data scientists are both intensely curious and great communicators. They answer important questions and tell good stories using data.

What is data infrastructure? 

Data scientists need specialized tools to manage and process large amounts of data. The minimum you need to get started is simple data access, usually via a database. Larger-scale or less uniform data may require a tool like Hadoop, an open source platform for distributed processing of large data sets across clusters of computers, as well as someone with the technical expertise to use it. Data stores like Cassandra are designed to perform well on very large datasets. These are some of the most commonly used tools, but there are many others for tasks such as streaming data collection, querying non-relational databases and job scheduling.

When do you need to hire a data scientist?

VentureBeat talked to data scientist Cathy O’Neil, who herself works for a startup (Intent Media), about when you need to hire a data scientist. If your data volume is growing, you don’t know if you are seeing noise or information in your data, or in general, if you are not running your business sufficiently quantitatively, then you may need to consider hiring. The following is our brief Q&A with O’Neil:

VentureBeat: How much data is enough for a startup to justify hiring a data scientist?

Cathy O’Neil: Too much to fit on an Excel spreadsheet. And it’s not just how much, it’s really about how high quality the data is; the best is for it to be clean and for it to not be public, or at least not generally used for the purpose that your business uses it for.

VB: Can a data scientist help you when your startup is still trying to find a product-market fit?

CO: Yes, in various ways. If your business model is itself quantitative, say you are trying to make money on some inefficiency of some market, a data scientist can help quantify the potential for the business. This is traditionally the job of a business person but can get pretty quantitative so is best done in cooperation with an analyst.

More critically, a data scientist can estimate how much and what kind of data will be needed for a given business model. Depending on the business, getting access to high quality data can be a big expense so this is important.

VB: When you won’t have “expected numbers” based on past performance and maybe your business model isn’t fixed yet, how can you make forecasts?

CO: This is tricky and a data scientist can do things like using proxy data, combined with domain expertise, to try to estimate things. In other words, if you have information about how other companies in related fields are doing, you can adjust their numbers to fit to yours, with quantitative estimates of how correlated the fields are.

VB: Data scientists sometimes use expensive proprietary software but startups don’t tend to have a lot of cash. Can they also use open source or low-cost tools?

CO: Yes, but you either have to hire a software engineer who has some data science skills, or you may need to have engineers spend quite a bit of time setting things up for the data scientists. In general data scientists don’t have the kind of technical skills that engineers do, and the engineers may need to develop some tools in-house to make things easy for the data scientist to do their thing.

VB: Is it ever useful to hire a data scientist part-time or for a particular project?

CO: Sure, you can advertise on LinkedIn for a data science consultant. This may be a good idea at the beginning of your business to be sure you’re collecting the right kind and enough good data to follow your business plan.

VB: Startup founders are often software developers, but few have expertise in data science. Any advice for hiring?

CO: My advice is, don’t be dismissive when data scientists are bad at programming, because that’s not what they’ve been focusing on. Make sure they know what they need to know about analytics, and figure out how to give them enough support to do those things.

VB: Any other advice on handling data scientists?

CO: Make sure you know what’s sexy about your data.