Data science demands elastic infrastructure 04-19

Data science demands elastic infrastructure

Those companies that try to run big data projects in data centers may be setting themselves up for failure. Matt Asay explains. 

As companies struggle to make sense of their increasingly big data, they're laboring to figure out the morass of technologies necessary to become successful. However, many will remain stymied, because they keep trying to fit a necessarily fluid process of asking questions of one's data with outmoded, rigid data infrastructure.

Or as Amazon Web Services (AWS) data science chief Matt Wood tells it, they need the cloud.

While the cloud isn't a panacea, its elasticity may well prove to be the essential ingredient to big data success.

How much cloud do I need?

The problem with trying to run big data projects within a data center revolves around rigidity. As Matt Wood told me in a recent interview, this problem "is not so much about absolute scale of data but rather relative scale of data."

In other words, as a company's data volume takes a step function up or down, enterprise infrastructure can't keep up. In his words, "Customers will tool for the scale they're currently experiencing," which is great... until it's not.

In a separate conversation, he elaborates:

"Those that go out and buy expensive infrastructure find that the problem scope and domain shift really quickly. By the time they get around to answering the original question, the business has moved on. You need an environment that is flexible and allows you to quickly respond to changing big data requirements. Your resource mix is continually evolving--if you buy infrastructure, it's almost immediately irrelevant to your business because it's frozen in time. It's solving a problem you may not have or care about any more."

Success in big data depends upon iteration, upon experimentation as you try to figure out the right questions to ask and the best way to answer them. This is hard when dealing with a calcified infrastructure.

A eulogy for the data center?

Of course, it's not quite so simple as "all cloud, all the time."

Data, it would seem, has to obey fundamental laws of gravity, as Basho CTO Dave McCrory told TechRepublic in an interview:

"Big data workloads will live in large data centers where they are most advantaged. Why will they live in specific places? Because data attracts data.

"If I already have a large quantity of data in a specific cloud, I'm going to be inclined to store additional quantities of large data in the same place. As I do this and add workloads that interact with this data, more data will be created."

Over time, enterprises will look to the public cloud for all the reasons Wood describes, but legacy data is unlikely to make the migration. There's simply no reason to try to house old data in new infrastructure. Not most of the time.

But some companies will find that they're more comfortable with existing data centers and will eschew the cloud. I'm not talking about hide-bound enterprise curmudgeons that shout "Phooey!" every time AWS is mentioned, either. No, sometimes the most data center-centric of companies will be the innovators like Etsy.

As Etsy CTO Kellan Elliott-McCrea informed TechRepublic, once Etsy had "gained confidence" in its ability to manage its Hadoop clusters (and other technology), they brought them in-house, netting a 10X increase in utilization and "very real cost savings."

Nor is Etsy alone. Other new-school web companies like Twitter have opted to run their own data centers, finding that this gives them greater control over their data.

You're no Twitter

As highly as you may estimate your abilities, the reality is that you're probably not an Etsy, Twitter, or Google. As painful as it is to say it, most of us are average. By definition.

This is what Microsoft's great genius was: rather than cater to the Übermensch of IT, Microsoft lowered the bar to becoming productive as a system administrator, developer, etc. In the process, Microsoft banked billions in profits, helping make a good sysadmin better or a decent developer good.

Regardless, all enterprises need to establish infrastructure that helps them to iterate. Some, like Etsy, may have figured out how to do this in their data centers--but for most of us, most of the time, Wood's advice rings true: "You need an environment that is flexible and allows you to quickly respond to changing big data requirements."

In other words, odds are that you're going to need the cloud.

View at the original source

How cloud adoption trends are driven by strategic imperatives 12-24

How cloud adoption trends are driven by strategic imperatives.

Picture credit: iStockPhoto

As this year comes to a close, let's take one last look at the most pervasive cloud computing trends - including increased usage across the different cloud service models, the key business drivers and the impact of agile innovation strategies.

Cloud computing adoption has matured, with 69 percent of survey respondents stating that at least a portion of their computing infrastructure is in the cloud. However, 56 percent of companies are still identifying IT operations that could potentially move to the cloud, according to the latest market study by IDG Enterprise.

Survey respondents believe that business technology is a game changer, and cloud solutions are providing advantages from increasing IT agility (63 percent), IT innovation (61 percent) and improving the ability to access critical business data and digital service applications (58 percent).

Removing the barriers to cloud adoption

That said, IT leaders perspective on barriers to bringing these advantages to fruition differs from their Line of Business (LOB) counterparts. There is agreement, however, that a company's biggest challenge to implementing a cloud platform is ensuring security.

But there's a significant disconnect on the second most important barrier. IT leaders are concerned about integration (46 percent).  In contrast, LOB leaders believe measuring return-on-investment is a more important challenge (37 percent).

"As use of cloud solutions mature, more than half of companies surveyed are shifting from adoption to upgraded services," said Brian Glynn, chief revenue officer of IDG Enterprise. "This opens the door for new and existing solution providers as businesses continue to look for ways to improve agility and innovation while balancing enterprise security and risk."

Cloud service preferences are evolving

Three-quarters of companies are confident that the assets they have placed in the cloud are secure. To help companies have a sense of control, 80 percent have already created, or will create, a governance policy in the next year.

Also, public cloud (60 percent) and private cloud (57 percent) solutions remain the preferred environments compared to hybrid cloud (19 percent). As more workloads move into the cloud, the amount of data stored in private and public clouds will each increase to 25 percent and 21 percent, respectively, in the next 18 months.

Since 2012, cloud investments have increased by 19 percent, with large enterprises spending on average $3.3 million a year, compared to SMBs spending $400,000. Moreover, spending on cloud solutions will account for almost a quarter of IT budgets in the coming year.

Cloud deployment motivation is strategic

Current estimates show that 23 percent of spending on cloud solutions happens outside of the IT department - with marketing, sales and human resources most often investing in solutions.

Besides, even when a cloud solution is purchased by LOB leaders, the IT team can still be involved in the management of the project. And, in instances where IT does not lead the project, 45 percent of the time IT is still called upon to take over the project.

No matter who initiates the move to cloud, one thing will always be certain, CEOs and other senior executives that approve the budget are not enamored by the technology - they have a strategic imperative and they seek a meaningful competitive advantage in the marketplace.

View at the original source