Applications Drive The Biggest Money In Big Data 01-09

Applications Drive The Biggest Money In Big Data

The real money in Big Data has nothing to do with selling Hadoop.

We're still fixating on all the wrong Big Data startups. Hortonworks, one of the primary companies behind Hadoop, recently went public to great fanfare and a $1.2 billion valuation. But Hortonworks and the rest of the so-called Big Data startups are actually some of the least interesting Big Data companies.

In fact, of the current crop of 40 startups valued at more than $1 billion, virtually none of them sell Big Data technology like Hadoop. But all of them make heavy use of data - lots of it - to deliver a wide array of services.

As consultant Peter Goldmacher declared back in 2013, the biggest winners in Big Data are the "business people that have identified opportunities to use data to create new opportunities or disrupt legacy business models." As we enter 2015, expect to see data double the number of billion-dollar startups even as public companies learn to grow through data, as well.

Do-It-Yourself Software Loses Its Luster

It used to be enough for a vendor to ship software and abandon the customer to figure it out (or pay hefty sums of money in consulting fees). SAP, for example, has made billions in revenue by shipping complex software and having customers shell out multiples of the software license fee for high-priced consultants to make sense of its Byzantine software.

That sort of strategy doesn't work very well anymore.

Forget startups for a moment. If we look at the stock prices of various data-related companies, investors are paying a premium for companies like Tableau and Qlik that make data easy to consume:

The companies rising the most include Tableau, Qlik and MicroStrategy, which provide tools to visualize data, while companies that tend to sell infrastructure like IBM and Teradata largely skidded through the year. (In fact, IBM is on its second year as one of the Dow Jones worst performers.)

Data Begets Billions

The analysis isn't perfect, of course. For example, though IBM sells a lot of core infrastructure it also has a Business Intelligence business. Oracle, for its part, plays in many camps, with a strong applications business to make up for its stalling database business.

But where the shift to Big Data really becomes apparent is in the Wall Street Journal's burgeoning billion-dollar startup club. As the Journal's Christopher Mims points out, "2014 was the year tech startup valuations went on a tear without precedent." 

It was also the year that tech startups put data to use at unprecedented levels.

No, not in the old-school Big Data way. When you review most lists of the "top 10 Big Data companies" they focus on those that sell Big Data technology. Among the top-15 most valuable startups, only Cloudera (and maybe Palantir) counts as a Big Data startup in this old sense of the word. 

Source: Wall Street Journal

Comb through the rest of the top-40 most valuable startups and you add MongoDB and Good Data. At face value this seems to suggest that Big Data really isn't that big of a deal.

Back to Goldmacher.

In Goldmacher's world, the "Big winners" in Big Data are "infrastructure providers like the Hadoop vendors and the NoSQL vendors," the "Bigger winners" are "the Apps and Analytics vendors that abstract the complexity of working with very complicated underlying technologies into a user friendly front end."

There's An App For That

But the "Biggest winners," as noted above, are companies like Uber, Stripe and Airbnb that have figured out how to "leverage data as an asset," thereby up-ending old industries and setting themselves apart. Look through the list of the top-40 most valuable startups and nearly all of them have this in common: they understand and leverage Big Data.

As we enter 2015, data will become more important than ever. It won't, however, be easy to track, because there's no meaningful "Big Data" category of vendors. Instead, data will transform industries as diverse as retail and healthcare, crowning multitudes of billion-dollar startups and billion-dollar revenue streams along the way.

View at the original source

Spark Just Passed Hadoop in Popularity on the Web–Here’s Why 01-09

Spark Just Passed Hadoop in Popularity on the Web–Here’s Why

Interest in Apache Spark surpassed Apache Hadoop for the first time last month, according to Google Trends. While it’s not a definitive statement of Spark’s actual impact on big data processing in the real world, it does indicate the enormous momentum the in-memory analytics software has garnered during a phenomenal run in 2014.

As you can see from the Google Trends graphic–which compares the relative popularity of search terms that people enter and prevalence of news articles about given topics–Apache Spark went from a relative unknown to big data superstar in a matter of months. Coming out of Cal Berkeley’s AMPLab, Spark was just one of a handful of promising distributed computing frameworks at this time last year.

In October Apache Spark (blue line) passed Apache Hadoop (red line) in popularity according to Google Trends

But from January through October of this year, interest in Spark skyrocketed. It went from next to nothing into the most talked-about and search-for big data technology in the land—even bigger than big data’s big daddy, Apache Hadoop.

So why has Spark risen so far and so fast? Spark’s popularity has largely been driven by developers who are tired of the complexity of MapReduce and who want an easier and faster way to build big data applications, primarily for Hadoop.

Setting up data flows is critical in today’s big data analytic apps, and Spark simply makes that task easier. When you get Spark, you get a variety of processing engines, including the Spark SQL capability, Spark Streaming, and the MLlib machine learning library (additional capabilites like GraphX for graph analytics, the SparkR capability for running R-based applications, and hte BlinkDB capability are also in the wings). Moving data from one processing engine to the next is more easily done under Spark than if you had to cobble together multiple Hadoop-based

processing engines, such as MapReduce, Impala, Storm, Mahout, and Giraph.

Spark also has speed on its side. Last month Databricks, the company behind Spark,released benchmark results that demonstrated Spark running three times faster than MapReduce on a 100TB sort workload, using 10 times less computing power. It also out-sorted MapReduce by a factor of four on a 1PB workload, using significantly less hardware. In some instances, Spark can run upwards of 100 times faster than MapReduce applications, Spark backers have claimed.

In the beginning of the year, there were about 200 contributors to Spark, which made it a more active project than Hadoop MapReduce. Since then, more than 100 additional contributors have signed on to help develop Spark.

Where will it go in 2015? Don’t bet against it from continuing the rise. “Spark is a fast moving project,” Databricks Head of Engineering, Ali Ghodsi, told Datanami last month. “It’s actually the most active big data project now out there. There’s a lot happening to it.”

The community of Hadoop software vendors are also increasingly turning to Spark to power big data analytic applications. Platfora, ClearStory Data, and Alpine Data Labs have all committed to using Spark in their Hadoop-based applications, while Trifacta and Paxata are also counting on Spark’s speed to power big data transformation solutions. Datameer is re-architecting its solution with Spark in mind, and Glassbeam, which builds a NoSQL-based product that helps companies make sense of big data generated by devices connected to the Internet of Things (IoT), recently adopted Spark running under Cassandra.

Spark is outpacing Hadoop itself as the hottest big data technology at the moment. The fact that Spark doesn’t need Hadoop creates an interesting tension in the marketplace. The folks at Databricks will tell you that they learned from Hadoop’s early mistakes and are seeking to eliminate much of the complexity that plagues first-gen Hadoop and MapReduce implementations.

Unfortunately, getting Spark running on-premise is still hard, Databricks says. That’s where its Databricks Cloud implementation of Spark comes in handy. And while Databricks Cloud doesn’t use Hadoop, many of its early customers analyze HDFS-resident data.

In many ways, Spark is helping to fulfill the big data promises and dreams of Hadoop and that has forced the Hadoop distributors to take notice.Cloudera was an early supporter of Spark and has been shipping the Spark software with its latest Hadoop distribution CDH 5, since it was in beta over a year ago. MapR Technologies has also supported the entire Apache stack since April.

It took a little longer for Hortonworks to catch the Spark wave. Hortonworks, which sticks closer to the core trunk of open source Hadoop and has openly questioned Spark’s readiness and ability to scale at the high end, started talking about making Spark a first-class citizen on Hadoop 2.0 and YARN during the summer. As it stands, Spark will be fully supported in Hortonworks Data Platform version 2.2, which it unveiled in October and is currently in tech preview.

Hortonworks is fully on board Spark bandwagon now and is ramping up efforts to fully integrate the technology with the rest of the Hadoop stack, which Hortonworks VP of strategic marketing John Kreisa recently described as a “shared vision for Apache Spark on Hadoop.”

“We’ve seen this unbridled excitement around Spark really over the past eight months,” Hortonworks Director of Product Marketing Jim Walker told Datanami recently. “It’s fascinating how quickly this is picking up in the broader enterprise.”

View at the original source

Ray Wang: Cloud Is The 'Foundation For Digital Transformation' 12-24

Ray Wang:

 Cloud Is The 'Foundation For Digital Transformation'

Here’s a startling little factoid: Since 2000, 52% of the companies in the Fortune 500 have either gone bankrupt, been acquired, ceased to exist, or dropped off the list.

That’s a massive shift, and it’s only going to increase, says Ray Wang, founder and principal analyst at Constellation Research.

Speaking at CloudWorld Boston, Wang examined how the convergence of a host of technologies is powering digital disruption. Mobile, social, analytics, connected devices, 3D printing and cloud computing are all part of a process enabling businesses to digitize their processes and their business models.

“Digitalization of business is a key factor in this accelerated pace of change. Information flows faster,” he said.

The technologies themselves are essentially table stakes—widely available and easy to use. Businesses differentiate themselves by using the technology to create new strategies of such disruptive power that they can wipe out entire markets.

“It’s not technology alone, it’s how that technology is being applied to business models,” said Wang.

For example, look at the iPhone, which has obviated the demand for items from flashlights and wristwatches to alarm clocks, cameras, and CDs. When Apple introduced its iconic device in 2007, who could imagine that one device could destroy 27 business models, as Wang asserts that it does?

“That’s why you have to get to digital, and that’s why we need to talk about cloud—it’s the foundation for digital transformation,” said Wang.

Indeed, Wang points to the cloud as the single most disruptive of all the new technologies – if only because of its ubiquity and ease of adoption. Cloud not only provides a source of unlimited and dynamic capacity, but also helps users consume innovation faster. For example, rather than having to wait for IT to upgrade an in-house application, SaaS users get the latest software innovation as a matter of course.

should consider to reinvent themselves for the digital economy and build a new digital identity.

1. Design new experiences and business models. Yes,

 cloud removes many of the barriers of on-premise systems. But companies still need to spend time thinking about what a digital user experience is and design new processes or businesses around that. Customers seek outcomes and experiences while many organizations continue to think in terms of selling products and services, said Wang. Businesses can use the cloud as a platform for change. “The worst thing you can do is replicate the same crap you had into the cloud,” he said.

2. Develop a culture of digital DNA. 

Digital transformation requires skills across the board, and companies must rethink the attributes that will help a digital business thrive. Balance the quant jocks and engineers with people with digital visual skills, for example.

3. Apply new technologies to existing infrastructure. 

Tech convergence of these digital disruptors is a starting point, but that doesn’t mean you hold a yard sale for your existing systems. Instead, figure out how to combine the new with the old to further digital transformation. Can you use PaaS and SaaS together, for example, to make applications globally portable or simplify your company’s ability to consume innovation via seamless upgrades? (It helps if you’re using a PaaS that supports Java and SaaS apps also built using Java.)

4. Move from gut to data-driven decisions. 

Data is at the heart of digital transformation, said Wang, and companies must figure out how to turn the data they collect from every online touchpoint into information that delivers real-time context and relevancy for decision-making.  Information needs to flow freely across platforms in order to build the foundation that allows companies to quickly move from data to insight to decision.

5. Co-create and co-innovate with new partners. 

In today’s always on, interconnected business world, digital business models work best when drawn up as an ecosystem, not a monolith. You’ll need strong leadership to find the right partners, from vendors to suppliers to customers, to create an ecosystem that supports your strategy. Again, cloud is key. “Very few companies today can open up their systems and have their partners build on top of them,” said Wang. “Cloud helps you do that.”

View at the original source

How cloud adoption trends are driven by strategic imperatives 12-24

How cloud adoption trends are driven by strategic imperatives.

Picture credit: iStockPhoto

As this year comes to a close, let's take one last look at the most pervasive cloud computing trends - including increased usage across the different cloud service models, the key business drivers and the impact of agile innovation strategies.

Cloud computing adoption has matured, with 69 percent of survey respondents stating that at least a portion of their computing infrastructure is in the cloud. However, 56 percent of companies are still identifying IT operations that could potentially move to the cloud, according to the latest market study by IDG Enterprise.

Survey respondents believe that business technology is a game changer, and cloud solutions are providing advantages from increasing IT agility (63 percent), IT innovation (61 percent) and improving the ability to access critical business data and digital service applications (58 percent).

Removing the barriers to cloud adoption

That said, IT leaders perspective on barriers to bringing these advantages to fruition differs from their Line of Business (LOB) counterparts. There is agreement, however, that a company's biggest challenge to implementing a cloud platform is ensuring security.

But there's a significant disconnect on the second most important barrier. IT leaders are concerned about integration (46 percent).  In contrast, LOB leaders believe measuring return-on-investment is a more important challenge (37 percent).

"As use of cloud solutions mature, more than half of companies surveyed are shifting from adoption to upgraded services," said Brian Glynn, chief revenue officer of IDG Enterprise. "This opens the door for new and existing solution providers as businesses continue to look for ways to improve agility and innovation while balancing enterprise security and risk."

Cloud service preferences are evolving

Three-quarters of companies are confident that the assets they have placed in the cloud are secure. To help companies have a sense of control, 80 percent have already created, or will create, a governance policy in the next year.

Also, public cloud (60 percent) and private cloud (57 percent) solutions remain the preferred environments compared to hybrid cloud (19 percent). As more workloads move into the cloud, the amount of data stored in private and public clouds will each increase to 25 percent and 21 percent, respectively, in the next 18 months.

Since 2012, cloud investments have increased by 19 percent, with large enterprises spending on average $3.3 million a year, compared to SMBs spending $400,000. Moreover, spending on cloud solutions will account for almost a quarter of IT budgets in the coming year.

Cloud deployment motivation is strategic

Current estimates show that 23 percent of spending on cloud solutions happens outside of the IT department - with marketing, sales and human resources most often investing in solutions.

Besides, even when a cloud solution is purchased by LOB leaders, the IT team can still be involved in the management of the project. And, in instances where IT does not lead the project, 45 percent of the time IT is still called upon to take over the project.

No matter who initiates the move to cloud, one thing will always be certain, CEOs and other senior executives that approve the budget are not enamored by the technology - they have a strategic imperative and they seek a meaningful competitive advantage in the marketplace.

View at the original source

Building a private cloud, of sorts, with OwnCloud 12-12

Building a private cloud, of sorts, with Own Cloud

Getting cloudy has always been a mixed proposition in the IT world. Your users want the convenience of using a variety of devices and having their work accessible on all of those devices from wherever they are, whereas you still have to worry about data security, lost computers, federal regulation and the control that is necessary to ensure your organization's information technology resiliency. Add the fact that most cloud services like OneDrive, Google Drive or Dropbox -- let's face it, the ones that your users want -- are consumer-oriented services that lack the ability to be managed and controlled centrally, and you have a face-off for the ages.

OwnCloud purports to solve these problems with the notion of a private cloud, but not in the sense that a private cloud is simply a data center you own and control that has elements of automation including self-service delivery, redundancy and easy spin-ups and spin-downs of various offerings and services. Rather, OwnCloud is more like a Dropbox that is not under Dropbox's control or a Google Drive where Google is not reading all of the data; it is a service you, as either the administrator for a larger organization or you as the end user, control, where you can decide which items you want shared with others, which devices you can access that data on and what apps you want to access that data.

For the free community edition that operates under open-source licensing rules, you have two choices: Host the code yourself on hardware that you own, all the while retaining complete control over the files and folders you manage with the service, or find a hosting partner that will run the OwnCloud platform for you, with the implication that you then must store at least some of your data at that partner. 

OwnCloud makes available a list of these providers. All have different methods of pricing what they offer, how much storage comes with their various plans and so forth. The OwnCloud project does not appear to endorse or have an emphasized partnership with any of these providers.

In this piece, I want to dig into the technical details of the OwnCloud offering and highlight some of its plusses and minuses. Let's take a look.

OwnCloud security  Arguably the strongest reason for adopting your own off-device storage and synchronization program is the ability to secure the contents of your users' files and folders.

OwnCloud relies on encryption to do most of the heavy lifting to protect the data under its control. OwnCloud uses Transport Layer Security (TLS) to protect data as it is in transit from device to server and vice versa, and then a separate encryption app to encrypt and decrypt the data. The encryption app uses a long, random key protected by the password for each user's account to encrypt user data.However, the decryption and encryption always take place on the server, leaving you as the administrator with control over what is stored within your OwnCloud deployment since you can control the encryption on the server. Also, if encryption were performed on each user device, the server would not have the key available to decrypt files and allow a Web interface to function.

What is especially interesting about the OwnCloud encryption app is that it can work with Dropbox, Google Docs, external servers accessed through the  WebDav HTTP protocol extension and more. This adds a layer of security to these other services and lets you informally extend the amount of storage that is available to you and your users while still providing a measure of protection during their use.

With memorable Dropbox security failures like the "everyone can read all of everyone else's data" incident of a few years ago and recent password integrity issues, this can be comforting to all manner of users. The main feature OwnCloud provides here is its seamless encryption.

The free community edition also includes support for integrating identities with LDAP directories and in particular Active Directory, so users can take advantage of their existing user accounts when using the service.

Synchronization and collaboration featuresPerhaps the most useful user-oriented feature of OwnCloud is the ability to synchronize data stored under the software's purview to a tablet, a phone or a central server through the use of a Web interface and the user's laptop or desktop computer. OwnCloud comes with clients for both the iOS and Android platforms (there is no Windows Phone support), although you have to pay extra for those unless you are using the paid enterprise edition of OwnCloud, in which case the mobile clients come included in your subscription.

In addition, in the community edition, users can edit their documents through a Web interface and they also get a nice photo gallery and photo-sharing capability. There is also calendar support, though for any shop on Microsoft Exchange or other groupware, it is difficult to see how the OwnCloud calendar really adds much value on top of that existing investment.

As far as collaboration goes, you can share files with users of other OwnCloud installations, and you can also set up alerts so that your users know when others have accessed the files that they have shared. It also includes support for versioning, so in case of a mistake, users can roll back files they have changed to earlier versions.

Installing Own Cloud

You can research a lot about setting up the OwnCloud service, but it is best to just accept one fact from the beginning: OwnCloud really prefers installing itself on Linux distributions, primarily because it relies on both PHP and MySQL for its core functioning, and neither is really built for running well and trouble-free on the Windows Server platform.

Installing on Linux

To install on Ubuntu Linux or its variants, you can get all of the prerequisites in place by entering the following three commands in the shell:

apt-get install apache2 mysql-server libapache2-mod-php5

apt-get install php5-gd php5-json php5-mysql php5-curl

apt-get install php5-intl php5-mcrypt php5-imagick



Next, from the installation page, click the blue "Archive File for server owners" button in the "Install OwnCloud Server" section, and then click Download Unix to download the package to your server system. You will get a file with a .tar.bz extension, which represents the core module of OwnCloud.

Now, just pop out the archive file with the following command:

tar -xjf owncloud-x.y.z.tar.bz2



Next, copy these files to the document root directory of your Web server. For Apache on Ubuntu, this is usually the /var/www directory. Here is a command that will do the heavy lifting:

cp -r owncloud /path/to/webserver/document-root



Make sure the Web user account on your system owns the Owncloud directory. On Ubuntu systems, this is the www-data user. On other Linux systems, this might be the apache account or the webuser account.

Your PHP configuration file will tell you the right user account for your system. The chown command shown below will assign the right ownership to these subdirectories:

chown -R www-data:www-data /var/www/owncloud



Now, enable SSL in Apache. On Ubuntu, again four simple commands will take care of this for you, even when using the default self-signed certificate you get in the base Apache package:

a2enmod ssl

a2ensite default-ssl

service apache2 reload

a2enmod rewrite



Then you can run the installation wizard by navigating to https://servername/owncloud. Since you are using a self-signed certificate, you might have to accept whatever warning your Web browser throws up. Then simply follow the wizard.

For more detailed information on installing OwnCloud on a variety of Linux distribution variants, visit the OwnCloud online documentation
.

Installing on Windows Server 2008 or higher

The set up on Windows looks different. Here's the process in a nutshell.

1. From the Start menu, choose Control Panel, Programs and, then under Programs and Features, Turn Windows Features On and Off.
2. Server Manager appears. Click Roles, then click Add Roles, and then add the Web server role. Make sure CGI support is enabled, FTP support is disabled and WebDAV publishing is disabled. (OwnCloud needs to own WebDAV at the application level, so disabling Windows WebDAV here will prevent conflicts.)
3. When the Add Roles Wizard is finished, reboot your server and then install PHP by going to the PHP for Windows download page and grabbing the PHP 5.3 “VC9 Non Thread Safe” version installer (near the bottom of the page) in either 32-bit or 64-bit editions.
4. Once the download finishes, run the installer, read the license agreement, agree, select an install directory and select IIS FastCGI as the install server.
5. Choose the defaults in the remainder of the wizard and then finish it out.
6. Next up, we need to install MySQL, so head over to http://dev.mysql.com/downloads/ . then download the MySQL Community Server edition and run the installer in the Typical Installation configuration.
7. Once the installation completes, choose to launch the MySQL Instance Configuration Wizard, choose a standard configuration, click to install MySQL as a Windows service and enable the Launch the MySQL Server Automatically button.
8. Configure a user account on the next page and then click Execute and Finish once the wizard is done trundling.
9. Next, install OwnCloud itself by downloading the blue “Archive File for server owners” package from http://owncloud.org/download. Unzip the .tar.bz file by using WinRAR or some other utility that supports the format, since Windows does not natively know how to read it, then copy the files to C:\inetpub\wwwroot, the Windows Website default directory.
10. Then simply follow the installation wizard, as explained in the next section.
11. The installation wizard

Even after tediously getting the OwnCloud code on your system, you are still not done. You now have to correctly set up the environment. The wizard, however, makes this a little easier than the initial setup steps.

Create an administrator account, including a username and password.

Under advanced options, you can choose a different data directory on the server storage for the service, and also you can set up either SQLite or MySQL for the service's database needs. Note you will need to install SQLite if you want to use it prior to launching the installation wizard.

Then click Finish Setup, and OwnCloud will set up its environment.

An analysis

For shops with a lot of Linux expertise, OwnCloud makes a lot of sense. It was built to live on those platforms. It uses PHP and MySQL extensively. As an open-source product, it relies a lot on community support and expertise to handle feature requests and issues.

For smaller shops without much built-in IT knowledge, you can run OwnCloud on most Web hosting platforms that run a few bucks a month, so it could be a good way to get a ton of functionality for a very low price, with some security built in, too. OwnCloud is a great fit for tiny businesses with hosting plans and large businesses with the necessary expertise to run OwnCloud servers in a good security posture.

For Windows organizations, however, it becomes a much murkier picture. Using a package that was built for Linux on Windows is always a daunting task. PHP and MySQL on Windows probably do not get enough testing (as compared to native Windows apps); you would have to do a lot to convince me otherwise. Patching and updating for both new features and security holes are much tougher propositions; you generally need a third-party engine to perform this sort of maintenance, since Windows Server Update Services generally ignores PHP and MySQL.

There is a paid support offering for businesses in OwnCloud Enterprise, which aims to solve many of these shortcomings. It also supports authentication integration using SAML -- a common standard -- and provides logging and backup systems and support for Microsoft SQL Server. There's also a product that allows admins to connect OwnCloud to Windows network drives, and manage and secure files in that way. The software supports network drives hosted on Windows Server 2008, 2012, Windows 7 or 8, as well as Linux-based Samba (emulating Windows). But this setup requires the paid Enterprise edition of OwnCloud running on a Linux server.

For Windows shops, I would not recommend the community edition of OwnCloud and would head either to the enterprise edition -- which involves paying some money, and paying even more for 24 x 7 support -- or go to another service entirely.

View at the original source