Your Data Should Be Faster, Not Just Bigger 02-08

Your Data Should Be Faster, Not Just Bigger

It’s universally acknowledged that Big Data is now a fact of life, but while large enterprises have spent heavily on managing largevolumes and disparate varieties of data for analytical purposes, they have devoted far less to managing high velocity data. That’s a problem, because high velocity data provides the basis for real-time interaction and often serves as an early-warning system for potential problems and systemic malfunctions.

Moreover, data proliferation has been accelerating. EMC recently reported that data volumes can be expected to double every two years, with the greatest growth coming from the vast amounts of new data being produced by intelligent devices and sensors. Oracle president Mark Hurd has predicted that the number of devices connected to the Internet will grow from 9 billion to 50 billion by the end of this decade.

What makes device data, sensor data, and other forms of “fast data” distinctive is that, unlike historical data, it is live, interactive, automatically generated, and often self-correcting. Historical data is used to identify patterns that inform future decision-making, while fast data is designed for real-time decisions and real-time responses. Think of fast data as the continuous processing of events and data in order to gain instantaneous insight and take instantaneous action.

While fast data is not really new, it has been largely restricted to a couple of high-value uses: complex event processing (CEP) activities that operate on event streams, examples of which include algorithmic trading and fraud monitoring in financial services; and event correlation, which includes the systems that monitor and manage complex industrial components such as jet engines.

So, what has changed? First, the explosion of fast data has driven the demand for instant action. Second, innovators in social media and services like Uber have shown that businesses can be differentiated based on their ability to act on data instantly. Uber knows where you are, where you’re going, and how you will pay to get there because it can capture, analyze, and act on data in real time. The availability of lower-cost memory is making fast data accessible for a broader set of applications, including:

• First responder systems that rely on integrating fast response data collection and analysis
• Network usage systems that respond instantly based on traffic patterns
• Customer-experience management systems that analyze vast amount of behavioral data in real time to tailor interactions and support self-service

Organizations that know how to use fast data will be more nimble, adaptive, and competitive. What can companies do differently to prepare for the opportunities created by high-velocity data? Here are a couple of suggestions:

1. Automate decision-making to increase customer engagement. Monitor customer activities to identify—and respond to—patterns, thresholds, and triggers. One major retailer is seeking to engage with customers in real time while they are online, but is hampered by traditional systems and batch processing environments. They are now creating an environment that marries customer and inventory data with streaming data so they can, for example, report to the customer whether a product is in stock and address any inventory gaps immediately.
2. Integrate machine-generated data to personalize interactions.EMC projects that soon nearly two-thirds of all data will be generated by machines, not people. That presents a technical challenge: how can companies capture and analyze many flows of data concurrently? The good news is that next-generation data systems that prioritize real-time data are in the early stages of adoption. By integrating new sources of machine-generated data, and combining it with traditional data sources, firms can further personalize their customer interactions. Ericsson, the mobile broadband company, has developed real-time visibility into its system performance, using device and user data as it is generated. This enables Ericsson to identify and improve slow performance as needed and to serve customers with programming tailored to them.

Firms that take these steps now will be well positioned to improve their operations and better serve their customers using high-speed data.

View at the original source

Apache Spark vs. Tez: What's the Difference? 02-01

Apache Spark vs. Tez: What's the Difference?

We’ve pointed out that Apache Spark and Hadoop MapReduce are two different Big Data beasts: the former being a high-performance in-memory data-processing framework; and the latter a mature batch-processing platform for the petabyte scale.

But what about Spark vs. Tez?

On paper, Spark and Tez have a lot in common: both possess in-memory capabilities, can run on top of Hadoop YARN and support all data types from any data sources. So, what’s the difference?

Apples vs. Oranges

This is how each framework brands itself:

“Apache Spark is a fast and general engine for large-scale data processing.” (source)

“The Apache Tez project is aimed at building an application framework which allows for a complex directed-acyclic-graph of tasks for processing data. It is currently built atop Apache Hadoop YARN.” (source)

Considering the fact that Spark also uses directed-acyclic-graphs, don’t they sound a bit similar? Maybe. Nonetheless, in an interview with Shaun Connolly, Hortonworks product strategy vice president, he differentiates between the two by saying that Spark is a general purpose engine with APIs for mainstream develowpers, while Tez is a framework for purpose-built tools such as Hive and Pig.

Pigging Out

Although Spark boasts an ease of use with friendly APIs that are available for Python, Scala and Java, there are some caveats when writing Spark jobs. For instance, if you work with flat files, you’ll have to write your own functions to do aggregations. That’s because Spark can only pull rows, but not separate them into columns.

Also, both frameworks claim to support Pig and Hive. We tried running Pig on Spark using the Spork project, but we had some issues. This may mean that full Pig support for Spark is still under construction—we’ll try again in the near future.

The YARN Spin

One major difference, is that Spark can run as a standalone or on top of Hadoop YARN, while Tez can only run on top of YARN—Tez was designed to do so. Spark is YARN compatible, but in practice, it can’t run alongside other YARN applications. At least at the moment.

Gopal V, one of the Tez developers, wrote an extensive post about why he likes Tez. He concludes that:

“Between the frameworks I've played with, that is the real differentiating feature of Tez - Tez does not require containers to be kept running to do anything, just the Application Manager running in the idle periods between different queries. You can hold onto containers, but it is an optimization, not a requirement during idle periods for the session.”

By “frameworks” he also means Spark—its containers need to keep running and hog resources even when they aren’t processing any data. Tez containers, however, can shut down as soon as they are finished and release the resources.

Most chances are that you use Hadoop-based applications anyway like Hive, HBase or even classical MapReduce. So you can install Spark on any Hadoop cluster, but you may run into resource management issues. On the other hand, Tez could fit quite nicely into your YARN architecture, resource management included.

Performance According to Whom

Spark claims to run 100× faster than MapReduce. Benchmarks performed at UC Berkeley’s Amplab show that Spark runs much faster than Tez (Spark is noted in the tests as Shark, which is the predecessor to Spark SQL). However, Berkeley invented Spark. Also, these benchmarks were made over a year ago with Hive 0.12, which runs over MapReduce, while Hive 0.13, which runs over Tez, has significant performance improvements.

Hortonworks did their own benchmarks. They found that Hive 0.13 running over Tez works up to 100× faster than the previous Hive version (though quite a few test queries mysteriously disappeared). 100× faster? Sound familiar?

So Spark and Tez both have up to 100× better performance than Hadoop MapReduce. That’s nice, but is Spark faster than Tez or vice versa? Who knows. If you ask someone who works for IBM they’ll say that the answer is neither: IBM Big SQL is the fastest. We really need a third party to run independent performance tests and settle the score, once and for all.

Conclusion?

Perhaps it comes down to politics and popularity, a clash of the Big Data titans with Hortonworks rooting for Tez and Cloudera rooting for Spark. The latter is more widespread since it’s available in various distributions, while the former is only available in Hortonworks’ distro. The users may decide the frameworks’ fate. At the moment, Spark is winning the race by far, at least according to Google Trends:

Maybe after the hype has faded, after people have gained more experience working with both Spark and Tez, we’ll finally be able to tell who will become the heir to the MapReduce crown.

(Image by andrewfhart, some rights reserved)

View at theoriginal source