Build a CEP App on Apache Spark and Drools Combining CDH with a business execution engine can serve as a solid foundation for complex event processing on big data. Event processing involves tracking and analyzing streams of data from events to support better insight and decision making. With the recent explosion in data volume and diversity of data sources, this goal can be quite challenging for architects to achieve. Complex event processing (CEP) is a type of event processing that combines data from multiple sources to identify patterns and complex relationships across various events. The value of CEP is that it helps identify opportunities and threats across many data sources and provides real-time alerts to act on them. Finance: Trade analysis, fraud detectionAirlines: Operations monitoringHealthcare: Claims processing, patient monitoringEnergy and Telecommunications: Outage detection Like all problems in the analytic world, CEP is also complicated by the exponential growth of data. Architecture and Design In the above picture:
Spark SQL for Real-Time Analytics Apache Spark is the hottest topic in Big Data. This tutorial discusses why Spark SQL is becoming the preferred method for Real Time Analytics and for next frontier, IoT (Internet of Things). By Sumit Pal and Ajit Jaokar, (FutureText). This article is part of the forthcoming Data Science for Internet of Things Practitioner course in London. If you want to be a Data Scientist for the Internet of Things, this intensive course is ideal for you. Overview This is the 1st part of a series of 3 part article which discusses SQL with Spark for Real Time Analytics for IOT. Introduction In Part One, we discuss Spark SQL and why it is the preferred method for Real Time Analytics. Objectives and Goals of Spark SQL While the relational approach has been applied to solving big data problems, it is in-sufficient for many big data applications. As they say, “The fastest way to read data is NOT to read it” at all. Spark SQL helps this philosophy by Spark SQL Spark SQL can support Batch or Streaming SQL. Related:
Deep Dive into Spark SQL’s Catalyst Optimizer Spark SQL is one of the newest and most technically involved components of Spark. It powers both SQL queries and the new DataFrame API. At the core of Spark SQL is the Catalyst optimizer, which leverages advanced programming language features (e.g. Scala’s pattern matching and quasiquotes) in a novel way to build an extensible query optimizer. We recently published a paper on Spark SQL that will appear in SIGMOD 2015 (co-authored with Davies Liu, Joseph K. Bradley, Xiangrui Meng, Tomer Kaftan, Michael J. To implement Spark SQL, we designed a new extensible optimizer, Catalyst, based on functional programming constructs in Scala. At its core, Catalyst contains a general library for representing trees and applying rules to manipulate them. Trees The main data type in Catalyst is a tree composed of node objects. As a simple example, suppose we have the following three node classes for a very simple expression language: Rules Applying this to the tree for x+(1+2) would yield the new tree x+3.
Software Suites for Data Mining, Analytics, and Knowledge Discovery commercial | free 11Ants Model Builder, a desktop predictive analytics modeling tool, which includes regression, classification and propensity models. AdvancedMiner from Algolytics, provides a wide range of tools for data transformations, Data Mining models, data analysis and reporting. Alteryx, offering Strategic Analytics platform, including a free Project Edition version. Angoss Knowledge Studio, a comprehensive suite of data mining and predictive modeling tools; interoperability with SAS and other major statistical tools. BayesiaLab, a complete and powerful data mining tool based on Bayesian networks, including data preparation, missing values imputation, data and variables clustering, unsupervised and supervised learning. Free and Shareware ADaM, Algorithm Development and Mining version 4.0 toolkit ADAMS: Advanced Data mining And Machine learning System, a flexible workflow engine for quickly building and maintaining real-world, complex knowledge workflows, released under GPLv3.
Using Apache Spark DataFrames for Processing of Tabular Data This post will help you get started using Apache Spark DataFrames with Scala on the MapR Sandbox. The new Spark DataFrames API is designed to make big data processing on tabular data easier. A Spark DataFrame is a distributed collection of data organized into named columns that provides operations to filter, group, or compute aggregates, and can be used with Spark SQL. DataFrames can be constructed from structured data files, existing RDDs, tables in Hive, or external databases. In this post, you’ll learn how to: Load data into Spark DataFramesExplore data with Spark SQL This post assumes a basic understanding of Spark concepts. This tutorial will run on the MapR v5.0 Sandbox, which includes Spark 1.3 You can download the code and data to run these examples from here: examples in this post can be run in the spark-shell, after launching with the spark-shell command. The sample data sets How many auctions were held? Summary
Getting Started with Spark on MapR Sandbox At MapR, we distribute and support Apache Spark as part of the MapR Converged Data Platform, in partnership with Databricks. This tutorial will help you get started with running Spark applications on the MapR Sandbox. Prerequisites HARDWARE REQUIREMENTS8GB RAM, multi-core CPU20GB minimum HDD spaceInternet accessSOFTWARE REQUIREMENTSA hypervisor. Starting up and Logging into the Sandbox Install and fire up the Sandbox using the instructions here: Logging in to the Command Line Before you get started, you'll want to have the IP address handy for your Sandbox VM. $ ssh root@192.168.48 “How to” for a Spark Application Next, we will look at how to write, compile, and run a Spark word count application on the MapR Sandbox. Example Word Count App in Java You can download the complete Maven project Java and Scala code here: Get a text-based dataset Pull down the text file: .