Cloudera to Support Your IOT Strategy 1 - Xpand IT

Cloudera to Support Your IOT Strategy 1

https://www.facebook.com/Xpand-IT-206508579394292/

https://twitter.com/xpandit

https://www.linkedin.com/company/xpand-it


1 Introduction

2 Why Companies Need an IoT Strategy? A Bit of Background

Data is Everywhere

How IoT is Transforming Business

Typical Architecture for IoT Solutions

3 Powering IoT with Cloudera Why Cloudera as IoT Solution?

Reason 1 • Scalable Solution

Reason 2 • Flexible Storage & Analysis

Reason 3 • The Right Components & Framework

Reason 4 • Open Standards

Reason 5 • Robust Partner Ecosystem

4 Exploring Top Use Cases by Industry

5 Conclusion

Content




3Cloudera to Support Your IOT Strategy

Introduction

In the coming decades, all the existing devices will be connected, intensifying what is known as the Internet of Things (IoT). This e-book explains the data impact of IoT and introduces Cloudera as a reference in Data Management, creating value from billions of data points.

So, is your company ready to build and manage an effective IoT Strategy in the Real World?

This e-book will also show you how Apache Hadoop can enable real-time analytics to drive value from your investments.Today, Cloudera is powering IoT in key areas such as insurance, manufacturing, connected cars/homes, smart cities and many others.

“The range of industries that can benefit from using Big Data technologies is increasing, as well as the number of its use cases.Naturally, companies that traditionally manage high (structured) data volumes are the ones that can more easily realize the potential: retail, banking, and telecommunication lead the pack.”

NUNO BARRETO Partner & Big Data Lead at Xpand IT


It’s All About Data

IoT (Internet of Things) is the idea that everyday objects such as wearable devices, industrial machinery and many others can use built-in sensors to collect data and take action according to it, across the network.

Can you imagine how many objects can be connected to the Internet, and how much we can benefit from examining the results of the data streams?

The IOT Ecosystem

Consumer

Industrial

Sensors / Things

IoT Gateway





Why Companies need an IoT Strategy?

2





“Internet of Things” is a growing topic of debate since 1999, both in the office and outside of it. This new concept has the potential to change the way we live and work. “By 2020, there will be approximately 20.8 billion devices on the Internet of Things. And it will not only be devices with sensory abilities, but also devices that are able to act.” - Gartner, Inc.

Currently, IoT technology can be found in many industries: energy, agriculture, building management, healthcare and transportation.

A Bit of Background





Data is Everywhere

Everything that can be connected, will be in the future. Let´s imagine, as an example, which you are on your way to work; if your car has access to your schedule, it could calculate the best route to ensure you don’t arrive late. And if by chance you have a scheduled meeting and the traffic is too dense, your car might send a text to the other party notifying them that you will be late.

What if it’s your alarm clock that wakes you up at 7 am, while your blinds start rising by themselves?

That is why IoT is a hot topic nowadays; this will certainly open the door to a bunch of opportunities and an increasing number of challenges such as Safety, Privacy and Data Sharing.

Another concern that many companies will face, is the huge amount of data that all of these devices will produce. Corporations will need to find ways to store, track, analyse and extract value from massive volumes of data.





It’s not unknown that data is improving business performance and driving top level initiatives. Hadoop is the key tool for supporting this situation and helping companies achieve actual business value. IoT initiatives have three key areas:

How IoT is Transforming Business

Global Customer Vision

Data-Driven Products

Business Risk


This area involves collecting all data related to what customers do since their first interaction (website/store) to the moment of the purchase.

There are countless advantages in collecting information from clients. Especially if the company has a way to capture, store, and analyse all the data collected from a large number of different sources at an unbelievable speed. And this can be applied to customers, vendors, partners, etc.

Global Customer Vision

Cloudera to Support Your IOT Strategy 1010Cloudera to Support Your IOT Strategy

Most companies are interested in using data to improve their products (for example, connect homes/cars) or to use data to develop new solutions/services. Some are even creating predictive analytics to notify their customers of a problem, update or discount or suggest a complementary product.

Data-Driven Products


Business Risks

Companies are challenged every day on how to protect their business from internal and external risks. By analysing, without boundaries, a huge quantity of events, things become simple. This wasn’t possible when you only looked at it for a few hours or days.

These new capabilities can be directly used to solve issues, such as defending your network against cyber-attacks, understanding the customer, identifying trends, improving efficiency and meeting requirements.

Instead of looking at a small data sample, save it, from all your users and all your sources, and keep it available for analysis for years. This amount of data, combined with modern analytic tools, will give you the ability to spot issues across a much larger surface area than ever before - making your existing systems more effective.





There is no ‘typical’ technology stack for IoT solutions.

Due to this situation, different vendors and developers have felt the need to create their own specialized stacks, tailored to their needs. As this evolves, we are seeing a few different patterns arising, which we will show you next. An IoT solution consists of the following tiers: Edge and Cenral processing.

Typical Architecturalfor IoT Solutions

Devices withsensors & actuators

Devices withsensors & actuators

Centralized platform fordata storage,

analytics and orchestration

Gateway

Gateway





Currently, there are three primary kinds of architectural patterns being used for IoT solutions:

1 Device-centric

This involves devices that are smart enough to make some type of choice at their level, without having to reach back the central control plane.

2 Gateway-centric

End devices are usually linked to control hubs or gateways, driving some part of the application logic to the IoT solution. In gateway-centric architectures, the system drives the devices without having to reach back to the central control plane.

3 Centralized control plane-centric

These kind of architectures have a centralized control plane at their core. The data is stored, treated and choices are made at the central level. Afterward, activities are sent out to the end devices or gateways.

IoT solutions, in most cases, are a combination of the architectural patterns above.The central control plane is where the long term data is stored, processed and that’s where an enterprise-grade, scalable data platform is essential.

Typical Architecturalfor IoT Solutions





Powering IoTwith Cloudera

3





Why Cloudera as IoT solution?

One of the major advantages of technologies such as Cloudera is the fact that it allows companies to store (and process) high fidelity data at a very low cost. The decision if whether there is value or not, can be taken later after the information has been properly analysed and correlated. Companies should first ensure they store all the data, so they can focus on analysing and extracting value from it.

This aligns perfectly with Cloudera Enterprise, the fastest, easiest and most Secure Hadoop distribution. In the last few years, Cloudera has been working hard to improve its platform’s capabilities to fulfil such requirements. There are 5 key benefits that make Cloudera the right platform for IoT.





Cloudera is based on the Apache Hadoop platform and includes flexible storage substrate in HDFS, along with scale out ingestion and processing frameworks that make the overall platform immensely scalable. Therefore, it can handle the speed and volume of data that IoT use cases demand.

Scalable Solution

Fault-Tolerant Hadoop Distributed File System (HDFS)

1

2

3

4

5

245

125

134

235

135

HDFS breaks incoming �les into blocks and stores them redundantly across the cluster

Provides reliable, scalable, low-cost storage.





Flexible Storage & Analysis

Cloudera Enterprise offers significant flexibility to users in several different ways.

In terms of the infrastructure, Cloudera can be deployed on bare metal industry standard servers or (non) public cloud environments.

It also provides agility with the types of access patterns supported, which can encompass batch processing, iterative in-memory processing, low latency analytics, stream processing, search or low latency data serving.

This completely solves data issues that come along with IoT use cases, since they require a diversity of different access patterns to be handled equally well.





The Right Components& Framework

New powerful components have emerged makingHadoop faster and improving next-generation capabilities. Bellow you will find highlighted the key components and frameworks that are most relevant for IoT use cases.

PROCESS, ANALYZE, SERVE

STORE

INTEGRATE

RESOURCESMANAGEMENT

YARN

SECURITYSentry, RecordService

RDB MS Sqoop

REAL-TIMEKafka, Flume

FILESYSTEMHDFS

RELATIONALKUDU

NosqlHBase

OtherObject Store

OPERATIONSCloudera ManagerCloudera Director

DATA MANAGEMENTCloudera Navigator

Encrypt and KeyTrustee Optimizer

Spark Impala SOLR

UNIFIEDSERVICES

sTREAM SQL SearchotherPig MapReduce

Spark, Hive

BATCH





Spark

Either used solely on data streams or leveraging other components (e.g. HDFS), Spark delivers high-speed processing and high-level APIs for Machine Learning, Graph, SQL and streaming. This allows data to be loaded in-memory and queried repeatedly, making it particularly apt for IoT scenarios.

Kafka

This framework is a distributed publish-subscribe messaging system that was initially developed at LinkedIn. A few years later became a part of the Apache project and now is a fast, scalable, distributed, partitioned and replicated commit log service, that is rapidly becoming the standard for event queuing. Kudu

The first native Hadoop storage engine that supports both high-throughput analytics and low-latency random access, radically make Hadoop architectures easier for increasingly common real-time analytics use cases that are prevalent on IoT scenarios.





Impala

As an integrated part of CDH, Impala is the open source, analytic MPP database that provides the fastest time-to-insight on large volumes of data. Hundreds of companies are using Impala to power their BI and SQL analytic workloads.

Hive

This framework was originally developed by Facebook. It is a data warehouse infrastructure built using Hadoop that provides a simple, SQL-like language called HiveQL, while still maintaining full support for MapReduce. Cloudera Hadoop enables Hive to run on Spark rather than Map Reduce, making it radically faster. HBase

NoSQL wide-column database, designed to run on top of HDFS. It has been modelled after Google’s BigTable and written in Java. The objective was to provide BigTable-like capabilities to Hadoop, such as storage for sparse data and wide-column data storage.





Open Standards

The set of data storage and processing technologies which define the Apache Hadoop ecosystem is expanding and ever-improving, covering a very diverse set of customer use cases, including the most demanding mission-critical enterprise applications.

Cloudera is the top solution, 100% supported by open source Hadoop distribution (CDH). Therefore, it’s responsible for shaping the evolution of the Hadoop platform, making it faster, easier to work with, and more secure.

Cloudera contributes, supports and helps in the creation of new enterprise-focused capabilities that include SQL analytics on Hadoop, encryption, and fine-grained access control.





Robust Partner Ecosystem

It comes as no surprise that Cloudera has a very dynamic group of partners, and therefore a rich ecosystem: Independent Software Vendors, Hardware and Cloud vendors’ and Global System Integrators.

With these solutions, customers have the option to choose their technology stack and rest assured that it will work as it was a single solution, as it’s supported by their specific vendors.

Also, in order to drive innovation across the Hadoop ecosystem and ensure that its customers have constant access to the leading, production-ready apps built on the most popular tools in Hadoop, such as Apache Spark, Impala, Kudu, Kafka and many others.


ExploringTop Use Casesby Industry

4





Use Cases Customer Case Study – Description

Connected Vehicles

With the objective of improving uptime and reducing fleet maintenance costs by 30-40%, one of the leading auto manufacturers in North America is using Cloudera as their data management platform. It allows them to monitor the condition of 150,000+ trucks in real-time.

Predictive Maintenance – Industrial IoT

Cloudera is being used by a leading industrial automation company based in North America as an IoT setting to gather, store and examine petabytes of sensor data. This data is streaming from thousands of different manufacturing systems in real-time and is helping with predictive maintenance and in eliminating machine downtime.

Predictive Maintenance – Heavy Machinery

With Cloudera, one of the biggest heavy equipment fleet constructers in North America is being capable of analysing large volumes of data at a high velocity. This data is collected from sensors and is being used to continuously monitor performance of their fleet and to do predictive maintenance as well as advanced defect detection.

Telematics & Usage Based Insurance

Using Cloudera and the power of Hadoop, a large European auto insurance company is now able to collect, store, and study its data in real-time from millions of black box devices installed in their clients’ vehicles. The main objective is to personalize auto insurance coverage & reduce claims by 30%.

Exploring Top Use Cases by Industry





Use Cases Customer Case Study – Description

Smart Buildings

One of the busiest airports in Europe is using Cloudera on Azure to gather, secure, and correlate sensor (IoT) data. This is collected from equipment’s within the airport: escalators, elevators, and baggage carousels and the objectives are to improve airport efficiency and customers’ safety.

Connected Homes

Together with Intel, Michael J Fox Foundation is using Cloudera as a data management platform. They gather and study over 300 readings per second, collected from thousands of wearables used by Parkinson patients. This in an effort to accelerate the discovery of a cure for this disease.

Smart Healthcare

A top home automation and security company in the US is looking to integrate data from over 20+ sensors, from millions of customers’ homes. This resulted in useful information regarding safety and savings.

Utility Analytics

Opower, a customer engagement platform tailor-made for utilities, has joined Cloudera to bring together utility consumption data. They collect the data from smart meters along with weather, consumer behaviour, and other disparate sources of information to help save over $500 Million for subscribers.

Exploring Top Use Cases by Industry


Conclusion

All devices/machines that can be connected, will be connected. So, companies need to consider the opportunity, strategy, and the technology.

Currently, the Internet of Things has been gaining power in a variety of industries, as well as the need for data management platforms to process, store, manage and analyse large volumes of data from IoT deployments.

The technological piece that is centric to any IoT strategy is Hadoop. Its versatility (it keeps being pushed and pushed), low cost and ease of growth, are key factors.

In terms of skills, organizations will need several experts in the entire spectrum, from infrastructure architecture, setup, and deployment, including security setup to application development using Spark, Kafka, Impala, Hive, HBase, and MapReduce to name a few.





How Can We Help?

Big Data Consulting

Big Data Development

Our Big Data experts with mastery in Cloudera Hadoop and NoSQL Databases can help you develop NRT (near real-time) and batch data pipelines for all types of data.

Big Data Operations

We help customers define, install, configure, manage and tune a distributed data environment. We can help you install and adopt Big Data software such as Cloudera Hadoop, Mongo DB, and Datastax/Cassandra.

You know your business. We know DATA and we strive to work with you to envision all possible ways to extract value from it, in order to define and develop a Big Data execution plan.





External References

[CLOUDERA] "Cloudera for Internet of Things".

http://pt.slideshare.net/cloudera/cloudera-for-internet-of-things.

September 1, 2016

[MORGAN JACOB] “A Simple Explanation Of 'The Internet Of Things'”.

http://www.forbes.com/sites/jacobmorgan/2014/05/13/simple-explanation-

internet-things-that-anyone-can-understand/#1ee097a86828

May 14, 2014

[RAJA VIJAY] “IoT – It’s all about the Data”.

https://vision.cloudera.com/hadoop-the-data-management-platform-for-iot/

May 5, 2016

[RAJA VIJAY] “Hadoop – A key enabler for IoT”.

https://vision.cloudera.com/using-apache-hadoop-to-derive-value-from-iot/

July 13, 2016

[KURANA AMANDEEP] “What is the ‘Internet of Things’?” .

https://vision.cloudera.com/what-is-the-internet-of-things/

September 25, 2015




http://pt.slideshare.net/cloudera/cloudera-for-internet-of-things

http://www.forbes.com/sites/jacobmorgan/2014/05/13/simple-explanation-internet-things-that-anyone-ca

http://www.forbes.com/sites/jacobmorgan/2014/05/13/simple-explanation-internet-things-that-anyone-ca

https://vision.cloudera.com/hadoop-the-data-management-platform-for-iot/

https://vision.cloudera.com/using-apache-hadoop-to-derive-value-from-iot/

https://vision.cloudera.com/what-is-the-internet-of-things/

www.xpand-it.com

LONDON

1 Primrose Street

London, EC2A 2EX

United Kingdom

+44 845 867 0875

LISBON

Rua do Mar Vermelho nº2

Fração 2.3

1990-152 Lisbon, Portugal

+351 218 967 150

VIANA DO CASTELO

Rua de Fornelos nº7

Viana do Castelo

4900-709 Portugal

+351 218 967 150

http://www.xpand-it.com

http://www.xpand-it.com/si-internacionalizacao/

Cloudera to Support Your IOT Strategy 1 - Xpand IT

Documents

Transcript of Cloudera to Support Your IOT Strategy 1 - Xpand IT