Skip to main content

Basics Of Hadoop Ecosystem - 2

Basics of Hadoop Ecosystem – 2


Apache Ambari is a web-based tool for provisioning, managing, and monitoring Apache Hadoop clusters. It has a very simple yet highly interactive UI to install various tools and perform various management, configuration and monitoring tasks. 

Introduction:

In part 2 of the blog series I will cover the other core components of a Hadoop Framework including querying part, external integration, data exchange, co-ordination and managing as well as monitoring of Hadoop Clusters. Please refer to Basic’s of Hadoop Ecosystem Part 1...

QUERYING:
Pig seems quite useful however; I am more of an SQL person. For those of us who still like SQL, we have SQL for Hadoop.

HIVE
Hive is a distributed data warehouse built on atop of HDFS to manage and organize large amounts of data. It provides a query based on SQL semantics (HiveQL) which is translated by the runtime engine to Map Reduce jobs for querying the data.
  • Schematized data store for housing large amounts of raw data.
  • SQL-like environment to execute analysis and querying tasks on raw data in the HDFS.
  • Integration with the outside RDBMS applications.


apache data storage
EXTERNAL INTEGRATION:

Flume 
A distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data into HDFS. Flume's transports large quantities of event data using a streaming data flow architecture that is fault tolerant and failover recovery ready.
  • Transports large amount of event data (network traffic, logs, email messages)
  • Streams data from multiple sources into HDFS
  • Guarantees a reliable real-time data streaming to the Hadoop applications.

DATA EXCHANGE:

Sqoop 
Apache Sqoop is a tool designed for efficiently transferring bulk data between Apache Hadoop and external data stores such as relational databases, and enterprise data warehouses. Sqoop is widely used in most of the Big Data companies to transfer data between relational databases and Hadoop. Sqoop works with relational databases such as Teradata, Netezza, Oracle, MySQL, Postgres etc.
  • Sqoop automates most of the process, depending on the database to describe the schema of the data to be imported.
  • Sqoop uses Map Reduce framework to import and export the data, which provides parallel mechanism as well as the fault tolerance.
  • It provides Connectors for all the major RDBMS Databases.
  • It supports full/incremental load, parallel export/import of data and data compression.
  • It supports Kerberos Security Integration.

CO-ORDINATION:

Zookeeper
Apache Zookeeper is a coordination service for distributed application that enables synchronization across a cluster. It is a centralized repository where distributed applications can put in data and get data out of it.
  • Zookeeper is a Hadoop Admin tool used for managing the jobs in a cluster.
  • Zookeeper Hadoop is akin to a watch guard as it notifies any change of data in one node to the other.

PROVISIONING, MANAGING & MONITORING HADOOP CLUSTERS:

Ambari
Apache Ambari is a web-based tool for provisioning, managing, and monitoring Apache Hadoop clusters. It has a very simple yet highly interactive UI to install various tools and perform various management, configuration and monitoring tasks. Ambari provides a dashboard for viewing cluster health such as heat maps and ability to view MapReduce, Pig and Hive applications visually along with the features to diagnose their performance characteristics in a user-friendly manner.
  • Master services mapping with Nodes.
  • We can always choose the services that we want to install.
  • Stack Selection made easy. We can customize our services.
  • Apache Ambari provides us a simpler interface and saves lot of our efforts on installation, monitoring and managing so many components and their different installation steps along with monitoring controls at ease.

Conclusion:
Hadoop has been a very effective solution for companies dealing with extremely huge data. It is a much sought after tool in the industry for data management in the distributed systems. As it is an open source, it is easily available for the companies to leverage it for their use.

These are some highlights on Apache’s Hadoop ecosystem.  Documentation for these is available on Apache Software Foundation Website.
Hadoop and its ecosystem is expected to grow further and take on new roles even as other systems fill important roles.

Comments

Post a Comment

Popular posts from this blog

Machine Learning & Deep Learning

Machine Learning & Deep Learning Understanding the latest advancements in artificial intelligence can seem overwhelming, but it really boils down to two concepts you’ve likely heard of before:  machine learning  and  deep learning . These terms are often thrown around in ways that can make them seem like interchangeable buzzwords, hence why it’s important to understand the differences. And those differences should be known! Examples of machine learning and deep learning are everywhere. It’s how Netflix knows which show you’ll want to watch next or how Facebook knows whose face is in a photo. And it’s how a customer service representative will know if you’ll be satisfied with their support before you even take a customer satisfaction (CSAT) survey. So what are these concepts that dominate the conversations about artificial intelligence and how exactly are they different? What is machine learning? Here’s a basic definition of machine learn...
Post a Comment
Read more

Hadoop As Big Data

Introduction: In this blog, I will discuss Big Data, its characteristics, different sources of Big Dataand some key components of Hadoop Framework. In the two part blog series, I will cover the basics of Hadoop Ecosystem. Let us start with Big Data and its importance in Hadoop Framework. Ethics, privacy, security measures are very important and need to be taken care while dealing with the challenges of Big Data. Big Data: When the Data itself becomes the part of the problem. Data is crucial for all organizations. It has to be stored for future use. We can refer the term Big Data as the data, which is beyond the storage capacity and the processing power of an organization. What are the sources of this huge data? There are different sources of data such as the social networks, CCTV cameras, sensors, online shopping portals, hospitality data, GPS, automobile industry etc., that generate data massively. Big Data can be characterized as: The Volume of the Data Velocit...
Post a Comment
Read more

Automatic Builds With GCP Cloud Build

Automatic Builds With GCP Cloud Build If you are looking for an easy way to automatically build your application in the cloud, then maybe Google Cloud Platform (GCP) Cloud Build is for you. In this post, we will build a Spring Boot Maven project with Cloud Build, create a Docker image for it, and push it to GCP Container Registry. 1. Introduction Cloud Build is the build server tooling of GCP, something similar as Jenkins. But, Cloud Build is available out-of-the-box in your GCP account and that is a major advantage. The only thing you will need is a build configuration file in your git repository containing the build steps. Each build step is running in its own Docker container. Several cloud builders which can be used as a build step are generally available. You can read more about Cloud Build on the  overview  and  concepts  website of GCP. There are three categories of build steps: Official  cloud builders provided by GCP; Community  cloud ...
Post a Comment
Read more