Skip to main content

Hadoop Framework and Ecosystem

Hadoop Framework and Ecosystem
In the previous blog on Hadoop Tutorial, we discussed about Hadoop, its features and core components. Now, the next step forward is to understand Hadoop Ecosystem. It is an essential topic to understand before you start working with Hadoop. 
Hadoop Ecosystem is neither a programming language nor a service, it is a platform or framework which solves big data problems. You can consider it as a suite which encompasses a number of services (ingesting, storing, analyzing and maintaining) inside it. Let us discuss and get a brief idea about how the services work individually and in collaboration.
Below are the Hadoop components, that together form a Hadoop ecosystem, I will be covering each of them in this blog:



  • HDFS -> Hadoop Distributed File System
  • YARN -> Yet Another Resource Negotiator
  • MapReduce -> Data processing using programming
  • Spark -> In-memory Data Processing
  • PIG, HIVE-> Data Processing Services using Query (SQL-like)
  • HBase -> NoSQL Database
  • Mahout, Spark MLlib -> Machine Learning
  • Apache Drill -> SQL on Hadoop
  • Zookeeper -> Managing Cluster
  • Oozie -> Job Scheduling
  • Flume, Sqoop -> Data Ingesting Services
  • Solr & Lucene -> Searching & Indexing 
  • Ambari -> Provision, Monitor and Maintain cluster

I hope this blog is informative and added value to you. If you are interested to learn more, you can go through this blog which tells you how Big Data is used in Industries and How Hadoop Is Revolutionizing Analytics.

Comments

Post a Comment

Popular posts from this blog

Machine Learning & Deep Learning

Machine Learning & Deep Learning Understanding the latest advancements in artificial intelligence can seem overwhelming, but it really boils down to two concepts you’ve likely heard of before:  machine learning  and  deep learning . These terms are often thrown around in ways that can make them seem like interchangeable buzzwords, hence why it’s important to understand the differences. And those differences should be known! Examples of machine learning and deep learning are everywhere. It’s how Netflix knows which show you’ll want to watch next or how Facebook knows whose face is in a photo. And it’s how a customer service representative will know if you’ll be satisfied with their support before you even take a customer satisfaction (CSAT) survey. So what are these concepts that dominate the conversations about artificial intelligence and how exactly are they different? What is machine learning? Here’s a basic definition of machine learn...
Post a Comment
Read more

Hadoop As Big Data

Introduction: In this blog, I will discuss Big Data, its characteristics, different sources of Big Dataand some key components of Hadoop Framework. In the two part blog series, I will cover the basics of Hadoop Ecosystem. Let us start with Big Data and its importance in Hadoop Framework. Ethics, privacy, security measures are very important and need to be taken care while dealing with the challenges of Big Data. Big Data: When the Data itself becomes the part of the problem. Data is crucial for all organizations. It has to be stored for future use. We can refer the term Big Data as the data, which is beyond the storage capacity and the processing power of an organization. What are the sources of this huge data? There are different sources of data such as the social networks, CCTV cameras, sensors, online shopping portals, hospitality data, GPS, automobile industry etc., that generate data massively. Big Data can be characterized as: The Volume of the Data Velocit...
Post a Comment
Read more

Automatic Builds With GCP Cloud Build

Automatic Builds With GCP Cloud Build If you are looking for an easy way to automatically build your application in the cloud, then maybe Google Cloud Platform (GCP) Cloud Build is for you. In this post, we will build a Spring Boot Maven project with Cloud Build, create a Docker image for it, and push it to GCP Container Registry. 1. Introduction Cloud Build is the build server tooling of GCP, something similar as Jenkins. But, Cloud Build is available out-of-the-box in your GCP account and that is a major advantage. The only thing you will need is a build configuration file in your git repository containing the build steps. Each build step is running in its own Docker container. Several cloud builders which can be used as a build step are generally available. You can read more about Cloud Build on the  overview  and  concepts  website of GCP. There are three categories of build steps: Official  cloud builders provided by GCP; Community  cloud ...
Post a Comment
Read more