Skip to main content

Hadoop Framework and Ecosystem

Hadoop Framework and Ecosystem
In the previous blog on Hadoop Tutorial, we discussed about Hadoop, its features and core components. Now, the next step forward is to understand Hadoop Ecosystem. It is an essential topic to understand before you start working with Hadoop. 
Hadoop Ecosystem is neither a programming language nor a service, it is a platform or framework which solves big data problems. You can consider it as a suite which encompasses a number of services (ingesting, storing, analyzing and maintaining) inside it. Let us discuss and get a brief idea about how the services work individually and in collaboration.
Below are the Hadoop components, that together form a Hadoop ecosystem, I will be covering each of them in this blog:



  • HDFS -> Hadoop Distributed File System
  • YARN -> Yet Another Resource Negotiator
  • MapReduce -> Data processing using programming
  • Spark -> In-memory Data Processing
  • PIG, HIVE-> Data Processing Services using Query (SQL-like)
  • HBase -> NoSQL Database
  • Mahout, Spark MLlib -> Machine Learning
  • Apache Drill -> SQL on Hadoop
  • Zookeeper -> Managing Cluster
  • Oozie -> Job Scheduling
  • Flume, Sqoop -> Data Ingesting Services
  • Solr & Lucene -> Searching & Indexing 
  • Ambari -> Provision, Monitor and Maintain cluster

I hope this blog is informative and added value to you. If you are interested to learn more, you can go through this blog which tells you how Big Data is used in Industries and How Hadoop Is Revolutionizing Analytics.

Comments

Post a Comment

Popular posts from this blog

Data Engineering - Tools & Intro

Data Engineering - Tools & Intro So I just realized that I am here after a month or so. I was busy at work and traveling. I am starting a kind of new series, I say it Data Engineering Series in which I will be discussing different tools. Of course, I am not able to discuss the entire concept of Data Engineering neither I know it as I will be learning myself. What is Data Engineering? Data Engineering is all about developing, maintaining systems that are responsible for transferring data in large volumes and make it available for analysts and data scientists to use it for analyzing and data modeling. Data engineering is a superset of Data Science or the subset, not clear to me but the collaboration of data engineers and scientists fruits useful data-driven solutions. Data Engineering tools It consists of several tools. Some are dealing with data storage while others with analysis and ETL. Ofcourse, Apache Kafka is one of them. The others tools that I might be covering are Apache...
Post a Comment
Read more

Creating REST API With Spring Boot

Let's Create a REST API With Spring Boot In this post, we will learn how to create REST API with Spring Boot, JPA, Hibernate, and MySQL. Create the Spring Boot Project. Define Database configurations. Create an Entity Class. Create JPA Data Repository layer. Create Rest Controllers and map API requests. Create Unit Testing for API requests and run the unit testing. Build and run the Project. Requirement We need to create a simple REST API to store user data to MySQL database with that API so we can create, update, delete, and get users information. Create the Spring Boot Project First, go to  Spring Initializr  and create a project with below settings Web  — Full-stack web development with Tomcat and Spring MVC DevTools  — Spring Boot Development Tools JPA  — Java Persistence API including spring-data-JPA, spring-orm, and Hibernate MySQL  — MySQL JDBC driver Actuator  — Production-ready features to help you monitor...
Post a Comment
Read more

Machine Learning & Deep Learning

Machine Learning & Deep Learning Understanding the latest advancements in artificial intelligence can seem overwhelming, but it really boils down to two concepts you’ve likely heard of before:  machine learning  and  deep learning . These terms are often thrown around in ways that can make them seem like interchangeable buzzwords, hence why it’s important to understand the differences. And those differences should be known! Examples of machine learning and deep learning are everywhere. It’s how Netflix knows which show you’ll want to watch next or how Facebook knows whose face is in a photo. And it’s how a customer service representative will know if you’ll be satisfied with their support before you even take a customer satisfaction (CSAT) survey. So what are these concepts that dominate the conversations about artificial intelligence and how exactly are they different? What is machine learning? Here’s a basic definition of machine learn...
Post a Comment
Read more