Skip to main content

Hadoop As Big Data

Introduction:
In this blog, I will discuss Big Data, its characteristics, different sources of Big Dataand some key components of Hadoop Framework.
In the two part blog series, I will cover the basics of Hadoop Ecosystem.
Let us start with Big Data and its importance in Hadoop Framework. Ethics, privacy, security measures are very important and need to be taken care while dealing with the challenges of Big Data.

Big Data: When the Data itself becomes the part of the problem.
Data is crucial for all organizations. It has to be stored for future use. We can refer the term Big Data as the data, which is beyond the storage capacity and the processing power of an organization.

What are the sources of this huge data?
There are different sources of data such as the social networks, CCTV cameras, sensors, online shopping portals, hospitality data, GPS, automobile industry etc., that generate data massively.
Big Data can be characterized as:
  • The Volume of the Data
  • Velocity of the Data
  • The Variety of Data being processed
Volume of Data à Data is increasing rapidly in GB, TB, PB and so on, and requires huge disk space to store it.
Velocity of Data à Huge Data is stored in Data Centres to cater to the organizational needs. In order to get data to the local workstation high-speed data processors are needed.
Variety of Data à Data can be broadly classified into the following types-Structured, Unstructured & Semi structured.
Big Data = (Volume + Velocity + Variety) of Data
big data consulting services

Comments

Post a Comment

Popular posts from this blog

Data Engineering - Tools & Intro

Data Engineering - Tools & Intro So I just realized that I am here after a month or so. I was busy at work and traveling. I am starting a kind of new series, I say it Data Engineering Series in which I will be discussing different tools. Of course, I am not able to discuss the entire concept of Data Engineering neither I know it as I will be learning myself. What is Data Engineering? Data Engineering is all about developing, maintaining systems that are responsible for transferring data in large volumes and make it available for analysts and data scientists to use it for analyzing and data modeling. Data engineering is a superset of Data Science or the subset, not clear to me but the collaboration of data engineers and scientists fruits useful data-driven solutions. Data Engineering tools It consists of several tools. Some are dealing with data storage while others with analysis and ETL. Ofcourse, Apache Kafka is one of them. The others tools that I might be covering are Apache...
Post a Comment
Read more

Creating REST API With Spring Boot

Let's Create a REST API With Spring Boot In this post, we will learn how to create REST API with Spring Boot, JPA, Hibernate, and MySQL. Create the Spring Boot Project. Define Database configurations. Create an Entity Class. Create JPA Data Repository layer. Create Rest Controllers and map API requests. Create Unit Testing for API requests and run the unit testing. Build and run the Project. Requirement We need to create a simple REST API to store user data to MySQL database with that API so we can create, update, delete, and get users information. Create the Spring Boot Project First, go to  Spring Initializr  and create a project with below settings Web  — Full-stack web development with Tomcat and Spring MVC DevTools  — Spring Boot Development Tools JPA  — Java Persistence API including spring-data-JPA, spring-orm, and Hibernate MySQL  — MySQL JDBC driver Actuator  — Production-ready features to help you monitor...
Post a Comment
Read more

Basics Of Hadoop Ecosystem - 2

Basics of Hadoop Ecosystem – 2 Apache Ambari is a web-based tool for provisioning, managing, and monitoring Apache Hadoop clusters. It has a very simple yet highly interactive UI to install various tools and perform various management, configuration and monitoring tasks.  Introduction: In part 2 of the blog series I will cover the other core components of a Hadoop Framework including querying part, external integration, data exchange, co-ordination and managing as well as monitoring of Hadoop Clusters. Please refer to Basic’s of Hadoop Ecosystem  Part 1 ... QUERYING: Pig seems quite useful however; I am more of an SQL person. For those of us who still like SQL, we have SQL for Hadoop. HIVE Hive is a distributed data warehouse built on atop of HDFS to manage and organize large amounts of data. It provides a query based on SQL semantics  (HiveQL)  which is translated by the runtime engine to Map Reduce jobs for querying the data. Sche...
Post a Comment
Read more