![\"\"\/](\"https:\/\/lh4.googleusercontent.com\/hl_oFvZyR7esbGxzB_cLxNl4TUl5TaylI980JbL75_iBCO3Yss05AoGMlsAf_iFJvJwWrhDr1Jw0L26-vhiHG7pZo3YHlQEo0qjU-kjLXu2WS60QCDtmnSnY2EfV_BvKgaEaomd1\")
<\/figure>\n\n\n\n- Hadoop Common<\/strong>: The common utilities that support the other Hadoop modules.<\/li>
- Hadoop Distributed File System (HDFS™)<\/strong>: A distributed file system that provides high-throughput access to application data.<\/li>
- Hadoop YARN<\/strong>: A framework for job scheduling and cluster resource management.<\/li>
- Hadoop MapReduce<\/strong>: A YARN-based system for parallel processing of large data sets.<\/li>
- Hadoop Ozone<\/strong><\/a>: An object store for Hadoop<\/li>
- Ambari™<\/strong><\/a>: A web-based tool for provisioning, managing, and monitoring Apache Hadoop clusters which includes support for Hadoop HDFS, Hadoop MapReduce, Hive, HCatalog, HBase, ZooKeeper, Oozie, Pig and Sqoop. Ambari also provides a dashboard for viewing cluster health such as heatmaps and ability to view MapReduce, Pig and Hive applications visually alongwith features to diagnose their performance characteristics in a user-friendly manner.<\/li>
- Avro™<\/strong><\/a>: A data serialization system.<\/li>
- Cassandra™<\/strong><\/a>: A scalable multi-master database with no single points of failure.<\/li>
- Chukwa™<\/strong><\/a>: A data collection system for managing large distributed systems.<\/li>
- HBase™<\/strong><\/a>: A scalable, distributed database that supports structured data storage for large tables.<\/li>
- Hive™<\/strong><\/a>: A data warehouse infrastructure that provides data summarization and ad hoc querying.<\/li>
- Mahout™<\/strong><\/a>: A Scalable machine learning and data mining library.<\/li>
- Pig™<\/strong><\/a>: A high-level data-flow language and execution framework for parallel computation.<\/li>
- Spark™<\/strong><\/a>: A fast and general compute engine for Hadoop data. Spark provides a simple and expressive programming model that supports a wide range of applications, including ETL, machine learning, stream processing, and graph computation.<\/li>
- Submarine<\/strong><\/a>: A unified AI platform which allows engineers and data scientists to run Machine Learning and Deep Learning workload in distributed clusters.<\/li>
- Tez™<\/strong><\/a>: A generalized data-flow programming framework, built on Hadoop YARN, which provides a powerful and flexible engine to execute an arbitrary DAG of tasks to process data for both batch and interactive use-cases. Tez is being adopted by Hive™, Pig™ and other frameworks in the Hadoop ecosystem, and also by other commercial software (e.g. ETL tools), to replace Hadoop™ MapReduce as the underlying execution engine.<\/li>
- ZooKeeper™<\/strong><\/a>: A high-performance coordination service for distributed applications.<\/li><\/ul>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
HDFS<\/h2>\n\n\n\n
- Breaks files into blocks<\/li>
- Stores across several commodity computers and each block is replicated more than once to ensure data reliability in case one node goes down. <\/li><\/ol>\n\n\n\n
Name Node – is like an index to the files(blocks) stored and data nodes. (Namenode )
Data Node – it actually stores the data<\/p>\n\n\n\nClient node asks for a file to Name Node and then goes to get it from the data nodes.<\/p>\n\n\n\n
<\/figure>\n\n\n\n
Writing – Client node <\/strong>hands over to Name node<\/strong> and it hands over it the name of data node.<\/strong> The client talks to Data node which inturn talk to other data nodes to get the file saved across them along with replication. They all respond back to the Client node and to Name Node .
<\/p>\n\n\n\n
NameNode Resilience<\/p>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
MapReduce<\/h2>\n\n\n\n
<\/figure>\n\n\n\n
- Mapper – maps to key value pairs<\/li>
- Shuffle and sort – groups and sorts the key value pairs<\/li>
- Reducer – Reduces the input to a single value <\/li><\/ol>\n\n\n\n
Below is example to find the no. of movies rated by each user<\/p>\n\n\n\n<\/figure>\n\n\n\n
What’s happening in the system. Client node talks to YARN which talks to NodeManager which in turn talks to Node and they output it on HDFS. If Node goes down then NodeManager shifts\/restarts process on another Node, if NodeManager goes down, its handled by YARN, if YARN goes down then High availability (HA) replica of it takes over (as discussed previously).
<\/p>\n\n\n\n
An example where we want to display Movie id and count of ratings for the movie. (Sort movies according its rating)<\/p>\n\n\n\n
<\/figure>\n\n\n\n
The need for a second reducer is that we needed sorted output (as per the count of ratings). The map step only maps but the framework does the \u201csort and shuffle\u201d on it according to the key. Note – framework takes everything as string so we need to zero pad the numbers for proper sorting.<\/strong><\/p>\n\n\n\n
The key being movieId in mapper would sort according to it. So, we change the order of key and value in the reducer. This ensures that <count,movieId> sorts in the order of count.
1 , abc(assume string as movieid)<\/p>\n\n\n\n2, bef, gef<\/p>\n\n\n\n
100, ded<\/p>\n\n\n\n
Now to display it in sorted order according to no of ratings, another reducer which reverses the key and value, and in this sort is introduced by sorting algo and not by framework.<\/p>\n\n\n\nAbc, 1<\/p>\n\n\n\n
Bef , 2<\/p>\n\n\n\n
Gef, 2<\/p>\n\n\n\n
Ded, 100<\/p>\n\n\n\n
Pig<\/h2>\n\n\n\n
Pig is a data flow language and execution engine. Pig has a scripting language pig-latin in which you can write map reduce steps\/functions. They give a good performance too and it’s easier to write it than map-reduce using Python\/Java.<\/p>\n\n\n\n
<\/figure>\n\n\n\n
Built on top of MapReduce and Tez. Use Pig by 1. Grunt 2.Ambari\/Hue 3. Script <\/p>\n\n\n\n
Example of getting top rated movies<\/p>\n\n\n\n
<\/figure>\n\n\n\n
If you execute this with Tez integrated , it runs faster.<\/p>\n\n\n\n
Apache Spark<\/h2>\n\n\n\n
A fast and general engine for large scale data processing.<\/p>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
Creating RDDs<\/p>\n\n\n\n
RDD is a Resilient Distributed Dataset. It is a data set but specialized on which is used by the spark framework to distribute it across various nodes. <\/p>\n\n\n\n
<\/figure>\n\n\n\n
Hive<\/h2>\n\n\n\n
Apache Hive<\/strong> is a data warehouse software project built on top of Apache Hadoop for providing data query and analysis. <\/p>\n\n\n\n
<\/figure>\n\n\n\n
When to use<\/p>\n\n\n\n
<\/figure>\n\n\n\n
Why not hive?<\/p>\n\n\n\n
<\/figure>\n\n\n\n
How Hive Works?<\/h3>\n\n\n\n
Let’s say you create a table. So it will store the table data into HDFS and will be stored as a delimited text file. There is no structure information saved here.<\/p>\n\n\n\n
Besides this it maintains a \u201cmetastore\u201d which stores the metadata about this.<\/p>\n\n\n\n
Hive uses HQL to get data.<\/p>\n\n\n\n
<\/figure>\n\n\n\n
Load data deals with BigData. Load Data local is non big data.
Managed table – hive takes ownership of the table. So if you write a drop table then it will drop data and structure.
External table- hive does not take ownership of it. So if you write a drop table then it will drop data but not<\/strong> its structure. This is useful in cases where some other Big Data applications want to use the structure of table<\/p>\n\n\n\nPartitioning<\/p>\n\n\n\n
Hive works very well with partitioning. It will manage the partitions separately. So lets say in case of address it can very well partition with the country name. This will improvise the query execution time.<\/p>\n\n\n\n
Alternative to Hive – IMPALA <\/strong>(Cloudera) – faster than Hive<\/p>\n\n\n\n
SQOOP<\/h2>\n\n\n\n
Ingestion framework . Helps to import from SQL based DBs to HDFC\/Hive and export to SQL as well. It also uses map reduce . When you give it a job it uses several mappers to get the task executed.<\/p>\n\n\n\n
Import data from sql to HDFS<\/p>\n\n\n\n
<\/figure>\n\n\n\n
Add –hive-import to import to hive<\/p>\n\n\n\n
<\/figure>\n\n\n\n
Scaling up<\/p>\n\n\n\n
<\/figure>\n\n\n\n
Sample Architecture with NoSQL and BigData<\/p>\n\n\n\n
<\/figure>\n\n\n\n
HBase<\/h2>\n\n\n\n
No SQL DB made from Google\u2019s BigTable<\/p>\n\n\n\n
<\/figure>\n\n\n\n
HMaster stores the partitions and information on where is what (on which region server). <\/p>\n\n\n\n
ZooKeeper knows about the master and can contact another master if one goes down.<\/p>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
Column family can store multiple columns.
A cell can have multiple versions of them based on timestamp.<\/p>\n\n\n\nOne can use Java\/Python to populate into HBase. If the file already exists on HDFS then one can use PIG to load data into Hive.<\/p>\n\n\n\n
Alternative to Hbase – Accumulo (better security – cell based access control)<\/p>\n\n\n\n
Cassandra<\/h2>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
MongoDB <\/h2>\n\n\n\n
Managing HuMONGOus data<\/p>\n\n\n\n
It favours consistency over availability. It has a Document based data Model so you can store JSON\/XML.<\/p>\n\n\n\n
No schema is enforced.If you want it can be enforced but not a mandate.<\/p>\n\n\n\n
MongoDB Terminology<\/p>\n\n\n\n
1 .Databases<\/p>\n\n\n\n
2. Collections – Tables<\/p>\n\n\n\n
3. Documents – Rows<\/p>\n\n\n\n
<\/figure>\n\n\n\n
Sharding<\/p>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
Query Engines <\/h2>\n\n\n\n
Apache Drill<\/h2>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
It’s like SQL for the entire ecosystem. You can get data from MongoDB and join it with Hive and so on. There is no learning curve as it uses SQL data.<\/p>\n\n\n\n
So why would one want to use Hive, MongoDB etc? Well it’s not very efficient with Big Joins.<\/p>\n\n\n\n
<\/figure>\n\n\n\n
Apache Phoenix<\/h2>\n\n\n\n
It’s a query engine especially for HBase. So you can use SQLs to query into HBase instead of writing a Java\/Python program. It’s quick. Also connects to Pig etc<\/p>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
Presto <\/h2>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
It’s made by Facebook. <\/p>\n\n\n\n
Presto can query Cassandra <\/strong>too besides others !<\/p>\n\n\n\n
Also, the UI is very good compared to Drill.<\/p>\n\n\n\n
<\/figure>\n\n\n\n
YARN – Yet Another Resource Negotiator<\/h2>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
Apache Tez<\/h2>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
Parallelizes steps or skips steps or removes redundant steps to improve the performance over map Reduce.<\/p>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
Mesos<\/h2>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
ZooKeeper<\/h2>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
Zookeeper favours availability over consistency <\/p>\n\n\n\n
Oozie<\/h2>\n\n\n\n
It is used to make workflows. So lets say you need coordination between various individual jobs like SQL, Hive, MongoDB etc then we can write an xml for the steps to be performed. Each of these steps will be coordinated by oozie.<\/p>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
Zeppelin<\/h2>\n\n\n\n
It provides an interactive way to write BigData programs. It can connect to various technologies and works like a workbook.<\/p>\n\n\n\n
<\/figure>\n\n\n\n
HUE – Hadoop User Experience <\/h2>\n\n\n\n
<\/figure>\n\n\n\n
It is not open source and managed by cloud era. It has an inbuilt Oozie editor to write workflows. <\/p>\n\n\n\n
Kafka <\/h2>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
Alternative to Kafka – Amazon Kinesis<\/p>\n\n\n\n
Flume<\/h2>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
Spark Streaming<\/h2>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
Structured Streaming<\/p>\n\n\n\n
Uses DataSets instead of DataFrame<\/p>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
Apache Storm<\/h2>\n\n\n\n
Storm vs Spark Streaming<\/p>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
Sliding windows – have a common \/overlapping part of the processing. Example 10 secs is window size so every sec 9 secs will be common to the previous one.<\/p>\n\n\n\n
Tumbling windows – it has no overlap.<\/p>\n\n\n\n
Flink<\/h2>\n\n\n\n
Can manage both event based and batch based both<\/p>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n\n
<\/figure>\n\n\n
\n <\/div>\n","protected":false},"excerpt":{"rendered":"Hadoop The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. Hadoop Common: The common utilities that support the other Hadoop modules. Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data. Hadoop […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[97],"tags":[],"class_list":["post-1785","post","type-post","status-publish","format-standard","hentry","category-tech-learnings","has-post-title","has-post-date","has-post-category","has-post-tag","has-post-comment","has-post-author",""],"yoast_head":"\n
Big Data Udemy Course Notes » Gaurav Wadhwani<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Big Data Udemy Course Notes » Gaurav Wadhwani\" \/>\n<meta property=\"og:description\" content=\"Hadoop The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. Hadoop Common: The common utilities that support the other Hadoop modules. Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data. Hadoop […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/\" \/>\n<meta property=\"og:site_name\" content=\"Gaurav Wadhwani\" \/>\n<meta property=\"article:published_time\" content=\"2021-04-10T08:43:35+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-04-10T08:43:41+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/lh4.googleusercontent.com\/hl_oFvZyR7esbGxzB_cLxNl4TUl5TaylI980JbL75_iBCO3Yss05AoGMlsAf_iFJvJwWrhDr1Jw0L26-vhiHG7pZo3YHlQEo0qjU-kjLXu2WS60QCDtmnSnY2EfV_BvKgaEaomd1\" \/>\n<meta name=\"author\" content=\"Gaurav Wadhwani\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Gaurav Wadhwani\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"25 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/\"},\"author\":{\"name\":\"Gaurav Wadhwani\",\"@id\":\"https:\/\/gauravw.com\/blog\/#\/schema\/person\/9a05a9c3487f35f6b4577c6956cf252e\"},\"headline\":\"Big Data Udemy Course Notes\",\"datePublished\":\"2021-04-10T08:43:35+00:00\",\"dateModified\":\"2021-04-10T08:43:41+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/\"},\"wordCount\":1578,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/gauravw.com\/blog\/#\/schema\/person\/9a05a9c3487f35f6b4577c6956cf252e\"},\"image\":{\"@id\":\"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/lh4.googleusercontent.com\/hl_oFvZyR7esbGxzB_cLxNl4TUl5TaylI980JbL75_iBCO3Yss05AoGMlsAf_iFJvJwWrhDr1Jw0L26-vhiHG7pZo3YHlQEo0qjU-kjLXu2WS60QCDtmnSnY2EfV_BvKgaEaomd1\",\"articleSection\":[\"Tech Learnings\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/\",\"url\":\"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/\",\"name\":\"Big Data Udemy Course Notes » Gaurav Wadhwani\",\"isPartOf\":{\"@id\":\"https:\/\/gauravw.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/lh4.googleusercontent.com\/hl_oFvZyR7esbGxzB_cLxNl4TUl5TaylI980JbL75_iBCO3Yss05AoGMlsAf_iFJvJwWrhDr1Jw0L26-vhiHG7pZo3YHlQEo0qjU-kjLXu2WS60QCDtmnSnY2EfV_BvKgaEaomd1\",\"datePublished\":\"2021-04-10T08:43:35+00:00\",\"dateModified\":\"2021-04-10T08:43:41+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/#primaryimage\",\"url\":\"https:\/\/lh4.googleusercontent.com\/hl_oFvZyR7esbGxzB_cLxNl4TUl5TaylI980JbL75_iBCO3Yss05AoGMlsAf_iFJvJwWrhDr1Jw0L26-vhiHG7pZo3YHlQEo0qjU-kjLXu2WS60QCDtmnSnY2EfV_BvKgaEaomd1\",\"contentUrl\":\"https:\/\/lh4.googleusercontent.com\/hl_oFvZyR7esbGxzB_cLxNl4TUl5TaylI980JbL75_iBCO3Yss05AoGMlsAf_iFJvJwWrhDr1Jw0L26-vhiHG7pZo3YHlQEo0qjU-kjLXu2WS60QCDtmnSnY2EfV_BvKgaEaomd1\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/gauravw.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Big Data Udemy Course Notes\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/gauravw.com\/blog\/#website\",\"url\":\"https:\/\/gauravw.com\/blog\/\",\"name\":\"Gaurav Wadhwani\",\"description\":\"Where I write \/ scribble\",\"publisher\":{\"@id\":\"https:\/\/gauravw.com\/blog\/#\/schema\/person\/9a05a9c3487f35f6b4577c6956cf252e\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/gauravw.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/gauravw.com\/blog\/#\/schema\/person\/9a05a9c3487f35f6b4577c6956cf252e\",\"name\":\"Gaurav Wadhwani\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/gauravw.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/88929454012064ffbe95370287faa36b?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/88929454012064ffbe95370287faa36b?s=96&d=mm&r=g\",\"caption\":\"Gaurav Wadhwani\"},\"logo\":{\"@id\":\"https:\/\/gauravw.com\/blog\/#\/schema\/person\/image\/\"},\"sameAs\":[\"http:\/\/gauravw.com\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Big Data Udemy Course Notes » Gaurav Wadhwani","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/","og_locale":"en_US","og_type":"article","og_title":"Big Data Udemy Course Notes » Gaurav Wadhwani","og_description":"Hadoop The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. Hadoop Common: The common utilities that support the other Hadoop modules. Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data. Hadoop […]","og_url":"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/","og_site_name":"Gaurav Wadhwani","article_published_time":"2021-04-10T08:43:35+00:00","article_modified_time":"2021-04-10T08:43:41+00:00","og_image":[{"url":"https:\/\/lh4.googleusercontent.com\/hl_oFvZyR7esbGxzB_cLxNl4TUl5TaylI980JbL75_iBCO3Yss05AoGMlsAf_iFJvJwWrhDr1Jw0L26-vhiHG7pZo3YHlQEo0qjU-kjLXu2WS60QCDtmnSnY2EfV_BvKgaEaomd1","type":"","width":"","height":""}],"author":"Gaurav Wadhwani","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Gaurav Wadhwani","Est. reading time":"25 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/#article","isPartOf":{"@id":"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/"},"author":{"name":"Gaurav Wadhwani","@id":"https:\/\/gauravw.com\/blog\/#\/schema\/person\/9a05a9c3487f35f6b4577c6956cf252e"},"headline":"Big Data Udemy Course Notes","datePublished":"2021-04-10T08:43:35+00:00","dateModified":"2021-04-10T08:43:41+00:00","mainEntityOfPage":{"@id":"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/"},"wordCount":1578,"commentCount":0,"publisher":{"@id":"https:\/\/gauravw.com\/blog\/#\/schema\/person\/9a05a9c3487f35f6b4577c6956cf252e"},"image":{"@id":"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/#primaryimage"},"thumbnailUrl":"https:\/\/lh4.googleusercontent.com\/hl_oFvZyR7esbGxzB_cLxNl4TUl5TaylI980JbL75_iBCO3Yss05AoGMlsAf_iFJvJwWrhDr1Jw0L26-vhiHG7pZo3YHlQEo0qjU-kjLXu2WS60QCDtmnSnY2EfV_BvKgaEaomd1","articleSection":["Tech Learnings"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/","url":"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/","name":"Big Data Udemy Course Notes » Gaurav Wadhwani","isPartOf":{"@id":"https:\/\/gauravw.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/#primaryimage"},"image":{"@id":"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/#primaryimage"},"thumbnailUrl":"https:\/\/lh4.googleusercontent.com\/hl_oFvZyR7esbGxzB_cLxNl4TUl5TaylI980JbL75_iBCO3Yss05AoGMlsAf_iFJvJwWrhDr1Jw0L26-vhiHG7pZo3YHlQEo0qjU-kjLXu2WS60QCDtmnSnY2EfV_BvKgaEaomd1","datePublished":"2021-04-10T08:43:35+00:00","dateModified":"2021-04-10T08:43:41+00:00","breadcrumb":{"@id":"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/#primaryimage","url":"https:\/\/lh4.googleusercontent.com\/hl_oFvZyR7esbGxzB_cLxNl4TUl5TaylI980JbL75_iBCO3Yss05AoGMlsAf_iFJvJwWrhDr1Jw0L26-vhiHG7pZo3YHlQEo0qjU-kjLXu2WS60QCDtmnSnY2EfV_BvKgaEaomd1","contentUrl":"https:\/\/lh4.googleusercontent.com\/hl_oFvZyR7esbGxzB_cLxNl4TUl5TaylI980JbL75_iBCO3Yss05AoGMlsAf_iFJvJwWrhDr1Jw0L26-vhiHG7pZo3YHlQEo0qjU-kjLXu2WS60QCDtmnSnY2EfV_BvKgaEaomd1"},{"@type":"BreadcrumbList","@id":"https:\/\/gauravw.com\/blog\/2021\/04\/big-data-udemy-course-notes\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/gauravw.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Big Data Udemy Course Notes"}]},{"@type":"WebSite","@id":"https:\/\/gauravw.com\/blog\/#website","url":"https:\/\/gauravw.com\/blog\/","name":"Gaurav Wadhwani","description":"Where I write \/ scribble","publisher":{"@id":"https:\/\/gauravw.com\/blog\/#\/schema\/person\/9a05a9c3487f35f6b4577c6956cf252e"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/gauravw.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/gauravw.com\/blog\/#\/schema\/person\/9a05a9c3487f35f6b4577c6956cf252e","name":"Gaurav Wadhwani","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/gauravw.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/88929454012064ffbe95370287faa36b?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/88929454012064ffbe95370287faa36b?s=96&d=mm&r=g","caption":"Gaurav Wadhwani"},"logo":{"@id":"https:\/\/gauravw.com\/blog\/#\/schema\/person\/image\/"},"sameAs":["http:\/\/gauravw.com"]}]}},"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","builder_content":"","_links":{"self":[{"href":"https:\/\/gauravw.com\/blog\/wp-json\/wp\/v2\/posts\/1785","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/gauravw.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/gauravw.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/gauravw.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/gauravw.com\/blog\/wp-json\/wp\/v2\/comments?post=1785"}],"version-history":[{"count":1,"href":"https:\/\/gauravw.com\/blog\/wp-json\/wp\/v2\/posts\/1785\/revisions"}],"predecessor-version":[{"id":1786,"href":"https:\/\/gauravw.com\/blog\/wp-json\/wp\/v2\/posts\/1785\/revisions\/1786"}],"wp:attachment":[{"href":"https:\/\/gauravw.com\/blog\/wp-json\/wp\/v2\/media?parent=1785"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/gauravw.com\/blog\/wp-json\/wp\/v2\/categories?post=1785"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/gauravw.com\/blog\/wp-json\/wp\/v2\/tags?post=1785"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
- Hadoop Distributed File System (HDFS™)<\/strong>: A distributed file system that provides high-throughput access to application data.<\/li>