Course Content
Hadoop Introduction
- Move computation not data
- Hadoop performance and data scale facts
- Hadoop in the context of other data stores
- The Apache Hadoop Project
- Hadoop - an inside view: MapReduce and HDFS
- The Hadoop Ecosystem
- What about NoSQL?
- Comparison with Other Systems
- RDBMS
- Grid Computing
- Volunteer Computing
- A Brief History of Hadoop
- Apache Hadoop and the Hadoop Ecosystem
- Hadoop Releases
MapReduce
- Analyzing the Data with Hadoop
- Map and Reduce
- Java MapReduceScaling Out
- Data FlowCombiner Functions
- Running a Distributed MapReduce Job
- Hadoop Streaming
- Ruby
- Python
- Hadoop Pipes
- Constructing the basic template of a MapReduce program
- Counting things
- Adapting for Hadoop's API changes
- Streaming in Hadoop
- Streaming with Unix commands
- Streaming with scripts
- Streaming with key/value pairs
- Streaming with the Aggregate package
- Improving performance with combiners
Distributing Data with HDFS
- The Design of HDFS
- HDFS Concepts
- Blocks
- Namenodes and Datanodes
- HDFS Federation
- HDFS High-Availability
- The Command-Line Interface
- Basic Filesystem Operations
- Hadoop Filesystems
- Interfaces
- The Java Interface
- Reading Data from a Hadoop URL
- Reading Data Using the FileSystem API
- Writing Data
- Directories
- Querying the Filesystem
- Deleting Data
- Data Flow
- Anatomy of a File Read
- Anatomy of a File Write
- Coherency Model
- Parallel Copying with distcp
- Keeping an HDFS Cluster Balanced
- Hadoop Archives
- Using Hadoop Archives
- Limitations
Understanding Hadoop I/O
- Data Integrity
- Data Integrity in HDFS
- LocalFileSystem
- ChecksumFileSystem
- Compression
- Codecs
- Compression and Input Splits
- Using Compression in MapReduce
- Serialization
- The Writable Interface
- Writable Classes
- Implementing a Custom Writable
- Serialization Frameworks
- Avro
- File-Based Data Structures
- SequenceFile
- MapFile
Advanced MapReduce
- Chaining MapReduce jobs
- Chaining MapReduce jobs in a sequence
- Chaining MapReduce jobs with complex dependency
- Chaining preprocessing and postprocessing steps
- Joining data from different sources
- Reduce-side joining
- Replicated joins using DistributedCache
- Semijoin: reduce-side join with map-side filtering
- Creating a Bloom filter
- What does a Bloom filter do?
- Implementing a Bloom filter
- Bloom filter in Hadoop version 0.20+
Writing Map-Reduce Applications
- The Configuration API
- Configuring the Development Environment
- Running Locally on Test Data
- Cluster Specs
- Cluster Setup and Installation
- Hadoop Configuration
- YARN Configuration
- Benchmarking a Hadoop Cluster
- Hadoop in the Cloud
- Tuning
- MapReduce Workflows
- Monitoring and debugging on a production cluster
- Tuning for performance
Map-Reduce Internals
- Anatomy of a MapReduce Job Run
- Classic MapReduce (MapReduce 1)
- YARN (MapReduce 2)
- Failures
- Failures in Classic MapReduce
- Failures in YARN
- Job Scheduling
- The Fair Scheduler
- The Capacity Scheduler
- Shuffle and Sort
- The Map Side
- The Reduce Side
- Configuration Tuning
- Task Execution
- The Task Execution Environment
- Speculative Execution
- Output Committers
- Task JVM Reuse
- Skipping Bad Records
Managing Hadoop
- Setting up parameter values for practical use
- Checking system's health
- Setting permissions
- Managing quotas
- Enabling trash
- Removing DataNodes
- Adding DataNodes
- Managing NameNode and Secondary NameNode
- Recovering from a failed NameNode
- Designing network layout and rack awareness
- Map-Reduce Features
- Counters
- Sorting
- Joins
- Side Data Distribution
- Map-Reduce Library
Map-Reduce Ecosystem
- Pig
- Thinking like a Pig
- Data flow language
- Data types
- User-defined functions
- Installing Pig
- Running Pig
- Managing the Grunt shell
- Learning Pig Latin through Grunt
- Speaking Pig Latin
- Data types and schemas
- Expressions and functions
- Relational operators
- Execution optimization
- Hive
- Installing and configuring Hive
- Example queries
- HiveQL in details
- Hive Sum-up
- Hbase
- Intoduction
- Concepts
- Clients
- Hbase vs RDBMS
- Thinking like a Pig
