Apache Oozie
Foreword
Preface
1. Introduction to Oozie
Big Data Processing
A Recurrent Problem
A Common Solution: Oozie
A Simple Oozie Job
Oozie Releases
Some Oozie Usage Numbers
2. Oozie Concepts
Oozie Applications
Oozie Workflows
Oozie Coordinators
Oozie Bundles
Parameters, Variables, and Functions
Application Deployment Model
Oozie Architecture
3. Setting Up Oozie
Oozie Deployment
Basic Installations
Requirements
Build Oozie
Install Oozie Server
Hadoop Cluster
Start and Verify the Oozie Server
Advanced Oozie Installations
Configuring Kerberos Security
DB Setup
Shared Library Installation
Oozie Client Installations
4. Oozie Workflow Actions
Workflow
Actions
Action Execution Model
Action Definition
Action Types
MapReduce Action
Java Action
Pig Action
FS Action
Sub-Workflow Action
Hive Action
DistCp Action
Email Action
Shell Action
SSH Action
Sqoop Action
Synchronous Versus Asynchronous Actions
5. Workflow Applications
Outline of a Basic Workflow
Control Nodes
`<start>` and `<end>`
`<fork>` and `<join>`
`<decision>`
`<kill>`
`<OK>` and `<ERROR>`
Job Configuration
Global Configuration
Job XML
Inline Configuration
Launcher Configuration
Parameterization
EL Variables
EL Functions
EL Expressions
The job.properties File
Command-Line Option
The config-default.xml File
The `<parameters>` Section
Configuration and Parameterization Examples
Lifecycle of a Workflow
Action States
6. Oozie Coordinator
Coordinator Concept
Triggering Mechanism
Time Trigger
Data Availability Trigger
Coordinator Application and Job
Coordinator Action
Our First Coordinator Job
Coordinator Submission
Oozie Web Interface for Coordinator Jobs
Coordinator Job Lifecycle
Coordinator Action Lifecycle
Parameterization of the Coordinator
EL Functions for Frequency
Day-Based Frequency
Month-Based Frequency
Execution Controls
An Improved Coordinator
7. Data Trigger Coordinator
Expressing Data Dependency
Dataset
Example: Rollup
Parameterization of Dataset Instances
current(n)
latest(n)
Parameter Passing to Workflow
dataIn(eventName)
dataOut(eventName)
nominalTime()
actualTime()
dateOffset(baseTimeStamp, skipInstance, timeUnit)
formatTime(timeStamp, formatString)
A Complete Coordinator Application
8. Oozie Bundles
Bundle Basics
Bundle Definition
Why Do We Need Bundles?
Bundle Specification
Execution Controls
Bundle State Transitions
9. Advanced Topics
Managing Libraries in Oozie
Origin of JARs in Oozie
Design Challenges
Managing Action JARs
Supporting the User’s JAR
JAR Precedence in classpath
Oozie Security
Oozie Security Overview
Oozie to Hadoop
Oozie Client to Server
Supporting Custom Credentials
Supporting New API in MapReduce Action
Supporting Uber JAR
Cron Scheduling
A Simple Cron-Based Coordinator
Oozie Cron Specification
Emulate Asynchronous Data Processing
HCatalog-Based Data Dependency
10. Developer Topics
Developing Custom EL Functions
Requirements for a New EL Function
Implementing a New EL Function
Supporting Custom Action Types
Creating a Custom Synchronous Action
Overriding an Asynchronous Action Type
Implementing the New ActionMain Class
Testing the New Main Class
Creating a New Asynchronous Action
Writing an Asynchronous Action Executor
Writing the ActionMain Class
Writing Action’s Schema
Deploying the New Action Type
Using the New Action Type
11. Oozie Operations
Oozie CLI Tool
CLI Subcommands
Useful CLI Commands
Oozie REST API
Oozie Java Client
The oozie-site.xml File
The Oozie Purge Service
Job Monitoring
JMS-Based Monitoring
Oozie Instrumentation and Metrics
Reprocessing
Workflow Reprocessing
Coordinator Reprocessing
Bundle Reprocessing
Server Tuning
JVM Tuning
Service Settings
Oozie High Availability
Debugging in Oozie
Oozie Logs
Developing and Testing Oozie Applications
Application Deployment Tips
Common Errors and Debugging
MiniOozie and LocalOozie
The Competition
Index
