Adi Wijaya | Data Engineering with Google Cloud Platform. A guide to leveling up as a data engineer by building a scalable data platform with Google Cloud (2024) [PDF] [EN]

Beyond its foundational lessons, this new edition delves into the essential realm of data governance within Google Cloud, providing you with invaluable insights into managing and optimizing data resources effectively. Written by a Data Strategic Cloud Engineer at Google, this book helps you stay ahead of the curve by guiding you through the latest technological advancements in the Google Cloud ecosystem. You’ll cover essential aspects, from exploring Cloud Composer 2 to the evolution of Airflow 2.5. Additionally, you’ll explore how to work with cutting-edge tools like Dataform, DLP, Dataplex, Dataproc Serverless, and Datastream to perform data governance on datasets.

By the end of this book, you'll be equipped to navigate the ever-evolving world of data engineering on Google Cloud, from foundational principles to cutting-edge practices.

What you will learn
Load data into BigQuery and materialize its output
Focus on data pipeline orchestration using Cloud Composer
Formulate Airflow jobs to orchestrate and automate a data warehouse
Establish a Hadoop data lake, generate ephemeral clusters, and execute jobs on the Dataproc cluster
Harness Pub/Sub for messaging and ingestion for event-driven systems
Apply Dataflow to conduct ETL on streaming data
Implement data governance services on Google Cloud
Who this book is for
Data analysts, IT practitioners, software engineers, or any data enthusiasts looking to have a successful data engineering career will find this book invaluable. Additionally, experienced data professionals who want to start using Google Cloud to build data platforms will get clear insights on how to navigate the path. Whether you're a beginner who wants to explore the fundamentals or a seasoned professional seeking to learn the latest data engineering concepts, this book is for you.

Table of Contents
Fundamentals of Data engineering with GCP
Big Data Capabilities on GCP
Building a data warehouse in BigQuery
Build Orchestration for Batch Data Loading Using Cloud Composer
Building a Data Lake using Dataproc
Process Streaming Data with Datastream, Pub/Sub and Dataflow
Visualizing Data for Making Data-Driven Decisions with Looker Studio
Build machine learning solutions on GCP
User and Project Management on GCP
Data Governance in GCP
Cost Strategy in GCP
CI/CD on Google Cloud Platform for Data Engineers
Boost your confidence as a Data Engineer

Part 1: Fundamentals

1. Fundamentals of Data Engineering
Understanding the data life cycle
Understanding the need for a data warehouse
Start with knowing the roles of a data engineer
A data engineer versus a data scientist
The focus of data engineers
Going through the foundational concepts for data engineering
ETL concept in data engineering
The difference between ETL and ELT
What is not big data?
A quick look at how big data technologies store data
A quick look at how to process multiple files using MapReduce
Summary
Exercise
Further Reading

2. Big Data Capabilities on GCP
Technical requirements
Understanding what the cloud is
The difference between the cloud and non-cloud era
The on-demand nature of the cloud
Getting started with GCP
Introduction to the GCP console
Practicing pinning services
A quick overview of GCP services for data engineering
Understanding the GCP serverless service
Service mapping and prioritization
The concept of quotas on GCP services
User account versus service account
Summary

Part 2: Build Solutions with GCP Components

3. Building a Data Warehouse in BigQuery
Technical requirements
Introduction to GCS and BigQuery
BigQuery data location
Introduction to the BigQuery console
Creating a dataset in BigQuery using the console
Loading the local CSV file into the BigQuery table
Using public data in BigQuery
Data types in BigQuery compared to other databases
Timestamp data in BigQuery compared to other databases
Preparing the prerequisites before developing our data warehouse
Step 1 – Accessing Cloud Shell
Step 2 – Checking the current setup using the command line
Step 3 – Initializing the gcloud init command
Step 4 – Downloading example data from Git
Step 5 – Uploading data to GCS from Git
Practicing developing a data warehouse
Data warehouse in BigQuery – Requirements for scenario 1
Steps and planning for handling scenario 1
Data warehouse in BigQuery – Requirements for scenario 2
Using the GCP console versus the code-based approach
Steps and planning for handling scenario 2
BigQuery's useful features
BigQuery console sub-menu options
BigQuery partitioned table
Summary
Exercise – Scenario 3
See also

4. Building Workflows for Batch Data Loading Using Cloud Composer
Technical requirements
Introduction to Cloud Composer
Understanding the working of Airflow
Introducing the Airflow web UI
Cloud Composer bucket directories
Exercise – build data pipeline orchestration using Cloud Composer
Level 1 DAG – creating dummy workflows
Deploying the DAG file into Cloud Composer
Cloud Composer 1 vs Cloud Composer 2
Provisioning Cloud Composer in a GCP project
Level 2 DAG – scheduling a pipeline from Cloud SQL to GCS and BigQuery datasets
Level 3 DAG – parameterized variables
Level 4 DAG – Guaranteeing task idempotency in Cloud Composer
Level 5 DAG – handling DAG dependency using an Airflow dataset
Summary

5. Building a Data Lake Using Dataproc
Technical requirements
Introduction to Dataproc
A brief history of the data lake and Hadoop ecosystem
A deeper look into Hadoop components
How much Hadoop-related knowledge do you need on GCP?
Introducing the Spark RDD and DataFrame concepts
Introducing the data lake concept
Hadoop and Dataproc positioning on GCP
Introduction to Dataproc Serverless
Exercise – Building a data lake on a Dataproc cluster
Creating a Dataproc cluster on GCP
Using GCS as an underlying Dataproc filesystem
Exercise – Creating and running jobs on a Dataproc cluster
Preparing log data in GCS and HDFS
Developing a Spark ETL job from HDFS to HDFS
Developing a Spark ETL job from GCS to GCS
Developing a Spark ETL job from GCS to BigQuery
Understanding the concept of an ephemeral cluster
Practicing using a workflow template on Dataproc
Building an ephemeral cluster using Dataproc and Cloud Composer
Submitting a Spark ETL Job from GCS to BigQuery using Dataproc Serverless
Summary

6. Processing Streaming Data with Pub/Sub and Dataflow
Technical requirements
Processing streaming data
Introduction to Pub/Sub
Introduction to Dataflow
Exercise – publishing event streams to Pub/Sub
Creating a Pub/Sub topic
Creating and running a Pub/Sub publisher using Python
Creating a Pub/Sub subscription
Exercise – using Dataflow to stream data from Pub/Sub to GCS
Creating a HelloWorld application using Apache Beam
Creating a Dataflow streaming job without aggregation
Creating a streaming job with aggregation
Introduction to CDC and Datastream
What is Datastream?
Exercise – Datastream ETL streaming to BigQuery
Step 1 – create a CloudSQL MySQL table
Step 2 – create a GCS bucket
Step 3 – create a GCS notification to the Pub/Sub topic and subscription
Step 4 – create a BigQuery dataset
Step 5 – configure a Datastream job
Step 6 – run a Dataflow job from the Dataflow template
Step 7 – insert a value in MySQL and check the result in BigQuery
Summary

7. Visualizing Data to Make Data-Driven Decisions with Looker Studio
Technical requirements
Unlocking the power of your data with Looker Studio
Don't confuse Looker Studio with Looker
From data to metrics in minutes with an Illustrative use case
Understanding what BigQuery INFORMATION_SCHEMA is
Exercise – accessing the BigQuery INFORMATION_SCHEMA table using Looker Studio
Exercise – creating a Looker Studio report using data from a bike-sharing data warehouse
Understanding how Looker Studio can impact the cost of BigQuery
What kind of table could be 1 TB in size?
How can a table be accessed 10,000 times in a month?
Creating Materialized Views and understanding how BI Engine works
Understanding BI Engine
Summary

8. Building Machine Learning Solutions on GCP
Technical requirements
A quick look at ML
Exercise – practicing ML code using Python
Preparing the ML dataset by using a table from the BigQuery public dataset
Training the ML model using Random Forest in Python
Creating a batch prediction using the training dataset's output
The MLOps landscape in GCP
Understanding the basic principles of MLOps
Introducing GCP services related to MLOps
Exercise – leveraging pre-built GCP models as a service
Uploading the image to a GCS bucket
Creating a detect text function in Python
Exercise – using GCP in AutoML to train an ML model
Exercise – deploying a dummy workflow with Vertex AI Pipelines
Creating a dedicated regional GCS bucket
Developing the pipeline on Python
Monitoring the pipeline on the Vertex AI Pipelines console
Exercise – deploying a scikit-learn model pipeline with Vertex AI
Creating the first pipeline, which will result in an ML model file in GCS
Running the first pipeline in Vertex AI Pipelines
Creating the second pipeline, which will use the model file from the prediction results as a CSV file in GCS
Running the second pipeline in Vertex AI Pipelines
Summary

Part 3: Key Strategies for Architecting Top-Notch Solutions

9. User and Project Management in GCP
Technical requirements
Understanding IAM in GCP
Planning a GCP project structure
Understanding the GCP organization, folder, and project hierarchy
Deciding how many projects we should have in a GCP organization
Controlling user access to our data warehouse
Use-case scenario – planning BigQuery ACLs on an eCommerce organization
Practicing the concept of IaC using Terraform
Exercise – creating and running basic Terraform scripts
Self-exercise – managing a GCP project and resources using Terraform
Summary

10. Data Governance in GCP
Technical requirements
Introduction to data governance
A deeper understanding of data usability
Exercise – implementing metadata tagging using Dataplex
A deeper understanding of data security
Example – BigQuery data masking
Exercise – finding PII using SDP
A deeper understanding of being accountable
Clear traceability
Clear data ownership
Data lineage
Clear data quality process
Exercise – practicing data quality using Dataform
Summary

11. Cost Strategy in GCP
Technical requirements
Estimating the cost of your end-to-end data solution in GCP
Comparing BigQuery on-demand and editions
An example – an estimating data engineering use case
Tips to optimize BigQuery using partitioned and clustered tables
Partitioned tables
Clustered tables
An exercise – optimizing BigQuery on-demand cost
Summary

12. CI/CD on GCP for Data Engineers
Technical requirements
An introduction to CI/CD
Understanding the data engineer's relationship with CI/CD practices
Understanding CI/CD components with GCP services
Exercise – implementing CI using Cloud Build
Creating a GitHub repository using a Cloud Source Repository
Developing the code and Cloud Build scripts
Creating a Cloud Build trigger
Pushing the code to the GitHub repository
Exercise – deploying Cloud Composer jobs using Cloud Build
Preparing the CI/CD environment
Preparing the cloudbuild.yaml configuration file
Pushing the DAG to our GitHub repository
Checking the CI/CD result in the GCS bucket and Cloud Composer
CI/CD best practices in data engineering
Summary
Further reading

13. Boosting Your Confidence as a Data Engineer
Overviewing the Google Cloud certification
Exam preparation tips
Extra GCP service materials
Quiz – reviewing all the concepts you've learned about
Questions
Answers
The past, present, and future of data engineering
Boosting your confidence and final thoughts
Summary

Index

Other Books You May Enjoy