Accelerating Data Engineering Pipelines Training
Data engineering is the foundation of data science and lays the groundwork for analysis and modeling. In order for organizations to extract knowledge and insights from structured and unstructured data, fast access to accurate and complete datasets is critical. Working with massive amounts of data from disparate sources requires complex infrastructure and expertise. Minor inefficiencies can result in major costs, both in terms of time and money, when scaled across millions to trillions of data points.
In this workshop, we’ll explore how GPUs can improve data pipelines and how using advanced data engineering tools and techniques can result in significant performance acceleration. Faster pipelines produce fresher dashboards and machine learning (ML) models, so users can have the most current information at their fingertips.
Duration
Prerequisites
- Intermediate knowledge of Python (list comprehension, objects)
- Familiarity with pandas a plus
- Introductory statistics (mean, median, mode)
Skills Gained
- How data moves within a computer. How to build the right balance between CPU, DRAM, Disk Memory, and GPUs.
- How different file formats can be read and manipulated by hardware.
- How to scale an ETL pipeline with multiple GPUs using NVTabular.
- How to build an interactive Plotly dashboard where users can filter on millions of data points in less than a second.
- Introduction
- Data on the Hardware Level
- Explore the strengths and weaknesses of different hardware approaches to data and the frameworks that support them:
- Pandas
- CuDF
- Dask
- ETL with NVTabular
- Learn how to scale an ETL pipeline from 1 GPU to many with NVTabular through the perspective of a big data recommender system.
- Transform raw json into analysis-ready parquet files.
- Learn how to quickly add features to a dataset, such as Categorify and Lambda operators.
- Data Visualization
- Step into the shoes of a meteorologist and learn how to plot precipitation data on a map.
- Learn how to use descriptive statistics and plots like histograms in order to assess data quality.
- Learn effective memory usage, so users can quickly filter data through a graphical interface.
- Final Project: Data Detective
- Users are complaining that the dashboard is too slow. Apply the techniques learned in class to find and eliminate efficiencies in the backend code.
- Final Review
