Applied Data Pipelines using Azure Data Factory Training
In this Azure Data course, participants explore Azure Data Factory (ADF), Microsoft's cloud-based data integration service. Participants learn ETL (Extract-Transform-Load) fundamentals, pipeline building, and external service integration. Through hands-on exercises, participants master data transformation techniques, orchestration, and monitoring. They also explore the similarities and differences among ADF, Synapse Pipelines, and Fabric.
After completing this course, participants confidently deploy efficient data workflows, adhering to best practices for performance optimization and cost management.
Duration
Prerequisites
Target Audience
Skills Gained
- Understand Azure Data Factory's role in modern data integration.
- Design and build data pipelines with Azure Data Factory.
- Integrate ADF with external services like Azure Blob Storage and Azure SQL.
- Master data orchestration and workflow management.
- Learn to monitor, manage, and optimize data pipelines for efficiency.
- Azure Data Factory Overview
- Understanding ETL
- Understanding Data Pipeline and Data Flow
- Evolution of data integration from on-premises to cloud-based solutions
- Understanding Azure Data Factory
- Key concepts in Azure Data Factory: pipelines, activities, datasets, triggers
- Building Data Pipelines in Azure Data Factory
- Introduction to Data Movement activities
- Using Copy Data activity
- Using Data Flow activity
- Using Custom activity
- Introduction to Data Transformation activities
- Using Mapping Data Flow
- Using Wrangling Data Flow
- Using Stored Procedure activity
- Working with Variables
- Working with Lookup
- Working with Wait
- Integration with External Services
- Integration with Azure services: Azure Blob Storage, Azure SQL, and Azure Synapse Analytics
- Integration with external services: Amazon S3, Google Cloud Storage, Salesforce
- Data Transformation Techniques
- Working with Multiple inputs/outputs
- Working with Schema modifier
- Working with Formatters
- Mapping Data Flow
- Wrangling Data Flow
- Implementing data cleansing, enrichment, and aggregation
- Data Orchestration and Workflow Management
- Managing dependencies and scheduling in Azure Data Factory
- Implementing parameterized pipelines for dynamic data processing
- Configuring triggers
- Working with parameters
- Using dynamic content in data pipelines
- Working with Iteration & Conditionals
- Using ForEach and Until
- Using Filter
- Using If Condition & Switch
- Monitoring and Management in Azure Data Factory
- Monitoring data pipelines using Azure Data Factory Monitoring Hub
- Understanding pipeline run status, triggers, and alerts
- Managing data factory resources, including linked services, datasets, and pipelines
- Implementing best practices for performance optimization and cost management
- Synapse Pipelines Integration
- Leveraging Azure Synapse Analytics for scalable data processing
- Integrating Synapse pipelines with Azure Data Factory for end-to-end data workflows
- Understanding Synapse Pipelines and their role in data processing
- Configuring Data Movement activities between Azure Data Factory and Azure Synapse Analytics
- Utilizing Synapse Linked Services and Datasets in Azure Data Factory pipelines
- Pipelines in Fabric
- Understanding and implementing Pipelines in Fabric for large-scale data processing
- Leveraging Data Flows in Pipelines in Fabric for complex data transformations
- Creating and managing Data Marts within Pipelines in Fabric for optimized data storage and retrieval
- Utilizing Datasets in Pipelines in Fabric for defining data structures and schemas
- Implementing best practices for designing and orchestrating Pipelines in Fabric
- Hands-on exercises: Building and optimizing data pipelines using Pipelines in Fabric
- Best Practices in Azure Data Factory
- Performance tuning for data pipelines
- Parallel execution and partitioning strategies for improved throughput
- Data skew management and load balancing techniques
- Pipeline design best practices: modularization, reusability, and maintainability
- Error handling and exception management strategies
- Version control and deployment best practices
- Resource optimization
- Leveraging serverless compute options and auto-scaling capabilities.
Partner Registration
The course you are registering for is being delivered by our sister company - ExitCertified. All logistics related to course delivery will be managed by the ExitCertified team. If you have a dedicated Web Age representative, please feel to reach out to them with any questions/concerns you may have.
You'll now be redirected to https://www.exitcertified.com to complete the enrollment process.
Partner Registration
The course you are registering for is being delivered by our sister company - ExitCertified. All logistics related to course delivery will be managed by the ExitCertified team. If you have a dedicated Web Age representative, please feel to reach out to them with any questions/concerns you may have.
You'll now be redirected to https://www.exitcertified.com to complete the enrollment process.
Partner Registration
The course you are registering for is being delivered by our sister company - ExitCertified. All logistics related to course delivery will be managed by the ExitCertified team. If you have a dedicated Web Age representative, please feel to reach out to them with any questions/concerns you may have.
You'll now be redirected to https://www.exitcertified.com to complete the enrollment process.
Partner Registration
The course you are registering for is being delivered by our sister company - ExitCertified. All logistics related to course delivery will be managed by the ExitCertified team. If you have a dedicated Web Age representative, please feel to reach out to them with any questions/concerns you may have.
You'll now be redirected to https://www.exitcertified.com to complete the enrollment process.
