Course Catalog  > Data Engineering and Big Data  > Streaming Analytics with Kafka
WA2920

Confluent Kafka for System Administrators Training

This Kafka course teaches system administrators how to manage Confluent Kafka, a powerful platform for real-time data processing. Attendees gain expertise in installing, configuring, and optimizing Kafka clusters, ensuring high-performance messaging for your microservices and cloud-native applications.

Request Information
Request On-Site or Customized Course Info
Course Details

Duration

3 days

Prerequisites

Basic understanding of messaging, cloud, Linux systems operation, architecture and virtualization would be beneficial.

Target Audience

Systems Administrator who are responsible for planning, installing, operating and troubleshooting a Confluent Kafka installation.

Skills Gained

  • Understand the use of Kafka for high performance messaging
  • Identify the usages for Kafka in Microservices
  • Explain the benefits of Kafka patterns
  • Differentiate between messaging and message brokers
  • Summarize characteristics and architecture for Kafka
  • Install and configure a Kafka cluster
  • Design distributed high throughput systems based on Kafka
  • Describe the built-in partitioning, replication and inherent fault-tolerance of Kafka
Course Outline
  • Introduction to Confluent Kafka
    • Messaging Architectures – What is Messaging?
    • Messaging Architectures – Steps to Messaging
    • Messaging Architectures – Messaging Models
    • What is Kafka?
    • Kafka Overview
    • Need for Kafka
    • When to Use Kafka?
    • Kafka Architecture
    • Core concepts in Kafka
    • Kafka Topic
    • Architecting Topics
    • Kafka Partitions
    • Kafka Producer
    • Kafka Consumer
    • Kafka Broker
    • Kafka Cluster
    • Why Kafka Cluster?
    • Sample Multi-Broker Cluster
    • Overview of ZooKeeper
    • Kafka Cluster & ZooKeeper
    • Who Uses Kafka?
  • Installing and Configuring Confluent Kafka
    • Planning for Kafka - Platform
    • Planning for Kafka - OS
    • Planning for Kafka - Java
    • Planning for Kafka - System Requirements
    • Installing Confluent Platform (Kafka)
    • Downloading Confluent Platform
    • Setting up for Development and Testing
    • Setting up for Production
    • Running in Docker Containers
    • Configuration Files
    • Starting Kafka
    • Using Kafka Command Line Client Tools
    • Setting up a Multi-Broker Cluster
    • Using Multi-Broker Cluster
    • Kafka Cluster Planning
    • Kafka Cluster Planning – Producer/Consumer Throughput
    • Kafka Cluster Planning – Number of Brokers (and ZooKeepers)
    • Kafka Cluster Planning – Sizing for Topics and Partitions
    • Kafka Cluster Planning – Sizing for Storage
    • Kafka Connect
    • Kafka Connect – Configuration Files
    • Using Kafka Connect to Import/Export Data
  • Data Pipelines Overview
    • Building Data Pipelines
    • Best Practices for Designing Pipelines
    • Considerations When Building Data Pipelines
    • Timeliness
    • Reliability
    • High and Varying Throughput
    • Data Formats
    • Transformations
    • Security
    • Security Best Practices
    • Failure Handling
    • Coupling and Agility
    • Ad-hoc Pipelines
    • Loss of Metadata
    • Extreme Processing
    • Kafka Connect Versus Producer and Consumer
  • Integrating Kafka with Other Systems
    • Introduction to Kafka Integration
    • Kafka Connect
    • Running Kafka Connect
    • Key Configurations for Connect workers:
    • Kafka Connect Example – File Source
    • Kafka Connect Example – File Sink
    • Kafka Connector Example – PostgresSQL to Elasticsearch
    • PostgresSQL to Elasticsearch Process
    • Write Data to Elasticsearch
    • Kafka Connector Scenario – SOAP Web Service to REST
    • Building Custom Connectors
    • Kafka Connect – Connectors
    • Kafka Connect - Tasks
    • Kafka Connect - Workers
    • Kafka Connect - Converters and Connect’s data model
    • Kafka Connect - Offset management
    • Alternatives to Kafka Connect
    • Introduction to Hadoop
    • Hadoop Components
    • Integrating Hadoop with Kafka
    • Hadoop Consumers
    • Hadoop Consumers – Produce Topic
    • Hadoop Consumers – Fetch Generated Topic
    • Kafka at Uber
    • Kafka at LinkedIn
    • Kafka at LinkedIn – Core Kafka Services
    • Kafka at LinkedIn – Libraries
    • Kafka at LinkedIn – Monitoring and Stream Processing
  • Schema Management, Streams and KSQL
    • Evolving Schema
    • Protobuf (Protocol Buffers) Overview
    • Avro Overview
    • Managing Data Evolution Using Schemas
    • Confluent Platform
    • Confluent Schema Registry
    • Schema Change and Backward Compatibility
    • Collaborating over Schema Change
    • Handling Unreadable Messages
    • Deleting Data
    • Segregating Public and Private Topics
    • What is KSQL?
    • Starting KSQL
    • Using the KSQL CLI
    • KSQL Data Types
  • Managing Kafka
    • Monitoring Kafka
    • Confluent Control Center
    • Control Center - Screen Shot
    • Control Center - Running
    • Control Center - Monitoring
    • Control Center - Management
    • Control Center - Alerts
    • Control Center - Development
    • Troubleshooting basics
    • Zookeeper CLI Shell
    • kafka-topics Command Line Script
    • console-consumer Command Line Script
    • Viewing logs
    • Inspecting Message Queue
    • Viewing Messages with Control Center
    • Viewing messages with REST Proxy
    • Maintaining Kafka Cluster
    • Updating Kafka version
    • Backup and Restore
    • Disaster recovery
  • Lab Exercises
    • Lab 1. Confluent Kafka Basics
    • Lab 2. Confluent Kafka Multiple Brokers and Import/Export Messages
    • Lab 3. Securing Confluent Kafka
    • Lab 4. Integrating Confluent Kafka with PostgreSQL
    • Lab 5. Confluent Kafka Monitoring using Graphite
    • Lab 6. KSQL Basics
    • Lab 7. Troubleshoot a Kafka Application