Job Description

DevOps Engineer - IBM/USAA
Location: San Antonio, TX

Tools & Technologies:

• Apache Kafka (Self-managed or MSK)

• AWS Managed Apache Flink

• Amazon EC2, S3, RDS, and VPC

• Terraform/CloudFormation

• Docker, Kubernetes (EKS)

• ELK Stack, CloudWatch

• Python, Bash

Skills & Expertise:

1.   AWS Managed Services:

◦ Proficiency in AWS services like Amazon MSK, Kinesis, Lambda, S3, EC2, RDS, VPC, and IAM.

◦ Experience with infrastructure management using Terraform or AWS CloudFormation.

2.   Apache Flink:

◦ Understanding of real-time stream and batch data processing with Apache Flink.

◦ Familiarity with Flink's integration with Kafka and managing Flink clusters on AWS.

3.   Kafka Broker (Apache Kafka):

◦ Strong knowledge of Kafka architecture, brokers, topics, partitions, producers, and consumers.

◦ Experience with Kafka management, monitoring, scaling, and optimization, particularly with Amazon MSK or self-managed Kafka clusters.

4.   DevOps & Automation:

◦ Expertise in automating deployments and infrastructure provisioning.

◦ Familiarity with CI/CD pipelines using tools like Jenkins, GitLab, GitHub Actions, CircleCI.

◦ Experience with Docker and Kubernetes for containerizing and orchestrating applications in the cloud.

5.   Programming & Scripting:

◦ Strong scripting skills in Python, Bash, or Go for automation tasks and data pipeline integrations.

6.   Monitoring & Performance Tuning:

◦ Proficiency with monitoring tools like CloudWatch, Prometheus, Grafana.

◦ Expertise in optimizing data pipelines for scalability, fault tolerance, and performance.

Responsibilities:

1.   Infrastructure Design & Implementation:

◦ Design and deploy scalable, fault-tolerant real-time data processing pipelines using Apache Flink and Kafka on AWS.

2.   Platform Management:

◦ Manage and optimize Kafka clusters (MSK or self-managed) and Flink jobs on AWS infrastructure for low-latency, high-throughput processing.

3.   Automation & CI/CD:

◦ Automate infrastructure provisioning and deployment using Terraform, CloudFormation, or other tools.

◦ Integrate new applications into CI/CD pipelines for real-time data processing.

4.   Collaboration with Data Engineering Teams:

◦ Work with Data Engineers, Data Scientists, and DevOps teams to ensure smooth integration and high performance of real-time data systems.

5.   Security and Compliance:

◦ Implement security for Kafka and Flink clusters, ensuring compliance with regulatory standards (e.g., GDPR, HIPAA).

6.   Optimization & Troubleshooting:

◦ Optimize Kafka and Flink performance, troubleshoot issues related to message delivery, job failures, or AWS service outages.

Job Tags

Similar Jobs