Job Description
AWS Senior Data Engineer (Kafka & PySpark)
Experience: 10-15 Years
Location: Chennai, Bengaluru, Hyderabad
We are seeking an experienced AWS Data Engineer with strong expertise in Kafka, PySpark, Spark SQL, Python, and AWS to build and optimize large-scale batch and real-time data platforms.
Key Skills
- PySpark, Spark SQL, Python, SQL
- Apache Kafka (Kafka Connect, Schema Registry, Streaming)
- AWS: S3, Glue, EMR, Lambda, IAM, Redshift, Step Functions, CloudWatch
- ETL/ELT & Real-Time Data Pipelines
- Spark Structured Streaming
- Data Modeling, Data Lakes & Lakehouse Architecture
- CI/CD, Git, Jenkins/GitHub Actions
- Data Quality, Governance & Security
Responsibilities
- Design and develop scalable batch and streaming data pipelines on AWS.
- Build high-performance PySpark and Spark SQL applications.
- Implement Kafka-based event-driven and real-time streaming solutions.
- Integrate and optimize AWS data services for reliability and cost efficiency.
- Implement data quality, governance, monitoring, and security best practices.
- Build CI/CD pipelines and support production deployments.
- Troubleshoot performance bottlenecks and production issues.
Good to Have
- Amazon MSK, Kinesis
- Databricks, Delta Lake, Unity Catalog
- Airflow / MWAA
- Snowflake, DBT
- Terraform
Primary Skills: AWS Data Engineering, Kafka, PySpark, Spark SQL, Python, Streaming Data Engineering.
About this job listing
This job opportunity is provided through our
external job listing network. MyJobAlerts helps
you discover job opportunities and redirects you
to the original listing to apply.