Job Description
Job Description
Experience: 6+ Years
Role: Senior Data Engineer
Location: Coimbatore- Work from Office
Employment Type: Full-time
Availability: Immediate Joiner Preferred
Role Summary
We are seeking an experienced Data Engineer with 6+ years of experience in designing, developing, and maintaining scalable data pipelines and data warehouse solutions. The ideal candidate should have strong expertise in ETL/ELT, Python, dbt, Snowflake, workflow orchestration, job scheduling, and Git.
Key Responsibilities
* Design, develop, test, and maintain scalable ETL/ELT data pipelines.
* Build and manage enterprise data warehouse solutions using Snowflake.
* Develop reusable data transformation models using dbt.
* Write efficient and maintainable Python programs for data ingestion, transformation, validation, and automation.
* Extract and integrate data from databases, APIs, flat files, and other structured or semi-structured sources.
* Develop dimensional data models using concepts such as:
* Star and snowflake schemas
* Fact and dimension tables
* Slowly Changing Dimensions
* Surrogate keys
* Incremental and historical data loads
* Create and maintain ETL jobs, schedules, dependencies, alerts, and recovery procedures.
* Work with orchestration and scheduling tools such as Apache Airflow, Control-M, Azure Data Factory, AWS Glue, or similar platforms.
* Implement data-quality checks, reconciliation processes, audit controls, and exception handling.
* Monitor pipeline performance and troubleshoot failed or delayed data-processing jobs.
* Optimize Snowflake queries, storage, warehouses, and overall pipeline performance.
* Use Git for source-code management, branching, merging, pull requests, code reviews, and release management.
* Support CI/CD processes for deploying data pipelines and dbt models across environments.
* Collaborate with business analysts, data analysts, reporting teams, architects, and other stakeholders to understand data requirements.
* Prepare and maintain technical documentation, data mappings, lineage, and operational support procedures.
* Participate in Agile ceremonies, sprint planning, estimation, peer reviews, and production-support activities.
Required Skills and Qualifications
* 6+ years of professional experience in data engineering, ETL development, or data warehousing.
* Strong understanding of ETL/ELT processes and data warehousing concepts.
* Hands-on experience with Python for data processing and automation.
* Strong hands-on experience with dbt, including:
* Models and materializations
* Sources and seeds
* Tests and documentation
* Macros and Jinja
* Incremental models
* Snapshots
* Hooks and packages
* Strong experience with Snowflake, including:
* Databases, schemas, tables, and views
* Virtual warehouses
* Stages and file formats
* Snowpipe
* Streams and Tasks
* Time Travel and zero-copy cloning
* Query and warehouse performance optimization
* Role-based access control
* Advanced SQL skills, including joins, CTEs, window functions, stored procedures, and query optimization.
* Experience creating, scheduling, monitoring, and troubleshooting ETL jobs.
* Experience with workflow orchestration tools such as Apache Airflow or an equivalent platform.
* Strong working knowledge of Git, including branching strategies, merge conflict resolution, pull requests, and code reviews.
* Experience working with structured and semi-structured data formats such as CSV, JSON, XML, Avro, and Parquet.
* Knowledge of data-quality frameworks, logging, monitoring, reconciliation, and error-handling practices.
* Good understanding of software development life cycle, Agile methodologies, and production-support processes.
* Strong analytical, troubleshooting, communication, and problem-solving skills.
Preferred Qualifications
* Experience with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform.
* Experience implementing CI/CD pipelines using GitHub Actions, GitLab CI, Jenkins, Azure DevOps, or similar tools.
* Familiarity with BI and reporting tools such as Power BI, Tableau, or Looker.
* Understanding of data governance, metadata management, lineage, security, and access control.
* Relevant Snowflake, dbt, cloud, or data-engineering certifications would be an advantage.
Expected Competencies
* Ability to independently design and deliver end-to-end data pipelines.
* Strong ownership of data accuracy, reliability, and performance.
* Ability to analyse complex data issues and provide sustainable solutions.
* Effective collaboration with technical and business stakeholders.
* Ability to mentor junior engineers and contribute to engineering best practices.