Senior/Principal/Senior Principal DevOps Engineer - Fully in OfficeThis job is for a Senior Principal DevOps Engineer specializing in DataOps. You will oversee data pipeline building, analysis, and governance using AWS services and managing ingestion from various databases.
Salary:RM 10K - RM 21K
Location:Midvalley
Job Type:Full-Time
Job DescriptionSUMMARY OF RESPONSIBILITIES
As a Principal DevOps Engineer supporting DataOps, you will play a crucial role in enabling efficient and reliable data pipeline building, data analysis, and data governance. Your primary responsibility will be to ensure smooth data ingestion into the data lake or data warehouse, enabling downstream data consumption for Data Scientists, Analysts, and other stakeholders. You will utilize various AWS services such as MSK/Kafka, S3, EMR or Glue, Lakeformation, Athena, QuickSight, and manage ingestion from different databases like MS SQL, MySQL, Oracle, and PostgreSQL.
Job RequirementsCollaborate with Data Engineers to design, implement, and optimize data pipelines.
Ensure efficient and reliable data ingestion processes from various sources into the data lake or data warehouse.
Implement data validation and quality checks to maintain data integrity throughout the pipeline.
Data Analysis Support:Assist Data Scientists and Analysts in their routine analytical work by providing data access and query support.
Optimize data storage and retrieval mechanisms to enhance performance and efficiency.
Collaborate with stakeholders to identify and implement solutions for data analysis and reporting requirements.
Data Governance and Security:Utilize AWS Lakeformation and other relevant tools to establish and enforce data governance policies.
Implement security measures and access controls to protect sensitive data.
Collaborate with stakeholders to ensure compliance with data privacy regulations.
Technical Expertise:Strong expertise in AWS services such as MSK/Kafka, S3, EMR or Glue, Lakeformation, Athena, and QuickSight. Proficient in database management systems like MS SQL, MySQL, Oracle, and PostgreSQL.
Familiarity with Change Data Capture (CDC) mechanisms and practices.
Solid understanding of data pipeline architecture, ETL processes, and data integration techniques.
Automation and Infrastructure as Code (IaC):Proficiency in infrastructure automation using tools like Terraform and CloudFormation.
Experience in managing and administering Kubernetes on-premises or EKS in AWS.
Experience with version control systems (e.g., Git) and CI/CD pipelines for automated deployments.
Understanding of Infrastructure as Code (IaC) principles to maintain reproducible and scalable environments.
Proficient in programming languages such as Python, Golang, and Java.
SkillsDevOps
Amazon Web Services
Docker Container
Terraform
Infrastructure as Code (IaC)
AWS CloudFormation
CI/CD
Company BenefitsWe encourage you to take time off to recharge. To combat the increasing cost of living, we provide commute allowance.
Company ActivityLast active - few hours ago
Experience Level5 - 12 Years of Experience
Manager / Team Lead
Job SpecialisationGeneral IT
Company ProfileTRIIIOWith more than 600 successful placements since our founding, you can trust us to provide you with good guidance in your job search.#J-18808-Ljbffr