Telecommut
Data Engineer
Arcgate is a dynamic and rapidly growing team of 2500+ professionals passionate about data and technology. We deliver cutting-edge solutions to some of the world’s most innovative startups to market leaders across application development, quality engineering, AI data preparation, data enrichment, search relevance, and content moderation.
Responsibilities
- Design, build, and optimize Python based data pipelines that handle large, complex, and messy datasets efficiently.
- Develop and manage scalable data infrastructures, including databases and data warehouses such as Snowflake, Azure Data Factory etc. ensuring reliability and performance.
- Build, maintain, and optimize CDC processes that integrate data from multiple sources into the data warehouse.
- Collaborate closely with data scientists, analysts, and operations teams to gather requirements and deliver high-quality data solutions.
- Perform data quality checks, validation, and verification to ensure data integrity and consistency.
- Support and optimize data flows, ingestion, transformation, and publishing across various systems.
- Work with AWS infrastructure (ECS, RDS, S3), manage deployments using Docker, and package services into containers.
- Use tools like Prefect, Dagster and dbt to orchestrate and transform data workflows.
- Implement CI/CD pipelines using Harness and GitHub Actions.
- Monitor system health and performance using DataDog.
- Manage infrastructure orchestration with Terraform and Terragrunt.
- Stay current with industry trends, emerging tools, and best practices in data engineering.
- Coach and mentor junior team members, promoting best practices and skill development.
- Contribute across diverse projects, demonstrating flexibility and adaptability.
- Bachelor’s degree in Computer Science, Engineering, Mathematics, Physics, or a related field.
- 5+ years of demonstrable experience building reliable, scalable data pipelines in production environments.
- Strong experience with Python, SQL programming, and data architecture.
- Hands-on experience with data modeling in Data Lake or Data Warehouse environments (Snowflake preferred).
- Familiarity with Prefect, Dagster, dbt, and ETL/ELT pipeline frameworks.
- Experience with AWS services (ECS, RDS, S3) and containerization using Docker.
- Knowledge of TypeScript, React, Node.js is a plus for collaborating on the application platform.
- Strong command of GitHub for source control and Jira for change management.
- Strong analytical and problem-solving skills, with a hands-on mindset for wrangling data and solving complex challenges.
- Excellent communication and collaboration skills; ability to work effectively with cross- functional teams.
- A proactive, start-up mindset, adaptable, ambitious, responsible, and ready to contribute wherever needed.
- Passion for delivering high-quality solutions with meticulous attention to detail.
- Enjoy working in an inclusive, respectful, and highly collaborative environment where every voice matters.
- Competitive salary package.
- Opportunities for growth, learning, and professional development.
- Dynamic, collaborative, and innovation-driven work culture.
- Work with cutting-edge technologies and leading-edge startups.
Click the Apply button below and become an Arcgatian!
Apply
