Senior Data Engineer

We are looking for a hands-on Data Engineer to build and scale the data infrastructure powering Encardio’s analytics, AI/ML, and decision intelligence for critical infrastructure and sensor-based monitoring systems.

Description: As a Data Engineer in the Edge of Technology Center, you will design and maintain robust ETL/ELT pipelines, real-time data ingestion systems, and distributed cloud-native data platforms. You will work with modern tools like Argo Workflows, Apache Airflow, Apache Iceberg, Kafka, and AWS EKS to transform edge device data (from sensors like accelerometers, strain gauges, and tilt meters) into reliable, analytics-ready datasets. This role requires strong system design skills, a deep understanding of data operations, and a collaborative mindset to support multi-disciplinary teams.

Responsibilities

• Design & Development
• Design, implement, and maintain ETL/ELT pipelines using Argo Workflows or Apache Airflow.
• Develop CI/CD pipelines with GitHub Actions for reliable deployments.
• Schema & Query Management
• Manage database schema changes using Alembic or Liquibase.
• Configure and optimize Trino and AWS Athena for distributed querying.
• Containerization & Cloud Deployment
• Deploy containerized workloads on AWS EKS using Docker.
• Use Helmfile and Argo CD for GitOps-based Kubernetes deployments.
• Data Lake & Storage Architecture
• Build and manage data lakes on AWS S3.
• Implement performant storage formats using Apache Iceberg.
• Infrastructure Automation
• Use Terraform (or similar IaC tools) for infrastructure provisioning and automation.
• Streaming & Real-time Systems
• Architect real-time event-driven systems using Kafka.
• Monitoring & Reliability
• Monitor and troubleshoot data systems using Prometheus and Grafana.
• Collaboration & Documentation
• Work with cross-functional teams to define data solutions.
• Maintain internal documentation and runbooks for knowledge sharing.
  
  

Key Deliverables

• Fully functional, scalable ETL/ELT pipelines.
• Zero-downtime, consistent schema migrations.
• Optimized distributed query environments with low-latency access.
• Production-grade Kubernetes deployments via Github action/ GitOps.
• Real-time Kafka pipelines with strict schema enforcement.
• Infrastructure managed and versioned with Terraform.
• Automated and reliable CI/CD pipelines.
• Delivery of data projects aligned with business goals.
• Proactive observability via monitoring dashboards and alerts.
• Clear and maintainable technical documentation.
  
  

Technologies

Data Pipelines & Orchestration

• Argo Workflows
• Apache Airflow
  
  

Database Migration

• Alembic
• Liquibase
  
  

Query Engines

• Trino
• AWS Athena
  
  

Containerization & Kubernetes

• Docker
• AWS EKS
• GCP GKE
• Helmfile
• Argo CD
  
  

Storage & Data Lake

• AWS S3
• Apache Iceberg
  
  

Databases

• PostgreSQL
• MySQL
• Aurora
  
  

Infrastructure as Code (IaC)

• Terraform
  
  

CI/CD & GitOps

• GitHub Actions
• Argo CD
• Helmfile
  
  

Streaming & Serialization

• Apache Kafka
• Apicurio
• AVRO
  
  

Monitoring & Observability

• Prometheus
• Grafana
  
  

Languages

• Python
• Bash