Skip to main content
A

Senior Data Engineer | Data Integration Developer

Active Recruitment
10 days ago
Full-time
On-site
Pretoria, South Africa

Introduction

Centurion  |  Gauteng  | Permanent

7+ years experience required - Senior

Hands on Engineer  | Data Integration Developer

This role is for someone that is not afraid to get their hands dirty.

 

Hands-on role that owns the full data engineering lifecycle:

  • from ingestion and transformation through to platform architecture and production deployment – designing, building and running complex data pipelines on AWS.

 

Technical Skills Required:

  • Python (advanced) – data pipeline development, automation and scripting.
  • SQL (advanced) – complex query writing, performance optimisation and stored procedures.
  • AWS (mid–advanced) – S3, Glue, Lambda, Redshift, RDS, IAM, CloudWatch, Athena.
  • Data warehousing and data modelling – star/snowflake schemas, dimensional and analytics-oriented design.
  • ETL/ELT pipeline development – batch, incremental and near-real-time patterns.
  • Data lakehouse architecture on cloud-native platforms.
  • CI/CD and Git – version control, automated testing and deployment pipelines.
  • Power BI – supporting BI teams, data model integration and semantic layer understanding.
  • Experience ingesting data from relational databases (e.g. SQL Server, Oracle), cloud storage, APIs, SFTP, MongoDB and file-based sources.

 

Highly Valued (Differentiating Skills):

  • Apache Spark / PySpark for large-scale distributed processing.
  • dbt for analytics engineering and transformation layer management.
  • Apache Airflow or AWS Step Functions for orchestration.
  • Terraform or CloudFormation (infrastructure-as-code).
  • Real-time streaming – Kafka, Kinesis or similar.
  • Machine learning workflow support – feature engineering and model data pipelines.
  • Microsoft Azure or GCP as secondary cloud exposure.

 

Senior Data Engineer, AWS, Python, SQL, Data Engineering, ETL/ELT, Data Lakehouse, Data Warehousing, Power BI, CI/CD, Agile, AI-enabled engineering, Financial Services, Centurion, Hybrid

Duties & Responsibilities

Delivery Ownership within Agile Execution

  • Contribute to data engineering outcomes aligned to sprint goals and programme-level commitments.
  • Take part in Agile ceremonies – planning, estimation, prioritisation and continuous improvement.
  • Decompose data features into implementable tasks with reliable effort estimates.
  • Ensure outputs meet agreed functional, performance and data quality expectations.

Data Engineering & Pipeline Development

  • Design and implement scalable ingestion and transformation pipelines across multiple systems and data domains, supporting batch and incremental processing.
  • Build in error handling, monitoring and alerting to keep pipelines robust.
  • Implement data validation and reconciliation mechanisms to maintain confidence in data assets.

Platform Design & Architecture (AWS)

  • Design solutions aligned to the data platform strategy and target architecture, and contribute to the evolution of the cloud-based data lakehouse environment.
  • Assess the impact of design choices on security, performance, cost and supportability.
  • Identify integration points, upstream/downstream dependencies and risks early in the delivery lifecycle.

Technical Leadership, Coaching & Standards

  • Provide guidance and technical oversight to less experienced data engineers.
  • Support analytics, BI and data science teams with clarity on data structures, availability and pipeline behaviour.
  • Review code and configuration, and help define shared standards, patterns and best practices.
  • Optimise and simplify existing pipelines and data models.

AI-Enabled Engineering (Governance-Led)

  • Promote responsible use of AI to improve engineering productivity and solution quality, within group and business-unit governance frameworks.
  • Use only enterprise-approved AI tooling, and ensure no sensitive, proprietary or client information is exposed to public or unapproved AI platforms.
  • Validate all AI-assisted outputs and retain accountability for correctness, compliance and production readiness.
  • Apply AI to understand data flows and lineage, improve pipeline logic, support query formulation and test scenario creation, and accelerate documentation.

DevOps, Operational Support & Incident Response

  • Apply version control, automated testing, CI/CD and structured releases.
  • Contribute to operational readiness, deployment pipelines and monitoring.
  • Diagnose and resolve data incidents, perform root cause analysis and implement corrective actions.

Security, Risk, Compliance & Governance

  • Build solutions with privacy, access control and regulatory considerations embedded by design.
  • Identify and mitigate data risks, and support remediation of audit findings and compliance gaps.
  • Maintain accurate documentation of pipelines, data models and operational considerations, and keep work tracking up to date in Jira.
  • Adhere to delivery governance, change management and release processes.

Stakeholder Engagement

  • Communicate technical progress, constraints and decisions clearly to business and technical stakeholders.
  • Collaborate across technology and analytics teams and contribute positively to team culture.

Desired Experience & Qualification

Skills & Experience:

  • 7+ years’ professional experience in data engineering or related roles.
  • Demonstrated delivery of production-ready, end-to-end data pipelines and cloud data platforms.
  • Strong exposure to AWS-based data architectures.
  • Experience across the full stack – ingestion, transformation, modelling and serving – not just a single layer.
  • Hands-on experience working within Agile or SAFe delivery environments.
  • Experience in financial services, insurance, investment management or other regulated data environments (preferred).
  • Practical experience applying AI tooling in engineering workflows – including using AI responsibly and validating its outputs – to support pipeline understanding, optimisation, testing or documentation.
  • Experience coaching or guiding other engineers.

Interested?