Rajashekar Raja — Senior Data Engineer

Skills

Core: Python, SQL, PySpark, Apache Spark, Apache Kafka, Apache Airflow
Cloud: AWS (S3, EMR, Lambda, Glue, ECS, Athena), GCP (GCS)
Platforms: Databricks, Delta Lake · Kubernetes, Docker
Quality / Ops: Great Expectations, OpenTelemetry, Grafana, CloudWatch, Prometheus
AI tools: Cursor/Codex, GitHub Copilot, custom AI agent skills, GenAI pipelines

Senior Data Engineer (Acting Tech Lead) · 2024 – Current

Led Snowplow pipeline modernization and Airflow DAG re-architecture for real-time behavioral tracking.
Built incremental processing with lookback and enrichment failure validation (PySpark/Python).
Data quality pipelines for web/mobile enrichment failures; AWS Lambda ingestion with GDPR-aligned governance.
OpenTelemetry / Grafana observability; team AI skills that shortened time to production.

Senior Data Engineer · 2022 – 2024

OpenTelemetry tracing across data platform services; dynamic sampling to cut collector cost.
Great Expectations + PySpark DQ library via Airflow; Type 1/Type 2 DQ datasets in GCS and Databricks catalog.
Migrated Ingestion API onto Istio with Argo CD rollouts for zero-downtime deployments.

Senior Data Engineer · 2017 – 2022

Spark/Python on Databricks for ML training pipelines under Airflow SLAs.
PySpark sales aggregation for inventory planning; Airflow jobs to S3 for Tableau.

Data Engineer · 2016 – 2017

Migrated DB2 batch jobs to Spark; broadcast joins and RDD lookups for performance.
Kafka real-time ingestion for downstream recommendation products.

Bachelor of Engineering (BE), Electrical and Electronics

Visvesvaraya Technological University, Bengaluru · 2004 – 2008