Senior Engineer - IT

**Job Description:** -------------------- ### **Key Responsibilities** * Design, develop, and maintain scalable data pipelines using **Databricks and PySpark** * Build robust **ETL/ELT workflows** for batch and near real‑time data processing * Write optimized and efficient **SQL queries** for data transformation, validation, and analysis * Work with large-scale distributed datasets using **Spark-based processing frameworks** * Perform data ingestion from multiple sources (APIs, files, databases) into data lake/lakehouse architecture * Ensure **data quality, integrity, and consistency** through validations and monitoring * Optimize pipeline performance and cost (partitioning, caching, query tuning, etc.) * Collaborate with cross-functional teams (Data Ops, Business, Analytics) to deliver data solutions * Troubleshoot pipeline failures and implement proactive monitoring and alerting * Maintain proper **documentation** for data models, transformations, and workflows ### **Required Skills & Experience** * Strong hands-on experience in: * **Databricks (Lakehouse platform, notebooks, jobs, clusters)** * **PySpark / Apache Spark** * **Advanced SQL (joins, window functions, performance tuning)** * Solid understanding of **ETL concepts, data warehousing, and data modeling** * Experience building **scalable data pipelines** for structured and semi-structured data * Familiarity with **data lake / delta lake architecture** * Good understanding of **performance optimization techniques** in Spark and SQL * Experience with **version control (Git)** and deployment practices * Strong problem-solving and analytical skills ### **Good to Have** * Experience with **cloud platforms (AWS preferably)** * Exposure to **CI/CD pipelines (Jenkins)** * Knowledge of **Delta Lake, Unity Catalog, or similar governance tools** * Understanding of **data quality frameworks / monitoring tools** * Experience working in **Agile / Scrum teams** **Location:** ------------- This position can be based in any of the following locations: Chennai **Current Guardian Colleagues: Please apply through the internal Jobs Hub in Workday**