Key Skills:
Strong hands-on expertise in Apache Spark 3.5.x or later Solid experience with Spark DataFrames, Datasets, and RDDs Strong proficiency in Spark SQL Experience working in cloud environments (AWS and/or Azure) Hands-on experience with cloud storage solutions such as Amazon S3 and Azure Data Lake Storage (ADLS) Strong understanding of ETL pipeline design and data modeling Proven experience in Spark job performance tuning and query optimization
Preferred / Good to Have:
Experience with Snowflake for data warehousing and analytics Exposure to real-time/streaming data processing Familiarity with distributed systems and large-scale data architectures Scala Programming Language
Roles & Responsibilities:
Design, develop, and maintain large-scale batch and real-time data processing solutions using Apache Spark (v3.5.x or later) Work extensively with Spark DataFrames, Datasets, and RDDs for high-volume data processing Develop and optimize queries using Spark SQL Build and manage ETL pipelines to support analytics and downstream data consumption Perform Spark job tuning and performance optimization to improve efficiency and scalability Collaborate with data engineers, architects, and analytics teams to deliver end-to-end data solutions Ensure data quality, reliability, and adherence to best practices across pipelines
Skills: Informatica Powercentre~Digital : BigData and Hadoop Ecosystem – MapR~Digital : NOSQL~Digital : Scala~Digital : PySpark