Data Pipeline Engineering: Design, develop, and deploy robust and scalable data pipelines using Databricks, incorporating data extraction from diverse sources (databases, APIs, streaming platforms), transformation and cleansing using Spark, and loading into target systems (data lakes, data warehouses, etc.). * Programming Skills: Strong programming skills in Python or Scala, with experience in data manipulation libraries (e.g., PySpark, Spark SQL).
meer