Franchesca Metz

IDR is seeking a Sr. Databricks Architect & Developer to join one of our top clients for a remote opportunity with occasional travel to IA. This role involves designing and developing scalable data pipelines and managing the Databricks platform within a dynamic data migration project in the government services industry.

Position Overview for the Sr. Databricks Architect & Developer:

• Develop high-performance scalable ETL pipelines using Databricks (PySpark, Python, Delta Lake, Databricks Workflows) for data migration from legacy source databases to PostgreSQL.
• Proficiently configure and manage Databricks Landing and Staging schemas to ensure secure and efficient data flows.
• Apply advanced database knowledge (SQL, Databricks SQL, and Postgres) to optimize load performance and large-volume cutovers.
• Contribute to data mapping, conceptual and technical design, and application testing with Databricks Notebooks.
• Implement data masking, support defect resolution, and actively contribute to successful data migration outcomes.

Requirements for the Sr. Databricks Architect & Developer:

• Experience with Databricks, Data Factory, PySpark, Pandas, SQL, PL/SQL, Spark Structured Streaming, AWS (S3, Glue, Lambda, Redshift, EMR), and Python.
• Skilled in ETL pipeline design, DevOps automation, data validation, and transformation.
• Knowledge of SFTP, DoDSAFE, NIPRGPT, and data visualization tools.
• Proven ability to optimize and monitor clusters, manage costs, and use cloud-native monitoring tools like Azure Monitor and CloudWatch.
• Strong background in designing and building high-performance ETL pipelines for data migration, including experience migrating data from VSAM files to PostgreSQL.

What's in it for you?

• Competitive compensation package
• Full Benefits; Medical, Vision, Dental, and more!
• Opportunity to get in with an industry leading organization.

Why IDR?

• 25+ Years of Proven Industry Experience in 4 major markets
• Employee Stock Ownership Program
• Dedicated Engagement Manager who is committed to you and your success.
• Medical, Dental, Vision, and Life Insurance
• ClearlyRated's Best of Staffing® Client and Talent Award winner 12 years in a row.