Nashville, Tennessee, United States
IDR is seeking a Databricks Developer to join one of our top clients for a remote opportunity with occasional travel to IA. This role involves designing and developing scalable data pipelines and managing the Databricks platform within a dynamic data migration project in the government services industry.
Position Overview for the Databricks Developer:
- Develop high-performance scalable ETL pipelines using Databricks (PySpark, Python, Delta Lake, Databricks Workflows) for data migration from legacy source databases to PostgreSQL.
- Proficiently configure and manage Databricks Landing and Staging schemas to ensure secure and efficient data flows.
- Apply advanced database knowledge (SQL, Databricks SQL, and Postgres) to optimize load performance and large-volume cutovers.
- Contribute to data mapping, conceptual and technical design, and application testing with Databricks Notebooks.
- Implement data masking, support defect resolution, and actively contribute to successful data migration outcomes.
Requirements for the Sr. Databricks Architect & Developer:
- Experience with Databricks, Data Factory, PySpark, Pandas, SQL, PL/SQL, Spark Structured Streaming, AWS (S3, Glue, Lambda, Redshift, EMR), and Python.
- Skilled in ETL pipeline design, DevOps automation, data validation, and transformation.
- Knowledge of SFTP, DoDSAFE, NIPRGPT, and data visualization tools.
- Proven ability to optimize and monitor clusters, manage costs, and use cloud-native monitoring tools like Azure Monitor and CloudWatch.
- Strong background in designing and building high-performance ETL pipelines for data migration, including experience migrating data from VSAM files to PostgreSQL.
What's in it for you?
- Competitive compensation package
- Full Benefits; Medical, Vision, Dental, and more!
- Opportunity to get in with an industry leading organization.
Why IDR?
- 25+ Years of Proven Industry Experience in 4 major markets
- Employee Stock Ownership Program
- Dedicated Engagement Manager who is committed to you and your success.
- Medical, Dental, Vision, and Life Insurance
- ClearlyRated's Best of Staffing® Client and Talent Award winner 12 years in a row.
