Note: The job is a remote job and is open to candidates in USA. Beacon Hill is a premier National Information Technology Staffing Group, and they are seeking a Senior Data Engineer to build and support production-grade data pipelines. The role involves migrating existing data assets to Databricks Unity Catalog, optimizing pipelines, and ensuring data quality and reliability.

Responsibilities

• Migrate existing data assets and pipelines to Databricks Unity Catalog
• Refactor and optimize existing pipelines using modern engineering standards and best practices
• Build and maintain scalable PySpark-based data pipelines
• Develop new source-of-truth datasets and modernize legacy data workflows
• Design and implement workflow orchestration using Databricks Jobs
• Deploy and manage pipelines using Databricks Asset Bundles
• Ensure data pipelines are reliable, observable, and production-ready
• Implement data quality controls, monitoring, and operational alerting
• Support platform migrations, testing, validation, and production cutovers
• Partner with engineering teams to improve platform stability, scalability, and data reliability

Skills

• Strong hands-on experience building and supporting production-grade data pipelines using PySpark and Apache Spark
• Extensive experience working within Databricks environments
• Experience with Databricks Unity Catalog migrations and modernization initiatives
• Experience with Databricks Unity Catalog governance and security models
• Strong understanding of ETL/ELT design, data modeling, and incremental data processing
• Experience modernizing legacy data platforms and data migration projects
• Experience working with large-scale datasets in cloud-based data platforms
• Experience with workflow orchestration, scheduling, and pipeline automation
• Knowledge of cloud data platforms such as Snowflake and/or Google Cloud
• Experience implementing data quality checks, monitoring, logging, and alerting
• Strong troubleshooting, collaboration, and problem-solving skills
• Familiarity with advertising, analytics, measurement, or large-scale data ecosystems
• Experience with CI/CD and DataOps best practices for data engineering environments

Benefits

• Medical, dental, vision, and federal and state leave programs as required by applicable agency regulations to those that meet eligibility
• Upon successfully being hired, details will be provided related to our benefit offerings

Company Overview