Military Spouse Connection Jobs

Description

The Core Data & Services team within the Data organization is seeking a Senior Data Engineer. In this role, you will be instrumental in establishing foundational data pipelines and datasets that are critical for enabling numerous engineering and analytical teams to harness data for key business decisions. You will provide vital information to engineering, analytics, and operational teams, supporting the scaling of the largest streaming service.

Your efforts to expand, scale, and standardize core principles through observability, lineage, data quality, logging, and alerting across all engineering teams are essential for creating a unified data platform. We are looking to grow our team with world-class Data Engineers who share our passion for making a positive impact!

NO C2C, Relocation won't be considered. Must live in New York.

Key Responsibilities:

• Maintain, update, and expand existing Core Data platform data pipelines using Scala and Python/Spark, ensuring strict uptime SLAs.

• Extend current Core Data platform functionality, including metadata parsing, extending the Metastore API, and building new integrations with internal and external APIs.

• Implement new batch and streaming data pipelines using Scala, Databricks, and Airflow.

• Develop the Lakehouse architecture, collaborating with customers, partners, and stakeholders to shift towards a Lakehouse-centric data platform.

• Create shared libraries in Scala and Python to abstract complex business logic, ensuring consistent functionality across all data pipelines.

• Utilize tech stack including Airflow, Spark, Databricks, Delta Lake, Snowflake, Scala, and Python.

• Collaborate with product managers, architects, and engineers to drive Core Data platform success.

• Develop and document internal and external standards and best practices for pipeline configurations, naming conventions, partitioning strategies, and more.

• Ensure high operational efficiency and quality of Core Data platform datasets to meet SLAs and ensure reliability and accuracy for all stakeholders.

• Actively participate in agile/scrum ceremonies to improve team processes.

• Engage with and understand customers, forming relationships to prioritize innovative new offerings and platform improvements.

• Maintain detailed documentation to support data quality and data governance requirements.

  Requirements

Basic Qualifications:

• 5+ years of data engineering experience developing large data pipelines.

• Strong algorithmic problem-solving skills.

• Proficient in Scala and Python programming.

• Basic understanding of AWS or other cloud provider resources (S3).

• Strong SQL skills with the ability to create queries to analyze complex datasets.

• Hands-on experience with distributed processing systems such as Spark in a production environment.

• Experience with data pipeline orchestration systems such as Airflow.

• Familiarity with scripting languages.

• Willingness and ability to learn new skill sets.

• Self-starter with an eye for detail and excellent analytical and communication skills.

Preferred Qualifications:

• Experience with major MPP or cloud database technologies (Snowflake, Redshift, Big Query).

• Experience developing APIs with GraphQL.

• Deep understanding of AWS or other cloud providers and infrastructure as code.

• Familiarity with data modeling techniques and data warehousing best practices.

• Familiarity with Scrum and Agile methodologies.

Required Education:

• STEM Bachelor’s or Master’s Degree

  Technology Doesn't Change the World, People Do.®

Robert Half is the world’s first and largest specialized talent solutions firm that connects highly qualified job seekers to opportunities at great companies. We offer contract, temporary and permanent placement solutions for finance and accounting, technology, marketing and creative, legal, and administrative and customer support roles.

Robert Half works to put you in the best position to succeed. We provide access to top jobs, competitive compensation and benefits, and free online training. Stay on top of every opportunity - whenever you choose - even on the go. Download the Robert Half app (https://www.roberthalf.com/us/en/mobile-app) and get 1-tap apply, notifications of AI-matched jobs, and much more.

All applicants applying for U.S. job openings must be legally authorized to work in the United States. Benefits are available to contract/temporary professionals, including medical, vision, dental, and life and disability insurance. Hired contract/temporary professionals are also eligible to enroll in our company 401(k) plan. Visit roberthalf.gobenefits.net for more information.

© 2024 Robert Half. An Equal Opportunity Employer. M/F/Disability/Veterans. By clicking “Apply Now,” you’re agreeing to Robert Half’s Terms of Use (https://www.roberthalf.com/us/en/terms) .