Lead Data Engineer - Etobicoke, Canada - National Consultants Professionals Ltd

Description

The Lead Data Engineer will be responsible for leading a team of data engineers that build and deliver the components of an effective Data Warehouse, leveraging Azure Data Lake, Azure Synapse, Snowflake for data storage, utilising Azure Data Factory and Synapse Pipelines for data integration, Databricks for data transformation and PowerBI for end-user data analytics and reporting. This role will play a critical part in the architectural design and implementation of these elements - capable of meeting a wide range of financial and underwriting reporting/analytics needs. This role will encompass a mix of hands-on technical leadership, team management, strategy development, and stakeholder relations; supporting Sunwing to continue its journey to develop its data platform on next-generation technology.

The Lead Data Engineer will:

• data engineering through both day-to-day data management (BAU) activities and strategic development projects, defining and orchestrating tasks and deliverables.
• develop, and hold accountability for core data engineering services, encompassing the end-to-end data lifecycle journey from ingestion through to the Warehouse.
• as a senior custodian of the Data Platform; administering CICD pipelines, completing peer/code reviews, ensuring the integrity of data, and holding accountability for access and security permissions.
• and develop ETL/ELT processes, data lake storage architecture, data warehouse, data marts; in-line with the company data management framework and enterprise data strategy.
• and develop models and databases both relational and dimensional to support reporting requirements.
• reporting solutions in line with standards, architectural principles, and practices.
• change impact assessment on live systems and processes.
• in an agile, proactive manner, delivering high quality solutions in agreed timescale.
• and support junior data engineers/developers in the team Principal Accountabilities:
• as a senior custodian of the Data Platform, with performance, security, and data integrity at the forefront of all decisions, ensure that adequate steps are taken to protect data and the data platform.
• in collaboration with Data Architects and act as a Senior contributor on the design, build, and management of the Data Platform; taking direct ownership of data engineering activities and responsibilities and ensure that all new data requirements are met using the most appropriate architecture components and engineering practices.
• leadership of both senior and general data engineers.
• ownership of incidents and define remediation plans, with the support of the wider team, to ensure production issues are resolved in line with expected RTO.
• and enforce SDLC best practices, coding standards and design principles covering all aspects of data lifecycle management and BI solutions.
• business requirements and build data solutions to meet business needs.
• in developing data management framework policies, processes, and standards.
• abreast of technology trends, state-of-the-art platforms, data engineering techniques/standards and industry trends to provide input on strategic roadmap.
• maintains a low technical debt & identify area(s) for automation and continuous improvement.
• Proof of Concepts to prove that business requirements can be met by technical solutions.
• with building the team and hiring activities.

Education, Qualifications, Knowledge, Skills, and Experience:

Required Skills & Knowledge

• hands-on experience with Azure Data Lake (incl. Blob Storage, Azure Synapse, Azure Data Factory, Azure Synapse and Snowflake
• experience of SQL/Azure SQL data warehousing
• experience of data engineering activities such as release management, environment controls, CICD pipeline orchestration
• working with Python and Spark
• ability of Microsoft BI software development including database / reporting development, full development cycle
• data modelling, problem solving, information analysis, attention to detail, flexibility of approach.
• ability to deal directly with business functions and interpret their thoughts into successful data & analytics solutions.
• ability to design and develop complex BI solutions in line with agreed architectural principles and toolsets.
• of working closely with the business and delivering complex requirements to tight timescales.
• interest in latest data engineering technologies, best practises, techniques and trends in the data industry Highly Desirable Skills and Knowledge
• experience in the design and build of data warehouse/data lake.
• of Synapse.
• development techniques (SCRUM, Kanban etc.) Required Toolsets.
• Data Lake Storage, Analysis Services (Multidimensional/Tabular), Power BI, Azure Data Factory, Synapse pipelines, Azure Databricks, Spark.
• SQL Server or Azure SQL DB.
• knowledge of data modelling techniques is Desirable Toolsets.
• working with Snowflake.
• with API, Connectors, etc. is desirable.