GCP Data Architect(Concord, CA - Toronto, Ontario
11 hours ago
Job description
Role : GCP Data Architect
Location :
Concord, CA (Onsite)
Long Term
Job Description
What You'll Do (Responsibilities):
Architect & Design: Design and implement robust, scalable, and cost-effective data solutions on Google Cloud, serving as the target architecture for migrated workloads.
Develop Reusable Frameworks & Accelerators: Design, build, and maintain reusable frameworks, templates, and code libraries to standardize and accelerate data engineering work.
This includes creating boilerplate pipeline structures, generic data validation modules, and automated deployment patterns that other engineers will leverage.
Migrate & Modernize: Lead the hands-on migration of data and processes from on-premises systems like Teradata and Hadoop to Google Cloud services, with a primary focus on BigQuery, Google Cloud Storage (GCS), Dataflow, and Dataproc. ETL/ELT Transformation: Analyze, deconstruct, and translate complex legacy ETL logic from tools like Informatica and Teradata BTEQ/Stored Procedures into modern, cloud-native pipelines, leveraging the frameworks/tooling you help create.
Pipeline Development: Build and automate new data pipelines for batch and streaming data using Python, SQL, and GCP's core services, ensuring all new development contributes to and benefits from our shared engineering frameworks.
Performance & Cost Optimization: Proactively optimize BigQuery performance through effective partitioning, clustering, and query tuning.
Data Validation & Governance: Develop and implement rigorous data validation frameworks to ensure data integrity and accuracy post-migration. Collaborate with governance teams to apply data security, lineage, and cataloging using tools like Google Cloud Data Catalog and Dataplex.
Collaboration & Mentorship: Work closely with on-premises data experts, business analysts, and other engineers to understand requirements, ensure a smooth transition, and act as a subject matter expert and mentor for GCP and internal framework best practices.
Required Qualifications (Must-Haves):
Professional Experience: 5+ years of professional experience in a data engineering role, with a proven track record of building and maintaining large-scale data systems.
Framework Design & Development: Demonstrable experience designing, building, and promoting the adoption of reusable data engineering frameworks such as Ingestion, Transformation, Validation etc
On-Premises Data Warehouse/Big Data Expertise: Deep, hands-on experience with at least one of the following on-premise ecosystems:
Teradata: Strong understanding of the Teradata architecture, utilities (BTEQ,
TPT, FastLoad/MultiLoad), and advanced SQL/Stored Procedure development.
Hadoop: Experience with the Hadoop ecosystem (HDFS, YARN, Hive) and hands-on proficiency with PySpark for large-scale data processing.
Enterprise ETL: Demonstrable experience designing and building complex workflows in an enterprise ETL tool like Informatica PowerCenter.
Google Cloud Platform (GCP) Proficiency: Demonstrable hands-on experience designing, building, and operating solutions with a comprehensive set of GCP data services, including:
Core Data Processing & Warehousing: Google BigQuery (including data modeling, performance tuning, cost management), Google Cloud Storage
(GCS), Cloud Dataflow, and Cloud Dataproc. Orchestration & Event-Driven Architecture: Cloud Composer (Managed
Airflow) for complex workflow orchestration, and Pub/Sub and Cloud Functions for building streaming and event-driven data pipelines.
Data Governance & Management: Practical experience using Dataplex for unified data management, security, and governance across data lakes and warehouses.
Core Engineering & Migration Skills:
Expert-level proficiency in SQL, including complex joins, window functions, and
performance tuning across different database engines.
Strong programming skills in Python, applying software engineering best practices.
Hands-on experience with Google's migration assessment tools (e.g.,
BigQuery Migration Service, Database Migration Service) to analyze on-premise workloads and accelerate migration.
Deep understanding of data warehousing concepts, ETL/ELT patterns, data modeling, and database design.
Preferred Qualifications (Nice-to-Haves):
Proven Migration Experience: Direct, hands-on experience successfully completing at
least one large-scale on-premise (Teradata, Hadoop, etc.) to Google Cloud Platform.
Certifications: A Google Cloud Professional Data Engineer certification is a plus.
Similar jobs
+Nos buscamos un Data Architect para liderar la concepción y el desarrollo de nuestras plataformas y productos de datos modernos. · ...
1 month ago
This job is with Danaher, an inclusive employer and a member of myGwork – the largest global platform for the LGBTQ+ business community. Please do not contact the recruiter directly. · Bring more to life. · Are you ready to accelerate your potential and make a real difference wi ...
18 hours ago
We're hiring an experienced Vision Systems Developer to develop applications that support the automation of predictive decision-making efforts. · Lead the design and development of next-generation vision system features and tooling. · Own the critical vision software and hardware ...
3 days ago
+We're hiring an experienced Vision Systems Developer to develop applications that support the automation of predictive decision-making efforts. · +Lead the design and development of next-generation vision system features and tooling, · A own critical vision software and hardware ...
1 month ago
We are seeking a passionate Jr. Soil Product Specialist to work closely with the Sales Manager and Product Specialist. This position has the potential to advance to a leadership role to facilitate Earthco's strategic growth. · Evaluate soil products through sample collection and ...
1 month ago
We're hiring an experienced Vision Systems Developer to develop applications that support the automation of predictive decision-making efforts. Lead the design and development of next-generation vision system features and tooling, · Own the critical vision software and hardware t ...
2 weeks ago
We are seeking an enthusiastic full-time individual with a design background who will be the front line person in our interior design firm. This role includes providing crucial support to our design team, conducting research on design trends and materials, drafting designs using ...
2 weeks ago
The Electrical Engineering Technologist is responsible for assisting Electrical Designers/Engineers in the development and qualification of research and engineering-grade electrical hardware. · ...
2 weeks ago
The IT Compliance Analyst is responsible for acting as the intermediary between internal or external regulatory bodies and IT. · Translate compliance into action: use a risk-based approach to identify and articulate actionable information security and regulatory requirements for ...
4 weeks ago
Bring more to life. Are you ready to accelerate your potential and make a real difference within life sciences, diagnostics and biotechnology? · ...
1 month ago
We are seeking a Patent Agent/Junior IP Counsel to partner with our Research Scientists and Engineers and work with the Intellectual Property (IP) Legal team to build and manage the SCIEX patent portfolio. · Serve as a trusted colleague to Research & Development (R&D) teams on al ...
1 month ago
As part of SCIEX, you will help to influence life changing research and outcomes. · ...
1 month ago
We're hiring an experienced Vision Systems Developer to develop applications that support the automation of predictive decision-making efforts. · Lead the design and development of next-generation vision system features and tooling, · Owning critical vision software and hardware ...
3 days ago
· We're hiring an experienced Vision Systems Developer to develop applications that support the automation of predictive decision-making efforts.Lead the design and development of next-generation vision system features and tooling, · Owning critical vision software and hardware ...
1 month ago
At SCIEX, one of Danaher's companies, we're committed to saving lives through life sciences and diagnostics. · Evaluate and qualify new commercial off-the-shelf components for SCIEX Mass Spectrometry instruments. · Audit technology and supplier capability lifecycle availability q ...
1 month ago
Toromont Cat seeks a Senior Network Architect to join our team in Concord The role provides technical vision and leadership and translates business objectives into scalable secure and high-performing network solutions. · ...
1 month ago
The Staff Software Developer is responsible for designing and developing advanced hardware and software solutions for one of the world's top life sciences companies.This position reports to the Software Development Manager, Chief Software Architect, and is part of the Software De ...
1 month ago
Bring more to life. · Accelerate your potential and make a real difference within life sciences, diagnostics and biotechnology?You'll thrive in a culture of belonging where you and your unique viewpoint matter. · In this role you will advance your hardware development skills as p ...
2 weeks ago
Job summaryAccelerate your potential and make a real difference within life sciences. · ResponsibilitiesAct as intermediary between internal or external regulatory bodies and IT. · ...
1 month ago
The IT Compliance Analyst is responsible for acting as the intermediary between internal or external regulatory bodies and IT, using a deep understanding of regulatory requirements, business processes, and technology to ensure desired outcomes are achieved.This position reports t ...
4 weeks ago