beBee background
Professionals
>
Toronto
aidan west

aidan west

Data Engineer

Engineering / Architecture

Toronto, Ontario

Social


About aidan west:

3 years’ experience building distributed systems, CI/CD pipelines, and ETL workflows. Proficient in cloud computing (GCP, Azure), programming (Go, Python, Bash), and data modelling with SQL (Postgres, BigQuery). Experience with Infrastructure as Code (Terraform), networking (Wireguard, Tailscale), virtualisation (Proxmox), and containerization (Docker). 

Experience

  • Designed and built a pipeline to extract, decompress and load multiple terabytes of data from Google Cloud Storage to Google BigQuery. Orchestrated in Google Workflows, with Cloud Functions written in Python and a Firestore database to track state. The distributed architecture allowed for a ~40% reduction in processing time, whilst ensuring idempotence and zero duplication.
  • Designed and implemented a CI/CD (continuous-development-continuous-integration) pipeline to automate the provisioning of GCP resources. Successful pull requests to Git branches would trigger Cloud Build to run Terraform commands. This enabled the creation of production and testing environments, ensuring that deployments to production would be safe.
  • Oversaw a cloud-to-hybrid migration, integrating a newly acquired on-premises server into our infrastructure. Administered with Proxmox. Set-up a Postgres server and secured database access by routing all clients through a proxy. This was undertaken to optimise costs and enhance control over internal data storage.
  • Integrated Evidence for cost-free data exploration, reducing BigQuery query spikes and cutting cloud costs by 14%.
  • Used Go to write an application to detect unexpected shifts in model outputs after quarterly input updates, ensuring consistency. This reduced detection time by ~11% and enabled faster error resolution. 
  • Built a containerized infrastructure automation pipeline to provision Proxmox VMs using Terraform and cloud-init. Configured cloud-init for automatic VM enrolment into the company tailnet, enabling access via Tailscale SSH. This streamlined deployment by eliminating the need for manual SSH key management.
  • Elevated the technical proficiency of the analytics team through the creation and delivery of Python training sessions, enhancing their programming capabilities. This allowed them to better leverage our data.

Education

Durham University, England - MSc Data Science – Distinction - OCTOBER  2020 - OCTOBER 2021

  • Dissertation focussed on training an Optical Character Recognition (OCR) model to digitise and analyse foreign office documents.

Durham University, England - BA History - Upper Second Class Honours - SEPTEMBER  2017 - JUNE 2020

  • Final year project focussed on mapping the shifting territories of the Dukes of Burgundy, using Python as tool to collate and map data.

Professionals in the same Engineering / Architecture sector as aidan west

Professionals from different sectors near Toronto, Ontario

Other users who are called aidan

Jobs near Toronto, Ontario

  • Hays

    Data Engineer

    1 week ago


    Hays Toronto, ON

    Your New Company · Join a leading organization that specializes in delivering innovative data solutions using Microsoft Fabric. Our team is dedicated to driving transformation and achieving excellence through cutting-edge technology and expert consulting services. · Your New Ro ...


  • Themesoft Inc. Toronto

    Job Title: Azure Data Engineer · Location: Toronto, Canada · Experience Level: 10+ years · Contract Position: · Key Responsibilities: · Design and implement scalable data pipelines using Azure Data Factory (ADF) to automate data integration and transformation processes. · Utilize ...


  • TransUnion LLC Burlington

    About Us: · At TransUnion, we're dedicated to finding ways information can be used to help people make better and smarter decisions. As a trusted provider of global information solutions, our mission is to help people around the world access opportunities that lead to a higher qu ...