Build N8N server on AWS and automate tasks including scraping and email
2 days ago
Job description
Project OverviewWe currently use + Apify to scrape product data. While functional, the process is too slow for large datasets.
We need to move to a faster, scalable architecture built in N8N, capable of efficiently scraping:
-5,000 – 20,000 SKUs per batch
- Large datasets with parallelized execution
- Modular architecture for future scraper expansion
We believe the solution may require:
- Distributed workflows
- Multiple concurrent scraping actors/workers
- Queue based batching system
- Rate limiting + proxy handling
- Smart retry logic
- We are open to the best architectural recommendation from you.
Phase 1:
Infrastructure
- Deploy and configure N8N on our AWS environment
- Recommend AWS architecture (EC2, ECS, Docker, scaling strategy)
- Implement proper logging and error tracking
- Ensure system stability and scalability
Phase 2:
High-Performance Scraper (Initial Use Case)
- Build first large scale scraping workflow
- Accept large SKU list input (5k–20k SKUs)
- Parallelize requests efficiently
Implement:
- Proxy support (if required)
- Rotation logic
- Retry/backoff handling
- Captcha/anti bot mitigation strategy
- Structured output (JSON/CSV/database ready)
- Success Criteria
- Scraping 5,000+ SKUs reasonably fast
- Scalable architecture for future scrapers
- Clean, modular N8N workflows
- Clear documentation for internal team
Strong experience with:
- N8N (required)
- AWS infrastructure (EC2, ECS, Docker)
- High scale scraping architectures
- Proxy rotation systems
- Anti bot mitigation strategies
- Experience building distributed or parallelized workflows
- Understands rate limiting, queues, and fault tolerance
- Can recommend performance optimized solutions
- This is the first of multiple scraping projects
- Long term work available for the right candidate
- Must be comfortable working with large datasets
- Must be able to think architecturally, not just build a simple scraper
Please include:
- Examples of scraping systems you've built
- Scale handled (number of records per run)
- N8N experience (live examples preferred)
- Suggested high level architecture for this project
Contract duration of 1 to 3 months. with 30 hours per week.
Mandatory skills:
Automation, Data Scraping
Similar jobs
I need someone to help me scrape the internet to find emails for all of the NYC restaurants I have in an excel file. · ...
1 week ago
We are seeking a skilled freelancer to scrape emails from the website The ideal candidate will have experience in web scraping and data extraction ensuring accurate and efficient collection of email addresses. · ...
1 month ago
I have a list of P&C insurance agencies based in Texas. I need to find their website, key contacts of owners / CEOs / Principals, their emails, their phone numbers, and their addresses It should be relatively straightforward data scraping · ...
3 weeks ago
We're looking for a reliable and detail-oriented freelancer to research and compile a targeted contact list of music blogs and music editorial outlets. · ...
1 month ago
I'm looking for a detail-oriented researcher to build a Google Sheet with company names and contact persons for the first 100 results on Google for the search query: "behang". · ...
1 month ago
We are seeking a detail-oriented freelancer to help us scrape email addresses from a provided list of companies. · The project requires accuracy and attention to detail to ensure we gather valid and up-to-date email addresses. · ...
1 month ago
Data Enrichment Specialist for Email Scraping
Only for registered members
We are seeking a Data Enrichment Specialist to gather verified business contact emails and names from a list of domains.Data Mining · Data Scraping · List Building · ...
1 month ago
We are seeking a skilled freelancer to scrape verified email addresses of real estate agents. · Lead Generation · Data Scraping · Data Entry · ...
1 month ago
I am seeking a detail-oriented freelancer to help me scrape my email inbox for receipts and invoices. · Data Entry<li Microsoft Excel<li Virtual Assistance<li Email Communication · ...
1 week ago
Email Address Scraping for Medical Businesses
Only for registered members
I need someone to scrape medical colleges, nursing programs, dentists, optometrists, · surgery centers, assisted living facilities, home health care workers, · and medical businesses in a 10 square mile radius of my locations · to get email addresses. · ...
1 month ago
We manage 20+ clients in the loan and finance space. · I'm looking for someone with a backend POV on backlink scraping for outreach. · This role is research and data extraction only - no emailing, · no pitching, · no follow-ups.Scrape · & identify relevant SaaS / real business we ...
1 month ago
Email and Data Scraping for UGC TikTok Campaign
Only for registered members
I am looking for a skilled freelancer to assist with email and data scraping for an upcoming Gen Z User-Generated Content (UGC) TikTok campaign. · Data scraping techniques and tools · ...
1 month ago
Email Scraping for Real Estate Companies in Nigeria
Only for registered members
I need someone to scrape business emails of real estate companies based in Nigeria. · This job is focused on collecting valid company emails and delivering them in a clean format. · ...
3 weeks ago
Email and Data Scraping for UGC TikTok Campaign
Only for registered members
Lead Generation · Data Scraping · Data Entry · ...
1 month ago
Email Scraping and Data Collection Expert Needed
Only for registered members
We are seeking a detail-oriented freelancer to assist with email scraping and data collection. · Data Scraping · Data Mining · Data Entry · ...
2 weeks ago
We are looking for a skilled web scraper to collect realtor email addresses from various online sources across five specific states. The goal is to compile a comprehensive list to support our cold email campaign. · Data extraction · Familiarity with real estate platforms · ...
2 weeks ago
Python Developer for Email Archive Data Scraping
Only for registered members
We need you to scrape each URL and extract the 3 main article blocks from each email. · I will provide:the CSV of campaign URLs (+ campaign IDs / sent dates) · 5–10 example URLs to calibrate the extraction · clarification on what counts as the "3 main articles" vs other links (so ...
2 weeks ago
Email Scraping and Verification for Automotive Repair Shops
Only for registered members
We are seeking a detail-oriented freelancer to scrape email addresses from automotive repair shops. · Data Scraping · ...
2 weeks ago
Email scraping for UGC Creators and Micro Influencers
Only for registered members
I am looking for a Lead Generation expert to help me build large, high-quality creator outreach lists. · ...
1 month ago
Email Data Scraping from Public Online Directories
Only for registered members
We are currently seeking a qualified freelancer to assist with the extraction of email data from various publicly available online directories. · A strong attention to detail and the ability to deliver results within established timelines are essential for this position. · Initia ...
3 days ago
Data Scraping and Automated Email Outreach for Romanian Doctors
Only for registered members
We are seeking a skilled freelancer to assist in gathering a comprehensive list of doctors based in Romania through data scraping techniques. · Data Scraping · Web Automation · Email Marketing · ...
3 weeks ago