Selenium Web Scraping Project From Scratch using Replit and AWS Lambda

Code: https://github.com/aakashns/selenium-... Note: Chromedriver & chromium doesn't come pre-installed with Replit after a new update. Please follow these steps to add chromedriver & chromium to replit: https://jovian.ai/birajde9/replit-add... Web scraping is a great way to extract public information from websites and create datasets for data analysis and machine learning. In this live hands-on workshop, we will walk through the process of building and deploying a web scraping project from scratch using Python, Selenium, and AWS Lambda. Chapters 00:00:00 - Project setup, execution, and first attempt 00:31:15 - Extracting Information using Selenium 01:07:30 - Sending results over email using SMTP 01:27:09 - Deploying the scraper using AWS Lambda 02:06:44 - Setting up a recurring job + Project Summary Objective 1. Scrape top 10 trending videos on YouTube using Selenium 2. Set up a recurring job on AWS Lambda to scrape every 30 minutes 3. Send the results as a CSV attachment over email (or to a spreadsheet) Prerequisites Python Topics Covered GitHub Replit Selenium AWS Lambda SMTP Step 1 - Create a GitHub repository Create a repository at https://github.com/new Add README, gitignore (Python) and license (Optional) Clone the repository locally References: Introduction to GitHub: https://lab.github.com/githubtraining... Git & GitHub tutorial:    • Git and GitHub for Beginners - Crash Course   Step 2 - Launch the repository on Replit Connect Replit with your GitHub account Launch the repository as a Replit project Set up the language and run command Create and execute a Python script Attempt to scrape the page using requests & Beautiful Soup References: Introduction to Replit: https://docs.replit.com/tutorials/01-... Replit + GitHub: https://docs.replit.com/tutorials/06-... YouTube trending feed: https://www.youtube.com/feed/trending Beautiful soup tutorial: https://blog.jovian.ai/web-scraping-u... Step 3 - Extract information using Selenium Install selenium and create a browser driver Load the page and extract information Create a CSV of results using Pandas References: Selenium tutorial: https://www.browserstack.com/guide/py... Pandas tutorial: https://jovian.ai/learn/data-analysis... Step 4 - Set up a recurring job on AWS Lambda Create an AWS Lambda Python function Deploy a sample script and observe the output Add layers for Selenium and Chromium Set up recurring job using AWS CloudWatch References: Python on AWS Lambda tutorial: https://stackify.com/aws-lambda-with-... Chromium & Selenium on AWS Lambda: https://dev.to/awscommunity-asean/cre... Recurring AWS Lambda functions: https://docs.aws.amazon.com/lambda/la... Step 4 - Send results over email using SMTP Create email client using smtplib Set up SSL, TLS and authenticate with password Send a sample email with just text Send an email with text and attachment References: Sending Email with Python: https://stackabuse.com/how-to-send-em... Send email using Python: https://www.geeksforgeeks.org/send-ma... Environment variables on Replit: https://docs.replit.com/programming-i... https://docs.aws.amazon.com/lambda/la... Update Google sheets using Python: https://www.analyticsvidhya.com/blog/... Selenium Lambda Layers: https://github.com/aakashns/selenium-... Reference code: https://gist.github.com/aakashns/5393... The workshop lasts approximately 3 hours and all code will be written live during the workshop. You will be able to follow along with the recording to work on your own web scraping project. About the Speaker Aakash N S is the founder & CEO of Jovian, a learning platform for data science with a community of 100,000+ data science enthusiasts. Aakash is also an avid blogger & online educator. Courses taught by Aakash have over 1.5 million views on YouTube.