Build a Web Scraper in Ruby on Rails

Learn how to build a web scraper & monitoring tool with Ruby on Rails

Start Series

16m

8 Lessons | 2h 17m | Beginner

16m

Scraping Web Pages with Ruby

Feb 26, 2024 Pro

Kicking off our Web Scraper series, we start out with the bread and butter: scraping web pages with Ruby and Nokogiri.

14m

Storing Web Scraper Results in ActiveRecord models

We need to keep track of the web pages we want to monitor and scrape data from, so let's create our ActiveRecord model for storing this information and a Result model for storing the result each time we scrape a web page.

11m

Web Scraper UI Controllers & Actions

Feb 28, 2024 Pro

Building the controllers and actions for our Web Scrapper application lets us interact with Rails to add and remove Pages and view the results as they're generated.

42m

Designing our Web Scraper UI with TailwindCSS

Feb 29, 2024 Pro

TailwindCSS is an excellent CSS framework for letting us quickly iterate on design ideas to make our UI much cleaner and easier to use.

14m

Sending Emails For Successful Web Scraper Results

Mar 01, 2024 Pro

When a web page is scraped successfully, we need a way to notify the user. For example, when the Raspberry Pi 5 is finally in stock, we want to know right away with an email.

12m

Deploying the Web Scraper Rails app to Production & Setting Up Cron Jobs

Mar 04, 2024 Pro

The next step is deploying our web scraper to production and setting up our cron jobs. We'll deploy our Rails app to a server using Hatchbox.io and configure cron jobs to run our scraping tasks on an interval.

16m

Sending Emails with Rails in Production using Postmark

Mar 05, 2024 Pro

Now that our application is deployed to production, we can use a service like Postmark to send real emails.

Wrapping up our Web Scraper in Rails

Mar 06, 2024 Pro

Wrapping up our Web Scraper, we debug an issue in production with SolidQueue and some challenges for you to add more features to the web scraper.

Next Series

Finished this series? Continue learning in the next series

OpenAI Whisper Transcriptions with Rails

In this series, we'll build an automated transcription service that monitors a video hosting API, transcribes new videos, and uploads captions to those videos.

Chris Oliver

7 Lessons

1h 58m

Intermediate

Start Series

Courses

Rails for Beginners

Advanced Ruby: Behind the Magic

Payments with Rails Master Class

Refactoring Rails

Apps

Hatchbox.io

Jumpstart Rails SaaS Template

Other

Remote Ruby Podcast

GoRails Open Source

Rails Hackathon

Beginner Bounties

Ruby on Rails Job Board