What is web scraping? A new engineer tries to acquire data from a practice website

Introduction

Hi everyone! I'm Mirai, a new engineer who joined the company in July as a liberal arts graduate with no prior experience, and I've been working in the Systems Development Department since July

This is the second article following the previous one on validation

Click here for the previous article ↓

What is validation in Laravel? Explained from the perspective of a complete beginner by a junior engineer

Today's topic web scraping .

This article is written from the perspective of a complete beginner, following the structure below

1. What is web scraping? When can it be used?
2. Things to be careful about before use
3. The difference between static and dynamic pages
4. Try it out on the practice site

Languages/Tools used: Python 3.12 / requests / Beautiful Soup / Playwright (uv is used for package management)

I hope this will be helpful to new engineers with a liberal arts background like me, as well as those who are currently studying!

What exactly is web scraping?

Web scrapinguses programs to automatically extract only the necessary information from the information displayed on a websiteand organize it into a user-friendly format.

Normally, we open a webpage in a browser, look at it visually, and copy the parts we need, right? Web scraping is like having a program do this "open → search → extract" process for you

When we break down what the program does, it follows this process:

1. Access a web page and receive its contents (HTML)
2. Find the location of the target data within the HTML
3. Extract the data and save it to a file or similar

A similar term crawling . Crawling is "visiting pages by following links," while scraping is "extracting information from pages." Search engines collect pages through crawling and then analyze the content of those pages.

When can it be used?

For example, it can be useful in situations like this

• Data Analysis and Research: Collect and analyze a list of publicly available announcements and articles.
• Monitoring your own website: Regularly check the content displayed on your website and for broken links.
• Automating information gathering: Automating the recording of publicly available information that is currently copied manually every day.

Things to be careful about before use

Web scraping is convenient, but you can't just do it on any website you want. Before you start, make sure you check the following points

1. Some websites prohibit automated data acquisition methods such as scraping, as stated in their terms of service . It's essential to refrain from such activities on sites where they are prohibited by the terms of service
2. Check your robots.txt file. robots.txt is a file that a website uses to tell other sites, "Please don't crawl this page with your programs." You can check it at https://(website domain)/robots.txt. However, remember that robots.txt is just a request from the website, so consider it in conjunction with reviewing the terms of service.
3. that don't put a load on the server can be accessed much faster than a human. Considerations such as spacing out requests and accessing only necessary pages are necessary.
4. Be careful when handling acquired data. Acquired text and images may be protected by copyright and may contain personal information. Just because you have acquired them does not mean you can freely publish or redistribute them.

In this article, we will use publicly available websites designed for practicing web scraping , so you can practice without worrying about these points

Before we begin, let's look at the differences between "page types" that you should know about when scraping

Static pages and dynamic pages

Static pages are pages where the data is already written in the HTML received from the server. You can retrieve the data by receiving and reading the HTML.

Dynamic pages are pages where the data is not displayed in a screen format when the HTML is received; the browser must execute JavaScript to display the data on the screen.

Note that the terms "static" and "dynamic" are sometimes used to refer to whether or not the HTML is generated on the server side. In this article, we will refer to pages from which data can be retrieved without executing JavaScript as "static pages," and pages from which data cannot be retrieved without executing JavaScript as "dynamic pages."

The problem here lies with methods like Python's `requests`, which simply "receive HTML." Because `requests` does not execute JavaScript, it cannot find data in the HTML that should be visible to the browser on dynamic pages

That's where Playwright . Playwright is a tool that allows you to run a real browser from a program, and it can extract data from a page after JavaScript has been executed.

Reference: Playwright for Python official documentation (https://playwright.dev/python/)

In the next chapter, we'll verify that it's not possible to retrieve the data using `requests`, and then try retrieving it using Playwright

Let's try it ①: Preparation and confirmation that "requests cannot be used"

The site we'll be using for this tutorial is Quotes to Scrape ( https://quotes.toscrape.com/ ) , a website designed for practicing web scraping . It displays famous quotes and their authors, and also offers a JavaScript version ( https://quotes.toscrape.com/js/ ) that displays the same content

This site is part of "toscrape.com," a sandbox (practice area) provided by Zyte (formerly Scrapinghub), a company that offers web scraping services, for learning how to scrape data

This site is designed for practice, so we'll use it for this test. When trying out other sites, always be sure to check the terms of service and robots.txt file beforehand

Reference: toscrape.com (https://toscrape.com/)

Reference: Quotes to Scrape (https://quotes.toscrape.com/)

Create an environment

For Python package management, uv . uv is a single tool that handles everything from installing the Python executable to managing libraries.

The installation method varies depending on the operating system, so please refer to the official documentation

Reference: uv official documentation (https://docs.astral.sh/uv/)

Once you have installed uv, run the following in the terminal (Command Prompt or PowerShell on Windows):

uv init scraping-practice cd scraping-practice uv add requests beautifulsoup4 playwright uv run playwright install chromium

The last line is a command to download the browser (Chromium) that Playwright runs on

Try retrieving it with Requests

We'll retrieve both the static and dynamic versions using requests and count how many quotes (div.quote) we find

Create a file called compare.py in the scraping-practice folder

import time import requests from bs4 import BeautifulSoup urls = [ "https://quotes.toscrape.com/", # Static page "https://quotes.toscrape.com/js/", # Dynamic page ] for url in urls: res = requests.get(url, timeout=10) soup = BeautifulSoup(res.text, "html.parser") quotes = soup.select("div.quote") print(f"{url} → {len(quotes)} items") time.sleep(1) # Leave a 1 second gap to be considerate of the server

This is what happened when I ran it

https://quotes.toscrape.com/ → 10 items https://quotes.toscrape.com/js/ → 0 items

Even though it looks the same in a browser, the dynamic version shows 0 results. This is because requests do not execute JavaScript, so the quotes (div.quote) that should be displayed on the screen are not yet present in the received HTML. This really illustrates the difference between static and dynamic

Let's try it ②: Retrieving data from a dynamic page using Playwright

Now for the real thing. We'll retrieve the quotes and authors from all and save them to a JSON file.

Find the location to retrieve using developer tools

Before writing any code, find out where the data is located in the HTML. https://quotes.toscrape.com/js/ , right-click on a quote, and select "Inspect" to open the developer tools.

Upon closer inspection, this is what it looked like

• A single quote:div.quote
• Text of the quote:span.text
• Author name:small.author
• "Next →" button:li.next > a

Write the code

Create a file called scrape_quotes.py in the scraping-practice folder

import json import time from urllib.parse import urljoin from playwright.sync_api import sync_playwright START_URL = "https://quotes.toscrape.com/js/" def main(): results = [] with sync_playwright() as p: # Launch the browser without a screen (headless) browser = p.chromium.launch(headless=True) page = browser.new_page() page.goto(START_URL) while True: # Wait for quotes to be displayed using JavaScript page.wait_for_selector("div.quote") for quote in page.locator("div.quote").all(): text = quote.locator("span.text").inner_text() author = quote.locator("small.author").inner_text() results.append({"text": text, "author": author}) print(f"Retrieved up to {page.url} (total {len(results)} items)") # If there is no "Next" button, go to the last page. `next_link = page.locator("li.next > a")` `if next_link.count() == 0: break time.sleep(1)` `# Leave a 1-second gap to be considerate of the server` `next_url = urljoin(page.url, next_link.get_attribute("href"))` `page.goto(next_url)` `browser.close()` `with open("quotes.json", "w", encoding="utf-8")` `as f: json.dump(results, f, ensure_ascii=False, indent=2)` `print(f"Completed: {len(results)} items saved to quotes.json")` `if __name__ == "__main__": main()`

Once you've written it, run the following command

uv run python scrape_quotes.py

point

• `wait_for_selector`: In dynamic pages, there may not be any data available the moment the page is opened. By writing "wait until the quote is displayed," you can prevent data from being missed. This is a concept unique to dynamic pages that was not available in `requests`.
• headless=True: This setting allows the browser to run without displaying the web page. If you set it to False, you can see the browser actually turning pages, so it's recommended to try it out at least once.
• sleep(1): Waits 1 second after each page turn. Note that there is no fixed rule for the number 1 second, so adjust it according to the terms of service and robots.txt settings on your actual website.

result

When I ran it, the data retrieval process proceeded by turning the pages

Retrieved up to https://quotes.toscrape.com/js/ (total 10 items) Retrieved up to https://quotes.toscrape.com/js/page/2/ (total 20 items) (omitted) Retrieved up to https://quotes.toscrape.com/js/page/10/ (total 100 items) Completed: 100 items saved to quotes.json

Finally, quotes.json was created. Its contents look like this: (Some parts omitted)

[ { "text": "“The world as we have created it is a process of our thinking. ...”", "author": "Albert Einstein" }, ... ]

The text and author name of each quote are saved together. We were able to successfully retrieve data from pages that previously showed zero results in requests!

summary

Let's reflect on what we learned this time

• Web scraping is a technique that automatically extracts necessary information from web pages using a program
• Before you begin, please review the Terms of Service and robots.txt to avoid overloading the server
• Dynamic pages may not display data that can be retrieved using requests
• Playwright allows you to run the browser and extract the data after it has been displayed

Thank you for reading this far!

reference:

• Playwright for Python official documentation (https://playwright.dev/python/)

- uv official documentation (https://docs.astral.sh/uv/)

・Quotes to Scrape (https://quotes.toscrape.com/)

・toscrape.com (https://toscrape.com/)