Why Is R Worth Considering for Data Scraping?

"Use Python for scraping, R for analysis" is the default assumption on many teams. This division of labor made historical sense, but as the rvest package matured and the tidyverse ecosystem filled out, R's capability in data scraping has been seriously underestimated.

rvest's core advantage isn't that its scraping capability outperforms Python's BeautifulSoup or Scrapy — it's that scraping and analysis happen in the same environment. A pharmaceutical data collection task done in Python needs a CSV export, then a re-import into R for statistical analysis; done in rvest, you flow directly into dplyr for cleaning and ggplot2 for visualization, with no format conversion or environment switching overhead.

For teams already using R heavily for data analysis, rvest is the lowest learning-cost scraping option.

How Do You Install and Configure rvest?

rvest is part of the tidyverse ecosystem, installed like any other CRAN package:

# Install rvest
install.packages("rvest")

# Or install the whole tidyverse (which includes rvest)
install.packages("tidyverse")

# Load
library(rvest)

rvest depends on the following core packages:

Once installed, run a simple test to confirm the environment is working:

library(rvest)

# Read a web page
page <- read_html("https://example.com")

# Extract the title
page %>% html_element("title") %>% html_text()
# Expected output: "Example Domain"

If the code above returns a result correctly, rvest's full dependency chain is ready.

What Are rvest's Core Functions?

rvest's API is extremely lean — fewer than 10 core functions. Mastering the following 6 covers 90%+ of scraping needs:

Difference between html_text() and html_text2(): html_text2() simulates browser text-rendering behavior, automatically handling whitespace and line breaks so the output more closely matches what a human reader sees. In most scenarios, prefer html_text2().

CSS Selectors vs. XPath

rvest supports both CSS selectors and XPath for element location:

# CSS selector
page %>% html_elements("div.article h2")

# XPath
page %>% html_elements(xpath = "//div[@class='article']//h2")

How Do You Build a Complete Scraping Case?

Using a public tender listing page as an example, here's the full flow from page read to data cleanup.

Step 1: Read the Target Page

library(rvest)
library(dplyr)

# Read the listing page
url <- "https://example.com/bids/list"
page <- read_html(url)

Step 2: Locate Data Elements

Use the browser's dev tools to inspect the HTML structure and identify the target's CSS selector. Assume each tender listing sits inside a div.bid-item container:

# Extract all tender entries
items <- page %>% html_elements("div.bid-item")

# Check how many were matched
length(items)

Step 3: Extract Structured Fields

# Extract title, publish date, and amount for each record
bids <- tibble(
  title = items %>% html_element("h3.title") %>% html_text2(),
  date  = items %>% html_element("span.date") %>% html_text2(),
  amount = items %>% html_element("span.amount") %>% html_text2(),
  link  = items %>% html_element("a") %>% html_attr("href")
)

# View the first few rows
head(bids)

Step 4: Data Cleaning

bids_clean <- bids %>%
  mutate(
    date = as.Date(date, format = "%Y-%m-%d"),
    amount = as.numeric(gsub("[^0-9.]", "", amount))
  ) %>%
  filter(!is.na(title))

The entire flow from scraping to cleaning lives in the same R script; the cleaned bids_clean can be passed straight to downstream statistical analysis and visualization code.

How Do You Handle Table Data?

A lot of public data comes as HTML tables, and rvest's html_table() converts these directly into an R data.frame:

# Extract all tables on the page
tables <- page %>% html_table()

# Grab the first table
df <- tables[[1]]

# When the header sits in the first row
df <- page %>%
  html_element("table.data-table") %>%
  html_table(header = TRUE)

Common html_table() arguments:

In pharmaceutical data scraping scenarios, public drug catalogs and price disclosure tables are typically published as HTML tables. html_table() combined with fill = TRUE handles irregular table structures caused by merged cells.

How Do You Handle Paginated Scraping?

Most listing pages are paginated, and rvest doesn't build in pagination logic — you construct the paginated URL and loop manually:

library(purrr)

# Build paginated URLs
base_url <- "https://example.com/bids/list?page="
pages <- 1:10

# Define a single-page scraper function
scrape_page <- function(page_num) {
  url <- paste0(base_url, page_num)
  
  page <- tryCatch(
    read_html(url),
    error = function(e) {
      message(paste("Page", page_num, "failed:", e$message))
      return(NULL)
    }
  )
  
  if (is.null(page)) return(tibble())
  
  items <- page %>% html_elements("div.bid-item")
  
  tibble(
    title = items %>% html_element("h3.title") %>% html_text2(),
    date  = items %>% html_element("span.date") %>% html_text2(),
    amount = items %>% html_element("span.amount") %>% html_text2()
  )
}

# Batch scrape with a 2-second delay between pages
all_bids <- map_dfr(pages, function(p) {
  Sys.sleep(2)  # Control request interval
  scrape_page(p)
})

Key point: Sys.sleep(2) controls the request interval to avoid putting too much pressure on the target site. In website scraper scenarios, a reasonable request interval is a prerequisite for sustained, stable collection.

How Do You Handle Request Failures and Exceptions?

Network timeouts, temporary target-site unavailability, and page structure changes are all common during data collection. Robust scraping code needs solid exception handling.

Retry Logic

read_html_with_retry <- function(url, max_retries = 3, delay = 2) {
  for (i in seq_len(max_retries)) {
    result <- tryCatch(
      read_html(url),
      error = function(e) {
        message(paste("Attempt", i, "failed:", e$message))
        if (i < max_retries) Sys.sleep(delay * i)
        return(NULL)
      }
    )
    if (!is.null(result)) return(result)
  }
  warning(paste("All retries failed for:", url))
  return(NULL)
}

Handling Missing Elements

When page structures aren't consistent, certain elements may be absent from some pages:

# Safe extraction function
safe_text <- function(node, selector) {
  el <- html_element(node, selector)
  if (is.na(el)) return(NA_character_)
  html_text2(el)
}

# Usage
bids <- tibble(
  title = map_chr(items, ~safe_text(.x, "h3.title")),
  date  = map_chr(items, ~safe_text(.x, "span.date"))
)

Common Errors and Handling

How Do You Set Request Headers and a Proxy?

rvest uses httr under the hood for HTTP requests, so you customize headers and proxy settings through httr's configuration:

Setting User-Agent

library(httr)

session <- session("https://example.com",
  user_agent("Mozilla/5.0 (compatible; DataBot/1.0)")
)

page <- session %>% read_html()

Configuring a Proxy

# Configure an HTTP proxy
page <- read_html(
  "https://example.com",
  httr::use_proxy(
    url = "http://proxy.example.com",
    port = 8080,
    username = "user",
    password = "pass"
  )
)

Setting a Timeout

page <- read_html(
  "https://example.com",
  httr::timeout(30)
)

In scenarios like public-opinion monitoring or tender data collection where the pipeline needs to run stably long-term, reasonable header configuration and timeout settings are basic guarantees against collection interruption.

What Are rvest's Advanced Uses?

Form Submission

rvest supports simulating form submission, which fits scraping tasks that require search or filter conditions:

# Create a session
sess <- session("https://example.com/search")

# Get the form
form <- html_form(sess)[[1]]

# Fill in form fields
form <- html_form_set(form,
  keyword = "data scraping",
  region = "nationwide"
)

# Submit the form
result <- session_submit(sess, form)

# Parse the results page
result %>% read_html() %>% html_elements("div.result-item")

Session Persistence

The session object created by session() automatically manages cookies, which fits multi-step operations:

sess <- session("https://example.com")
sess <- session_jump_to(sess, "https://example.com/page2")
sess <- session_jump_to(sess, "https://example.com/page3")

Working with httr2

rvest 1.0+ recommends pairing with httr2, which provides a more modern HTTP request API:

library(httr2)

resp <- request("https://example.com") %>%
  req_headers("Accept-Language" = "zh-CN") %>%
  req_retry(max_tries = 3) %>%
  req_perform()

page <- resp %>% resp_body_html()

rvest vs. Python Scraping Tools — How Do You Choose?

The core difference between the two isn't scraping capability but ecosystem fit:

Conclusion: If R is your team's primary language, scraping scale is under tens of thousands of pages per day, and the target pages don't require JavaScript rendering, rvest is the more efficient choice. If scraping scale reaches the million-page level or you need to handle dynamically rendered pages, the Python ecosystem's toolchain is more mature.

FAQ

Q: Can rvest scrape JavaScript-rendered pages?

rvest itself doesn't execute JavaScript. If a page's content depends on JavaScript loading, there are two alternatives: first, analyze the XHR requests directly and use httr to call the API and get JSON data; second, pair with the RSelenium package to drive a browser, render the page, then parse the DOM with rvest. The former is more efficient; the latter has broader coverage.

Q: How do I optimize slow scraping speed?

rvest's scraping speed bottleneck is usually network I/O, not parsing. Optimization directions: use the furrr package for multi-threaded parallel scraping, reuse the session object to reduce TCP handshake overhead, enable local caching for unchanged pages. Note that even with parallel scraping, you still need to control concurrency against the same target site.

Q: How do I persist data scraped with rvest?

Scraped results are standard R data.frames or tibbles, and can be written directly to CSV, RDS, or a database. Common approaches: write_csv() for CSV, saveRDS() for R native format, DBI::dbWriteTable() for MySQL/PostgreSQL. RDS format preserves data-type information and is recommended as an intermediate storage format.

Q: How do I avoid getting throttled by the target site during scraping?

The core principle is to mimic normal user access behavior: set a reasonable User-Agent, control request intervals to 2–5 seconds, avoid repeated requests to the same URL in a short window, and respect robots.txt. For large-scale scraping, combining with rotating proxy IPs distributes request origins.

Q: What if html_table() output has garbled characters?

Usually an encoding issue. First specify the encoding at read time with read_html(url, encoding = "UTF-8"). If the page declares a different encoding, use that instead. You can also apply iconv() after reading for encoding conversion. The most common encodings for Chinese pages are UTF-8 and GBK.