How do I follow redirects automatically in Ruby web scraping?

In Ruby, when you're doing web scraping and you want to follow redirects automatically, you can use several HTTP client libraries that support this feature. The most popular libraries are Net::HTTP which is part of the Ruby standard library, and external gems like HTTParty and RestClient.

Using Net::HTTP

Net::HTTP does not follow redirects by default, so you need to handle them manually. Here's an example of how to do this:

require 'net/http'

def fetch(uri_str, limit = 10)
  # You should choose a better exception.
  raise ArgumentError, 'too many HTTP redirects' if limit == 0

  response = Net::HTTP.get_response(URI(uri_str))

  case response
  when Net::HTTPSuccess then
    response
  when Net::HTTPRedirection then
    location = response['location']
    warn "redirected to #{location}"
    fetch(location, limit - 1)
  else
    response.value
  end
end

print fetch('http://example.com')

Using HTTParty

HTTParty is a popular gem that simplifies HTTP requests and it automatically follows redirects by default. Here's how to use it:

First, install the gem:

gem install httparty

Then, you can use it in your script:

require 'httparty'

response = HTTParty.get('http://example.com')
puts response.body

Using RestClient

RestClient is another gem that can be used to make HTTP requests in Ruby. It also follows redirects by default. First, you need to install the gem:

gem install restclient

Then, use it as follows:

require 'rest-client'

response = RestClient.get('http://example.com')
puts response.body

In both HTTParty and RestClient, if you want to customize the redirect behavior, you can check their respective documentations for advanced usage. However, for most cases, the default behavior should suffice for following redirects during web scraping.

Remember to always respect the terms of service of the website you're scraping, and be aware that excessive requests can lead to your IP being blocked. It's also good practice to handle exceptions and check the robots.txt file of the website to ensure you're allowed to scrape their pages.