Search for '{{search_term}}'

Download an HTML page using Ruby

CMOS is the Code-Maven Open Source podcast that also includes video interviews. Subscribe to this feed RSS feed with your Podcast listener app or via iTunes iTunes.

While a page on a web-site is totally different from a file, several languages provide a way to read them as if they were regular files. I am not sure if this is a good idea, but it certainly works for some people.

In Ruby the open-uri modules provides this simplified interface.

After loading the module with require it overrides the standard open function so from now on, in addition to opening regular files, it will be able to 'open' URLs as well. Of course it can only open them read-only as we can only fetch pages cannot push them out, but it can get all kinds of additional parameters.


require 'open-uri'
url = ''
fh = open(url)
html =
puts html

The open in such cases will return an instance of StringIO. If we were printing out the contet of fh we would get:

puts fh   #  #<StringIO:0x007fc41c8bc238>

Once we get the object we can apply the same methods as on regular filehandle. For example we can use the read method to read in the content of the whole page. As opposed to the case when we read regular files, in this case there is no efficiency reason to read the content line-by-line. The way HTTP works it does not make much sense. By the time we start reading the page the whole document have arrived and is located in the memory of our program. We can as well copy it to our internal variable using the read method.

Lie about who are we

When a browser access a web site it tells the site what kind of browser is that, which version etc. The same happens when we "open" a web page using the open function supplied by the open-uri module.

By default opern-uri calls it 'browser' Ruby which does not say mutch. We can change it to whatever we want by passing "User-Agent" to the open call:


require 'open-uri'
url = ''
fh = open(url, 
   "User-Agent" => "Code-Maven-Example (see: )"
html =
puts html

This string will be written in the Access log of the web server we connect to.


In the comments, please wrap your code snippets within <pre> </pre> tags and use spaces for indentation.
comments powered by Disqus