Node.js
JavaScript
Python
Flask
Ruby

Search for '{{search_term}}'

Download a web page

CMOS is the Code-Maven Open Source podcast that also includes video interviews. Subscribe to this feed RSS feed with your Podcast listener app or via iTunes iTunes.

Building a web crawler starts by fetching one page. It can be an HTML page. The RSS or Atom feed of a web site. Some site provide API where get the response as a JSON string.

It can be downloading an image or some other binary file.

The task is quite simple. Given a URL fetch the thing that is behind it and put it in a variable.

For a variation on this, you can write a function that will get a URL and a filename and will save the content it received in the given file.

Solutions

Comments

In the comments, please wrap your code snippets within <pre> </pre> tags and use spaces for indentation.
comments powered by Disqus