Building a web crawler starts by fetching one page. It can be an HTML page. The RSS or Atom feed of a web site. Some site provide API where get the response as a JSON string.

It can be downloading an image or some other binary file.

The task is quite simple. Given a URL fetch the thing that is behind it and put it in a variable.

For a variation on this, you can write a function that will get a URL and a filename and will save the content it received in the given file.

Solutions