19 June 2016

This evening I came across a Google Search result for Game of Thrones Audiobooks and my immediate question was how can I download all these .mp3 files at once to my computer. As going through each of those directory and clicking on those files to download was, well a bit too boring and time consuming.

So I gave a thought to it and tried to play with the very famous shell command wget and that’s it! After few trials it gave me what I needed.

wget --execute="robots = off" --mirror --convert-links --no-parent --wait=5 <website-url>

Explanation with each options

  • wget: Simple Command to make CURL request and download remote files to our local machine.
  • --execute="robots = off": This will ignore robots.txt file while crawling through pages. It is helpful if you’re not getting all of the files.
  • --mirror: This option will basically mirror the directory structure for the given URL. It’s a shortcut for -N -r -l inf --no-remove-listing which means:
    • -N: don’t re-retrieve files unless newer than local
    • -r: specify recursive download
    • -l inf: maximum recursion depth (inf or 0 for infinite)
    • --no-remove-listing: don’t remove ‘.listing’ files
  • --convert-links: make links in downloaded HTML or CSS point to local files
  • --no-parent: don’t ascend to the parent directory
  • --wait=5: wait 5 seconds between retrievals. So that we don’t thrash the server.
  • <website-url>: This is the website url from where to download the files.

Happy Downloading :smiley:

Categories: Tags:

blog comments powered by Disqus