Download using curl

Question

Level 1

2 points

Download using curl

Can anyone tell me the command to download multiple files from a single webpage using curl.

I've tried curl -O http://www.domain.com/
but don't know what to append to the end for the following instances

1) all files of the same extension on a webpage
2) all files no matter the extension

i'm trying to grab mostly wmv and various video files but 20 of them at a time.

MacBook Pro 2.4 6 GB Ram, Mac OS X (10.5.8)

Posted on Jan 25, 2011 10:09 PM

Reply

Answer 1

Jan 26, 2011 4:59 AM in response to Bdayz

Curl will need to know the complete url for each file it downloads. I'm pretty sure there is no wild card expansion on the server side. If there are links to these files on the website, you might consider using wget to download the entire site:
wget -rkpq -l4 -nH -np <URL> 2>/dev/null
Or to see the links on a page you can use lynx:
lynx -dump http://www.domain.com | awk '/http/{print $2}'

Reply

Answer 2

Bdayz Author

Level 1

2 points

Jan 27, 2011 5:43 PM in response to Cole Tierney

Is there a way to get wget to only download certain files and exlcude others?

I want to just get most .wav files but all of them are on links on a html page. I'd rather not drag and drop each on into terminal obviously

I tried this:

wget -nd -r l1 --no-parent -A.wav http://domain.com/

It says error 403 forbidden

Reply

Answer 3

Bdayz Author

Level 1

2 points

Jan 28, 2011 2:41 PM in response to Bdayz

this finally worked after cobbling together all the misc info on the web....

wget -r -l1 -H -t1 -np -nd -A.wav -erobots=off http://domain.com

Reply