Downloading torrent results in weird results

This is the place for queries that don't fit in any of the other categories.

Downloading torrent results in weird results

Postby michaelz » Mon Sep 16, 2013 7:16 pm

Hi,

I tried searching through the forum, but couldn't find anything close to this.

When I dowload a torrent-file with the following code, the file can't be read;
Code: Select all
#fileUrl is the original file, example; http://cdimage.debian.org/debian-cd/7.1.0/ia64/bt-cd/debian-7.1.0-ia64-CD-1.iso.torrent
#outputFile is the filename to use for saving, example; debian-7.1.0-ia64.iso.torrent
#tempOutputDirectory is a global var and the temporary directory I want to use, example; /tmp
def downloadTorrentFile(fileUrl, outputFile):
   outputFile=tempOutputDirectory + "/" +outputFile
   req = urllib2.Request(fileUrl)
   response = urllib2.urlopen(req)
   the_page = response.read()
   
   f_out=open(outputFile,'wb')
   f_out.write(the_page)
   f_out.close()


So I started digging and it appears because it's gripped. So I changed to code a little bit to this
Code: Select all
def downloadTorrentFile(fileUrl, outputFile):
   outputFile=tempOutputDirectory + "/" +outputFile
   req = urllib2.Request(fileUrl)
   response = urllib2.urlopen(req)
   the_page = response.read()
   decompressed_data=zlib.decompress(the_page, 16+zlib.MAX_WBITS)
   
   f_out=open(outputFile,'wb')
   f_out.write(decompressed_data)
   f_out.close()


The file can be read now and even though it looks valid, the content isn't the way it should. It won't download anything (apparently the trackers ain't good) and the main file is in an additional sub-directory and rar-ed. If I use exactly the same download-url in firefox, it looks good so there's something wrong in the downloaded file.

The original would look like something like this
description.txt
debian-ia64.7.1.0.iso

The downloaded version with my code looks like it wants to download something like this (when opened in vuze / ktorrent);
description.txt
debian-ia64.7.1.0/part.rar
debian-ia64.7.1.0/part.r01
debian-ia64.7.1.0/part.r02
...

Any ideas?
michaelz
 
Posts: 4
Joined: Mon Sep 16, 2013 6:58 pm

Re: Downloading torrent results in weird results

Postby metulburr » Mon Sep 16, 2013 8:55 pm

When I dowload a torrent-file with the following code, the file can't be read;

are you talking about the actual torrent file or what the torrent file downloads?

The downloaded version with my code looks like it wants to download something like this

that looks like the iso, just not fully downloaded yet. Did you attempt to complete the process? If so, did you unrar the rar to see if the iso is inside?

I downloaded the torrent, and it is just the iso, so i have no idea where you are getting the text file from. The "part" in the name, looks like you just didn't let it finish:
I downloaded via python3.x, replacing ktorrent with whatever torrent client you are using:
Code: Select all
import urllib.request
import os
import subprocess

url = "http://cdimage.debian.org/debian-cd/7.1.0/ia64/bt-cd/debian-7.1.0-ia64-CD-1.iso.torrent"


filename = os.path.split(url)[1]
torrent = urllib.request.urlopen(url)
savefile = open(filename, "wb")
savefile.write(torrent.read())
savefile.close()

cmd = 'ktorrent {}'.format(filename)
proc = subprocess.Popen(cmd.split())
proc.wait()
New Users, Read This
OS Ubuntu 14.04, Arch Linux, Gentoo, Windows 7/8
https://github.com/metulburr
steam
User avatar
metulburr
 
Posts: 1476
Joined: Thu Feb 07, 2013 4:47 pm
Location: Elmira, NY

Re: Downloading torrent results in weird results

Postby michaelz » Tue Sep 17, 2013 5:59 pm

When using your code I get the following result;

~/scripts$ python test.py
Traceback (most recent call last):
File "test.py", line 1, in <module>
import urllib.request
ImportError: No module named request

I tried altering it to use simly urllib.urlopen but that resulted in more errors.

I'm using python 2.7.4 and when I download the torrent with my code it just doesn't match what I download through a browser.
diff python.torrent browser.torrent returns;
Binary files /tmp/python.torrent and ~/Download/browser.torrent differ

Even though downloaded from the same url.
michaelz
 
Posts: 4
Joined: Mon Sep 16, 2013 6:58 pm

Re: Downloading torrent results in weird results

Postby metulburr » Tue Sep 17, 2013 8:57 pm

oh sorry, i forgot to mention it was python3.x

this is python2.x version:
Code: Select all
import urllib2
import os
import subprocess

url = "http://cdimage.debian.org/debian-cd/7.1.0/ia64/bt-cd/debian-7.1.0-ia64-CD-1.iso.torrent"


filename = os.path.split(url)[1]
torrent = urllib2.urlopen(url)
savefile = open(filename, "wb")
savefile.write(torrent.read())
savefile.close()

cmd = 'ktorrent {}'.format(filename)
proc = subprocess.Popen(cmd.split())
proc.wait()
New Users, Read This
OS Ubuntu 14.04, Arch Linux, Gentoo, Windows 7/8
https://github.com/metulburr
steam
User avatar
metulburr
 
Posts: 1476
Joined: Thu Feb 07, 2013 4:47 pm
Location: Elmira, NY

Re: Downloading torrent results in weird results

Postby micseydel » Tue Sep 17, 2013 9:22 pm

How about urllib.urlretrieve for fetching the torrent? You get everything you do in several lines for free, plus if you had to deal with a larger file I assume it wouldn't read the whole thing into memory first.
Join the #python-forum IRC channel on irc.freenode.net!

Please do not PM members regarding questions which are meant to be discussed publicly. The point of the forum is so that others can benefit from it. We don't want to help you over PMs or emails.
User avatar
micseydel
 
Posts: 1390
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Re: Downloading torrent results in weird results

Postby snippsat » Wed Sep 18, 2013 12:00 am

urlretrieve as mention by micseydel,dos this simpler.
Code: Select all
try:
    # Python2
    from urllib import urlretrieve
except ImportError:
    # Python3
    from urllib.request import urlretrieve

url = "http://cdimage.debian.org/debian-cd/7.1.0/ia64/bt-cd/debian-7.1.0-ia64-CD-1.iso.torrent"
filename = url.split('/')[-1]
urlretrieve(url, filename)

plus if you had to deal with a larger file I assume it wouldn't read the whole thing into memory first.

When mention Just a demo of not reading read the whole thing into memory.
Can of course write this with urllib :?
Requests is very good,and make many thing urllib do look a little old and outdated.
Code: Select all
import requests

r = requests.get("http://download.thinkbroadband.com/10MB.zip", stream=True)
with open('10MB.zip.', 'wb') as handle:
    for block in r.iter_content(1024):
        if not block:
            break
        handle.write(block)
User avatar
snippsat
 
Posts: 231
Joined: Thu Feb 21, 2013 12:04 am

Re: Downloading torrent results in weird results

Postby michaelz » Wed Sep 18, 2013 8:41 am

I started originally with urlretrieve (didn't mention it here), but for some reason the file is still gzip encoded when downloading it with urlretrieve. That's why I started with urllib's request and zlib to decompress the stream. But some reason, the resulting downlaoded files are corrupted and don't give the same results as downloaded through the browser.

When just using urlretrieve, every application (libtorrent, ktorrent, vuze) say it's a badly bencoded file. When decompressing that file with gzip (or request stream with zlib), they seem to get a valid torrent file, but they don't have the right tracker and try to download 20Mb rar-ed files (this is how torrent works on a lower level, with 20Mb rar-ed files).

I did a new try with urlretrieve and vuze says the file is not a .torrent. libtorrent says gives; RuntimeError: invalid bencoding.
michaelz
 
Posts: 4
Joined: Mon Sep 16, 2013 6:58 pm

Re: Downloading torrent results in weird results

Postby michaelz » Wed Sep 18, 2013 9:18 am

I have settled for the simple way, using the magnet :)
http://danfolkes.com/tag/add_magnet_uri/
michaelz
 
Posts: 4
Joined: Mon Sep 16, 2013 6:58 pm


Return to General Coding Help

Who is online

Users browsing this forum: Google [Bot] and 4 guests

cron