Problems with ftplib opening and connection

Problems with ftplib opening and connection

Postby pow3rline » Mon Jul 11, 2016 6:59 pm

Hi,

I'm having problems writing an apparently simple code. I'm a total newbie with Python and I'm following a course from which this exercise is taken. The goal is to be able to read a file with a list of public ftp sites which allow anonymous connection and list the /pub directory content. This should be made using 5 threads. So far it seems no so difficult but for some reason which I don't know, it has been impossible for me to connect to any ftp site successfully.

This is my code:

Code: Select all
#!/usr/bin/env python

from  threading import Thread
import Queue
import time
import ftplib

class List_pub(Thread):

        def __init__(self,queue):
                Thread.__init__(self)
                self.queue=queue

        def run(self):
                while True:
                        address_ftp=self.queue.get()
                        print "Connecting to %s" %address_ftp
                        try:
                                ftp=ftplib.FTP(address_ftp)
                                ftp.login()
                                ftp.cwd('pub')
                                ftp.retrlines('LIST')
                        except ftplib.all_errors as e:
                                print "Fail to connect...\n"
                                print e

                        self.queue.task_done()
queue=Queue.Queue()
num_threads=5

for i in range(num_threads):
        worker=List_pub(queue)
        worker.setDaemon(True)
        worker.start()

for line in open('lista_ftps.txt','r'):
        line=line.rstrip('\n')
        queue.put(line)


And the content of "lista_ftps.txt" is (all of them are working, I checked them):

Code: Select all
rtfm.mit.edu
ftp.uni-stuttgart.de
ftp.x.org
ftp.crc.dk
ftp.rediris.es
ftp.heanet.ie
ftp.de.netbsd.org
ftp.iij.ad.jp
ftp.cica.es
ftp.cps.unizar.es
ftp.ua.es
ftp.udc.es
ftp.ugr.es
ftp.uvigo.es


When I execute the script, the result varies in each attempt:

- Sometimes no exit is shown
- In some occasions, it says:
Code: Select all
Exception in thread Thread-1 (most likely raised during interpreter shutdown):Exception in thread Thread-3 (most likely raised during interpreter shutdown):Exception in thread Thread-2 (most likely raised during interpreter shutdown):


Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_innerTraceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner

Traceback (most recent call last):  File "ftp_threads2.py", line 16, in run

  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner  File "/usr/lib/python2.7/Queue.py", line 179, in get
  File "ftp_threads2.py", line 16, in run
  File "/usr/lib/python2.7/threading.py", line 383, in notify
  File "/usr/lib/python2.7/Queue.py", line 179, in get
<type 'exceptions.TypeError'>: 'NoneType' object is not callable  File "ftp_threads2.py", line 16, in run


  File "/usr/lib/python2.7/Queue.py", line 179, in get
  File "/usr/lib/python2.7/threading.py", line 383, in notify  File "/usr/lib/python2.7/threading.py", line 383, in notify

<type 'exceptions.TypeError'>: 'NoneType' object is not callable<type 'exceptions.TypeError'>: 'NoneType' object is not callable

Exception in thread Thread-4 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
  File "ftp_threads2.py", line 16, in run
  File "/usr/lib/python2.7/Queue.py", line 179, in get
  File "/usr/lib/python2.7/threading.py", line 383, in notify
<type 'exceptions.TypeError'>: 'NoneType' object is not callable


At the beggining I thought that I was not passing the correct format fo the "address_ftp" string to FTP but at this point I am not so sure about what is the problem.

Any ideas?

Thanks.
Last edited by micseydel on Mon Jul 11, 2016 7:01 pm, edited 1 time in total.
Reason: Initial post lock.
pow3rline
 
Posts: 4
Joined: Mon Jul 11, 2016 6:04 pm

Re: Problems with ftplib opening and connection

Postby Ofnuts » Tue Jul 12, 2016 8:40 pm

Sort of works for me... First isolate the actual ftp connection in its own function that you can test on the side:

Code: Select all
def ftpList(address_ftp):
    print "Connecting to %s" %address_ftp
    try:
        ftp=ftplib.FTP(address_ftp)
        ftp.login()
        ftp.cwd('pub')
        ftp.retrlines('LIST')
        print 'Finished...'
    except ftplib.all_errors as e:
        print "Fail to connect...\n"
        print e

ftpList('rtfm.mit.edu')

Does this work for you?

Then your threading code has plenty of problems, but you aren't there yet :)
This forum has been moved to http://python-forum.io/. See you there.
User avatar
Ofnuts
 
Posts: 2659
Joined: Thu May 14, 2015 9:46 am
Location: Paris, France, EU, Earth, Solar system, Milky Way, Local Cluster, Universe #32987440940987

Re: Problems with ftplib opening and connection

Postby pow3rline » Wed Jul 13, 2016 3:13 pm

Hi, thanks for the attention.

Yeah, that code works like a charm for me.

Actually, I forgot to mention that I successfully tried to run the ftp function in an independent .py file. The code I tried was:

Code: Select all
from ftplib import FTP

for line in open('lista_ftps.txt','r'):

        address_ftp=line.rstrip('\n')
        print address_ftp
        ftp=FTP(address_ftp)
        ftp.login()
        ftp.cwd('pub')
        ftp.retrlines('LIST')
        ftp.quit()


Which works fine. So I guess the thread part is the key point of this mess, as you said :lol:
pow3rline
 
Posts: 4
Joined: Mon Jul 11, 2016 6:04 pm

Re: Problems with ftplib opening and connection

Postby Ofnuts » Wed Jul 13, 2016 4:52 pm

One problem as see with you code is that the main code queues things and exits without waiting for the threads. This ends the application before the threads have had a chance to connect... You have to join() all threads, something like:
Code: Select all
workers=[]
for i in range(num_threads):
        worker=List_pub(queue)
        workers.append(worker)
        # start the worker here

# queue the ftp sites here

# Now loop waiting for the workers to end
print 'Waiting workers end'
for worker in workers:
    worker.join()
This forum has been moved to http://python-forum.io/. See you there.
User avatar
Ofnuts
 
Posts: 2659
Joined: Thu May 14, 2015 9:46 am
Location: Paris, France, EU, Earth, Solar system, Milky Way, Local Cluster, Universe #32987440940987

Re: Problems with ftplib opening and connection

Postby pow3rline » Thu Jul 14, 2016 9:47 pm

Yeah, the .join() instruction pretty much solve the situation, thanks for the tip, so easy but so unknown for me at the same time. Now I can continue improving my script. I am going to investigate about thread locks in order to print the ftp name and the directory list in the correct order. Probably I will post future doubts :P
pow3rline
 
Posts: 4
Joined: Mon Jul 11, 2016 6:04 pm

Re: Problems with ftplib opening and connection

Postby Ofnuts » Fri Jul 15, 2016 1:03 pm

pow3rline wrote:I am going to investigate about thread locks in order to print the ftp name and the directory list in the correct order. Probably I will post future doubts :P


No point doing this... just remove the threads if you want things to happen in sequence. Or have each thread write its output in a list, and print that list right after you have join()ed that thread.
This forum has been moved to http://python-forum.io/. See you there.
User avatar
Ofnuts
 
Posts: 2659
Joined: Thu May 14, 2015 9:46 am
Location: Paris, France, EU, Earth, Solar system, Milky Way, Local Cluster, Universe #32987440940987

Re: Problems with ftplib opening and connection

Postby pow3rline » Fri Jul 15, 2016 3:04 pm

Ofnuts wrote:
pow3rline wrote:I am going to investigate about thread locks in order to print the ftp name and the directory list in the correct order. Probably I will post future doubts :P


No point doing this... just remove the threads if you want things to happen in sequence. Or have each thread write its output in a list, and print that list right after you have join()ed that thread.


I'm learning the language and this is just a random exercise in order to practice new skills :) I was considering the both options but, to be honest, I hadn't realize that saving the results in a list is the most obvious. Otherwise, as you have pointed out, the threading/concurrent part is pointless. Thanks for the guidance ;)
pow3rline
 
Posts: 4
Joined: Mon Jul 11, 2016 6:04 pm


Return to Networking

Who is online

Users browsing this forum: No registered users and 1 guest

cron