dnspython - Increasing occurrence of timeout exception

dnspython - Increasing occurrence of timeout exception

Postby srnec » Sun Mar 17, 2013 11:44 pm

I managed to write this script which tests list of domains. His task is to get answer from every domain if it has any NS record. If it has, then this script tests every obtained NS records if it has IPv6 address (AAAA record).
Problem is that script more and more triggers Timeout exception although the domain has NS records. In the last stages when is almost on the end of the list of domains is nearly every domain evaluated as without NS records.
I think problem is with threads or in main_thread part of the script.
I used threading because I need to process 280k domains and I need to process it fast. So, I created for every domain thread (main_thread) and within the main thread Subthreads (NS_getAAAA_thread) for every obtained NS record.

This is just the part of the whole scirpt where Im testing simultaneously glue, AAAA, MX and NS record for every domain in list. Glue and AAAA record for www.[domain] are evaluated correctly, problem is just occuring while getting MX and NS records. Some of them are properly evaluated but while I am closer to the end of the domain list the worse answers from resolver are.

Main code:

Code: Select all
resolver = dns.resolver.Resolver()
resolver.lifetime= 2.5
resolver.timeout= 2

lock = threading.Lock()
threads = []

queue = Queue.Queue()
MAX_THREAD_COUNT = 100


try:
    cursor.execute("SELECT * FROM domains")
    results = cursor.fetchall()

    for i in range(MAX_THREAD_COUNT):
        t = main_thread(queue)
        t.daemon = True
        threads.append(t)
        t.start()

    for row in results:
        queue.put((row[0],row[1]))

    queue.join()

    for thread in threads:
        thread.keepRunning = False
    threads = []

except Exception as ex:
    sys.stderr.write("Global Error: " + str(ex))


Main thread code:

Code: Select all
class main_thread(threading.Thread):
    def __init__(self,queue):
        threading.Thread.__init__(self)
        self.queue = queue

    def run(self):
        self.keepRunning = True
        while self.keepRunning:
            arg = self.queue.get()
            id_domain = arg[0]
            domain = arg[1]
            SubThreads = []

            #---------------- NS Records ----------------
            try:
                answers = resolver.query(domain, 'NS')
                for ns in answers:
                    t = NS_getAAAA_thread(id_domain, ns)
                    t.setDaemon(True)
                    SubThreads.append(t)
                    t.start()

            except NXDOMAIN as enx:
                print "NS - Error: NXDOMAIN - domain does not exist"
            except Timeout as etime:
                print "NS - Error: dns.exception.Timeout"
            except NoAnswer as ea:
                print "NS - Error: No Answer"
            except NoNameservers as ens:
                print "NS - Error: No non-broken nameservers are available"


            for SubThread in SubThreads:
                SubThread.join()

            self.queue.task_done()


NS thread code:

Code: Select all
class NS_getAAAA_thread(threading.Thread):
    def __init__(self,id_domain,ns):
        threading.Thread.__init__(self)
        self.id_domain = id_domain
        self.ns = ns

    def run(self):
        res = dns.resolver.Resolver()
        res.lifetime = 2
        res.timeout = 1

        try:
            answers = res.query(str(self.ns),'AAAA')
            ip_ns = str(answers[0])
        except DNSException:
            ip_ns = "N/A"
        except Exception:
            sys.stderr.write("Global Error: " + str(ex))

        with lock:
            sql = "INSERT INTO ns (id_domain,ns,ip_ns) VALUES ('" + str(self.id_domain) + "','" + str(self.ns) + "','" + str(ip_ns) + "')"
            try:
                cursor.execute(sql)
                db.commit()
                print "NS" , '>>' , str(ip_ns), ' :: ', str(self.ns)
            except Exception as ex:
                sys.stderr.write("Global Error: " + str(ex))
                db.rollback()
srnec
 
Posts: 1
Joined: Sun Mar 17, 2013 11:32 pm

Return to Networking

Who is online

Users browsing this forum: No registered users and 1 guest

cron