Subprocess Popen speed issues without output

This is the place for queries that don't fit in any of the other categories.

Subprocess Popen speed issues without output

Postby KHarvey » Tue Mar 19, 2013 5:28 pm

Looks like I had to re-register since Taos took over. Hopefully I can start helping out and asking questions again on here as it has been a while.

So I am working on some network monitoring scripts and I found a weird issue with subprocess.Popen that I have not been able to figure out how to handle.
Code: Select all
lpNetSeg = '10.1.1.1/24'
objNMapIPScan = subprocess.Popen('nmap ' + lpNetSeg +' -sP', stdin=None, stdout=None, stderr=None, shell=True)
objNMapIPScan.communicate()

Now that I look at that piece of code I see that I need to switch to parameters for the lpNetSeg.
Anyways, when I run that piece of code it takes approximately 1.7 minutes to run.
But I don't need any of the output from it, in fact I would like to suppress the output from it. So I changed the code to this:
Code: Select all
lpNetSeg = '10.1.1.1/24'
objNMapIPScan = subprocess.Popen('nmap ' + lpNetSeg +' -sP', stdin=None, stdout=subprocess.PIPE, stderr=None, shell=True)
objNMapIPScan.communicate()

When I run this code it takes approximately 3.3 minutes to run.

Is there a way for me to suppress the output of the command without drastically slowing down my code?

I am using nmap as a ping sweep to populate the ARP / MAC-Address tables of my networking gear. So I need to run the ping sweep and let it finish before my script continues, hence the objNMapIPScan.communicate(), and that is also why I don't care about the output of the command as it just needs to run. The rest of my script just goes through and pulls the MAC addresses by port on my network gear but it has to be prepopulated by my ping sweep else my switches timeout the data and clears the tables.
KHarvey
 
Posts: 34
Joined: Tue Mar 19, 2013 5:13 pm
Location: US

Re: Subprocess Popen speed issues without output

Postby setrofim » Tue Mar 19, 2013 5:48 pm

Erm, this may be a silly question, but have you tired redirecting the output to null in the command itself, i.e.
Code: Select all
command = ' '.join(['nmap', lpNetSeg, '-sP 1>/dev/null 2>&1'])
objNMapIPScan = subprocess.Popen(command, stdin=None, stdout=None, stderr=None, shell=True)

(replace '/dev/null' with 'NUL' if on Windows)?
setrofim
 
Posts: 288
Joined: Mon Mar 04, 2013 7:52 pm

Re: Subprocess Popen speed issues without output

Postby KHarvey » Tue Mar 19, 2013 6:03 pm

setrofim wrote:Erm, this may be a silly question, but have you tired redirecting the output to null in the command itself, i.e.


Nope not silly at all, I just never thought of trying that. It's an excellent suggestion. Also thank you for the prompt response.

I just ran through my same tests using 1>/dev/null and 2>&1, but the results are the same. With output to the screen it took 1.6 minutes and without output it took 3.3 minutes.

This is just weird, and I can't figure out why.

By the way, just in case my timings are taken into question here is my code I am using to determine script run time:
Code: Select all
#Define time constants
dteStartTime = time.localtime()

if(time.mktime(time.localtime()) - time.mktime(dteStartTime)) > 60:
intTimeFrame = (time.mktime(time.localtime()) - time.mktime(dteStarTime)) / 60
strTimeMeasurement = ' minutes'
else:
intTimeFrame = (time.mktime(time.localtime()) - time.mktime(dteStarTime))
strTimeMeasurement = ' seconds'
print str(round(intTimeFrame,1)) + strTimeMeasurement
KHarvey
 
Posts: 34
Joined: Tue Mar 19, 2013 5:13 pm
Location: US

Re: Subprocess Popen speed issues without output

Postby KHarvey » Tue Mar 19, 2013 7:26 pm

So I don't really like the way that I am doing this, but it appears to be working.
Code: Select all
lpNetSeg = '10.1.1.1/24'
objNMapIPScan = subprocess.Popen('nmap ' + lpNetSeg +' -sP', stdin=None, stdout=-1, stderr=None, shell=True)
strTest = objNMapIPScan.communicate()

This code runs for 2.4 minutes.

I also tried this:
Code: Select all
lpNetSeg = '10.1.1.1/24'
objNMapIPScan = subprocess.Popen('nmap ' + lpNetSeg +' -sP', stdin=None, stdout=-1, stderr=None, shell=True)
while 1:
strLine = objNMapIPScan.stdout.readline()
if not strLine: break

This code runs for 1.8 minutes.

I really do not understand what is happening and why. Also I thought that when using subprocess that I am supposed to the use the communicate() tuple, but it appears way faster and easier to use stdout.readline() like the old os.popen. Communicate() has that problem that if I am doing inline processing that I have to verify the type of the variable, else I get a ton of NoneType errors while using communicate(), so I may just go with the stdout.readline() unless someone has something better.
KHarvey
 
Posts: 34
Joined: Tue Mar 19, 2013 5:13 pm
Location: US

Re: Subprocess Popen speed issues without output

Postby metulburr » Tue Mar 19, 2013 7:26 pm

I need to run the ping sweep and let it finish before my script continues

Code: Select all
objNMapIPScan.wait()
New Users, Read This
OS Ubuntu 14.04, Arch Linux, Gentoo, Windows 7/8
https://github.com/metulburr
steam
User avatar
metulburr
 
Posts: 1470
Joined: Thu Feb 07, 2013 4:47 pm
Location: Elmira, NY

Re: Subprocess Popen speed issues without output

Postby KHarvey » Tue Mar 19, 2013 7:43 pm

metulburr wrote:
I need to run the ping sweep and let it finish before my script continues

Code: Select all
objNMapIPScan.wait()

That was how I used to do it, until I read the documentation for subprocess:
Popen.wait()
Wait for child process to terminate. Set and return returncode attribute.

Warning This will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.
KHarvey
 
Posts: 34
Joined: Tue Mar 19, 2013 5:13 pm
Location: US

Re: Subprocess Popen speed issues without output

Postby KHarvey » Wed Mar 20, 2013 2:58 pm

I have changed my subprocess code slightly
Code: Select all
objNMapIPScan = subprocess.Popen(["nmap", lpNetSeg, "-sP"], stdin=None, stdout=-1, stderr=None, shell=False)
while 1:
  strLine = objNMapIPScan.stdout.readline()
  if not strLine: break


This has not helped the speed at all, but I believe it is more programmatically correct.
Using communicate() runs the script at 3.3 minutes and reading each line runs the script at 2.4 mins.

I still don't know why there is such a large time discrepancy, but I will move forward with the readline code.

I would still like to know why it using communicate() is so slow, or maybe my code is just incorrect.
KHarvey
 
Posts: 34
Joined: Tue Mar 19, 2013 5:13 pm
Location: US

Re: Subprocess Popen speed issues without output

Postby setrofim » Wed Mar 20, 2013 3:17 pm

If I were to guess, the issue may be with the fact that communicate() buffers all of the output from the pipes in memory. It's possible that continually having to allocate a larger buffer to accommodate the growing output...

Another thing that may be worth trying, is to use wait() as was suggested by metulburr, but set the stdout/stderr to None and redirect the output to /dev/null in the command. Since you're not creating pipes to the process, deadlock won't be an issue.

Code: Select all
command = ' '.join(["nmap", lpNetSeg, "-sP >/dev/null 2>&1"])
objNMapIPScan = subprocess.Popen(command, stdin=None, stdout=-1, stderr=None, shell=True)
objNMapIPScan.wait()

Note that shell needs to be True since it's used for the redirection.
setrofim
 
Posts: 288
Joined: Mon Mar 04, 2013 7:52 pm

Re: Subprocess Popen speed issues without output

Postby KHarvey » Tue Mar 26, 2013 5:19 pm

setrofim wrote:If I were to guess, the issue may be with the fact that communicate() buffers all of the output from the pipes in memory. It's possible that continually having to allocate a larger buffer to accommodate the growing output...

Another thing that may be worth trying, is to use wait() as was suggested by metulburr, but set the stdout/stderr to None and redirect the output to /dev/null in the command. Since you're not creating pipes to the process, deadlock won't be an issue.

Code: Select all
command = ' '.join(["nmap", lpNetSeg, "-sP >/dev/null 2>&1"])
objNMapIPScan = subprocess.Popen(command, stdin=None, stdout=-1, stderr=None, shell=True)
objNMapIPScan.wait()

Note that shell needs to be True since it's used for the redirection.

Thanks setrofim.
I apologize about my late response I have been swamped.

I didn't think about the buffers. But you are correct, by dumping the command to /dev/null and using wait() it has decreased the running time to 2.3 minutes.

Thanks for the help. I will go ahead and update my code to use this.
KHarvey
 
Posts: 34
Joined: Tue Mar 19, 2013 5:13 pm
Location: US


Return to General Coding Help

Who is online

Users browsing this forum: Google [Bot] and 5 guests