How to run python scritps one after another and in parallel

This is the place for queries that don't fit in any of the other categories.

How to run python scritps one after another and in parallel

Postby ericrystal » Tue Jun 04, 2013 12:09 pm

Hello,
Could some one help to the question as the subject (better the python codes): some scripts should be executed one by one and some should be executed simultaneously.
By example, I have 5 scripts: 1.py, 2.py, 3.py, 4.py, 5.py (some of them executed with sys.argv[]).
2.py depends on 1.py; 5.py depends on the results of 3.py and 4.py; 3.py and 4.py are independent so they could be executed in the same time and when both of them are finished (the execute time of 3.py and 4.py is different), the 5.py is executed automatically. I'd like all is automatical. That is to say:
Firstly:
python 1.py 'a.txt' 9
Secondly:
python 2.py
Thirdly:
python 3.py 'b.txt' 5
python 4.py 'c.txt' 3
Finally:
python 5.py
Thanks in advance!
Last edited by ericrystal on Tue Jun 04, 2013 12:30 pm, edited 4 times in total.
ericrystal
 
Posts: 18
Joined: Thu Apr 11, 2013 8:56 am

Re: How to run python scritps one after another and in paral

Postby setrofim » Tue Jun 04, 2013 12:11 pm

Rather than using separate scripts, you can parallelize using the multiprocessing module.
setrofim
 
Posts: 288
Joined: Mon Mar 04, 2013 7:52 pm

Re: How to run python scritps one after another and in paral

Postby ericrystal » Tue Jun 04, 2013 12:21 pm

setrofim wrote:Rather than using separate scripts, you can parallelize using the multiprocessing module.

Thanks for your answer, but the documentation is so complicated for me, could you give me some codes based on my example?
ericrystal
 
Posts: 18
Joined: Thu Apr 11, 2013 8:56 am

Re: How to run python scritps one after another and in paral

Postby setrofim » Tue Jun 04, 2013 1:05 pm

Instead of invoking multiple scripts, put the functionality into functions in one script. Then in that script
Code: Select all
from multiprocessing import Process

def func_1(filename, value):
     # Your 1.py stuff goes here
     pass

def func_2():
     pass

# etc...

if __name__ == '__main__':
   p1 = Process(target=func_3, args=('b.txt', 5))
   p2 = Process(target=func_4, args=('c.txt', 3))
   func_1('a.txt', 9)
   func_2()
   p1.start()
   p2.start()
   p1.join()
   p2.join()
   func_5()
setrofim
 
Posts: 288
Joined: Mon Mar 04, 2013 7:52 pm

Re: How to run python scritps one after another and in paral

Postby ericrystal » Tue Jun 04, 2013 3:20 pm

setrofim wrote:Instead of invoking multiple scripts, put the functionality into functions in one script. Then in that script
Code: Select all
from multiprocessing import Process

def func_1(filename, value):
     # Your 1.py stuff goes here
     pass

def func_2():
     pass

# etc...

if __name__ == '__main__':
   p1 = Process(target=func_3, args=('b.txt', 5))
   p2 = Process(target=func_4, args=('c.txt', 3))
   func_1('a.txt', 9)
   func_2()
   p1.start()
   p2.start()
   p1.join()
   p2.join()
   func_5()

Thank you very much!
ericrystal
 
Posts: 18
Joined: Thu Apr 11, 2013 8:56 am

Re: How to run python scritps one after another and in paral

Postby Somelauw » Wed Jun 05, 2013 6:09 pm

ericrystal wrote:Hello,
Could some one help to the question as the subject (better the python codes): some scripts should be executed one by one and some should be executed simultaneously.
By example, I have 5 scripts: 1.py, 2.py, 3.py, 4.py, 5.py (some of them executed with sys.argv[]).
2.py depends on 1.py; 5.py depends on the results of 3.py and 4.py; 3.py and 4.py are independent so they could be executed in the same time and when both of them are finished (the execute time of 3.py and 4.py is different), the 5.py is executed automatically. I'd like all is automatical. That is to say:
Firstly:
python 1.py 'a.txt' 9
Secondly:
python 2.py
Thirdly:
python 3.py 'b.txt' 5
python 4.py 'c.txt' 3
Finally:
python 5.py
Thanks in advance!


From within python or outside python?
If from outside python, what os / shell are you using?
Join the #python-forum IRC channel on irc.freenode.net!
Somelauw
 
Posts: 75
Joined: Tue Feb 12, 2013 8:30 pm

Re: How to run python scritps one after another and in paral

Postby ericrystal » Thu Jun 06, 2013 8:22 am

Somelauw wrote:
ericrystal wrote:Hello,
Could some one help to the question as the subject (better the python codes): some scripts should be executed one by one and some should be executed simultaneously.
By example, I have 5 scripts: 1.py, 2.py, 3.py, 4.py, 5.py (some of them executed with sys.argv[]).
2.py depends on 1.py; 5.py depends on the results of 3.py and 4.py; 3.py and 4.py are independent so they could be executed in the same time and when both of them are finished (the execute time of 3.py and 4.py is different), the 5.py is executed automatically. I'd like all is automatical. That is to say:
Firstly:
python 1.py 'a.txt' 9
Secondly:
python 2.py
Thirdly:
python 3.py 'b.txt' 5
python 4.py 'c.txt' 3
Finally:
python 5.py
Thanks in advance!


From within python or outside python?
If from outside python, what os / shell are you using?


Thanks for your answer!
Within python.
ericrystal
 
Posts: 18
Joined: Thu Apr 11, 2013 8:56 am


Return to General Coding Help

Who is online

Users browsing this forum: cmajor28 and 4 guests