Help with Threading

This is the place for queries that don't fit in any of the other categories.

Help with Threading

Postby JdeBruin » Sat May 18, 2013 9:02 am

This is my first script where I want to use the python threading module. I have a large dataset which is a list of dict this can be as much as 200 dictionaries in the list. The final goal is a histogram for each dict 16 histograms on a page ( 4x4 ) - this already works.
What I currently do is a create a nested list [ [ {} ], [ {} ] ] each inner list contains 16 dictionaries, thus each inner list is a single page of 16 histograms. Iterating over the outer-list and creating the graphs takes to long. So I would like multiple inner-list to be processes simultaneously and creating the graphs in "parallel".
I am trying to use the python threading for this. I create 4 threads loop over the outer-list and send a inner-list to the thread. This seems to work if my nested lists only contains 2 elements - thus less elements than threads. Currently the scripts runs and then seems to get hung up. I monitor the resource on my mac and python starts off good using 80% and when the 4-thread is created the CPU usages drops to 0%.

My thread creating is based on the following : http://www.tutorialspoint.com/python/py ... eading.htm

Any help would be create!!!
JdeBruin
 
Posts: 2
Joined: Sat May 18, 2013 9:00 am

Re: Help with Threading

Postby setrofim » Sat May 18, 2013 1:30 pm

  1. You should not be accessing the same list from multiple threads. Each thread should have its own queue and your Main thread should put items into the queues for worker threads.
  2. You should not be using threading at all. Python threads do not run in parallel because of something called the Global Interpreter Lock.
  3. Multiprocessing module will allow you to get true parallelism. So what you really whan to be using is multiprocessing.Pool.map
setrofim
 
Posts: 288
Joined: Mon Mar 04, 2013 7:52 pm

Re: Help with Threading

Postby JdeBruin » Mon Jun 03, 2013 1:52 am

Thanks for the reply and comments.
JdeBruin
 
Posts: 2
Joined: Sat May 18, 2013 9:00 am


Return to General Coding Help

Who is online

Users browsing this forum: W3C [Linkcheck] and 2 guests