How to speed this up?

This is the place for queries that don't fit in any of the other categories.

How to speed this up?

Postby PyMD5 » Thu Jun 30, 2016 5:36 am

Hi, all. I'm utterly new to python, so I'm not sure if I'm even doing things right... I'm trying to compare an MD5 hash of an IP address to a known MD5 hash to see if it matches.

Here's the code I originally came up with:
Code: Select all
import hashlib
def ipv4_md5_search(hash, range_start=0, range_end=256):
  for a in range(range_start, range_end):
    print ('{}.{}.{}.{}'.format(a,0,0,0))
    for b in range(256):
      for c in range(256):
        for d in range(256):
          h = hashlib.md5(('%s.%s.%s.%s' % (a,b,c,d)).encode('utf-8')).hexdigest()
          if h == hash:
            print ('{}.{}.{}.{} {}'.format(a,b,c,d,hash))
            return True
  print ("No match found")
ipv4_md5_search('ccdd8d3d940a01b2fb3258c059924c0d')


This takes 35 seconds to run for each first-octet (from 0.0.0.0 to 1.0.0.0, from 1.0.0.0 to 2.0.0.0, etc.).

I tried again:
Code: Select all
import hashlib,sys
from datetime import datetime
def ipv4_md5_search(hash,range_start=0,range_end=256):
  print('{} {}'.format(hash,str(datetime.now())))
  [sys.exit(print('{}.{}.{}.{} {} {}'.format(i1,i2,i3,i4,hash,str(datetime.now())))) for i1 in range(range_start, range_end) for i2 in range(256) for i3 in range(256) for i4 in range(256) if hashlib.md5(('%s.%s.%s.%s' % (i1,i2,i3,i4)).encode('utf-8')).hexdigest() == hash]
  return True
  print("No match found")
ipv4_md5_search('ccdd8d3d940a01b2fb3258c059924c0d')


This also takes 35 seconds per first-octet.

So, I tried to parallelize it:
Code: Select all
import hashlib,sys,multiprocessing
from datetime import datetime
pool = multiprocessing.Pool(processes=8)
def ipv4_md5_search(hash, range_start=0, range_end=256):
  for a in range(range_start, range_end):
    print ('{}.{}.{}.{} {}'.format(a,0,0,0,str(datetime.now())))
    for b in range(256):
      for c in range(256):
        for d in range(256):
          h = hashlib.md5(('%s.%s.%s.%s' % (a,b,c,d)).encode('utf-8')).hexdigest()
          if h == hash:
            print ('{}.{}.{}.{} {} {}'.format(a,b,c,d,hash,str(datetime.now())))
            return True
  print ("No match found "+ str(datetime.now()))
r = pool.map(ipv4_md5_search('ccdd8d3d940a01b2fb3258c059924c0d',0,256))
pool.join()
pool.close()


This takes 39 seconds for each first-octet... it's slower than the single thread code! I'm running an 8-core Intel i7 CPU.

I also tried generating threads and putting each thread in its own process using import multiprocessing, splitting up the IP address pool into 8 separate chunks... it ran slightly faster, but wouldn't return a result even for a known MD5:IP address match, so I won't even bother posting that code.

So how do I speed this up?

Thanks for any help you can provide.
Last edited by micseydel on Thu Jun 30, 2016 5:48 am, edited 1 time in total.
Reason: Initial post lock.
PyMD5
 
Posts: 35
Joined: Thu Jun 30, 2016 5:19 am

Re: How to speed this up?

Postby Ofnuts » Thu Jun 30, 2016 1:00 pm

If you loop on the last 3 bytes, you still have 2^24 things, or roughly 16 million items... That's still a lot. Typically what is done these days it to generate all the hashes and keep them in a file or set of files that are intelligently looked up using the has as an entry (see "Rainbow table" on Wikipedia).
This forum has been moved to http://python-forum.io/. See you there.
User avatar
Ofnuts
 
Posts: 2659
Joined: Thu May 14, 2015 9:46 am
Location: Paris, France, EU, Earth, Solar system, Milky Way, Local Cluster, Universe #32987440940987

Re: How to speed this up?

Postby PyMD5 » Sat Jul 02, 2016 10:11 am

Ok, I tried again (for the fifth time), and I think I've got it. It's now doing 1 first-octet every 9.166 seconds, so almost 4 times faster... the multiprocessing overhead and more complex code slows it down so it's not 8 times faster, but I'll take anything I can get.

Code: Select all
import multiprocessing, hashlib, sys
from datetime import datetime

def worker(hash, range_start, range_end, e, key):
  for a in range(range_start, range_end):
    for b in range(256):
      for c in range(256):
        for d in range(256):
          h = hashlib.md5(('%s.%s.%s.%s' % (a,b,c,d)).encode('utf-8')).hexdigest()
          if h == hash:
            e[key] = ('{} {}.{}.{}.{} {}, {}'.format("Match Found:",str(a),str(b),str(c),str(d),str(hash),str(datetime.now())))
            return True
  e[key] = ("No match found. " + str(datetime.now()))
  exitcode = self._bootstrap()
  sys.exit(exitcode)

if __name__ == '__main__':
  hashinput = '01e8231d1c6f91ff34ce69dfbb41ae84'
  mgr = multiprocessing.Manager()
  e = mgr.dict()
  print("Starting: " + str(datetime.now()))
  jobs=[
    multiprocessing.Process(target=worker, args=(hashinput, 0, 33, e, 1)),
    multiprocessing.Process(target=worker, args=(hashinput, 33, 65, e, 2)),
    multiprocessing.Process(target=worker, args=(hashinput, 65, 97, e, 3)),
    multiprocessing.Process(target=worker, args=(hashinput, 97, 129, e, 4)),
    multiprocessing.Process(target=worker, args=(hashinput, 129, 161, e, 5)),
    multiprocessing.Process(target=worker, args=(hashinput, 161, 193, e, 6)),
    multiprocessing.Process(target=worker, args=(hashinput, 193, 225, e, 7)),
    multiprocessing.Process(target=worker, args=(hashinput, 225, 256, e, 8)),
  ]

  for j in jobs:
    j.start()
  for j in jobs:
    j.join()
  print("Results:")
  for elem in range(1,9):
    print (e[elem])
PyMD5
 
Posts: 35
Joined: Thu Jun 30, 2016 5:19 am

Re: How to speed this up?

Postby PyMD5 » Sun Jul 03, 2016 5:23 am

I tweaked the code a bit so that all the threads exit once a match between an MD5 hash and an IP address is found in any of the threads. This saves a lot of time, as once a match is found, the rest of the threads don't have to continue processing.

There is one issue... the parent process which spawns the worker threads doesn't die once all the worker threads have terminated... this appears to be a known issue with Python. Restarting the MD5 search will divorce the zombie parent process from the main Python process and make it a standalone process that can be killed in Task Manager.

Code: Select all
import multiprocessing, hashlib, sys
from datetime import datetime

def worker(hash, range_start, range_end, e, key):
  e[key] = ('{} {} {} {}'.format("Thread",str(key),"no match found.",str(datetime.now())))
  for a in range(range_start, range_end):
    match = str(e)
    if (match.find("Match Found:") >= 0):
      e[key] = ('{} {} {} {}'.format("Thread",str(key),"exited early.",str(datetime.now())))
      sys.exit(0)
    for b in range(256):
      e[key] = ('{} {} {} {}'.format("Thread",str(key),"no match found.",str(datetime.now())))
      for c in range(256):
        for d in range(256):
          if hash == hashlib.md5(('{}.{}.{}.{}'.format(a,b,c,d)).encode('utf-8')).hexdigest():
            e[key] = ('{} {}.{}.{}.{} {}, {}'.format("Match Found:",str(a),str(b),str(c),str(d),str(hash),str(datetime.now())))
            sys.exit(0)

if __name__ == '__main__':

  hashinput = '9f787273ca8769f2aec75fd993a026d5'
  mgr = multiprocessing.Manager()
  e = mgr.dict()
  print("Starting: " + str(datetime.now()))
  jobs=[
    multiprocessing.Process(target=worker, args=(hashinput, 0, 33, e, 1)),
    multiprocessing.Process(target=worker, args=(hashinput, 33, 65, e, 2)),
    multiprocessing.Process(target=worker, args=(hashinput, 65, 97, e, 3)),
    multiprocessing.Process(target=worker, args=(hashinput, 97, 129, e, 4)),
    multiprocessing.Process(target=worker, args=(hashinput, 129, 161, e, 5)),
    multiprocessing.Process(target=worker, args=(hashinput, 161, 193, e, 6)),
    multiprocessing.Process(target=worker, args=(hashinput, 193, 225, e, 7)),
    multiprocessing.Process(target=worker, args=(hashinput, 225, 256, e, 8)),
  ]

  for j in jobs:
    j.daemon = False
    j.start()
  for j in jobs:
    j.join()
  print("Results:")
  for elem in range(1,9):
    print (e[elem])
Last edited by PyMD5 on Mon Jul 04, 2016 1:02 am, edited 2 times in total.
PyMD5
 
Posts: 35
Joined: Thu Jun 30, 2016 5:19 am

Re: How to speed this up?

Postby Ofnuts » Sun Jul 03, 2016 9:01 am

Why are you using processes instead of using threads?
This forum has been moved to http://python-forum.io/. See you there.
User avatar
Ofnuts
 
Posts: 2659
Joined: Thu May 14, 2015 9:46 am
Location: Paris, France, EU, Earth, Solar system, Milky Way, Local Cluster, Universe #32987440940987

Re: How to speed this up?

Postby micseydel » Sun Jul 03, 2016 9:15 am

Ofnuts wrote:Why are you using processes instead of using threads?

This problem is CPU-bound, so given that the GIL makes threading unable to truly parallelize, using multiprocessing here makes sense to me. Although the rainbow table is definitely best.
Due to the reasons discussed here we will be moving to python-forum.io on October 1, 2016.

This forum will be locked down and no one will be able to post/edit/create threads, etc. here from thereafter. Please create an account at the new site to continue discussion.
User avatar
micseydel
 
Posts: 3000
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Re: How to speed this up?

Postby Ofnuts » Sun Jul 03, 2016 10:15 am

Aw, yes, the GIL. Btw why are your calling terminate()? When you get out of join() the process is already terminated, since you don't use a time-out.
This forum has been moved to http://python-forum.io/. See you there.
User avatar
Ofnuts
 
Posts: 2659
Joined: Thu May 14, 2015 9:46 am
Location: Paris, France, EU, Earth, Solar system, Milky Way, Local Cluster, Universe #32987440940987

Re: How to speed this up?

Postby PyMD5 » Sun Jul 03, 2016 11:14 pm

micseydel wrote:
Ofnuts wrote: Why are you using processes instead of using threads?

This problem is CPU-bound, so given that the GIL makes threading unable to truly parallelize, using multiprocessing here makes sense to me. Although the rainbow table is definitely best.


Exactly right. I tried to use threads, but it ran them consecutively, and it didn't spike the CPU like using Processes does (meaning it's searching as fast as it can).

I've got rainbow tables of MD5:IP address matches, as well (split into multiple files, one for each first-octet), but it takes about 45 minutes to search through all of them using Agent Ransack. With this, I'm splitting the search up into 8 different starting points equally spaced throughout the IPv4 space, rather than searching sequentially starting at 0.0.0.0... if it gets a match early on, it saves a lot of time.
Last edited by PyMD5 on Sun Jul 03, 2016 11:30 pm, edited 2 times in total.
PyMD5
 
Posts: 35
Joined: Thu Jun 30, 2016 5:19 am

Re: How to speed this up?

Postby PyMD5 » Sun Jul 03, 2016 11:17 pm

Ofnuts wrote: Aw, yes, the GIL. Btw why are your calling terminate()? When you get out of join() the process is already terminated, since you don't use a time-out.


That's an attempt at mitigating the zombie parent process (the one that spawns the worker processes) after the child processes terminate. I was thinking maybe the child processes weren't automatically join()'ing, so the parent process was waiting for them to do so, so it could terminate. It obviously didn't work.

I also tried .join()'ing the child processes from inside each child process just before calling sys.exit(0)... didn't work, either.
PyMD5
 
Posts: 35
Joined: Thu Jun 30, 2016 5:19 am

Re: How to speed this up?

Postby PyMD5 » Mon Jul 04, 2016 9:06 am

Ok, one more version... this one doesn't terminate the other processes early when a match is found in any one process, because it's searching on multiple MD5 hashes. This minimizes the amount of total time spent doing bulk searches.

Code: Select all
import multiprocessing, hashlib, sys
from datetime import datetime

def worker(hashin, range_start, range_end, e, key):
  e[key] = ""
  for a in range(range_start, range_end):
    for b in range(256):
      for c in range(256):
        for d in range(256):
          h = hashlib.md5(('{}.{}.{}.{}'.format(a,b,c,d)).encode('utf-8')).hexdigest()
          for entry in range(0,len(hashin)):
            if h == hashin[entry]:
              e[key] = e[key] + ('\n{} {}.{}.{}.{} {}, {}'.format("Match Found:",str(a),str(b),str(c),str(d),str(hashin[entry]),str(datetime.now())))

if __name__ == '__main__':
  hashinput = [
    '01e8231d1c6f91ff34ce69dfbb41ae84', # 1.0.0.0
    '1ea93a7558872d5f0b0220a57a6de7c4', # 34.0.0.0
    '5442f932d194e43c5bed950d108cf297', # 66.0.0.0
    'f519644a378c625ba6cd34602a8831e4', # 130.0.0.0
    '15f4faad810f7928e5a6a94ecefc7065', # 162.0.0.0
    '58b6425085816fd95f141f9b96c7b305', # 194.0.0.0
  ]

  mgr = multiprocessing.Manager()
  e = mgr.dict()
  print("Starting: " + str(datetime.now()))
  print('{} {} {}'.format("Checking for",str(len(hashinput)),"hashes:"))
  for entry in range(0, len(hashinput)):
    print(hashinput[entry])
  jobs=[
    multiprocessing.Process(target=worker, args=(hashinput, 0, 33, e, 1)),
    multiprocessing.Process(target=worker, args=(hashinput, 33, 65, e, 2)),
    multiprocessing.Process(target=worker, args=(hashinput, 65, 97, e, 3)),
    multiprocessing.Process(target=worker, args=(hashinput, 97, 129, e, 4)),
    multiprocessing.Process(target=worker, args=(hashinput, 129, 161, e, 5)),
    multiprocessing.Process(target=worker, args=(hashinput, 161, 193, e, 6)),
    multiprocessing.Process(target=worker, args=(hashinput, 193, 225, e, 7)),
    multiprocessing.Process(target=worker, args=(hashinput, 225, 256, e, 8)),
  ]

  for j in jobs:
    j.daemon = False
    j.start()
  for j in jobs:
    j.join()
  print("Results:")
  for elem in range(1,len(jobs)+1):
    if e[elem]:
      print (e[elem])
    else:
      print ('{} {} {} {}'.format("Thread",str(elem),"no match found.",str(datetime.now())))


My next project: getting Python to run on the video card to speed things up even more.
PyMD5
 
Posts: 35
Joined: Thu Jun 30, 2016 5:19 am

Re: How to speed this up?

Postby PyMD5 » Thu Aug 18, 2016 1:10 am

And yet another version. This one searches multiple hashes on multiple CPU cores (via multiprocessing), immediately returns a result when a match is found, uses the map function instead of a for loop, removes any hashes that a match is found for (so we no longer have to consume CPU cycles checking those hashes), employs several tricks to speed up the processing, searches through IPv6 IP addresses and displays '% Completion' on a single line that is rewritten with each update of the percentage.

This is meant to run in a CMD prompt, rather than in the IDLE shell... so you'd type: python.exe MD5crack_IPv6.py

Windows IDLE shell is quirky, functions called via multiprocessing aren't allowed to print to screen (which necessitated my use of [key] in the last version, and necessitated printing out the results after all the searching was done).

The original version of this code took ~48 hours to run through 1 billion IP addresses. Employing multiprocessing lowered that to ~26.5 hours. It's now doing just under a billion per hour. I had to run it with only 6 processes, since running it on all 8 cores of my CPU swamps the CPU, slows down the overall progress, and makes the computer nearly unusable.

If anyone sees any means of squeezing even more performance out of this code, let me know. I'm looking into OpenCL and CUDA as a means of speed increase, but none of the packages on pip support the latest version of Python.

Code: Select all
# =====================================================
import multiprocessing,hashlib,ipaddress
from datetime import datetime

def worker(hashin, starting_ip, range_start,range_end):
  ip6 = ipaddress.IPv6Address(starting_ip)+range_start
  hashlibmd5 = hashlib.md5

  def MD5hash(count):
    return hashlibmd5(bytes(str(ip6+count),'ascii')).hexdigest()

  for count,r in enumerate(map(MD5hash,range(range_start,range_end))):
    if (count%555555==0):
      print('Progress: {:.2%}\t{}'.format(count/(range_end-range_start),ip6+count), end='\r')
    for entry in hashin:
      if r == entry:
        print('{}\t{}'.format(r,ip6+count))
        hashin.remove(r)
        print('{} {} {}'.format("Checking for",str(len(hashin)),"hashes"), end='\r')

if __name__ == '__main__':
  starting_ip = '::1'

  hashinput = [
    '00000000000000000000000000000000', # TEST HASH
    '11111111111111111111111111111111', # TEST HASH
    '22222222222222222222222222222222', # TEST HASH
  ]

  mgr = multiprocessing.Manager()
  jobs=[
    multiprocessing.Process(target=worker, args=(hashinput, starting_ip, 0, 5000000000)),
    multiprocessing.Process(target=worker, args=(hashinput, starting_ip, 5000000000, 10000000000)),
    multiprocessing.Process(target=worker, args=(hashinput, starting_ip, 10000000000, 15000000000)),
    multiprocessing.Process(target=worker, args=(hashinput, starting_ip, 15000000000, 20000000000)),
    multiprocessing.Process(target=worker, args=(hashinput, starting_ip, 20000000000, 25000000000)),
    multiprocessing.Process(target=worker, args=(hashinput, starting_ip, 25000000000, 30000000000)),
  ]

  print('{} {} {} {}\n\n{} {} {}'.format("Starting",len(jobs),"processes",str(datetime.now()),"Checking for",str(len(hashinput)),"hashes:"))
  for entry in hashinput:
    print(entry)
  print('\nResults:')

  for j in jobs:
    j.daemon = False
    j.start()
  for j in jobs:
    j.join()

  print('\n{} {}'.format("SEARCH COMPLETE",str(datetime.now())))
# =====================================================
PyMD5
 
Posts: 35
Joined: Thu Jun 30, 2016 5:19 am

Re: How to speed this up?

Postby PyMD5 » Thu Aug 18, 2016 2:53 am

Oooh, just found something interesting... I was re-working the hashlib.md5 line in my IPv4 multi-processing cracker to inline some of the code (do a search for "Python avoid the dots" for a good explanation) to speed it up, and I messed it up such that it wasn't using hashlib.md5, but it was looping through the entire IPv4 IP address space... it took 3.23 seconds across all 8 cores used. So almost all of the CPU time is taken in computing the MD5 hashes... so now I'm searching for code that will calculate MD5 hashes faster.
PyMD5
 
Posts: 35
Joined: Thu Jun 30, 2016 5:19 am

Re: How to speed this up?

Postby wavic » Thu Aug 18, 2016 10:57 am

Look at this. I ran on it at the other day but my other laptop is not in my hands right now. Can't try it
wavic
 
Posts: 165
Joined: Wed May 25, 2016 8:51 pm

Re: How to speed this up?

Postby wavic » Thu Aug 18, 2016 11:16 am

I have a question. Can I use multiprocessing on only one function? It uses for loop but can't be cut into pieces. Heavy computations is involved. I want to speed it up but I am new in Python and I don't know how to do it. I know about threading, multiprocessing, and some others but didn't try it yet. First I want to understand how it works. I've tested it over a small pieces of data, but eventually ( if I finish it somehow ) that will be like a drop in a lake.
wavic
 
Posts: 165
Joined: Wed May 25, 2016 8:51 pm

Re: How to speed this up?

Postby PyMD5 » Sun Aug 21, 2016 6:31 am

wavic wrote:I have a question. Can I use multiprocessing on only one function? It uses for loop but can't be cut into pieces. Heavy computations is involved. I want to speed it up but I am new in Python and I don't know how to do it. I know about threading, multiprocessing, and some others but didn't try it yet. First I want to understand how it works. I've tested it over a small pieces of data, but eventually ( if I finish it somehow ) that will be like a drop in a lake.


Yes, you can. The "def worker(hashinput,range_start,range_end):" is split up between each core of the CPU to distribute the load and speed things up. All you have to do is figure out what part of the processing you want to split up and how.
PyMD5
 
Posts: 35
Joined: Thu Jun 30, 2016 5:19 am

Re: How to speed this up?

Postby PyMD5 » Sun Aug 21, 2016 6:51 am

Another interesting factoid I found... .format() is slower than the old %-style formatting.

I stripped out all other things from my IPv4 code except for the loops and the command that generates the MD5 hashes, inserted timing code, then tested.

The way I was doing it (using the new .format() command):
Code: Select all
h = hashlib.md5(bytes('{!s}.{!s}.{!s}.{!s}'.format(a,b,c,d),'utf-8')).hexdigest()

-or-
Code: Select all
h = hashlibmd5(bytes(str('{}.{}.{}.{}'.format(a,b,c,d)),'utf-8')).hexdigest()

(Both lines are functionally equivalent, and take the same amount of time.)

Single process, for the entire IPv4 address space, this took ~212 minutes.

The way I'm now doing it (using the old %-style formatting):
Code: Select all
h = hashlib.md5(bytes('%s.%s.%s.%s' % (a,b,c,d),'utf-8')).hexdigest()


Single process, for the entire IPv4 address space, this took ~168 minutes.

That's a time savings of ~44 minutes for the entire IPv4 address space.

It now searches through the entire IPv4 address space on 8 processes (8 cores of an 8-core CPU) with a single MD5 hash in 1 hour, 3 minutes. With 10 MD5 hashes, it takes 1 hour, 10 minutes.
Last edited by PyMD5 on Sun Aug 21, 2016 5:05 pm, edited 1 time in total.
PyMD5
 
Posts: 35
Joined: Thu Jun 30, 2016 5:19 am

Re: How to speed this up?

Postby PyMD5 » Sun Aug 21, 2016 7:00 am

Another thing I'm trying (although I haven't timed this yet)... I'm using a list comprehension instead of a for loop:

So I'm replacing this:
Code: Select all
for entry in hashinput:
  if h == entry:
    print('\n{} {}.{}.{}.{} {}, {}'.format("Match Found:",str(a),str(b),str(c),str(d),str(entry),str(datetime.now())))


with this:
Code: Select all
[print('{}\t{}.{}.{}.{}\t{}'.format("Match Found:",a,b,c,d,str(entry))) for entry in hashinput if entry == h]


I'll keep you apprised if it's faster.
PyMD5
 
Posts: 35
Joined: Thu Jun 30, 2016 5:19 am

Re: How to speed this up?

Postby PyMD5 » Mon Aug 22, 2016 1:42 am

I don't know why everyone says list comprehensions and maps are faster than for loops... I've run every conceivable variety of list comprehension and map and it's always been slower than the simple nested for loop.

I ran a test on the following:
Code: Select all
hashlibmd5 = hashlib.md5
[print('{}.{}.{}.{}\t{}'.format(i1,i2,i3,i4,str(entry))) for i1 in range(range_start, range_end) for i2 in range(0,256) for i3 in range(0,256) for i4 in range(0,256) for entry in hashinput if hashlibmd5(('%s.%s.%s.%s' % (i1,i2,i3,i4)).encode('utf-8')).hexdigest() == entry]


to replace this:
Code: Select all
hashlibmd5 = hashlib.md5
for a in range(range_start,range_end):
  print('Progress: {:.2%}'.format((a-range_start)/(range_end-range_start)), end='\r')
  for b in range(256):
    for c in range(256):
      for d in range(256):
        h = hashlibmd5(('%s.%s.%s.%s' % (a,b,c,d)).encode('utf-8')).hexdigest()
        for entry in hashinput:
          if h == entry:
            print('\n{} {}.{}.{}.{} {}, {}'.format("Match Found:",str(a),str(b),str(c),str(d),str(entry),str(datetime.now())))


... within the framework of my multi-processing setup as outlined above. Using for loops, it takes 1 hour 3 minutes to traverse the entire IPv4 address space (with one MD5 hash, with 8 processes on an 8-core CPU). Using the list comprehension above, it took far longer... I stopped it after it went longer than 2 hours. And there's no way of printing the progress as the original code does.
PyMD5
 
Posts: 35
Joined: Thu Jun 30, 2016 5:19 am

Re: How to speed this up?

Postby Ofnuts » Mon Aug 22, 2016 7:32 am

A list comprehension generates/allocates a list, given the size of your lists this becomes non trivial.
This forum has been moved to http://python-forum.io/. See you there.
User avatar
Ofnuts
 
Posts: 2659
Joined: Thu May 14, 2015 9:46 am
Location: Paris, France, EU, Earth, Solar system, Milky Way, Local Cluster, Universe #32987440940987

Re: How to speed this up?

Postby wavic » Mon Aug 22, 2016 9:51 pm

I don't know if this is going through faster but you can try. I was looking at numpy also but I've got headache
I look at itertools for the first time. Yummy :twisted:
Code: Select all
In [1]: import itertools

In [2]: for ip in itertools.product(range(3), repeat=2): print(ip) # it will be range(256), repeat=4
(0, 0)
(0, 1)
(0, 2)
(1, 0)
(1, 1)
(1, 2)
(2, 0)
(2, 1)
(2, 2)

In [3]:


In Numpy there is something called broadcasting but I didn't get it. It looks like it will do the job ( generating tuples of all possible octets )
May be here there is a better solution. I think broadcasting is not what's need
wavic
 
Posts: 165
Joined: Wed May 25, 2016 8:51 pm

Next

Return to General Coding Help

Who is online

Users browsing this forum: No registered users and 4 guests