PyMD5 wrote:Actually, I'm working on that. I found that with a list comprehension, it greatly sped up the process with a small range of IP addresses, but as the list grew, it'd consume all memory and slow to a crawl... so I'm combining my current code with a list comprehension that only runs on a million IP addresses at a time, wrapped in a for loop that increments by one million. If it works, that should just about double the speed of this thing.
Hm! All that indexing... Is there a way to avoid that? Doesn't this slow down the program?
Yeah, it does. In order to keep the list comprehension fast, it has to be really small, which means it's looping through the rest of the code more often, which slows it down... a Catch-22 I couldn't resolve.
I'm trying something else, creating a bunch of MD5 hashes all at once, chucking them into an array, then comparing the known hashes to that array. I'm testing that code now. It's doing about 2 million IP addresses per minute per process right now, but I've got more tweaking to do.
Today I learned how to make multiple commands in a single list comprehension:
[[print('%s\t%s' % (entry1,ipcount+int(entry2[:-32]))),hashinput.remove(entry1)] for entry1 in hashlocal for entry2 in h if entry1 == entry2[-32:]]
That only runs once every two million IP addresses, so it doesn't slow things down much.