New Features as of version 1.4.2 December 2012 Top-Level: * Added a thread-safe "Bag" data structure * Change so MidasTalker works with Jython * Using Py_ssize_t correctly (previously would have compile error on newer Pythons) Details: * MidasTalker can use non-blocking sockets to make sure an "open" can have a reasonable timeout, but the Jython socket module (by default) requires the non-blocking socket for select to work. There is a CPython compatible select (called cpython_compatible_select) in Jython that allows things to work. To see which select to use, we look for "Java" in the sys.version string. * Added several abstractions for a thread-safe Bag: the idea is that several threads are reaching into a bag, all at the same time, and we want to allow each thread to pull out one piece of work at a time. A simple idea, but making it thread-safe and thread-hot is the challenge. The first key aha! is that we only need to supply a bag of ints from [0..n) : By putting everything we want served out into an array, we simply grab the int we want from the bag and index into the array. With this key simplification, we can take advantage of atomic increment instructions to make Bags easy, thread-safe, quick, and (most important) avoid excessive collateral damage (collateral damage here would be preventing others from enetering the bag). iDrawer: The first abstraction is the primitive on which all other Bag abstractions are built-on: the iDrawer: it is a thread-safe bag that deals out integers from start to start+length. It has no notion of how many threads are looking in the bag: it simply locks a counter and only increments it if there are more ints to deliver. If FASTREFCNT is set, it uses the atomic instructions to implement the iDrawer so that it uses atomic increment. For small bags of ints with a few workers, this is probably the best abstraction to use. iCupboard: The second abstraction keeps a notion of how many workers are reaching into the bag. The metaphor is that you have a large cupboard where each worker gets his own drawer in the cupboard (where all the integers are scattered among the cupboards pretty evenly). For most of the life of the worker, he only looks in the one drawer assigned to him. It's only when his drawer becomes empty that he starts rummaging through other drawers. For larger numbers of workers, this is a better abstraction to use for a bag because it keeps the workers from bumping into each (i.e,, reduces collateral damage). This interface is slightly clumsy to use, because each worker must have it's own drawer number to start in. (It's not hard, just slightly more clumsy). Which bag to use? Probably start with the iDrawer, and if timing results tell you you can't pull fast enough from the bag, consider using an iCupboard. If you pulling VERY FAST from a bag (i.e., pull, do a small amount of work and pull again), it's possible no bag will work well: heap pressure (too many news or mallocs) can really kill thread performance. If you suspect this is an issue, you can set the iCupboard to set "protect" to false (which basically turns the iCupboard into a barrier sychronizer where NO sync has to be done) and the bag overhead is minimal. A slight rewriting of your work to avoid heap pressure can make a huge difference in performance. * Updated the Python C Extension module: fixed a compiler error on newer platforms (as I didn't use Py_ssize_t correctly) and a small warning on Python 2.7