The Stackless Python seems to provide the right answer to this troubles. It brings the concept of microthreads (or green threads), which offers a very elegant and efficient way to deal with large-scale parallelism. In fact, it's based on the idea of non-preemtive multitasking (tasks can self-interrupt and voluntarily give control to other tasks). The good news is that the Stackless Python is 100% compatible with the standard Python. Let's imagine the potential of hybrid C++/Stackless systems based on the Boost.Python! Take a look at the EVE Online, it's a real example of what is possible to achieve with this technology.
Check these papers to see the key concepts and examples:
Introduction to Concurrent Programming with Stackless Python by Grant Olson
Multithreaded Game Scripting with Stackless Python by Harry Kalogirou
Easy Concurrency with Stackless Python by Chris McAvoy
There are also some slides available:
CCP Games presentation at PyCon 2006
Stackless .pdf ebook Download
The code below shows how a game engine might take advantage of the Stackless Python. There are three agents running in parallel (each in its own microthread). The scheduler in the main loop executes only a specified number of instructions (10 in this case) in each iteration, which allows other sub-systems to execute as well. Isn't it great!
Code: Select all
import stackless class Agent: def __init__(self, id): self.id = id stackless.tasklet(self.run)() def run(self): # run Agent's loop n = 0 while 1: print(" I'm Agent #%d (%d)" % (self.id, n)) n +=1 stackless.schedule() # give control to others def main(): # create 3 Agents for id in range(3): Agent(id) # run the main loop while 1: # run agents in parallel, execute 10 instructions # and reinsert itself for later continuation print("Run agents:") stackless.run(10).insert() # do something else (e.g. rendering) print("Do something else...") if __name__ == "__main__": main()