Stackless Python - A first look

[fips]

The necessity of running thousands of parallel tasks is very common in all kinds of simulation problems, such as video games, virtual worlds or agent-based systems. The idea of involving traditional OS threads is not very attractive here. Not only they are difficult to code and debug but they are also pretty inefficient in such massively parallel systems. Serialization of the tasks, which is the traditional approach, leads to complicated state management and inefficiencies as well, so it isn't the right way to go, either.

The Stackless Python seems to provide the right answer to this troubles. It brings the concept of microthreads (or green threads), which offers a very elegant and efficient way to deal with large-scale parallelism. In fact, it's based on the idea of non-preemtive multitasking (tasks can self-interrupt and voluntarily give control to other tasks). The good news is that the Stackless Python is 100% compatible with the standard Python. Let's imagine the potential of hybrid C++/Stackless systems based on the Boost.Python! Take a look at the EVE Online, it's a real example of what is possible to achieve with this technology.

Check these papers to see the key concepts and examples:
Introduction to Concurrent Programming with Stackless Python by Grant Olson
Multithreaded Game Scripting with Stackless Python by Harry Kalogirou
Easy Concurrency with Stackless Python by Chris McAvoy

There are also some slides available:
CCP Games presentation at PyCon 2006
StacklessEuroPy.pdf
Stackless .pdf ebook Download

The code below shows how a game engine might take advantage of the Stackless Python. There are three agents running in parallel (each in its own microthread). The scheduler in the main loop executes only a specified number of instructions (10 in this case) in each iteration, which allows other sub-systems to execute as well. Isn't it great!

Code: Select all

import stackless

class Agent:
    def __init__(self, id):
        self.id = id
        stackless.tasklet(self.run)()

    def run(self):
        # run Agent's loop
        n = 0
        while 1:
            print(" I'm Agent #%d (%d)" % (self.id, n))
            n +=1
            stackless.schedule() # give control to others
            
def main():

    # create 3 Agents
    for id in range(3):
        Agent(id)
        
    # run the main loop
    while 1:
        
        # run agents in parallel, execute 10 instructions
        # and reinsert itself for later continuation
        print("Run agents:") 
        stackless.run(10).insert()
                        
        # do something else (e.g. rendering)
        print("Do something else...")

if __name__ == "__main__":
    main()

[fips]

In thinking about visual simulations, I came to the next big question:
What is the best way to separate the state management from the visual representation?

Consider a patrolling robot with this state logic:

Code: Select all

while 1:
    set_velocity(10)
    sleep(5) # seconds, much longer compared to the frame sampling period
    set_velocity(-10)
    sleep(5)

How to render the robot in every frame?

[fips]

Some more interesting resources:
What are the drawbacks of Stackless Python?
Using Stackless by Andrew Dalke
Erlang vs. Stackless python: a first benchmark by Muharem Hrnjadovic