"Stackless" XEmacs?

[Jerry James]

On Tue, Apr 29, 2008 at 11:47 PM, Stephen J. Turnbull
<stephen at xemacs.org> wrote:
> Another interesting idea from emacs-devel.
>
>  dhruva writes:
>   > If we are looking at concurrency, there is another paradigm based on
>   > maintaining multiple internal function call stacks which a scheduler
>   > can schedule in some fair fashion. I am talking of stackless Python
>   > implementation. You really do not have multiple threads but get
>   > simulated concurrency through stack switching. For IO intensive usage
>   > using async IO with stackless might make a good candidate.
>   >
>   > -dhruva

The word "stackless" appears to be a misnomer.  Any idea where that came from?

I wrote a package that does this some years ago.  Look for my name and
"BERT"; it's probably still floating around on the web somewhere.  I
don't think we really want to go in this direction.  Here's why.

1. You have to choose between two painful alternatives.
   a. A "fair" scheduler that uses a timer to trigger stack switches: this
      means pseudo-concurrency between operations on different stacks, which
      means you have to rewrite pretty much all of your code to use mutexes and
      condition variables.
   b. A scheduler that only runs inside of well-defined functions, meaning that
      a given stack can hog the CPU for arbitrary amounts of time.
2. If you choose 1a, then you wind up doing all the work you would have to do
   for a "real" thread library anyway.
3. There are several mature, well-debugged thread libraries out there.  Why
   ignore them and start the development and debugging process all over again?
4. XEmacs is already built on a Reactor pattern.  The "stackless" architecture
   is just another version of the Reactor.  It won't provide any performance
   gains over what we have now.  If we want better performance, we should look
   at switching to a Proactor pattern.  However, not all platforms support
   asynchronous I/O adequately, so we would really have to have a Reactor/
   Proactor hybrid.
5. The world is going multi-core.  Intel is talking about having 80-core
   processors available in 2 years.  The "stackless" architecture uses only 1
   core.  A "real" thread library can take advantage of multiple cores.

I am interested in having a discussion on what it would take to switch
from our Reactor-based architecture to a multithreaded one.  Such a
switch would have serious implications for the Lisp engine and the
garbage collector, for example.
-- 
Jerry James
http://loganjerry.googlepages.com/