Linux-kernel mailing list archive 1999-36,: Re: linux threads vs. solaris threads

# Re: linux threads vs. solaris threads

Larry McVoy (lm@bitmover.com)
Sat, 11 Sep 1999 02:50:04 -0700

- Messages sorted by: [ date ][ thread ][ subject ][ author ]
- Next message: Andreas Dilger: "Re: 2.3.18-ac1: double_lock() moved into <include/linux/dcache.h>"
- Previous message: manfreds@colorfullife.com: "Re: Kernel oops (2)"

On Sat, Sep 11, 1999 at 11:04:43AM +0000, Hrafnkell Eiriksson wrote:
> Few days ago in my concurrent programming class the teacher stated
> that the time it takes the OS to switch between threads (i.e. threads
> created with pthreads_create()/clone(CLONE_VM|CLONE_FILES etc)) was
> probably higher in Linux than in Solaris (and other commercial Unices)
> on the same hardware.
>
> (and no we were not confusing Linux threads and Solaris
> userspace threads, we were talking about kernel level threads in
> both systems).

In that case, I doubt it. Linux is a substantially lighter weight system
than Solaris. Given that you are talking about kernel supported threads,
the work that Linux does is pretty much the same as the work that Solaris
does.

> I found the lmbench benchmark that tries to measure the time it takes
> to do a context switch. Checking the lmbench source shows that it
> creates new processes with fork() and measures the time it takes
> to do a context switch between them. I don't think that this measurement
> is a fair measurement for the time it takes to switch between threads
> as there is no need to flush the TLB wich has to be done when doing a
> switch between processes if I have understood the purpose of a TLB
> correctly (but I might be wrong as I dont understand this well enough).

Almost all (maybe 100%) modern hardware has context ids in the TLB. The
context ID is basically the same a process id. So there is no need to
flush the TLB, the entries are different for each process.

> So I concluded that lmbench was not the right tool to measure this and
> answer my question.

Don't be so quick to dismiss it. Running lmbench's context switch code on
the same hardware but on different operating systems tells you something:
it tells you how fast each OS can switch the _heaviest_ weight process
it has. That is itself interesting. It's true that there can be lighter
weight objects in the system, but see below for a rationale why this is
not an issue.

> As far as I understand, switching between threads in Linux means
> changing the PC counter, the stackpointer and restoring the registers
> and FP state. Solaris probably does something similar.

That's about right. So if both OS's are saving and restoring exactly
the same state, why should there be any difference in performance?

The answer is that the state saving is really a very small part of the
context switch path in Solaris, at least. If that was not true then
the context switch times between user level threads and the context
switch times between kernel level threads would be close to identical.
They aren't.

So what's the cost difference? The difference is the cost of getting to
the point in the kernel where you decide to switch. That cost is
substantially higher in Solaris. Solaris is a fine grain threaded system
with 1000's of kernel level locks. A number of those locks are taken
on the way to the context switch code. That and other code present in
Solaris but not present in Linux accounts for the difference.

This is a clear example of why scaling can be viewed as "bad". It comes
with a cost. While it is true that a fine grain threaded OS can take
better advantage of more processors, it's also true that it takes worse
advantage of less processors. They are robbing Peter to pay Paul.

> Can anyone here help me determine if my teacher is right or wrong on
> this (or isn't there a right/wrong answer? :) or point me
> to webpages, articles, books or benchmarking tools?

You can take the lmbench timing harness and trivially create a
benchmark which uses threads instead of processes and see what
the numbers are.

> Also, is there any more "context switch" involved in switching
> between threads in Linux than in Solaris? A Linux thread
> created with the clone() system call has the same virtual
> memory and same file descriptors as other threads running
> with it within the same process so there isn't really any
> context to switch between except changing the state of
> the CPU.

Something to note: Linux actually shared page tables between processes
when the processes are sharing the same virtual address space. The
page tables are an attribute of the VM, not of the process. There's
normally a 1:1 mapping but when you clone there is a N:1 mapping.

That's really cool because it means that Linux cloned processes are
every bit as lightweight as Solaris threads, there is no technical
reason for that not to be true.

    --

    ---
    Larry McVoy            	   lm@bitmover.com           http://www.bitmover.com/lm

    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.rutgers.edu
    Please read the FAQ at http://www.tux.org/lkml/

     Next message: Andreas Dilger: "Re: 2.3.18-ac1: double_lock() moved into <include/linux/dcache.h>"
     Previous message: manfreds@colorfullife.com: "Re: Kernel oops (2)"