Comparing IPC and capability invocation

[Jonathan S. Shapiro]

On Fri, 2003-12-12 at 08:34, Joshua LeVasseur wrote:
> How does the EROS model support multiprocessor object invocation?

I will answer below, but I believe that this is entirely orthogonal to
the question of object-id bits. The proposed "object id" bits are merely
opaque bits that are carried in the descriptor. The "recipient server
bits" [note that I am avoiding the word thread intentionally] are still
present and their handling does not change.

> Given that L4 invocations directly address threads, a client can 
> address a server's CPU-local thread, independent of the object type.

I believe that you have the desirability of this exactly backwards. What
you are appear to be saying is:

  In order to address the CPU-local thread of a service, the thread-id
  of that thread should be revealed to the client.

I will describe below why this is unnecessary. First, let me identify
why it is undesirable:

  1. Migration of the client causes it to invoke the wrong service
     thread, because it now holds an inappropriate ID and has no
     way to discover this.

  2. The fact that the service is multithreaded (at all) is exposed
     to the client. This is a violation of encapsulation. It is
     unnecessary and unjustified.

  3. CPU scheduling information is unnecessarily revealed (which is
     a covert channel).

In short, this is exactly a case where a default policy *does* belong in
the kernel, because the client is not in a position to have up to date
knowledge without exposing a whole bunch of problems.

> Thus a client can 
> load-balance (by first choosing a client thread on the same CPU as the 
> server thread), or the client can let the server load-balance (and thus 
> choose the server thread which executes on the same CPU as the client 
> thread).

Hold on! I agree that the client should sometimes load balance, but it
must first be the server's decision to reveal that it is multithreaded
at all. If the server wishes to do so, it is free to provide multiple
descriptors/thread-ids to the client. If the server does not, it does
not reveal multiple IDs of this form. Note that NONE of the dispatching
depends on the bits of the thread-id per se. All of what you describe
can be equally well accomplished by invoking descriptors selectively.
Fundamentally, what is needed is a function held in client memory of the
form:

     f(descriptor-id) -> CPU ID

The descriptor-id can either be an address in the thread/descriptor
address space or it can be the thread-id. Precisely *because* it doesn't
matter, I would strongly advocate that it should NOT be the thread-id
because this violates encapsulation.

> If the kernel were to distribute object 
> invocations to a server's per-CPU threads, it would be implementing a 
> policy, and one which might not satisfy the user tasks.

This is regrettably true. However, I want to point out there is a strong
tension here between speed on the one hand and (portability,
encapsulation) on the other. You are making an argument that is entirely
motivated by performance. For many systems, this is NOT the primary
design consideration. Choosing speed uber alles is also policy, and it
is a failure of microkernel architecture to impose an interface that
prevents the efficient realization of systems that place other
priorities first.
 
> I'm highlighting this multiprocessor problem as an example of why one 
> might want to directly address thread IDs, rather than object IDs.

I hope my comments above reveal that (a) your goal is sound, but (b)
your goal does not in any way require revealing thread-ids.

> I can imagine solutions for EROS, such 
> as allocating one capability per processor, but this overloads the 
> concept of a capability, does it not?

In my view, it does not. A capability internally names a pair of the
form:

	(representation-state, interface signature)

It is perfectly okay to hold two capabilities that name the same
representation state and interface (method set). The two capabilities
might have different behavior with respect to the communication
transport, but if they have the same operational semantics, they are the
"same" capability (we are getting here into the question "what does it
mean to test two objects for equality?").

shap