On Thu, 2004-01-01 at 02:56, Rudy Koot wrote:
Some comments and questions to about 5 different e-mails:
Some answers, as best I can. :-)
===== Rudy
Some of Volkmar's arguments are based on the fact that L4 is a general purpose microkernel, I think the discussion would be more useful if we look at how a microkernel with the purpose of running a capability based system on top, should be design. Leaving the question wheter those capabilities should be implemented at kernel-level or user-level open.
We already have that kernel. It's called EROS. While I do know how to build a variant that is much closer in spirit to L4 (i.e. smaller -- we designed such a nucleus at IBM) I do not believe that its end to end performance will be any better.
However, I have been very carefully avoiding the direction you identify for three reasons. The first is that I deeply respect the L4 work. The second is that I attempted a successor to L4 after Jochen left IBM and it became a bit sensitive. The third is that the L4 group has not chosen to pursue capability-based design, and I am not interested in redirecting their work.
At this time, what I am trying to achieve is a common nucleus on which both teams can pursue their research. This will facilitate cross-fertilization of ideas, and I believe that this will benefit both groups. The plain fact is that the L4 and EROS communities can successfully proceed in independent directions, each lacking a critical mass of people. I do not believe that this is the most productive way to proceed.
And I should acknowledge that I disagree with Volkmar very fundamentally about the "performance uber alles" assumption. Performance is important, but from a research perspective it is *more* important to understand what the fundamental architectural issues are. Once we do, we can step back and decide what to take out. At the moment, we have no common base on which direct comparison is possible.
I don't think that the L4 kernel has all the answers, but I also think that the EROS kernel does not have all the answers. Each, I think, has important strengths. EROS lacks a certain elegance of minimality. L4 (as I understand it today) lacks a credible story about security, access control and denial of service. Perhaps it is time for both groups to step back and make a serious attempt to learn from each other.
===== Rudy
Any form of IPC in which the decission to allow or disallow a thread to send a message to another thread is vulnerable to a DoS attack. If the decission is mad by the kernel the IPC can be canceled immediately, but if it is made at user level another thread is blocked for a small amount of time.
This is exactly true, but it fails to take into account the fact that IPC indirection can possibly suppress this behavior. I do not understand the current IPC indirection proposal, and trying to reason without understanding it is very slippery. Also impolite. Something I am known for, but I do try not to do it on purpose. :-)
From the conversations on this list, however, it appears to me that the
L4 research groups do not agree universally on what the IPC control mechanism is, and this may be contributing to some difficulty in the discussion. One group clearly advocates thread spaces. Volkmar (if I understand him) currently advocates IPC indirection. The two designs have very different implications for performance, implementation, and security.
===== Shap
b) For some types of systems (capability systems), disclosure of the sender id is an absolute violation of design requirements, so any microkernel that relies exclusively on server-side security checks based on sender-id is not a universal microkernel.
Sorry, but you still owe me an example of a situation in which revealing the sender-id actually breaks the security of the system. The only thing I could think of was that it might reveal some details about the internal structure of an application, but if the system is secure that information hasn't got much use to a malicious program or user.
From the perspective of security you are reasoning backwards. There is
never a mandate to justify withholding information. The mandate is always the other way: if you wish to argue that a sender-id is desirable, you must justify why its disclosure is safe and also why it is required.
Observation: it is not, and it is not.
In answer to your question, here are some problems with disclosing sender ID:
1. The one you mentioned: it discloses client structure, which is something that the server is not entitled to know.
2. It mandates a client structure, because using the sender-id as a credential strongly discourages the application developer from breaking their applications into multiple address spaces and separate threads. This defeats robustness and security both.
3. It is inaccurate. The server can NEVER rely on the sender-id, because it cannot know if the immediate sender is acting in proxy for someone else. That is, sender-id can only provide false security.
Personally, I find (2) very ironic. Ultimately, the main justification for fast IPC is that it makes decomposed applications easier. I find it curious, given this, that L4 adopts an access checking model that defeats decomposition. I also find it very curious that no L4-based system (at least, none that I know about) has really pushed for fine-grain decomposition. Perhaps Mungi did and I do not understand it well enough.
As a backwards compatibility matter, and as a validation of the L4 nucleus, running Linux on an L4 is interesting. As a matter of future research, and as a matter of forward-looking system architecture, it is boring. The right forward-looking research question is:
*Given* a fast nucleus in the style of L4, what is the most effective way to structure a native operating environment? What fundamental and novel leverage do such kernels provide?
That is: do fast IPC and microkernels provide anything that is architecturally important, or are they just clever tricks that are good only for conference papers? Obviously, I believe that they are useful.
The problem I see is that *none* of our groups are big enough to successfully evaluate this question of "what native environment?" It requires a truly awesome amount of work.
I really think this is an important point. If revealing the sender-id is indeed a violation of security, there wouldn't be a need to discuss wheter or not thread-spaces need to be implemented. They just have to.
c) More specifically, any microkernel that requires checks based on sender-id is entirely unsuitable as a platform for EROS-NG.
Because (a) revealing sender-id is a security hole, and (b) EROS is a capability system. The latter is a matter of definition.
Assume you add 2 more TLB entries and 5 more L2 cache lines--your aftercosts for IPC go up by 2*80 + 5*80 = 560 cycles. Considering overall IPC costs of 1000 cycles on a P4 with all those nasty cache and TLB flushes you add an overhead of >50%.
I believe these costs are exagerated. To convert a capability into sender-id a simple table lookup can be used. This requires a single TLB entry and a single L2 cache line (worst case). If IPC is frequent, which is the case in which you want high performance, the TLB entry is very likely to be present and probably even the L2 cache line.
Volkmar is combining two things in one, and seems to be forgetting that he does so. You are correct. Once you have the capability, obtaining the target thread id is quite fast. No table lookup is required. The capability is kernel-protected state, and can simply contain a direct pointer to the recipient PCB.
I think that the costs that Volkmar is identifying come from the need to lookup the thread-id or capability in the thread mapping space. That is, these are the costs of the address space traversal. The costs are the same in either case (thread-id or capability).
If the kernel implementation uses flexpages, then I believe that in the usual case the reality will be only one TLB miss, but the number of L2 references seems plausible and problematic.
In practice, I would use a cache for an L1 software probe first in order to avoid all marginal TLB and cache misses in the usual case, but this would result in high variance of IPC times.
In our case we use an object identifier in the message, which is a handle to an object descriptor and do a reverse check (i.e. if the thread is allowed to invoke that object). Costs: boundary check (can be a simple AND) one MOV and a CMP. The permissions can go into the same cache line.
Volkmar's technique is quite elegant. The only problem I see is that it places the burden of verification on the wrong party -- it should be on the client. That is, the design invites DoS attacks.
However, Volkmar is making an assumption that is problematic. He is assuming that the passed ID is a pointer. From a security perspective this is VERY bad. First, it invites memory attacks. Second, it discloses information about the internal implementation of the server. Third, it makes selective rescind of authority very hard to do.
One of the features of EROS is an indirection object. This object can stand in front of a start capability transparently. The client invokes the indirection object capability rather than the start capability. Given this construction, a service can selectively rescind that client capability by destroying the indirection object.
This is something that L4 presently has no means to do. Of course, L4 does not need it, because it makes no assumption about controlling client invocations at all.
A possibility you did not mention is a hybrid thread id which has an thread and a descriptor part. The descriptor is kernel enforced. That is what we currently have in V4 (please not that V2 and X.0 are completely outdated!!!)--the version part of the thread id.
Sorry, but I do not see how the version part could be used as the server defined word. You can't have multiple versions (read server defined words) of the same thread-id at the same time.
I don't either, but I don't understand what Volkmar is proposing enough to comment intelligently. I would like to, and I hope he will respond with a pointer to a specification or more detail about where these references are occuring. I am not sure that I have understood him correctly.
shap