multithreaded tcp echo server problem & flips

[Martin Pohlack]

Christian Helmuth wrote:
> Hello David,
> 
> On Tue, Aug 01, 2006 at 04:55:53AM -0700, David Silcott wrote:
>> I've ported my multithreaded tcp server to L4 using
>> flips but I am having some problems getting it to run
>> properly. It is a quite a bit to explain so I've
>> written a simple tcp echoserver to demonstrate my
>> problem.
> 
> Ouch, already at this point I remember some limitations of the FLIPS
> experiment and L4VFS. FLIPS supports concurrent blocking RPCs, e.g.
> send() and recv(), only for calls using different L4VFS connections, see
> l4vfs_init_connection(). If one uses the same connection for two blocking
> calls the latter one is blocked in the underlying IPC in the microkernel.
> 
> L4VFS currently implements no extensive multi-threading support and
> therefore initializes only one connection to FLIPS per task, see
> l4/pkg/l4vfs/lib/libc_backends/socket_io/socket.c line 44 ff.

L4VFS *does* support multithreading applications, for example with the
term server.  Therefore many functions in the IDL interfaces have this
[allow_reply_only] flag set which basically allows the server to not
answer calls directly but delayed with the help of a worker thread.  In
between the session thread in the server is ready to receive new calls.

Grep in l4vfs/term_server for "DICE_NO_REPLY" for examples.  The problem
here seems to be that the current implementation of flips does not
utilize this (there are no worker threads, right?).

Even transparently supporting one session per client thread by L4VFS
would not always solve the problem always, as clients may use a
user-level thread package on top of L4-threads (e.g. Java) or might
interrupt threads with ex-regs, which could result in even one thread
calling several blocking operations at once.  The problem must be solved
in the servers.

>> The following happens when the server starts and the
>> client connects and send data.
>>
>> 1. Echoserver starts and listens on port 2222
>> 2. client connects, server creates a child_server
>> thread to service client and starts listening again
>> 3. client sends string to child_server thread which is
>> received. child_server however is unable to complete
>> send(echo), it seems to be blocked at this point.
> 
> I guess your server blocks in accept() and the child_server blocks during
> its "send()" IPC in the kernel.
> 
>> 4. The send is only completed when another client
>> attempts a connect(..and the server moves on from the
>> accept(..) statement).
> 
> Now the thread in FLIPS replies the accept() request and enters an
> ipc_wait(). The child_server "send()"-IPC is unblocked and the packet is
> echoed.
> 
> To fix this issue you should start in the before-mentioned socket.c file
> and init L4VFS connections to FLIPS per thread.

You could fix it this way but it would be an intransparent hack (which
might be perfectly ok for your situation).  You could also enhance flips
with worker threads.

If you fix the problem either way please send patches back!

Cheers,
Martin Pohlack