hello all,
I've ported my multithreaded tcp server to L4 using flips but I am having some problems getting it to run properly. It is a quite a bit to explain so I've written a simple tcp echoserver to demonstrate my problem. The following happens when the server starts and the client connects and send data.
1. Echoserver starts and listens on port 2222 2. client connects, server creates a child_server thread to service client and starts listening again 3. client sends string to child_server thread which is received. child_server however is unable to complete send(echo), it seems to be blocked at this point. 4. The send is only completed when another client attempts a connect(..and the server moves on from the accept(..) statement).
I have appended my code at the bottom of this mail maybe someone can spot my mistake and highlight it.
To check if I've used the threads properly I also wrote a program in which each thread prints the numbers from 0 to 9 and that runs fine. So I am pointing my finger and the recv(), accept() and send() functions in my echo server :-|
Any suggestions will be greatly appreciated.
cheers david
=============================================================================
/*** GENERAL INCLUDES ***/ #include <stdlib.h> #include <stdio.h> #include <unistd.h> #include <sys/socket.h> #include <sys/types.h> #include <netinet/in.h> #include <arpa/inet.h> #include <string.h>
/*** L4 INCLUDES ***/ #include <l4/names/libnames.h> #include <l4/thread/thread.h> #include <l4/log/l4log.h> #include <l4/dm_mem/dm_mem.h>
#define BUF_SIZE 100
l4_threadid_t srv; char LOG_tag[9] = "tcpecho"; int data_key;
struct th_data{ int c; //socket char foo[10]; //.... };
/* exit handler */ void child_exit(l4thread_t thread, void * data); void child_exit(l4thread_t thread, void * data){ struct th_data *ptr = (struct th_data *)data;
/* close the connection and free up memory used by data structure*/ close(ptr->c); free(ptr); }
/* declare exit function descriptor */ L4THREAD_EXIT_FN(exit_fn,child_exit);
void child_server(void *data); void child_server(void *data){ char databuf[BUF_SIZE]; struct th_data *dptr = (struct th_data *)data; int ret, sock, n; void *ret_val; l4thread_t id; sock = dptr->c; LOG("got socket\n"); /* register exit function */ ret = l4thread_on_exit(&exit_fn, // exit function descriptor data // data pointer ); if (ret < 0){ /* register error function failed */ LOG_Error("Register error function failed\n"); } id = l4thread_myself(); LOG("Server thread starting with id %u\n",id); /* we are up */ if (l4thread_started(ret_val) < 0){ /* startup notification failed */ LOG_Error("Startup notification failed\n"); } LOG("Entering loop to wait for requests on socket %d.... \n",sock); while (1){ LOG("At top of while loop\n"); if((n = recv(sock, databuf, BUF_SIZE-1, 0)<=0)){ LOG_Error("recv failed %d returned \n",n); break; } databuf[4] = '\0'; databuf[5] = '\n'; LOG("GOT MESSAGE\n"); LOG(">%s; sending ....\n",databuf); databuf[0] = 'X'; send(sock,databuf,6,0); memset(databuf,0,BUF_SIZE); LOG("Looping\n"); } /* Call on exit function to terminate */ l4thread_exit(); }
/* Start the server */ int main(int argc, char *argv[]){
int c, s,err, ret_val, errcount=0; struct sockaddr_in addr;
if ((argc < 2) || strcmp("--nowait", argv[1])) while (names_waitfor_name("ifconfig", &srv, 30000) == 0) { LOG("Client(main): "mini_ifconfig" not available, keep trying..."); }
memset(&addr, 0, sizeof(addr)); addr.sin_addr.s_addr = htonl(INADDR_ANY); addr.sin_family = AF_INET; addr.sin_port = htons(2222); LOG("try to open socket"); if ((s=socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)) < 0) { LOG("Couldn't create socket"); return -1; } if (bind(s, (struct sockaddr *)&addr, sizeof(struct sockaddr)) < 0) { LOG("Couldn't bind"); return -1; } LOG("bound to socket %d, call listen",s); if ((err=listen(s, 1)) < 0) { LOG("listen(%d): %d", s, err); return -1; }
LOG("echo server is up and accepting...");
names_register("echoserver"); while(1){ struct sockaddr dummyaddr; int addrlen; l4thread_t th; struct th_data *th_dptr; LOG("waiting for another connection\n"); if ((c=accept(s, &dummyaddr, &addrlen)) < 0) { LOG("accept(%d): %d", s, c); if (++errcount > 10) { LOG("maximum error count reached---giving up..."); exit(1); } continue; } LOG("Allocating thread data struct\n"); th_dptr = (struct th_data*)l4dm_mem_allocate( sizeof(struct th_data), L4DM_CONTIGUOUS); memset(th_dptr,0,sizeof(struct th_data)); th_dptr->c = c; LOG("Got a connection request, creating thread to handle client on socket %d\n", c); th = l4thread_create( child_server ,(void *)th_dptr ,L4THREAD_CREATE_SYNC); ret_val = l4thread_startup_return(th); /* Thread started ... listening for other requests */ LOG("thread created ... listening again\n"); } return 0; }
__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
Hello David,
On Tue, Aug 01, 2006 at 04:55:53AM -0700, David Silcott wrote:
I've ported my multithreaded tcp server to L4 using flips but I am having some problems getting it to run properly. It is a quite a bit to explain so I've written a simple tcp echoserver to demonstrate my problem.
Ouch, already at this point I remember some limitations of the FLIPS experiment and L4VFS. FLIPS supports concurrent blocking RPCs, e.g. send() and recv(), only for calls using different L4VFS connections, see l4vfs_init_connection(). If one uses the same connection for two blocking calls the latter one is blocked in the underlying IPC in the microkernel.
L4VFS currently implements no extensive multi-threading support and therefore initializes only one connection to FLIPS per task, see l4/pkg/l4vfs/lib/libc_backends/socket_io/socket.c line 44 ff.
The following happens when the server starts and the client connects and send data.
- Echoserver starts and listens on port 2222
- client connects, server creates a child_server
thread to service client and starts listening again 3. client sends string to child_server thread which is received. child_server however is unable to complete send(echo), it seems to be blocked at this point.
I guess your server blocks in accept() and the child_server blocks during its "send()" IPC in the kernel.
- The send is only completed when another client
attempts a connect(..and the server moves on from the accept(..) statement).
Now the thread in FLIPS replies the accept() request and enters an ipc_wait(). The child_server "send()"-IPC is unblocked and the packet is echoed.
To fix this issue you should start in the before-mentioned socket.c file and init L4VFS connections to FLIPS per thread.
HTH
Christian Helmuth wrote:
Hello David,
On Tue, Aug 01, 2006 at 04:55:53AM -0700, David Silcott wrote:
I've ported my multithreaded tcp server to L4 using flips but I am having some problems getting it to run properly. It is a quite a bit to explain so I've written a simple tcp echoserver to demonstrate my problem.
Ouch, already at this point I remember some limitations of the FLIPS experiment and L4VFS. FLIPS supports concurrent blocking RPCs, e.g. send() and recv(), only for calls using different L4VFS connections, see l4vfs_init_connection(). If one uses the same connection for two blocking calls the latter one is blocked in the underlying IPC in the microkernel.
L4VFS currently implements no extensive multi-threading support and therefore initializes only one connection to FLIPS per task, see l4/pkg/l4vfs/lib/libc_backends/socket_io/socket.c line 44 ff.
L4VFS *does* support multithreading applications, for example with the term server. Therefore many functions in the IDL interfaces have this [allow_reply_only] flag set which basically allows the server to not answer calls directly but delayed with the help of a worker thread. In between the session thread in the server is ready to receive new calls.
Grep in l4vfs/term_server for "DICE_NO_REPLY" for examples. The problem here seems to be that the current implementation of flips does not utilize this (there are no worker threads, right?).
Even transparently supporting one session per client thread by L4VFS would not always solve the problem always, as clients may use a user-level thread package on top of L4-threads (e.g. Java) or might interrupt threads with ex-regs, which could result in even one thread calling several blocking operations at once. The problem must be solved in the servers.
The following happens when the server starts and the client connects and send data.
- Echoserver starts and listens on port 2222
- client connects, server creates a child_server
thread to service client and starts listening again 3. client sends string to child_server thread which is received. child_server however is unable to complete send(echo), it seems to be blocked at this point.
I guess your server blocks in accept() and the child_server blocks during its "send()" IPC in the kernel.
- The send is only completed when another client
attempts a connect(..and the server moves on from the accept(..) statement).
Now the thread in FLIPS replies the accept() request and enters an ipc_wait(). The child_server "send()"-IPC is unblocked and the packet is echoed.
To fix this issue you should start in the before-mentioned socket.c file and init L4VFS connections to FLIPS per thread.
You could fix it this way but it would be an intransparent hack (which might be perfectly ok for your situation). You could also enhance flips with worker threads.
If you fix the problem either way please send patches back!
Cheers, Martin Pohlack
On Wednesday 02 August 2006 12:51, Martin Pohlack wrote:
Christian Helmuth wrote:
Hello David,
On Tue, Aug 01, 2006 at 04:55:53AM -0700, David Silcott wrote:
I've ported my multithreaded tcp server to L4 using flips but I am having some problems getting it to run properly. It is a quite a bit to explain so I've written a simple tcp echoserver to demonstrate my problem.
Ouch, already at this point I remember some limitations of the FLIPS experiment and L4VFS. FLIPS supports concurrent blocking RPCs, e.g. send() and recv(), only for calls using different L4VFS connections, see l4vfs_init_connection(). If one uses the same connection for two blocking calls the latter one is blocked in the underlying IPC in the microkernel.
L4VFS currently implements no extensive multi-threading support and therefore initializes only one connection to FLIPS per task, see l4/pkg/l4vfs/lib/libc_backends/socket_io/socket.c line 44 ff.
L4VFS *does* support multithreading applications, for example with the term server. Therefore many functions in the IDL interfaces have this [allow_reply_only] flag set which basically allows the server to not answer calls directly but delayed with the help of a worker thread. In between the session thread in the server is ready to receive new calls.
A small note: It is true that the answer can be delayed by the server, however, the genereated code for L4v2 clients generated by the current DICE implementation uses l4_ipc_call on the client side. Therefore the very same thread which received the request has to reply (as long as "send-as" is not implemented).
Frank
Thanks for the suggestions guys. I am going to have try at getting this thing to work.
Will let you know how it goes
David
__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
l4-hackers@os.inf.tu-dresden.de