Friday, April 10, 2015

Creating a muli-processes DTLS server.

I need to create an UDP server working with multiple clients over DTLS.  I found one note about net-snmp's DTLS Implementation Notes which almost has the same requirement as mime.  But it seems it falls back to use memory BIO and cache the data, and Campgnol VPN's solution is making use of NAT traversal technique that open each new connection with a new UDP port, which does not fit my requirement to only use one UDP port for server.

From the notes, I notice the author is almost succeeded with peeking the incoming traffic except the "Packets Pile Up" problem.  I think that is the right direction for me.

I tried on a single process server first. With recvmsg() with MSG_PEEK, I could get the client's address and port number, from which I could select the right context for the DTLS link.  I realized that I need a custom BIO module to avoid the"Packets Pile Up" problem.

I made a copy of crypto/bio/bss_dgram.c from OpenSSL's source and renamed it to my_udp.c, then stripped away anything unrelated to my usage, basically removed all platform specific and SCTP code except for Linux.  Then further removed all code related to connected filed flag.  Then changed the reading function to peek the packet first and compare the source IP and port number to the current one.  If it is not the current one, then fake an EAGAIN error.

Since server is listening on multiple interfaces, when sending data out, it needs to set its source address correspondingly.  But UDP does not directly give the local address used for the data.  With socket option IP_PKTINFO turn on for UDP server socket, local server address could be found with recvmsg() together with MSG_PEEK.  It is then set in my custom UDP BIO to be used when sending out data.  This seems to work fine with single server process.

But then I was facing the challenge to serve client on multiple processes on the same UDP port. After some tried and errors, I settled on the following method:

The parent creates socketpair() with each child for communication.  For the start, only the first child would listen on the UDP server port.  After peeking from the incoming data, calculate a child index from the client address and port. If the child index is its own, it proceeds to process the data, if not, it stops monitoring the server port for incoming data and send a command to parent and inform parent that the other child should handle the incoming data.  When parent receives the command, it will send the information to the other child, the other child gets the message and start monitoring server port and process incoming data and so on.

It seems working, but have not test out the performance, whether it is worth the trouble to have multiple processes.

Post a Comment