epoll Scalability Web Page

Introduction

Davide Libenzi wrote an event poll implementation and described it at the /dev/epoll home page here. His performance testing led to the conclusion that epoll scaled linearly regardless of the load as defined by the number of dead connections. However, the main hindrance to having epoll accepted into the mainline Linux kernel by Linus Torvalds was the interface to epoll being in /dev. Therefore, a new interface to epoll was added via three new system calls. That interface will hereafter be referred to as sys_epoll. Download sys_epoll here.


sys_epoll Interface


sys_epoll Man Pages

Testing

We tested using two applications:

dphttpd

Software

The http client is httperf from David Mosberger. Download httperf here. The http server is dphttpd from Davide Libenzi. Download dphttpd here. The deadconn client is also provided by Davide Libenzi. Download deadconn here.

Two client programs (deadconn_last and httperf) run on the client machine and establish connections to the HTTP server (dphttpd) running on the server machine. Connections established by deadconn_last are "dead". These send a single HTTP get request at connection setup and remain idle for the remainder of the test. Connections established by httperf are "active". These continuously send HTTP requests to the server at a fixed rate. httperf reports the rate at which the HTTP server replies to its requests. This reply rate is the metric reported on the Y-axis of the graphs below.

For the tests, the number of active connections is kept constant and the number of dead connections increased from 0 to 60000 (X-axis of graphs below). Consequently, dphttpd spends a fixed amount of time responding to requests and a variable amount of time looking for requests to service. The mechanism used to look for active connections amongst all open connections is one of standard poll(), /dev/epoll or sys_epoll. As the number of dead connections is increased, the scalability of these mechanisms starts impacting dphttpd's reply rate, measured by httperf.

dphttpd SMP

Server

Client

Results for dphttpd SMP

dphttpd UP

Server

Client

Results for dphttpd UP



Pipetest

David Stevens added support for sys_epoll to Ben LaHaise's original pipetest.c application. Download Ben LaHaise's Ottawa Linux Symposium 2002 paper including pipetest.c here. Download David Steven's patch to add sys_epoll to pipetest.c here.


Pipetest SMP

System and Configuration

Results for pipetest on an SMP

Results for Pipetest on a UP

Same Hardware and Configuration as with the SMP pipetest above
with CONFIG_SMP = n being the only change.

sys_epoll stability comparisons Oct 30, 2002


Following are performance results comparing version 0.14 of the (Download v0.14 here) to the version, v0.9, originally used for the performance testing outlined above. (Download v0.9 here) Testing was done using two measures: pipetest details here. and dphttpd details here..

Analysis and Conclusion

The system call interface to epoll performs as well as the /dev interface to epoll. In addition sys_epoll scales better than poll and AIO poll for large numbers of connections. Other points in favour of sys_epoll are:

Due to these factors sys_epoll should be seriously considered for inclusion in the mainline Linux kernel.

Acknowledgments

Thank You to: Davide Libenzi Who wrote /dev/epoll, sys_epoll and dphttpd.
He was an all around great guy to work with.
Also Thanks to the following people who helped with testing and this web site:
Shailabh Nagar, Paul Larson , Hanna Linder, and David Stevens.