Opened 11 years ago

Closed 7 years ago

Last modified 7 years ago

#635 closed task (fixed)

Replace use of select() in the I/O manager with epoll/kqueue/etc.

Reported by: simonmar Owned by: bos
Priority: normal Milestone:
Component: libraries/base Version: 6.13
Keywords: Cc: Bulat.Ziganshin@…, pho@…, johan.tibell@…, elijah.epifanov@…, 8mayday@…, cgibbard@…, morrow@…, lvh@…, gale@…, jystic@…, khaelin@…, nonowarn@…, kazu@…, jeffz_, 1@…, merehap@…, gabriel@…, v.dijk.bas@…
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: Runtime performance bug Test Case: N/A
Blocked By: Blocking:
Related Tickets: Differential Rev(s):
Wiki Page:

Description

Subject says it all. We've known about this problem for ever, but haven't got around to doing anything about it.

Here is a library that provides a unified API over the various event APIs: http://monkey.org/~provos/libevent/.

See also http://www.kegel.com/c10k.html.

1 week is an estimate for doing it properly on all relevant platforms, by someone who is familiar with the APIs and code.

Attachments (1)

base.diff (15.0 KB) - added by bos 7 years ago.
base.diff

Download all attachments as: .zip

Change History (44)

comment:1 Changed 11 years ago by vs@…

You can find my patches for a libeventified RTS for GHC 6.0 at: http://www-i2.informatik.rwth-aachen.de/~stolz/Haskell/libevent/

comment:2 Changed 11 years ago by simonmar

I should add that we'd prefer to do this using the I/O manager in the threaded runtime (libraries/base/GHC/Conc.lhs), rather than change the current select() machinery in the non-threaded runtime (ghc/rts/posix/Select.c). Volker's libevent patch is for the latter, I believe.

comment:3 Changed 10 years ago by igloo

Milestone: 6.8
Test Case: N/A

comment:4 Changed 10 years ago by guest

Cc: Bulat.Ziganshin@… added

i strongly prefer to see this in separate library which isn't part of RTS

comment:5 Changed 9 years ago by AudreyTang

Should the libevent-based approach be investigated in the future, http://software.schmorp.de/pkg/libev.html has a more efficient implementation of the same (actually emulated) API -- see http://libev.schmorp.de/bench.html for benchmarks.

comment:6 Changed 9 years ago by simonmar

Milestone: 6.8 branch_|_

comment:7 Changed 9 years ago by PHO

Cc: pho@… added

comment:8 Changed 8 years ago by simonmar

Architecture: UnknownUnknown/Multiple

comment:9 Changed 8 years ago by simonmar

Operating System: UnknownUnknown/Multiple

comment:10 Changed 8 years ago by tibbe

Cc: johan.tibell@… added

comment:11 Changed 8 years ago by hajile

Cc: elijah.epifanov@… added

comment:12 Changed 7 years ago by bsdemon

Cc: 8mayday@… added

comment:13 Changed 7 years ago by guest

Cc: cgibbard@… added

comment:14 Changed 7 years ago by morrow

Cc: morrow@… added

comment:15 in reply to:  description Changed 7 years ago by lvh

Cc: lvh@… added

comment:16 Changed 7 years ago by YitzGale

Cc: gale@… added

comment:17 Changed 7 years ago by jystic

Cc: jystic@… added

comment:18 Changed 7 years ago by galdor

Cc: khaelin@… added

comment:19 Changed 7 years ago by simonmar

difficulty: Difficult (1 week)Difficult (2-5 days)

comment:20 Changed 7 years ago by simonmar

difficulty: Difficult (2-5 days)Project (more than a week)
Type of failure: None/Unknown

comment:21 Changed 7 years ago by nwn

Cc: nonowarn@… added

comment:22 Changed 7 years ago by bos

Owner: set to bos
Type of failure: None/UnknownRuntime performance bug

I've started work on this, and have the threaded RTS's IO manager thread successfully using either poll or select, depending on the system.

This step alone is a big deal, because select has a low fixed limit on the number of file descriptors it can manage, usually 1,024. Using poll instead removes that limit.

There is a lot more work to do, but this seems like a good start.

Changed 7 years ago by bos

Attachment: base.diff added

base.diff

comment:23 Changed 7 years ago by bos

I've attached the current working patch to this ticket. It applies cleanly against both base 4.2.0.0 (as used in GHC 6.12.1) and the version used by HEAD.

Simon, I think this patch is worth applying on top of base-4.2.0.0 as is, since it removes that pesky dependency on select. This would be a nice fix to have in GHC 6.12.2.

For more substantial changes (use of epoll or kqueue), the amount of work involved is much higher.

comment:24 Changed 7 years ago by simonmar

This does look like a step in the right direction. How does this compare in performance to the select version? Glancing at the patch it looks like building the fdset will be more expensive than before. There's one withForeignPtr per FD. The INLINEs on the C functions don't have any effect (unfortunately), even with -fvia-C.

comment:25 Changed 7 years ago by bos

There is indeed one withForeignPtr per FD. ISTR I'd been told that was effectively free, so I didn't worry about it. Clearly, we could use a plain Ptr instead and just free the pollfd data by hand when shutting down.

I'll have benchmark numbers within a couple of days. I seem not to have the flu finally, so I have to do Real Work again.

comment:26 in reply to:  25 Changed 7 years ago by simonmar

Replying to bos:

There is indeed one withForeignPtr per FD. ISTR I'd been told that was effectively free, so I didn't worry about it. Clearly, we could use a plain Ptr instead and just free the pollfd data by hand when shutting down.

You're right - withForeignPtr should be free, as long as the ForeignPtr is strict. Probably wouldn't hurt to glance at the Core and check it is being inlined away though.

comment:27 Changed 7 years ago by kazu-yamamoto

Cc: kazu@… added

comment:28 Changed 7 years ago by tibbe

We've been working on a replacement for the I/O manager for a few weeks now. The latest code can be found at:

http://github.com/tibbe/event

comment:29 Changed 7 years ago by kazu-yamamoto

It seems to me that System.Event enables event-driven network programming on kqueue/epoll but does not enable thread programming (with Handle). Are there any plan to integrate the result of System.Event to GHC.Conc?

comment:30 in reply to:  29 Changed 7 years ago by tibbe

Replying to kazu-yamamoto:

It seems to me that System.Event enables event-driven network programming on kqueue/epoll but does not enable thread programming (with Handle). Are there any plan to integrate the result of System.Event to GHC.Conc?

Yes. System.Event.Thread implements the needed functionality.

comment:31 Changed 7 years ago by kazu-yamamoto

OK. Does patch to GHC.Conc already exist? If so, I would like to test it.

comment:32 in reply to:  31 Changed 7 years ago by tibbe

Replying to kazu-yamamoto:

OK. Does patch to GHC.Conc already exist? If so, I would like to test it.

Not yet. Bryan said he would look into integrating the work. The whole event library have to be moved into the GHC repo. You can test it out right now. Have a look at the examples in the benchmarks directory e.g. PongServer.

comment:33 Changed 7 years ago by kazu-yamamoto

I have already looked the benchmarks directory. "dead-conn" cannot be complied due to link problems on Mac (Leopard). "pong-server" causes resource exhauseted. (I carefully unlimited descriptors.) % ./pong-server pong-server: accept: resource exhausted (Too many open files)

"event-http" and "static-http" works well. "event-http" is faster than "static-http".

Anyway, since I want to write thread code using Handle on epoll/kqueue, not event-driven code using Socket, I cannot use System.Event for my program at this moment.

comment:34 in reply to:  33 ; Changed 7 years ago by tibbe

Replying to kazu-yamamoto:

I have already looked the benchmarks directory. "dead-conn" cannot be complied due to link problems on Mac (Leopard).

If you send me an email with the error I can try to reproduce it (I'm also on OS X).

"pong-server" causes resource exhauseted. (I carefully unlimited descriptors.) % ./pong-server pong-server: accept: resource exhausted (Too many open files)

There are a number of kernel settings that need to be changed to run some of these benchmarks. All of the settings are done automatically for you by the benchmark_shell.py script. It can be used like so:

sudo benchmarks/benchmark_shell.py benchmarks/pong-server

Anyway, since I want to write thread code using Handle on epoll/kqueue, not event-driven code using Socket, I cannot use System.Event for my program at this moment.

If you want to write production code you better wait for GHC 6.14 which is our goal for integrating the code into GHC.

If you want to play with the code using forkIO style concurrency (but using Sockets instead of Handles) you can you System.Event.Thread and benchmarks/EventSocket.hs.

comment:35 in reply to:  34 ; Changed 7 years ago by kazu-yamamoto

If you send me an email with the error I can try to reproduce it (I'm also on OS X).

The error is attached.

There are a number of kernel settings that need to be changed to run some of these benchmarks. All of the settings are done automatically for you by the benchmark_shell.py script. It can be used like so:

Thanks!

If you want to play with the code using forkIO style concurrency (but using Sockets instead of Handles) you can you System.Event.Thread and benchmarks/EventSocket.hs.

I read it before. But I still want to use Handle for buffering.

The error: (Note I upgrade my Mac to Snow Leopard and GHC to 6.12.1)

% make
ranlib ../dist/build/libHSevent-0.1.a
ghc -funbox-strict-fields -package bytestring -ignore-package event -fno-ignore-asserts -Wall -fno-warn-orphans -fno-warn-missing-signatures -i../dist/build -package-name event-0.1  -package network -package network-bytestring -package attoparsec -package bytestring-show -package mtl -threaded -o dead-conn Args.o EventUtil.o EventSocket.o DeadConn.o ../dist/build/libHSevent-0.1.a
gcc -O2 -o deadconn deadconn.o
ld: warning: in deadconn.o, file is not of required architecture
Undefined symbols:
  "_main", referenced from:
      start in crt1.10.6.o
ld: symbol(s) not found
collect2: ld returned 1 exit status
make: *** [deadconn] Error 1

comment:36 in reply to:  35 Changed 7 years ago by tibbe

Replying to kazu-yamamoto:

The error: (Note I upgrade my Mac to Snow Leopard and GHC to 6.12.1)

% make
ranlib ../dist/build/libHSevent-0.1.a
ghc -funbox-strict-fields -package bytestring -ignore-package event -fno-ignore-asserts -Wall -fno-warn-orphans -fno-warn-missing-signatures -i../dist/build -package-name event-0.1  -package network -package network-bytestring -package attoparsec -package bytestring-show -package mtl -threaded -o dead-conn Args.o EventUtil.o EventSocket.o DeadConn.o ../dist/build/libHSevent-0.1.a
gcc -O2 -o deadconn deadconn.o
ld: warning: in deadconn.o, file is not of required architecture
Undefined symbols:
  "_main", referenced from:
      start in crt1.10.6.o
ld: symbol(s) not found
collect2: ld returned 1 exit status
make: *** [deadconn] Error 1

I'm not quite sure what this is, looks like a 32 vs 64-bit issue. If you don't need deadconn (which creates idle connections useful when benchmarking scaling properties) you can build the target you want using e.g. make pong-server.

comment:37 Changed 7 years ago by jeffz_

Cc: jeffz_ added

comment:38 Changed 7 years ago by PeteC

Cc: 1@… added
patch: 0

comment:39 Changed 7 years ago by merehap

Cc: merehap@… added

comment:40 Changed 7 years ago by gwicke

Cc: gabriel@… added

comment:41 Changed 7 years ago by basvandijk

Cc: v.dijk.bas@… added

comment:42 Changed 7 years ago by tibbe

Resolution: fixed
Status: newclosed
Version: 6.4.16.13

Done. It took a little longer than one week.

If you're experience incorrect behavior that you think is related to the new I/O manager please file a bug and assign it to me.

Sat Jul 24 12:23:55 CEST 2010  Johan Tibell <johan.tibell@gmail.com>
  * Integrate new I/O manager, with signal support
Tue Aug 10 10:22:48 CEST 2010  Simon Marlow <marlowsd@gmail.com>
  * Integrated new I/O manager
  (patch originally by Johan Tibell <johan.tibell@gmail.com>, minor merging by me)

comment:43 Changed 7 years ago by simonmar

Thanks for closing the bug, and well done to both you and Bryan!

Note: See TracTickets for help on using tickets.