Opened 9 years ago

Last modified 7 days ago

#2189 new bug

hSetBuffering stdin NoBuffering doesn't work on Windows

Reported by: FalconNL Owned by:
Priority: normal Milestone:
Component: Core Libraries Version: 6.8.2
Keywords: hsetbuffering buffering buffer Cc: camio, igloo, Deewiant, ryani, sof, ddiaz, Artyom.Kazak@…, malaquias@…, dagitj@…, ryan.gl.scott@…, core-libraries-committee@…, Jedai, ekmett
Operating System: Windows Architecture: x86
Type of failure: None/Unknown Test Case:
Blocked By: Blocking:
Related Tickets: #11394, #13440 Differential Rev(s):
Wiki Page:

Description (last modified by bgamari)

The following program repeats inputted characters until the escape key is pressed.

import IO
import Monad
import Char

main :: IO ()
main = do hSetBuffering stdin NoBuffering
          inputLoop
          
inputLoop :: IO ()
inputLoop = do i <- getContents
               mapM_ putChar $ takeWhile ((/= 27) . ord) i

Because of the hSetBuffering stdin NoBuffering line it should not be necessary to press the enter key between keystrokes. This program works correctly in WinHugs (sep 2006 version). However, GHC 6.8.2 does not repeat the characters until the enter key is pressed. The problem was reproduced with all GHC executables (ghci, ghc, runghc, runhaskell), using both cmd.exe and command.com on Windows XP Professional.

Attachments (1)

read-vs-readfile.c (1.1 KB) - added by Deewiant 8 years ago.
read vs. ReadFile: handling of enter varies

Download all attachments as: .zip

Change History (53)

comment:1 Changed 9 years ago by igloo

difficulty: Unknown
Milestone: 6.8.3

Thanks for the report!

It works for me on Linux; I can't test on Windows at the moment.

comment:2 Changed 9 years ago by igloo

Priority: normallow

OK, I can reproduce the problem. At the Haskell level the problem is that asyncRead# isn't returning when a key is pressed; I haven't followed all the RTS stuff through to see where it goes wrong. Could be related to #806. I doubt we'll get to this for 6.8.3, so I'm marking it as low priority.

comment:3 Changed 9 years ago by simonmar

Priority: lownormal

This is important to folks here at Galois; I'll try to look at it before 6.8.3.

comment:4 Changed 9 years ago by judah

There is code in base/cbits/consUtils.c:set_console_buffering to disable Windows line buffering (by calling the Win32 function SetConsoleMode). However, the code in GHC.Handle which is supposed to call that function was #ifdef'ed out on mingw in the following commit:

http://cvs.haskell.org/cgi-bin/cvsweb.cgi/fptools/libraries/base/GHC/Handle.hs#rev1.14

From the comments in System.Posix.Internals:

-- 'raw' mode for Win32 means turn off 'line input' (=> buffering and
-- character translation for the console.) The Win32 API for doing
-- this is GetConsoleMode(), which also requires echoing to be disabled
-- when turning off 'line input' processing. Notice that turning off
-- 'line input' implies enter/return is reported as '\r' (and it won't
-- report that character until another character is input..odd.) This
-- latter feature doesn't sit too well with IO actions like IO.hGetLine..

I'm not that experienced with Windows programming, but maybe instead of changing the ConsoleMode, we could call the lower-level ReadConsoleInput/PeekConsoleInput when we want to read only one character at a time.

comment:5 Changed 9 years ago by igloo

Milestone: 6.8.36.10.1

comment:6 Changed 9 years ago by igloo

Component: GHCilibraries/base

comment:7 Changed 9 years ago by igloo

See also #2568.

comment:8 Changed 8 years ago by igloo

Milestone: 6.10.16.10.2

comment:9 Changed 8 years ago by camio

Cc: camio added

This is also very important for those of us working on FRP at Anygma since it precludes simple console-based FRP examples.

comment:10 Changed 8 years ago by igloo

Cc: igloo added
Priority: normalhigh

I'd also find this useful

comment:11 Changed 8 years ago by camio

Alistar Bayley noted this workaround in haskell-cafe

{-# LANGUAGE ForeignFunctionInterface #-}
import Data.Char
import Control.Monad (liftM)
import Foreign.C.Types

getHiddenChar = liftM (chr.fromEnum) c_getch
foreign import ccall unsafe "conio.h getch"
  c_getch :: IO CInt

comment:12 Changed 8 years ago by Deewiant

Cc: Deewiant added
Summary: hSetBuffer stdin NoBuffering doesn't seem to work in ghc 6.8.2 on Windows XPhSetBuffering stdin NoBuffering doesn't work on Windows

Unfortunately conio doesn't mix with ordinary IO, as I demonstrated in a response to that thread on glasgow-haskell-users.

As for the original problem, I think this'd take some work to solve: GHC would have to convert to the Win32 API for all of its IO on Windows. (That'd also help with #806.)

What currently happens is the following call chain, starting from hGetChar (a bit of documentation for any would-be fixers):

System.IO.hGetChar stdin

-- meanings of numbers: stdin FD, not a socket, offset 0, length 1
GHC.Handle.readRawBuffer "hGetChar" 0 0 <buffer> 0 1

GHC.Handle.asyncReadRawBuffer 0 0 <buffer> 0 1

GHC.Conc.asyncReadBA 0 0 1 0 <buffer>

-- offset got applied
GHC.Conc.asyncRead 0 0 1 <buffer>

-- asyncReadzh_fast in rts/PrimOps.cmm
asyncRead# 0 0 1 <buffer>

-- in rts/win32/AsyncIO.c
-- the new 0 signifies that this is a read and not a write
addIORequest(0, 0, 0, 1, <buffer>)

From there it ends up into the asynchronous IO work queue, whence it eventually gets picked up by IOWorkerProc (in rts/win32/IOManager.c). It notices that the workKind is WORKER_READ but not WORKER_FOR_SOCKET, so it does a plain read() call.

In the current situation that's basically fine, since the hSetBuffering never did anything and we're still in line buffered mode. In unbuffered mode, that'd be a problem, as the comment from System.Posix.Internals (which judah posted above) asserts: enter needs to be pressed twice. The fact that it gives '\r' instead of '\n' isn't such a big problem since it can be easily modified.

I'm attaching a C program which shows how the problem is avoided by using the Win32 API directly (in this case, ReadFile is fine) instead of the POSIX read: the latter requires two presses of enter, the former only one.

Since there seems to be no way of getting a Win32 HANDLE object from a C FILE* let alone a POSIX file descriptor, I believe that the only reasonable way of getting this and #806 to work reliably is to convert the whole IO subsystem to use the Windows API directly—starting from changing GHC.IOBase.Handle__.haFD from an FD to a HANDLE on Windows.

Changed 8 years ago by Deewiant

Attachment: read-vs-readfile.c added

read vs. ReadFile: handling of enter varies

comment:13 Changed 8 years ago by simonmar

First, I completely agree that we should be using the Win32 API directly instead of the POSIX compatibility layer. It's that way for historical reasons - I think it was easier to port the IO library to Windows in the first place by using the POSIX layer. As you say, #806 would be helped by that, but also the Windows side of #635, and #989.

The call sequence you posted only applies to the non-threaded RTS. With the threaded RTS, everything is in the IO library. We still end up calling read(), but directly from Haskell.

Incedentally, there is a way to get a HANDLE from a file descriptor: _get_osfhandle(fd). You might want to look at libraries\base\cbits\inputReady.c for some serious console hackery.

comment:14 Changed 8 years ago by Deewiant

Heh, I just noticed that _get_osfhandle is used about 30 lines down from the code I was looking at last time. Thanks for the heads up. Using it this might be fixable with a lot less work by just changing the read call in IOWorkerProc to a ReadFile.

The threaded RTS seems to end up in __hscore_PrelHandle_read in libraries/base/include/HsBase.h which is again just a read call and might be convertable to ReadFile without too much trouble.

comment:15 Changed 8 years ago by simonpj

Priority: highnormal

We'd like to see this solved, but it looks as if it'd be a lot of work for a relatively narrow case. Can anyone else help? Meanwhile, reducing priority.

Simon & Simon

comment:16 Changed 8 years ago by ryani

Cc: ryani added

comment:17 Changed 8 years ago by sof

I believe I have a trivial fix for this, and can forward for testing & commit, if still of interest.

comment:18 Changed 8 years ago by sof

Cc: sof added

comment:19 in reply to:  17 Changed 8 years ago by simonmar

Replying to sof:

I believe I have a trivial fix for this, and can forward for testing & commit, if still of interest.

Yes please!

comment:20 Changed 8 years ago by sof

ok, not futzing with making this a proper patch:

--- old-base/cbits/consUtils.c  2009-02-25 10:06:10.631125000 -0800
+++ new-base/cbits/consUtils.c  2009-02-25 10:06:10.646750000 -0800
@@ -25,10 +25,13 @@
     DWORD flgs = ENABLE_LINE_INPUT | ENABLE_ECHO_INPUT;

     if ( (h = (HANDLE)_get_osfhandle(fd)) != INVALID_HANDLE_VALUE ) {
+      /* Only for console-connected Handles */
+      if ( GetFileType(h) == FILE_TYPE_CHAR ) {
        if ( GetConsoleMode(h,&st) &&
-            SetConsoleMode(h, cooked ? (st | ENABLE_LINE_INPUT) : st & ~flgs) ) {
+            SetConsoleMode(h, cooked ? (st | flgs) : st & ~flgs)  ) {
            return 0;
        }
+      }
     }
     return -1;
 }

--- old-base/GHC/Handle.hs      2009-02-25 10:06:10.631125000 -0800
+++ new-base/GHC/Handle.hs      2009-02-25 10:06:10.646750000 -0800
@@ -1374,13 +1374,10 @@
           is_tty <- fdIsTTY (haFD handle_)
           when (is_tty && isReadableHandleType (haType handle_)) $
                 case mode of
-#ifndef mingw32_HOST_OS
-        -- 'raw' mode under win32 is a bit too specialised (and troublesome
-        -- for most common uses), so simply disable its use here.
+                   -- Note: we used to disable 'cooked' mode setting
+                   -- for mingw / win32 here, but it is now back on (and well
+                   -- behaved for Console-connected Handles.)
                   NoBuffering -> setCooked (haFD handle_) False
-#else
-                  NoBuffering -> return ()
-#endif
                   _           -> setCooked (haFD handle_) True

           -- throw away spare buffers, they might be the wrong size

i.e., reliably (un)setting the line buffering flags of the underlying Console handle (if there's one) takes care of this.

comment:21 Changed 8 years ago by simonmar

Owner: set to simonmar
Priority: normalhigh

comment:22 Changed 8 years ago by simonmar

Owner: changed from simonmar to igloo
Type: bugmerge

Patch applied:

Thu Mar  5 03:33:23 PST 2009  Simon Marlow <marlowsd@gmail.com>
  * FIX #2189: re-enabled cooked mode for Console-connected Handles on Windows

comment:23 Changed 8 years ago by igloo

Resolution: fixed
Status: newclosed

Merged

comment:24 Changed 8 years ago by igloo

Resolution: fixed
Status: closedreopened
Type: mergebug

We rolled back this patch: With it, it is not possible to type in ghci running from a cygwin window or a cmd window (but an msys window or SSHing into cygwin works fine).

Sigbjorn, any ideas on what the problem is and how to solve it?

comment:25 Changed 8 years ago by igloo

Milestone: 6.10.26.12.1
Owner: igloo deleted
Status: reopenednew

comment:26 Changed 8 years ago by simonmar

Priority: highnormal

comment:27 Changed 7 years ago by igloo

Milestone: 6.12.16.12 branch
Type of failure: None/Unknown

comment:28 Changed 7 years ago by igloo

Milestone: 6.12 branch6.12.3

comment:29 Changed 7 years ago by igloo

Milestone: 6.12.36.14.1
Priority: normallow

comment:30 Changed 6 years ago by ddiaz

Cc: ddiaz added

comment:31 Changed 6 years ago by igloo

Milestone: 7.0.17.0.2

comment:32 Changed 6 years ago by igloo

Milestone: 7.0.27.2.1

comment:33 Changed 6 years ago by Artyom.Kazak

Cc: Artyom.Kazak@… added

comment:34 Changed 6 years ago by romildo

Cc: malaquias@… added

comment:35 Changed 5 years ago by igloo

Milestone: 7.2.17.4.1

comment:36 Changed 5 years ago by igloo

Milestone: 7.4.17.6.1
Priority: lownormal

comment:37 Changed 5 years ago by pcapriotti

I've spent some time looking into this. To summarize the situation:

  1. setting a console handle to unbuffered mode breaks ghci input with cmd.exe and cygwin
  2. unbuffered console input doesn't work properly with newline characters when using the POSIX API
  3. I tried the simple C program above, and it works fine, but ESC characters (as in the original issue) are still not detected. Maybe the ESC key has some special behavior? Ctrl+[ does send a '\ESC' character with msys, but not with cmd.exe.

Possible solutions:

  • The best solution still seems to be to rewrite the whole IO subsystem using the Windows API. Is anyone working on this already, perhaps?
  • Alternatively, it might be possible to replace calls to read and write with ReadFile and WriteFile respectively. I think we need to replace both to ensure that buffering still works. The problem with this is that we need to carry around the windows handle, to avoid using _get_osfhandle at every call to read.

I'm not sure if any of those solutions would address problem 1 above.

comment:38 Changed 5 years ago by igloo

Milestone: 7.6.17.6.2

comment:39 Changed 4 years ago by dagit

Cc: dagitj@… added

comment:40 Changed 3 years ago by thoughtpolice

Milestone: 7.6.27.10.1

Moving to 7.10.1.

comment:41 Changed 2 years ago by thoughtpolice

Component: libraries/baseCore Libraries
Owner: set to ekmett

Moving over to new owning component 'Core Libraries'.

comment:42 Changed 2 years ago by thoughtpolice

Milestone: 7.10.17.12.1

Moving to 7.12.1 milestone; if you feel this is an error and should be addressed sooner, please move it back to the 7.10.1 milestone.

comment:43 Changed 2 years ago by RyanGlScott

Cc: ryan.gl.scott@… core-libraries-committee@… added

comment:44 Changed 2 years ago by ekmett

Owner: ekmett deleted

Relinquishing ownership of this ticket.

comment:45 Changed 2 years ago by Jedai

Cc: Jedai added

comment:46 Changed 19 months ago by thoughtpolice

Milestone: 7.12.18.0.1

Milestone renamed

comment:47 Changed 18 months ago by BalinKingOfMoria

Cc: ekmett added
Milestone: 8.0.17.10.3

comment:48 Changed 18 months ago by rwbarton

Milestone: 7.10.38.0.1

Definitely not going to happen for 7.10.3.

comment:49 in reply to:  48 Changed 18 months ago by BalinKingOfMoria

Replying to rwbarton:

Definitely not going to happen for 7.10.3.

In the meantime, are there any viable workarounds (that haven't since been rolled back)? Raw input would be very helpful on Windows.

comment:50 Changed 15 months ago by bgamari

Description: modified (diff)

comment:51 Changed 14 months ago by thomie

Milestone: 8.0.1

comment:52 Changed 7 days ago by simonpj

Note: See TracTickets for help on using tickets.