Support double-byte encodings (Chinese/Japanese/Korean) on Windows

localeEncoding uses the console code page for text file encoding/decoding for single-byte encoding environment on Windows. But GHC.IO.Encoding.CodePage.Table doesn't have double-byte encodings (Chinese/Japanese/Korean), now. Its current state often causes problem on double-byte encoding environment.

• http://hackage.haskell.org/trac/hackage/ticket/658
• http://hackage.haskell.org/trac/hackage/ticket/659

I know we can solve problem by using hSetEncoding with utf8 or othere UTF-* encodings. But it's not good solution.

According to previous Windows patch, GHC.IO.Encoding.CodePage.Table doesn't support double-byte encodings because Windows' shared library support doesn't work.

• http://www.haskell.org/pipermail/cvs-libraries/2009-September/011289.html
• http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=packages/base;a=commitdiff;h=20090913022126-9f663-2bb505cf915b18313bf41a25853d9d13d7444feb.gz

Currently we do not support double-byte encodings (Chinese/Japanese/Korean), since
including those codepages would increase the table size to 400KB.  It will be
straightforward to implement them once the work on library DLLs is finished.

I think Windows' shared library support works now. Because #3879 (closed) is closed.

So, how about add supporting double-byte encodings (Chinese/Japanese/Korean) on Windows?

Trac metadata

Trac field	Value
Version	6.12.1
Type	FeatureRequest
TypeOfFailure	OtherFailure
Priority	normal
Resolution	Unresolved
Component	libraries/base
Test case
Differential revisions
BlockedBy
Related
Blocking
CC	shelarcy@gmail.com
Operating system
Architecture

mentioned in issue #3498 (closed)

changed weight to 5

added Tfeature request Trac import core libraries labels

Trac metadata

Trac field	Value
Version	6.12.1 → 6.13

changed milestone to %7.0.2

changed milestone to %7.2.1

We could fall back on MultiByteToWideChar for code pages that CodePage.hs doesn't know about. This won't bloat the binaries at all, though it can involve more overhead than our native-Haskell CodePage decoder if the character size is not 2 bytes (i.e. something other than UTF-16 encoded Stings).

I don't know how to use MultiByteToWideChar to implement TextEncoding, other than doing binary-chop when it fails. This is why we had to make our own CodePage implementations. Some more details here:

http://www.haskell.org/pipermail/libraries/2009-July/012077.html

I see. Still, having a dog-slow binary chop for these double-byte encodings is better than having no support at all.

Replying to [ticket:3977#comment:50351 batterseapower]:

I see. Still, having a dog-slow binary chop for these double-byte encodings is better than having no support at all.

Maybe. It would be tricky to get working though, and getting the error semantics right might be a pain (if it's possible at all). Are double-byte encodings widely used?

IME it is very common for PCs in China to be using one of the double-byte code page settings, since there is still a ton of software out there that uses the legacy Windows APIs. I imagine the situation is the same in Japan/Korea but I have no direct experience.

Of course, even if DBCS is commonly used the localeEncoding is not as useful on Windows as it is on *nix in the context of GHC since we will mostly just call into the *W APIs and thus sidestep the locale entirely... which probably makes this ticket a low priority. In particular my upcoming patch set to implement PEP383 behaviour should support CJK without having a double-byte-aware localeEncoding.

Trac metadata

Trac field	Value
CC	shelarcy@gmail.com →

Oops, I mistook operation. Sorry for noise.

Trac metadata

Trac field	Value
CC	→ shelarcy@gmail.com

changed milestone to %7.4.1

mentioned in issue #5754 (closed)

Trac metadata

Trac field	Value
Related	- → #5754 (closed)

changed milestone to %7.6.1

changed weight to 3

Trac metadata

Trac field	Value
Priority	normal → low

changed milestone to %7.6.2

assigned to @trac-batterseapower

Fixed on the dbcs branch of the base library.

changed milestone to %7.8.1

Is this now fixed in HEAD by this?:

commit f982978eaa5d74c5dffe71c14a1555587b6a5e48
Author: Max Bolingbroke <batterseapower@hotmail.com>
Date:   Thu Apr 18 21:29:08 2013 +0100

    Support for Windows DBCS and new SBCS with MultiByteToWideChar

changed milestone to %7.10.1

Moving to 7.10.1

Moving over to new owning component 'Core Libraries'.

Trac metadata

Trac field	Value
Component	libraries/base → Core Libraries

closed

removed milestone

commit 2216b897

Author: Max Bolingbroke <>
Date:   Thu Apr 18 21:29:08 2013 +0100

    Support for Windows DBCS and new SBCS with MultiByteToWideChar
    
    Because MultiByteToWideChar/WideCharToMultiByte have a rather unhelpful
    interface, we have to use a lot of binary searching tricks to get them
    to match the iconv-like interface that GHC requires.
    
    Even though the resulting encodings are slow, it does at least mean that we
    now support all of Window's code pages. What's more, since these codecs are
    basically only used for console output there probably won't be a huge volume
    of text to deal with in the common case, so speed is less of a worry.
    
    Note that we will still use GHC's faster table-based custom codec for supported
    SBCSs.

commit 802e99a5

Author: Max Bolingbroke <>
Date:   Tue Apr 23 19:15:02 2013 +0100

    Add comprehensive test for codepage encodings+recovery for them

Trac metadata

Trac field	Value
Resolution	Unresolved → ResolvedFixed
CC	shelarcy@gmail.com → core-libraries-committee@haskell.org, shelarcy@gmail.com

added incorrect runtime result label

added Plow label