• sci.lang.japan FAQ
• 14. Computers

• Previous: How is Japanese input on a computer?
• Next: What is half-width katakana?

Encodings of Japanese

By Alexandre Elias (modified for the sci.lang.japan FAQ)

Contents

1. Introduction
1. Goal of this document
2. Japanese writing
3. Character set vs. encoding
4. Overview of the encoding schemes
2. JIS
1. JIS character sets
1. JIS X 0201
2. JIS X 0208
3. JIS X 0212
4. JIS X 0213
5. NEC/IBM proprietary extension
2. JIS-based Encodings
1. EUC-JP
2. Shift JIS
3. CP932
4. ISO-2022-JP
3. Unicode
1. Endianness and the BOM
2. Unicode-based encodings
1. UTF-8
2. UTF-16
3. UTF-32
4. Punycode
3. Dead Unicode encodings
1. UTF-7
2. SCSU
3. Other dead encodings
4. Encodings in the wild
1. Conversion issues
2. Conversion software
1. Microsoft Windows
2. iconv
3. Perl
4. nkf
3. On the web
1. Web page encodings
2. HTML entities
4. In e-mail
1. Email headers
5. Tables
1. JIS Japanese Character Sets
2. Encoding schemes for Japanese characters
3. Shift JIS conversions
6. Web links

1. Introduction

1.1. Goal of this document

This document introduces how Japanese text is encoded on computers.

1.2. Japanese writing

Japanese has three main writing systems: hiragana, katakana and kanji. Roman letters (a, b, c ...), arabic numerals (0, 1, 2 ...) and various punctuation marks (。, !, ?, 「, 」...) (see What are the names of the Japanese non-kana, non-kanji symbols?) are also found in Japanese writing. All of these different writing systems can be found in the same sentence. Here is an example:

UNIX での日本語文字コードを扱うために使用されている従来の EUC は次のようなものでした。
This means "The conventional EUC encoding used to handle Japanese character codes on Unix was as follows."

Each hiragana, katakana, or kanji character is square and of similar size.

Japanese was traditionally written in columns, from top to bottom, with succeeding columns going right to left. See Can Japanese be written right to left? Nowadays, Japanese is often written in rows going from left to right, like English. Apart from word processing and desktop publishing programs, left-to-right writing is almost always used on computer displays.

Hiragana and katakana together are known collectively as the "kana". There are about fifty hiragana and fifty katakana. Each hiragana has a katakana equivalent: they are analogous to small and capital letters. Unlike roman letters, their size does not vary. Hiragana have a smooth curvy appearance, whereas katakana are blocky and jagged. Each kana corresponds to one syllable. Here are a few kana:

The kanji are a complex ideographic writing system from China. See Kanji They remain very similar to Chinese to this day. The kanji are unified with Chinese and Korean in the Unicode character set. There are thousands of kanji in common use in Japan. The quantity of kanji makes Japanese writing difficult both to learn and to encode on computers. Kanji can be recognized by their complex, intricate appearance: for example, 通, 飛, and 治 are common kanji. But a few kanji look simple, such as 人 and 女.

Because the kana can express all possible sounds in Japanese, it is possible to write any Japanese sentence using only one of the two kana writing systems. Thus, the earliest standard Japanese character set for computers (JIS X 0201, described later) supported only katakana, which, although inconvenient, was sufficient.

1.3. Character set vs. encoding

A character set is a one-to-one mapping between a set of distinct integers and a set of written symbols. For example, define a new character set FOOBAR that maps the alphabet {A, B, C} to the digits 1, 2, and 3, respectively. A character set is an abstract concept that exists only in the mind of the programmer: computers do not directly manipulate character sets.

An encoding is a way characters are stored into 0s and 1s of computer memory. To implement FOOBAR support on a real computer, the most obvious way to encode data would be to represent one character per byte, following the usual way of encoding integers in binary. In this scheme, the string "AABC" would become:

00000001 00000001 00000010 00000011

01011011

1.4. Overview of the encoding schemes

In real life, encodings tend to multiply uncontrollably as implementers accommodate the quirks in their systems, but character sets remain few. In the English-speaking world, the only two character sets on the radar are ASCII and EBCDIC. Similarly, there are only two character sets used to write Japanese: the JIS (Japanese Industrial Standard) character set and Unicode. (And the Japanese part of Unicode is actually derived from JIS.)

Unicode is the new, superior standard. Japanese computers have been using JIS for decades, and Unicode has only appeared in the past few years. More and more Japanese text is written in Unicode, but it is still necessary to deal with both JIS and Unicode text.

As needs have evolved, both standards have undergone several revisions. This is mainly a problem with JIS, since its revision process has been somewhat chaotic. In contrast, Unicode follows a policy that each new revision must be a strict super-set of previous ones, so version conflicts rarely cause problems.

There are three JIS encodings (Shift JIS, EUC, ISO-2022-JP) and three Unicode encodings (UTF-8, UTF-16, UTF-32) in widespread use. In a nutshell:

• Shift JIS is the Microsoft encoding of JIS, standard on Windows and Mac systems. Almost all Japanese web pages used to be encoded in Shift JIS.
• EUC (EUC-JP) is the Unix encoding of JIS. It used to be standard on Linux/BSD systems, and is sometimes used on web pages.
• ISO-2022-JP is the 7-bit JIS encoding, that e-mails are usually sent in. It is rarely used in other contexts.
• UTF-8 is the Unicode encoding standard in Unix and on the internet, and UTF-16 the Unicode encoding standard in Windows. UTF-32 is mostly used for internal representation inside programs, and not for interchange.

2. JIS

JIS stands for "Japanese Industrial Standard", in Japanese Nihon Kōgyō Kikaku (日本工業規格). JIS is a blanket term used to describe all non-Unicode Japanese character sets and encodings, all of which are based on standards published by the Japanese Standards Association, Nihon Kikaku Kyōkai (日本規格協会) in Japanese.

2.1. JIS character sets

Unlike Unicode, there is no single JIS character set. A JIS encoding actually involves several standard character sets used in combination:

• JIS X 0201 - Roman characters (a, b, c ...) and half-width katakana only. Standard prefix: JG.
• JIS X 0208 - Core character set. 6355 kanji from level 1 and 2, 524 kana/punctuation/etc. Standard prefix: JH.
• JIS X 0212 - Supplemental characters. 5801 rare kanji, 266 non-English European characters. Standard prefix: JJ.
• JIS X 0213 - New unified character set. Sometimes called JIS2000, introduced in the year 2000, and meant to replace the 0208/0212 combination. It has no standard prefix.

The "X" in the names of these standards is a code for information technology related standards.

JIS encodings employ various methods to use these overlapping character sets together.

The above four JIS standards are the important ones, but for reference, here is the meaning of all the JIS X 200 codes:

JIS X 0201 - Roman/katakana (JG)
JIS X 0202 - ISO-2022-JP
JIS X 0203, 0204, 0205, 0206, 0207 - Obsolete/withdrawn standards
JIS X 0208 - Main kanji character set (JH)
JIS X 0209 - How to write JIS X 0211 characters
JIS X 0210 - How to encode numbers
JIS X 0211 - Standard ASCII control codes
JIS X 0212 - Supplemental character set (JJ)
JIS X 0213 - New unified JIS character set
JIS X 0218 - Definition of standard prefixes (JG, JH, JJ, JK, JL)
JIS X 0221 - Unicode (JK for UCS-2, JL for UCS-4)

2.1.1. JIS X 0201

JIS X 0201 is an eight-bit character set which supports only katakana and the ASCII characters. JIS X 0201 hexadecimal character codes can be prefixed with "JG" to distinguish them from other JIS character sets. It was designed in the 1960s (long before the other standards), when computers were not powerful enough to store kanji.

The 7-bit part (i.e. 0x00 to 0x7f) of JIS X 0201 is identical to ASCII, with two exceptions: the backslash character '\' (0x5c) is replaced by a yen symbol, and the tilde character '~' (0x7c) becomes an overline. Thus, displaying ASCII text in a Japanese font will work almost perfectly — except that all backslashes will turn into yens. This problem is so pervasive that it is no longer really a "problem" at all: Japanese people all know and expect to find yen symbols instead of backslashes in such things as DOS/Windows path separators.

This part of this encoding is also known as JIS Roman.

The 8-bit part is divided in this fashion:

0x80 to 0xa0 inclusive: Reserved
0xa1 to 0xdf inclusive: Japanese-style punctuation and half-width katakana
0xe0 to 0xff inclusive: Reserved

The katakana are "half-width", in other words tall and thin, because makes them the same size as roman letters, and thus easy to display on fixed-width terminals. See What is half-width katakana?

2.1.2. JIS X 0208

JIS X 0208 is the most important standard. When people say "the JIS standard", they mean JIS X 0208. JIS X 0208 hexadecimal character codes can be prefixed with "JH" to distinguish them from other JIS character sets. It has gone through four official versions from the Japanese Standards Association.

Revision history:

The 1983, 1990 and 1997 standards are essentially the same, being close supersets of each other. They are collectively known as "New-JIS", or mostly just "JIS". Thus it is possible to talk about "JIS X 0208" without mentioning the year. The kanji included in these standards are the JIS level one and two kanji. See What are JIS level one and two kanji?

JIS X 0208 is set up as a 2-dimensional, 94x94 grid. The position of a character on this grid is called its "kuten" (区点). The horizontal lines on the grid are:

This 94x94 grid fits between 33 and 126 inclusive, almost completely overlapping the non-control part of ASCII. The hexadecimal JIS code is a 16-bit code resulting from adding 32 to both the ku and the ten, and concatenating the two resulting bytes, the vertical coordinate becoming the high byte. For example, the kanji 私, watashi, "I", has a kuten code of 27-68. The "ku" part is 27, giving a byte value of 27 + 32 = 59, which is 3B in hexadecimal, and the "ten" part is 68, giving a byte value of 68+32 = 100, which is 64 in hexadecimal, thus the hexadecimal JIS code is 3B64.

The kanji in JIS X 0208 are enough for the vast majority of writing, but every so often a rarer kanji is needed (to write names especially). This is why the following standards exist.

2.1.3. JIS X 0212

JIS X 0212 was introduced in 1990 to accommodate the demand for rare kanji. JIS X 0212 hexadecimal character codes can be prefixed with "JJ" to distinguish them from other JIS character sets. It is meant to be used in the same encoding alongside JIS X 0208. It contains 5801 obscure level 3 kanji. Even educated native Japanese people will not be familiar with most of them.

Like JIS X 0208, it is organized in a 94x94 grid. Here is a description of each horizontal line on the grid:

Because of its overlap with JIS X 0208, encodings have to jump through hoops to use both JIS X 0208 and JIS X 0212 in the same text. Moreover, it leads to confusion. JIS kanji 0x6666 might be JH6666 or JJ6666. The only nice thing is that, though the kanji don't, at least the non-kanji parts of JIS X 0208 and JIS X0212 occupy disjoint code ranges.

2.1.4. JIS X 0213

JIS X 0213 is the successor to JIS X 0208. It is sometimes called JIS2000 (having been standardized in the year 2000). It is not yet in wide use: for now, it can be ignored. However, for future reference, here is a brief description.

The design of JIS X 0213 is clever. The new characters in JIS X 0213 are mostly kanji and a few miscellaneous other characters. The standard is divided into two parts. Level 3 kanji in JIS X 0213 are said to be in "Plane 1", and Level 4 kanji are said to be in "Plane 2". Both planes are a 94x94 grid.

Here is the trick: Plane 1 kanji occupy the unused codespace in JIS X 0208, whereas Plane 2 kanji occupy the unused codespace in JIS X 0212. To visualize what is happening, imagine printing out the JIS X 0208 grid, the JIS X 0212 grid, the JIS X 0213 plane 1 grid and the JIS X 0213 plane 2 grid on four separate sheets of transparent plastic. Then place the JIS X 0213 plane 1 sheet over the JIS X 0208 sheet without any overlapping characters, and place the JIS X 0213 plane 2 sheet over the JIS X 0212 sheet with no overlap.

To accommodate JIS X 0213, all 3 major JIS encodings have undergone revision. Their new MIME names are EUC-JISX0213, Shift_JISX0213, and ISO-2022-JP-3. Fortunately, because of the clever design of JIS X 0213, the new encodings are only slightly different from their original version.

2.1.5. NEC/IBM proprietary extension

NEC/IBM defined roughly a thousand new characters for use with CP932 (code page 932), Microsoft's proprietary extension of the Shift JIS encoding. These cause no end of problems, even having duplicates with standard characters. As these characters are tightly bound to CP932, they are described in CP932.

2.2. JIS-based Encodings

2.2.1. EUC-JP

EUC (Extended Unix Code), also known as UJIS, is an encoding which encodes all the characters of JIS X 0201, JIS X 0208 and JIS X 0212. It is compatible with ASCII, but not with JIS X 0201: EUC does not support 1-byte half-width katakana/punctuation (though it does support it in 2 bytes).

Similar to UTF-8, EUC has the property that ASCII characters are left as-is, and every other character has the top bit of each of its bytes set.

Each character set is encoded as follows:

Note, though, that it may be a bad idea to use characters in JIS X 0212 with EUC, as some software might not recognize them. Because almost all characters in EUC take up only 2 bytes, it is all too easy for careless programmers to build software that will break when it encounters a 3-byte EUC character.

EUC's standard MIME label is "EUC-JP".

2.2.2. Shift JIS

Shift JIS is an encoding of the JIS standard which was the standard encoding for Japanese on Microsoft and Apple computers before the advent of Unicode. The selling point of Shift JIS (a.k.a. SJIS) is that, unlike EUC, it is backwards-compatible with not only ASCII, but also JIS X 0201, so Shift JIS can be used to encode both JIS X 0201 and JIS X 0208 (but not JIS X 0212). One-byte half-width katakana/punctuation is valid Shift JIS. Unfortunately, this compatibility means that Shift JIS is the messiest encoding of all.

The first byte of a Japanese character in Shift JIS always has the top bit set and, to permit its use mingled with JIS X 0201 characters, avoids the high JIS X 0201 range of 0xA0 to 0xDF. The second byte's top bit is not necessarily set. It can go all the way from 0x40 to 0xFC (overlapping with ASCII).

The algorithm to convert JIS codes into Shift JIS is complicated, and can't be explained in a nutshell. It involves dividing things by 2 and adding arbitrary magic numbers. See Shift JIS conversions for a precise description.

Shift JIS's standard MIME label is "Shift_JIS".

2.2.3. CP932

CP932 (code page 932) is an extension of Shift JIS from Microsoft. It adds the NEC/IBM extended characters. This extension is:

NEC special characters (83 characters in SJIS row 13),
NEC-selected IBM extended characters (374 characters in SJIS rows 89..92),
and IBM extended characters (388 characters in SJIS rows 115..119).

2.2.4. ISO-2022-JP

The most widely supported encoding for e-mail is the 7-bit ISO-2022-JP standard, which has been used for Japanese e-mail since the beginning. This encoding is almost certain to be understood by a Japanese recipient. It has also been standardized by the JSA under the name "JIS X 0202".

ISO-2022-JP is essentially a mix of plain ASCII and raw, 7-bit JIS. Like all JIS encodings, 16-bit characters are encoded in big-endian byte order. To distinguish between conflicting character sets, escape codes are used.

There are 3 revisions of ISO-2022-JP. Their MIME labels are "ISO-2022-JP", "ISO-2022-JP-1", and "ISO-2022-JP-2".

ISO-2022-JP - Supports ASCII and JIS X 0208.
ISO-2022-JP-1 - Also supports JIS X 0212.
ISO-2022-JP-2 - Also supports other languages like Chinese and Greek.

The original ISO-2022-JP will be good enough most of the time. For maximum compatibility, prefer it to the others.

Here are the relevant escape sequences:

The text begins in ASCII by default, and it must be switched back to ASCII at the end. It is also recommended for newlines to always be encoded in ASCII. If this is e-mail, the escape codes must be used in the Subject: or From: lines if they contain Japanese, again switching back to ASCII when done.

ESC ( B should be preferred over ESC ( J. The latter is a legacy code whose use is discouraged today. Also, avoid ESC ( I unless half-width katakana are wanted.

By adding escape codes by hand, it is possible to easily send ISO-2022-JP encoded e-mail with any e-mail client, even if it has no support at all for Japanese.

ISO-2022-JP-2 has a variety of other escape codes, having been extended to support random other languages. See RFC1554 for details.

3. Unicode

Unicode is a character set begun circa 1990. It intends to assign a unique code to every character in every living writing system. Unicode, the character set, is sometimes called UCS (Universal Character Set). Encodings of Unicode are called UTF-something. Pedants make the distinction but both these terms are interchangeable in casual speech.

Many people, when they first hear about Unicode, assume that Unicode encoding is as simple and clean as ASCII, except that each character code maps to 2 bytes instead of just 1. This is wrong. Unicode encodings are much more complicated than this. Making it as simple as ASCII was the naive idea when Unicode was first conceived, but then reality caught up with the standard. There are now no less than 3 popular encodings for Unicode.

Unlike JIS, UCS is not organized in a grid. It is a flat, one-dimensional sequence, with character codes ranging from 0 to 0x10FFFF. A code in this range can almost, but not quite, fit into 20 bits of storage. Each writing system has its own range of codes.

The creators of Unicode thought when they started that 16 bits would be enough to contain all useful writing systems, but they were wrong. The latest Unicode standard goes up to (a little more than) 20 bits, and a kludge was designed to the new high-plane characters in what was previously 16-bit only text (UTF-16, described below). Unicode is now separated into 17 planes, from Plane 0 to Plane 16, the plane number coming from the value of the top 4 bits.

Here are the character ranges of interest to us:

3.1. Endianness and the BOM

Unlike all the JIS-based standards, which enforce big-endian encoding, Unicode also panders to the little-endian people. Although endianness problems don't appear in UTF-8, which is 8-bit based, they do in UTF-16 and UTF-32, both of which can be either big-endian or little-endian.

Endianness, or byte order, means the order of the bytes when a 16-bit or higher integer is changed into a series of 8-bit bytes. There are two commonly used orders:

Big-endian:
0x1234 -> 0x12, 0x34
0x12345678 -> 0x12, 0x34, 0x56, 0x78

Little-endian:
0x1234 -> 0x34, 0x12
0x12345678 -> 0x78, 0x56, 0x34, 0x12

Thus, both UTF-16 and UTF-32 have 3 MIME labels: "UTF-16", "UTF-16BE", "UTF-16LE", and "UTF-32", "UTF-32BE", "UTF-32LE". "BE" and "LE" stand for big-endian and little-endian, respectively.

When one of the qualified MIME labels is used (and this is preferable), there is no ambiguity, and that is the end of the discussion. But when the ambiguous "UTF-16" and "UTF-32" labels are used (or when the Unicode is found in a file with no external meta-data), the byte order can be either. How can they be told apart?

The answer is the BOM (Byte Order Mark), a two-byte (in UTF-16) or four-byte (in UTF-32) code which can optionally be put at the beginning of a serialization (i.e. file) to specify its endianness. The BOM is considered meta-data, and not part of the actual Unicode text.

The BOM has 4 different forms:

The code U+FEFF (what the BOM would be if it was a real Unicode character instead of meta-data) is the ZERO WIDTH NO-BREAK SPACE. Not by coincidence, this is an invisible character which does absolutely nothing, so that putting a BOM outside the first character of a file by mistake (for example, by naively concatenating two Unicode files together) will generally have no serious consequences. The code 0xFEFF is always interpreted as a BOM in the first position of the file, and always interpreted as a ZERO WIDTH NO-BREAK SPACE anywhere else.

If the file has no BOM, it is necessary to ask the user or use some other heuristic.

3.2. Unicode-based encodings

3.2.1. UTF-8

UTF-8 has the following properties:

• UCS characters U+0000 to U+007F (ASCII) are encoded as bytes 0x00 to 0x7F, thus it is compatible with ASCII. Files and strings which contain only 7-bit ASCII characters have the same encoding in ASCII and UTF-8.
• All UCS characters >U+007F are encoded as a sequence of several bytes, each of which has the most significant bit set. Therefore, no ASCII byte (0x00-0x7F) can appear as part of any other character.
• The first byte of a multibyte sequence that represents a non-ASCII character is always in the range 0xC0 to 0xFD and it indicates how many bytes follow for this character. All further bytes in a multibyte sequence are in the range 0x80 to 0xBF.
• All possible 2^31 UCS codes can be encoded.
• A UTF-8 encoded character may theoretically be up to six bytes long, however the 16-bit Basic Multilingual Plane characters are only up to three bytes long.
• The sorting order of big-endian UCS-4 byte strings is preserved.
• The bytes 0xFE and 0xFF are never used in the UTF-8 encoding.

The encoding is as follows. Take the UCS code in big-endian order and map its bits to the 'x'es in the following table:

U-00000000 - U-0000007F: 0xxxxxxx
U-00000080 - U-000007FF: 110xxxxx 10xxxxxx
U-00000800 - U-0000FFFF: 1110xxxx 10xxxxxx 10xxxxxx
U-00010000 - U-001FFFFF: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

UTF-8 has no endianness problems. UTF-8 uses "UTF-8" as standard MIME label.

3.2.2. UTF-16

UTF-16 (a.k.a. UCS-2) is the "obvious" encoding of Unicode. Each character code of each Plane 0 character is directly translated to a 16-bit integer.

Before the addition of the new planes, this was the whole story, but now to access them, the reserved codespace of Unicode found between 0xD800 and 0xDFFF inclusively is used. There are no characters in this range. There are eleven bits of free codespace available here.

If plane 0 is omitted, any Unicode character in the higher planes can be expressed in 20 bits. Thus any higher-plane character can be expressed using two 16-bit codes between 0xD800 and 0xDFFF, of which the 10 lowest bits are used in each to encode half the character — always in big-endian order (the high 10 bits are in the first code), regardless of the endianness of the file itself.

The Unicode character number is not encoded directly in these 2 sets of 10 bits: first subtract 0x10000. So Plane 0 cannot be used with this scheme, but Plane 16 becomes available.

The unused 11th bit (0x0400) is used to increase the robustness of UTF-16 text. In the first 16-bit code, the 11th bit is always 0, and in the second 16-bit code, it is always 1. That way, if a file is truncated in the middle of a higher-plane character, a program can tell that it is reading the second half, and not corrupt the character after it.

UTF-16 is affected by endianness issues. See Endianness and the BOM for details.

UTF-16 was adopted by Microsoft for use in their APIs.

UTF-16 has three possible MIME labels. They are "UTF-16BE" for big-endian byte order, "UTF-16LE" for little-endian, and just "UTF-16" for ambiguous (not recommended).

See RFC2781 for an official standard.

3.2.3. UTF-32

UTF-32 (a.k.a. UCS-4) is the simplest of all the Unicode encodings, but also the least efficient. In UTF-32, every character is a direct translation of its Unicode character code to a 32-bit integer. The only tricky part is the endianness issues (see Endianness and the BOM).

UTF-32 is not widely used for interchange because it is very wasteful of space. UTF-16 always outperforms or matches the space performance of UTF-32, usually by a factor of 2. UTF-8 can outperform it by a factor of 4. In any given UTF-32 file, most high bits will be all zeroes, because the vast majority of Unicode characters (including kanji) are in Plane 0, the one that can be encoded with only 16 bits. However, its simplicity makes it very appropriate for internal representation inside programs, which is the main justification for its existence.

UTF-32 has three possible MIME labels. They are "UTF-32BE" for big-endian byte order, "UTF-32LE" for little-endian, and just "UTF-32" for ambiguous (not recommended).

See the Unicode Standard Annex #19 for the official standard.

3.2.4. Punycode

Punycode is a method whereby Unicode characters can be encoded for use in domain names containing non-ASCII characters. See RFC3492.

3.3. Dead Unicode encodings

These encodings never really caught on.

3.3.1. UTF-7

UTF-7's major property and reason for existence is that it is 7-bit, and thus adequate for e-mail. But because MIME Content-Transfer-Encodings are good enough to use 8-bit encodings in e-mail, this never caught on.

UTF-7, like its JIS sibling ISO-2022-JP, has the property that most ASCII text can be plunked right into it without modification, and an escape character is used to indicate the beginning of "real" Unicode. But instead of using the ESC control code to do this, UTF-7 simply uses the + character. The text following the + is big-endian UTF-16 encoded in a close variant of Base64.

(In a nutshell, Base64 encodes 24 bits into 4 bytes, spreading it out to 6 bits per byte. Each of these bytes is mapped into the following alphabet:

       Value Encoding  Value Encoding  Value Encoding  Value Encoding
           0 A            17 R            34 i            51 z
           1 B            18 S            35 j            52 0
           2 C            19 T            36 k            53 1
           3 D            20 U            37 l            54 2
           4 E            21 V            38 m            55 3
           4 F            22 W            39 n            56 4
           6 G            23 X            40 o            57 5
           7 H            24 Y            41 p            58 6
           8 I            25 Z            42 q            59 7
           9 J            26 a            43 r            60 8
          10 K            27 b            44 s            61 9
          11 L            28 c            45 t            62 +
          12 M            29 d            46 u            63 /
          13 N            30 e            47 v
          14 O            31 f            48 w         (pad) =
          15 P            32 g            49 x
          16 Q            33 h            50 y

UTF-7's variant is different only insofar as the pad character '=' is not used.)

When the UTF-16 is finished, the - character acts as terminator. Any character not in the Base64 list will also work as terminator, but only - is swallowed. Therefore, UTF-7 encoded Japanese text will look something like +6shGa5Hp-, or just +6shGa5Hp. An example given in the standard is that the Unicode sequence "Hi Mom -<WHITE SMILING FACE>-!" becomes "Hi Mom -+Jjo--!".

As this encoding is not widely supported and won't ever be, it should not be used in outgoing mail.

UTF-7 uses "UTF-7" as standard MIME label.

See RFC2152 for the official standard.

3.3.2. SCSU

Another failed encoding scheme whose only strong point is efficiency. SCSU is actually more of a compression scheme than an encoding. The gist of it is that it switches into small 128-character "windows" for the duration of a string, within which characters are encoded only by their offset to the beginning of the window rather than their entire code. This method is worthless for encoding kanji, which are spread out all over the character set. SCSU didn't catch on because more efficient alternatives such as using gzip already existed.

3.3.3. Other dead encodings

"UTF-1" and "UTF-7,5" may crop up. UTF-1 is an inferior encoding proposed in the early days of Unicode but never much used, now completely superseded by UTF-8. "UTF-7,5" is a variant of UTF-8, proposed long after UTF-8 had already become standard. It offers a few small advantages over UTF-8, but not enough to merit switching over to it. Neither of these will ever be seen in the wild.

4. Encodings in the wild

4.1. Conversion issues

The list of Japanese characters in Unicode was ripped straight from JIS, so JIS can be converted into Unicode without many problems. But don't expect perfectly sensible conversions. CP932, Microsoft's proprietary extension of Shift JIS, causes the most trouble. It has duplicates and bad mapping tables that don't do proper round-tripping. If something is converted from Unicode to CP932 and back, the end result is something different from what was started with.

4.2. Conversion software

4.2.1. Microsoft Windows

When using any conversion software on Microsoft Windows, be sure to specify CP932 for the encoding, rather than "Shift_JIS". The kanji and kana parts of the text will be the same, but symbols, such as circled numbers like ①, have different values in Shift JIS and CP932.

4.2.2. iconv

One of the best conversion tools is the iconv program, part of the libiconv library, distributed freely by GNU at gnu.org under the LGPL. It supports all the important encodings described in this document, even ones as obscure as UTF-7 and the new JIS X 0213 encodings. It can convert from any encoding to any other encoding. It is available as a command-line tool for Unix or as a C library for both Windows and Unixes. For example, to convert kanjidic from EUC into UTF-8:

iconv -f EUC-JP -t UTF-8 < kanjidic > kanjidic_utf8

4.2.3. Perl

The Perl programming language supports most encodings via its Encode module. For example, to convert kanjidic from EUC into UTF-8:

perl -CO -MEncode -pe '$_=decode("euc-jp",$_)' < kanjidic > kanjidic_utf8

4.2.4. nkf

On Windows, consider also nkf, the "Network kanji filter". It supports EUC, Shift JIS, ISO-2022-JP, and Unicode. To convert kanjidic from EUC into UTF-8:

nkf -E -w8 < kanjidic > kanjidic_utf8

4.3. On the web

Japanese web pages are mainly encoded in Shift JIS, EUC or UTF-8. In the past, Shift JIS was the most popular. EUC is found on some sites. Unicode is gradually gaining in popularity. In principle it would be possible to see ISO-2022-JP on web pages, since Internet Explorer supports it, but it is rare.

4.3.1. Web page encodings

The MIME label for the encoding is normally specified by the web server inside the HTTP headers. A typical HTTP header's content type looks like

Content-Type: text/html; charset=UTF-8

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

In fact, it can't hurt to use such a META tag in all Japanese HTML even if the web server is configured to send UTF-8. That will make it easier to switch to another web server, and someone looking at the source will immediately know which encoding is being used, without needing to resort to the rules of thumb in this document.

4.3.2. HTML entities

In addition to the above encodings, HTML allows one to enter Unicode characters by number, using only plain ASCII: for example &#x5B57; (including the semicolon) for 字, the character with the hex code 5B57. Or a decimal: &#23383; These are known as HTML entities. This is the most reliable method and nice for a few Japanese characters in an otherwise English document, but it can be painful to type.

To automatically convert all non-ASCII characters in a document into HTML entities with Perl,

perl -C7 -pe 's/([^\x00-\x7f])/"&#".ord($1).";"/ge;' < in.html > out.html

perl -C7 -pe 's/([^\x00-\x7f])/sprintf "&#X%X;", ord $1/ge;'

To reverse this process,

perl -C7 -pe 's/&#(\d+);/chr $1/ge;' < in.html > out.html

perl -C7 -pe 's/&#x([0-9a-f]+);/chr hex $1/gei;' < in.html > out.html

4.4. In e-mail

ISO-2022-JP uses only 7 bits and was designed specially for e-mail.

E-mail presents special difficulties, because (apparently) some archaic e-mail relays support only transfers of 7-bit data. If a message contains a byte where the 8th bit is set, it could potentially be corrupted or rejected. Since most of the above methods exploit the 8th bit of each byte, it might not work to naively put Japanese text encoded using one of them directly in an email.

However, this restriction can be circumvented. Email already provides means of getting around the 7-bit problem, for example for binary attachments. The MIME standard for e-mail headers provides, in addition to a means of specifying the text encoding, a "Content-Transfer-Encoding", a standard means of re-encoding text to avoid using the 8th bit. For example, to send a Shift JIS encoded e-mail, specify this in the header:

Content-Type: text/plain; charset=Shift_JIS
Content-Transfer-Encoding: base64

The text of the mail would first be encoded in Shift JIS, and then this encoded text would in turn be encoded in Base64 (see section 3.3.1 on UTF-7 for an explanation of Base64). This method could potentially be used to send mail using any of the codings described above. A detailed discussion of MIME content transfer encodings is outside of the scope of this document; see RFC1521 for the MIME standard.

So, if someone doesn't support ISO-2022-JP, try using this method alongside Shift JIS for e-mail. This is a good bet for Windows and Mac users, since Shift JIS is the standard encoding of those systems.

4.4.1. Email headers

Email headers in Japanese email with Japanese text are usually encoded in a form something like

=?iso-2022-jp?B?GyRCRnxLXDhs?=

=?<charset>?<encoding>?<data>?=

Please see this regarding copyright permissions for this content.

5. Tables

5.1. JIS Japanese Character Sets

• See What is the origin of the gojūon kana ordering? for more about the Japanese pronunciation gojuuon order.
• See What are kanji radicals? and What are the names of the kanji radicals? for more about the ordering of kanji by radical.

5.2. Encoding schemes for Japanese characters

1. Use reserved points for Level 1 and Level 2 Kanji.
2. Use reserved points for Supplemental Kanji.
3. Unofficial.
4. In addition, GB 2312, KS C 5601, ISO 8859-1 and ISO 8859-7 are usable.
5. Includes JIS X 0208 characters.
6. Incompatible with ISO-2022-JP-1 and ISO-2022-JP-2.
7. Non ISO-2022 system.
8. Excluding 0xXX7F.
9. Shifted JIS X 0213 Plane.2 entries only. No space for JIS X 0212.

5.3. Shift JIS conversions

This is included here for reference. Note, however, that it is better not to use the algorithmic method given here for actual conversions, but a table-based lookup, due to incompatibilities introduced in CP932.

Hi = (0x101 + Row) / 2;

if (Row & 1) {
    if (Cell < 64) {
        Lo = 0x3F + Cell;
    } else {
        Lo = 0x40 + Cell;
    }
} else {
    Lo = 0x9E + Cell;
}

Hi = (0x181 + Row) / 2;

if (Row < 8) {
    Hi = (0x1DF + Row) / 2;
} else {
    Hi = (0x1D9 + Row) / 2;
}

Hi = (0x19B + Row) / 2;

Above tables copyright (C) 2001 イオ (smile4me@ps.ksky.ne.jp), All Rights Reserved.

Please see this regarding copyright permissions for this content.

6. Web links

Requests For Comments (RFCs)

• RFC1468: Japanese Character Encoding for Internet Messages
• RFC1521: MIME (Multipurpose Internet Mail Extensions) Part One: Mechanisms for Specifying and Describing the Format of Internet Message Bodies
• RFC1554: ISO-2022-JP-2: Multilingual Extension of ISO-2022-JP
• RFC2047: MIME (Multipurpose Internet Mail Extensions) Part Three: Message Header Extensions for Non-ASCII Text
• RFC2152: A Mail-Safe Transformation Format of Unicode
• RFC2616: Hypertext Transfer Protocol -- HTTP/1.1
• RFC2781: UTF-16, an encoding of ISO 10646
• RFC3492: Punycode: A Bootstring encoding of Unicode for Internationalized Domain Names in Applications (IDNA)

Software

• Unicode Utilities at unicode.org
• libiconv at gnu.org (Free Software Foundation).
• Perl programming language.
• nkf (network kanji filter) at Sourceforge.
• An example of using the iconv library (in C)
• Detect Shift-JIS bytes in a stream of bytes (in C)

General

Unicode

References

• Understanding Japanese Information Processing by Ken Lunde

• sci.lang.japan FAQ
• 14. Computers

• Previous: How is Japanese input on a computer?
• Next: What is half-width katakana?

Copyright © 1994-2022 Ben Bullock

If you have questions, corrections, or comments, please contact Ben Bullock or use the discussion forum / Privacy policy

Book reviews	Convert Japanese numbers	Handwritten kanji recognition	Stroke order diagrams	Convert Japanese units