KANJIDIC Project: Difference between revisions

From EDRDG Wiki
Jump to navigation Jump to search
Line 39: Line 39:


{| class="wikitable"
{| class="wikitable"
|+ Caption: Kanjidic Information Fields
|+ Kanjidic Information Fields
|-
|-
! Field
! Field
! Description
! Kanjidic Code<br/>(if any)
! Kanjidic Code<br/>(if any)
! Group Entity
! Group Entity
Line 49: Line 48:
|-
|-
| Kanji
| Kanji
| The character itself
| none
| none
| literal
| literal
Line 56: Line 54:
|-
|-
| JIS code-point
| JIS code-point
|
|
|
| codepoint
| codepoint
Line 63: Line 60:
|-
|-
| Unicode code-point
| Unicode code-point
|
| U
| U
|codepoint
|codepoint
| cp_value cp_type="ucs"
| cp_value cp_type="ucs"
|  
|  
|-
| Radical (KangXi Zidian)
| B/C
| radical
| rad_value rad_type="classical"
| Where Nelson uses the KangXi radical this has a "B" code, otherwise it has a "C" code
|-
| Radical (Nelson)
| B
| radical
| rad_value rad_type="nelson_c"
|
|-
| Grade
| G
| misc
| grade
| The "grade" of the kanji. <br/>- G1 to G6 indicates the grade level as specified by the Japanese Ministry of Education for kanji that are to be taught in elementary school (1006 Kanji). These are sometimes called the ''kyōiku'' (education) kanji and are part of the set of ''jōyō'' (daily use) kanji;<br/>- G8 indicates the remaining ''jōyō'' kanji that are to be taught in secondary school (additional 1130 Kanji);<br/>- G9 and G10 indicate ''jinmeiyō'' ("for use in names") kanji which in addition to the ''jōyō'' kanji are approved for use in family name registers and other official documents. G9 (649 kanji, of which 640 are in KANJIDIC) indicates the kanji is a "regular" name kanji, and G10 (212 kanji of which 130 are in KANJIDIC) indicates the kanji is a variant of a ''jōyō'' kanji.
|-
| Frequency-of-use ranking
| F
| misc
| freq
| The 2,501 most-used characters have a ranking which expresses the relative frequency of occurrence of a character in modern Japanese. The data is based on an analysis of word frequencies in the Mainichi Shimbun over 4 years by Alexandre Girardi.  Note: (a) these frequencies are biased towards words and kanji used in newspaper articles, and (b) the relative frequencies for the last few hundred kanji so graded is quite imprecise.
|}
|}



Revision as of 07:20, 6 September 2018

The KANJIDIC Project

(Note that this page in the process of being rewritten, so be patient with any aspects that seems incomplete.)

Introduction

The KANJIDIC project, which began in 1991, has the goal of compiling and distributing comprehensive information on the kanji used in Japanese text processing. It covers the 13,108 kanji in three main Japanese standards:

Three data files are distributed by this project:

  • the KANJIDIC2 file, which is in XML format and Unicode/UTF-8 coding, and contains information about all 13,108 kanji. For this file the following information is available:
  • the KANJIDIC file, which in in EUC-JP coding and covers the 6,355 kanji in JIS X 0208. For this there is the

There is also a combined overview of the KANJIDIC/KANJD212 files.

Content & Format

The database and distributed data files contain an entry for each of the kanji, with each entry containing a number of fields of data about the kanji. The data is described in the following table. The format of the distributed files as as follows:

  • the KANJIDIC and KANJD212 files are text files with one line per kanji and the information fields separated by spaces. The format of each line is:
    • the kanji itself followed by the hexadecimal form of the JIS "ku-ten" coding, e.g. "亜 3021";
    • information fields beginning with one or two-letter codes as per the table below. For example "S10" indicates a stroke count of 10;
    • the Japanese readings of the kanji. ON readings (音読み) are generally in katakana and KUN readings (訓読み) in hiragana. An exception is the set of kokuji for measurements such as centimetres, where the reading is in katakana. Hyphens are used to indicate prefixes/suffixes, and '.' indicates the portion of the reading that is okurigana. There may be several classes of reading fields, with ordinary readings first, followed by members of the other classes, if any. The current other classes, and their tagging, are:
      • where the kanji has special nanori (i.e. name) readings, these are preceded the marker "T1";
      • where the kanji is a radical, and the radical name is not already a reading, the radical name is preceded the marker "T2".
    • the meanings (usually in English). Each field begins with an open brace '{' and ends at the next close brace '}'.
  • the KANJIDIC2 file is in XML and is structured according to its DTD (Document Type Definition). The DTD contains extensive annotations and is intended to be the primary documentation for the file. This sample illustrates the structure of a typical entry. Information fields are grouped by type within entities such as <dic_number> and <query_code>, with specific values indicated by an attribute code. For example the kanji 亜 has the number 43 in the original Nelson kanji dictionary and 81 in the New Nelson. This is recorded in the XML file as:
<dic_number>
<dic_ref dr_type="nelson_c">43</dic_ref>
<dic_ref dr_type="nelson_n">81</dic_ref>
....
</dic_number>
Kanjidic Information Fields
Field Kanjidic Code
(if any)
Group Entity Entity/Attribute Comment
Kanji none literal
JIS code-point codepoint cp_value cp_type="jis208" e.g. 亜 is "3021" in KANJIDIC and
"16-01" in KANJIDIC2
Unicode code-point U codepoint cp_value cp_type="ucs"
Radical (KangXi Zidian) B/C radical rad_value rad_type="classical" Where Nelson uses the KangXi radical this has a "B" code, otherwise it has a "C" code
Radical (Nelson) B radical rad_value rad_type="nelson_c"
Grade G misc grade The "grade" of the kanji.
- G1 to G6 indicates the grade level as specified by the Japanese Ministry of Education for kanji that are to be taught in elementary school (1006 Kanji). These are sometimes called the kyōiku (education) kanji and are part of the set of jōyō (daily use) kanji;
- G8 indicates the remaining jōyō kanji that are to be taught in secondary school (additional 1130 Kanji);
- G9 and G10 indicate jinmeiyō ("for use in names") kanji which in addition to the jōyō kanji are approved for use in family name registers and other official documents. G9 (649 kanji, of which 640 are in KANJIDIC) indicates the kanji is a "regular" name kanji, and G10 (212 kanji of which 130 are in KANJIDIC) indicates the kanji is a variant of a jōyō kanji.
Frequency-of-use ranking F misc freq The 2,501 most-used characters have a ranking which expresses the relative frequency of occurrence of a character in modern Japanese. The data is based on an analysis of word frequencies in the Mainichi Shimbun over 4 years by Alexandre Girardi. Note: (a) these frequencies are biased towards words and kanji used in newspaper articles, and (b) the relative frequencies for the last few hundred kanji so graded is quite imprecise.

(This table will be extended as time goes on.)

History

(some comments by Jim Breen)

KANJIDIC began around 1991 as two files: jis1detl.lst and jis2detl.lst, which were later merged into a single file.

The first file was compiled initially from the file "kinfo.dat" supplied by Stephen Chung, who in turn compiled his file from a file prepared by Mike Erickson. I originally added about 1900 "meanings" by James Heisig keyed in by Kevin Moore from the book "Remembering The Kanji". I later added the meanings from Rik Smoody's files, compiled when he was working for Sony in Japan. These appear to have been based on Nelson.

The second file was compiled from a complete JIS2 list with Bushu and stroke counts kindly supplied to me by Jon Crossley, to which I added Nelson numbers, yomikata and meanings extracted from Rik Smoody's file.

Theresa Martin was an early assister with this file, particularly with tracking down and correcting many mistranscribed yomikata (the old zu/dzu, oo/ou, ji/dji, etc. problems).

Jeffrey Friedl did a major overhaul in September-October 1992, in which he added the original frequency rankings, Halpern codes, SKIP patterns, updated the grading ("G" fields) to reflect the modern Jouyou lists, corrected radical numbers, corrected stroke counts and readings to fall in line with modern usage.

Magnus Halldorsson corrected some erroneous Halpern numbers, and provided them for a lot of the radicals. He provided the list of Heisig indices, which he originally compiled himself, then verified and expanded using lists from Richard Walters and Antti Karttunen. He also passed on to me the list of Gakken indices compiled by Antti Karttunen.

Lee Collins provided the Unicode mappings.

Iain Sinclair has provided the yomikata, meanings and S&H indices of many of the obscure JIS2 kanji.

Christian Wittern, a Sinologist working at Kyoto University, sent me a monster file prepared by Dr Urs App from Hanazono College. From this I have extracted the Four Corner and Morohashi information. Christian also provided the original Pinyin details, which were later replaced. I am very grateful for these significant contributions.

In March 1994 the Morohashi indices were proof-read and corrected by Christian.

Alfredo Pinochet supplied all the Henshall numbers.

Ingar Holst has provided considerable assistance in regularizing the Bnnn and Cnnn radical classifications to remove some errors that were in the original JIS2 file, and to make it all conform to Nelson's classification.

In mid-1993 I withdrew the SKIP codes from the distributed file as it appeared that their presence violated Jack Halpern's copyright on these codes. Jeffrey Friedl contacted Jack about this, and Jack obtained permission from his publisher for the codes to be included subject (initially) to copyright and usage restrictions. In March 1994 the Halpern indices and SKIP codes were checked against an extract from Jack's files, and the "Z" mis-classification codes added, again from his files. Jack has also made a lot of useful comments and suggestions about the content and format of the file. I am most grateful to Jack for his permission and assistance, and also to Jeffrey for making the contact.

In May 1995, a number of updates took place. Jeffrey Friedl established contact with James Heisig, and obtained a further set of his indices. I contacted Mark Spahn (via the "honyaku" mailing list) and he kindly provided most of the missing S&H descriptors, and Jack Halpern released to me the SKIP codes of the kanji not in the New Japanese-English Character Dictionary. For all this material I am most grateful.

In August 1995, I added the O'Neill index numbers. These were compiled by Jenny Nazak, David Rosenfeld and myself. Thanks to Jenny & David for their assistance.

In January and February 1996 the Morohashi numbers were checked thoroughly against two important sources: a file of Unicode-Morohashi data (Uni2Dict) which was prepared by Koichi Yasuoka from the allocation in the JIS X 0221 standard, and the review draft of the proposed revision of the JIS X 0208 standard, which was prepared by the INSTAC Committee, and made available in a text file, thus enabling comparisons. All the mismatches between the three files were examined against the Morohashi text, and extensive corrections made to all three files. I am grateful to Koichi Yasuoka and Masayuki Toyoshima for their considerable assistance in this task.

In March 1996 the Korean readings were added. They were provided by Dr Charles Muller, then of of Toyo Gakuen University, to whom I am most grateful. Chuck's compilation of Korean readings is extremely thorough and scholarly, and I am pleased to be able to incorporate them.

In April 1996 the readings of all the kanji were compared with those in the JIS X 0208 draft, and a number of corrections and additions made.

In May 1996 I carried out a "unification" of the readings of the KANJIDIC and KANJD212 files, wherein all the readings of the "itaiji" were brought into line. The identification of these itaiji was drawn from a file posted to the fj.kanji group by Taichi Kawabata (kawabata@is.s.u-tokyo.ac.jp), which was compiled at the ETL from the itaiji identification in the JIS X 0208 and JIS X 0212 standards. I corrected a few errors, and added some extra sets which were indicated in the JIS X 0208-1996 draft.

In July 1996 the Pinyin details were completely replaced by a new set. The original Pinyin were from an earlier compilation by Christian Wittern, and and contained many errors. Two more reliable sources had become available: the Uni2Pinyin file compiled by Koichi Yasuoka, which is based in part on the TONEPY.tit by Yongguang Zhang; and the PYCHAR set of readings of Big5 hanzi compiled by Christian Wittern. The Pinyin currently in the KANJIDIC file is a combination of the two, following the order in the Uni2Pinyin file.

In August 1996 I corrected a few more missing and erroneous Nelson numbers, using a massive Nelson list prepared by Wolfgang Cronrath. He also flagged the kokuji, so I added these to the readings fields as "{(kokuji)}".

Also in August 1996 I deleted the handful of former "XJxxxx" cross-references, and replaced them with a much more comprehensive set, so that they now represent all the recognized "itaiji". The file I used for this was the corrected itaiji file mentioned above.

In April 1997 I corrected a large number of bushu codes. Many of these had been identified as errors by Jean-Luc Leger who analyzed and examined all the Nelson bushu. I also identified and added a large number of missing Cnnn codes.

Also in April 1997 I added the S&H "Kanji & Kana" indices. These had been keyed by Olivier Galibert (Olivier.Galibert@mines.u-nancy.fr). (There must be an outbreak of kanji interest on Nancy.)

In February 1998, the long-awaited inclusion of the "New Nelson" numbers took place. I had been waiting for the editor of the New Nelson, John Haig, to supply a list (as he had agreed some years before), but in the meantime, Jean-Luc Leger keyed a list, so they are now available.

Also between December 1997 and February 1998 a large number of Level 2 kanji had their stroke counts corrected to bring them into line with the counting principles used in the Level 1 kanji. This usually aligned the counts with those used in the New Nelson and in S&H. Appendix E of this document was amended to reflect this. The leg-work in tracking this material down was done by Wolfgang Cronrath.

During December 1998 & Jan 1999 I updated the stroke counts of many of the Level 2 kanji, using an analysis of them carried out by Wolfgang Cronrath. I also added the De Roo codes, which had been keyed by Jasmin Blanchette, who also typed the explanatory material. I contacted Fr De Roo in Tokyo who readily agreed to the inclusion of the codes.

The extension of the S&H Kana & Kanji numbers to the 2nd edition was done by Enrique Sanchez Rosa.

The Hangul versions of the Korean readings (which only appear in the XML version) were provided by Francis Bond and Kyonghee Paik.

I did the Tuttle card numbers myself.

James Rose provided the numbers from Crowley's "The Kanji Way to Japanese Language Power", Sakade's "A Guide To Reading and Writing Japanese", and also for that book's 3rd Edition edited by Henshall, Seeley & De Groot.

The "Kodansha's compact Kanji guide" codes were provided by Richard Fremmerlid.

The "Kanji in Context" codes were provided by Randy Foreman.

The Spanish kanji meanings (which appear in the XML format, and may also appear in special versions of KANJIDIC) were compiled by Francisco Gutierrez and provided by Gabriel Sanroman.

Alain Thierion translated the meanings of the kanji into French, and also provided the Maniette numbers.

Andrew Slater provided updates to the JLPT numbers, and additional numbers for the Japanese Flashcards series.