View source for Tanaka Corpus

=Introduction=
This page provides some brief documentation for the Tanaka Corpus of parallel Japanese-English sentences, and in particular the modification and editing that has been carried out to enable use of the corpus as a source of examples in the WWWJDIC dictionary server and other systems.

The corpus was compiled by Professor Yasuhito Tanaka at Hyogo University and his students, as described in his [http://www.edrdg.org/projects/tanaka/tanaka.pdf Pacling2001] paper (it is described in the paper as the "Past Compilation Method".) At Pacling2001 Professor Tanaka released copies of the corpus, and stated that it is in the public domain. According to Professor Christian Boitet, Professor Tanaka did not think the collection was of a very good standard. (Sadly, Prof. Tanaka died in early 2003.)

At the 2002 Papillon workshop in Tokyo, Professor Boitet included a copy of the corpus in a CD, distributed to participants, and suggested that it may serve as examples in a dictionary. Jim Breen realised it had the potential to be a source of example sentences in the WWWJDIC server. He edited, reformatted and indexed the corpus and linked it at the word level to the dictionary function in the server. (see below) 

The inclusion of the Corpus in the WWWJDIC server exposed it to a wide audience, and a number of other systems incorporated the corpus into their operation. It also began to be used in some research projects in natural language processing.

In 2006 the Corpus was incorporated into the [http://tatoeba.org/home Tatoeba Project] being developed by Trang Ho to provide a sentence-based multi-lingual resource. That project is now the "home" of the Corpus.

=Compilation=
Professor Tanaka's students were given the task of collecting 300 sentence pairs each. After several years, 212,000 sentence pairs had been collected.

From inspection, it appears that many of the sentence pairs have been derived from textbooks, e.g. books used by Japanese students of English. Some are lines of songs, others are from popular books and Biblical passages.

The original collection contained large numbers of errors, both in the Japanese and English. Many of the errors were in spelling and transcription, although in a significant number of cases the Japanese and English contained grammatical, syntactic, etc. errors, or the translations did not match at all.

The original file can still be downloaded (see below.)
=Initial Modifications to the Corpus=
As mentioned above, the Tanaka Corpus was edited and adapted to be used within the [http://www.edrdg.org/cgi-bin/wwwjdic/wwwjdic?1C WWWJDIC] dictionary server as a set of example sentences associated with words in the dictionary. In order to adapt the corpus for this role, it was edited as follows:
# an initial regularization of the punctuation of the Japanese and English sentences was carried out, then duplicate pairs were removed, reducing the original file from 210,000 pairs to 180,000 pairs;
# sentences which differed only by differences in orthography (e.g. kana/kanji usage, okurigana differences), numbers, proper names, minor grammatical points such as plain/polite verb usage, etc. were reduced to single representative examples;
# sentences where the Japanese consisted of a short Japanese statement in kana were removed;
# sentences with spelling errors, kana-kanji conversion errors, etc. were corrected;
# sentences where the English version did not match the Japanese were edited to make the two versions agree;
# where the sentences contain gender-specific language or words, the English portion has been tagged with [M] or [F] respectively;
# sentences where the Japanese was too garbled to derive a valid English equivalent were removed.

The process described above has continued, and at present the edited corpus has just over 150,000 sentence pairs.
=Incorporation into the WWWJDIC Server=
(The initial incorporation of the Tanaka Corpus in the WWWJDIC server is described in a [http://www.edrdg.org/~jwb/paperdir/dicexamples.html paper] presented to the 2003 [http://www.papillon-dictionary.org/papillon/Home.po Papillon workshop].)
In order to facilitate the linking of sentences in the Corpus to words in the online dictionary, a list of Japanese words and phrases was extracted from each sentence. This was carried out using the Chasen morphological analysis program. Compound words which had dictionary entries were recombined as necessary. At present about 30,000 unique Japanese words and phrases are indexed.

The list of words associated with each sentence is used by the WWWJDIC server to select examples of the usage of the words. In addition, users of the WWWJDIC server can search the Corpus using text strings in Japanese and/or English, and using regular expressions. 
More information on the WWWJDIC use of the corpus is in the [http://www.edrdg.org/wwwjdic/wwwjdicinf.html#examp_tag documentation].

To see an example of the sentence linking, [http://www.edrdg.org/cgi-bin/wwwjdic/wwwjdic?1Q%C2%E7%B3%D8%C0%B8_1_%A4%C0%A4%A4%A4%AC%A4%AF%A4%BB%A4%A4 here] is the sentence display for 大学生. There is also a function for [http://www.edrdg.org/cgi-bin/wwwjdic/wwwjdic?10 browsing] the sentences.

The indices used to link dictionary headwords and sentences are now maintained within the Tatoeba project, and are [[Sentence-Dictionary_Linking|described here]].

=Subset=
An automatically-generated subset of the edited corpus is also available. The subset contains those sentences with one or more words marked with a "~" (see above.)
=JMdict Incorporation=
From June 2021 the sentences in the subset mentioned above have been included as examples in a special version of the JMdict Japanese dictionary. The approx. 31,400 sentences have been embedded in the 28,700 entries in the dictionary to which they have term-level linkages (some  sentences are associated with different senses of a term.) See [[JMdict-EDICT_Dictionary_Project#CURRENT_VERSION_.26_DOWNLOAD|here]] for the download location.

=Tatoeba Project=
The Corpus is now maintained within the [http://tatoeba.org/home Tatoeba Project]. This project has extended the file to include many other languages, and many sentences are available in multiple languages. The project WWW site has extensive facilities for searching and editing the sentences, and has an active community of people entering and editing sentences.

=Copyright Issues=
Professor Tanaka originally placed the Corpus in the Public Domain, and that status was maintained for the versions used by WWWJDIC. In late 2009 the Tatoeba Project decided to move it to a Creative Commons [http://creativecommons.org/licenses/by/2.0/fr/deed.en_GB CC-BY] licence (that project is in France, where the concept of public domain is not part of the legal framework.) It can be freely downloaded and used provided the source is attributed.
=Downloads=

The current edited version used in the WWWJDIC servers can be downloaded from:
* [ftp://ftp.edrdg.org/pub/Nihongo/examples.gz complete version] (EUC-JP). Each time it is updated a [http://www.edrdg.org/examples/examples_date date-stamp] is set;
* [ftp://ftp.edrdg.org/pub/Nihongo/examples.utf.gz complete version] (UTF-8)
* [ftp://ftp.edrdg.org/pub/Nihongo/examples_s.gz subset] (in EUC-JP).

In addition the last [ftp://ftp.edrdg.org/pub/Nihongo/examples_pd.gz Public Domain version] (Oct 2008) is available.

Downloads can also be made from the [http://tatoeba.org/eng/pages/download-tatoeba-example-sentences Tatoeba database].

The original file is available in [ftp://ftp.edrdg.org/pub/Nihongo/tanakacorp_utf8.gz UTF8 coding] or [ftp://ftp.edrdg.org/pub/Nihongo/tanakacorp_euc.gz EUC-JP coding]. Please note that these versions are only provided for historical interest. '''DO NOT''' use them in projects, as they are full of errors.

=Credits=
Many people have played a significant role in making the Tanaka Corpus available and useful:
* Christian Boitet, who alerted Jim Breen to its existence;
* Paul Blay, who for several years maintained and extended the Corpus, and did extensive work on the indices;
* Trang Ho and her team of collaborators, who made a home for the WWWJDIC indices in the Tatoeba Project, and has greatly enhanced the Corpus;
* Francis Bond, who has been a recent contributor and user of the Corpus for NLP research.

=Warning/Disclaimer=
The Corpus is a useful and interesting collection of matched Japanese and English sentence pairs, however it cannot be regarded as containing natural or representative examples of text in either language. This is because of the way it was originally compiled and the artificial nature of the sources. Also it still contains a large number of errors and repetitions. It certainly should not be used for any statistical analyses of the text.
While the Corpus appears to be adequate and useful as a source of examples of word usage, the user is advised to be cautious and critical. The following points should be considered:
# the sentences were typed in by students in order to meet a work requirement. Initially there were many mistakes both in the Japanese and English. While many have been corrected some still remain.
# some clearly contain translations into Japanese of English sentences, and often do not represent the most natural way things are said in Japanese (overuse of pronouns, etc.).
# others contain English translations which are very literal translations of the Japanese, and perhaps came from simple machine translation systems.
# it should be noted that many of the sentences are of the kinds found in older "study for entrance exam books", and thus are likely  to have contrived examples of grammar usage or slightly archaic English examples passed down from generation to generation by (Japanese) English teachers. They are not examples of normally-used  modern English, and should not always be regarded as suitable for English study.
# please DO NOT use the original file without exercising considerable caution. It contains thousands of errors and duplications in both the Japanese and English sentences. If any project wants to use the Tanaka material, it is STRONGLY recommended that the updated file is used.