Contents
Page
Abstract 2
Acknowledgements 3
Chapter 1 Introduction
1.1 Monolingual
and Multilingual Dictionaries 4
1.2 Aims 5
1.3 Thesis
overview 5
Chapter 2 Past work
2.1
Dictionaries 6
2.2 Electronic
dictionaries 8
2.3
Information retrieval 9
Chapter 3 Data, concepts and tasks
3.1
Unicode 13
3.2 Limitations
with multilingual dictionaries 14
3.3 Operations, requirements and tradeoffs 16
Chapter 4 Internal representation of the XML
document
4.1 XML 21
4.2 XML
information access 24
4.3 DOM
(Document Object Model) 24
4.4 SAX (Simple
API for XML) 26
4.5
Object orienting the XML file 28
4.6
Other XML tools 29
4.7
Future XML tools 30
Chapter 5 Indexing and entry retrieval
5.1
Limitations with JMdict 33
5.2
Indexing techniques 33
5.3
Databases 35
5.4
Hashing 38
5.5 Additional considerations 38
Chapter 6
Interface to an electronic dictionary
6.1 Query interface options 40
6.2 Viewing search results 41
6.3 XSL 42
Chapter 7
Conclusion 44
Appendices 45
Bibliography 53
Abstract
Multilingual
Electronic Dictionary Project
The
purpose of the research was to investigate and analyse the creation of a
multilingual dictionary application using JMdict as a study target. The JMdict
file is an XML document containing multilingual dictionary entries of
Japanese-English-German. A dictionary application is made up several major
programming components: user input of query information, query search and
retrieval and display of information. These aspects of multilingual dictionary
application creation were investigated with a major focus on internal
representation of dictionary entries in the application. Current XML parsers
have been found to be inappropriate as a form of data representation and
retrieval on their own. The SAX parser could possibly be used along with the most
favorable method, the indexing technique. Investigations on multilingual
dictionary files, the XML format, handling of foreign characters using Unicode
and input of foreign characters has been carried out. Furthermore, processing
of information retrieval such as stemming techniques and spelling suggestions
have also been investigated. This research can be a guide for programmers
wishing to find out about creating multilingual dictionary software.
Acknowledgements
First and foremost, my
biggest thanks goes to my supervisor, Associate Professor Jim Breen. His advice
and inspiration has been invaluable throughout the thesis. In addition, without
his hard work in the JMdict, this project would not be possible. Thanks for all
your help.
Thanks to my friends, they
know who they are, for their encouragement, criticisms and also for being there
almost every step of the way with me on this journey.
Last but not least, I
would like to give a sincere thanks to my family and relatives, especially my
Mother Maria for helping with proofreading and providing moral support.
Chapter
1
Introduction
1.1
Monolingual and Multilingual Dictionaries
Dictionaries are an essential part of learning and education and are a centuries old teaching tool used all over the world in a multitude of different languages. The Oxford English Dictionary defines it as “A book dealing with the individual words of a language (or certain specified classes of them), so as to set forth their orthography, synonyms, derivation, and history…”[1] Dictionaries initially began life as topic specific dictionaries and also as bilingual dictionaries (Roman-Latin and also Chinese dictionaries) as early as the 16th century. So the use of bilingual dictionaries is not a new concept. Over the years, dictionaries also broadened their scope becoming more field-specific, for example, music, medical and scientific dictionaries. Soon, translations from several different languages to another language emerged, resulting in the creation of multilingual or interlingual dictionaries. Indeed, multilingual dictionaries provide a powerful tool for students to enhance their learning of another language.
Despite the range of dictionary functionalities, the structure of the print dictionary has always been constant, closely following the definition of a dictionary. Being able to search through one dictionary would enable readers to search through all dictionaries in the same way. English dictionaries are all laid out from A-Z (or in some cases A-K and L-Z). To find a word in these dictionaries simply requires the user to search alphabetically to find the entry. Asian language dictionaries, indexed by ideographic characters, may have a slightly different format in that they may order the dictionary via stroke or radical order. In any case, the reader has to manually sift through pages in a set pattern to find their definition or translation. There was no other known way to search for information within a dictionary [Atkins, 1985].
As computers have grown more powerful and widespread, the opportunity
for automated processes and digital storage drew closer. Programmers,
developers and lexicographers are now no longer constrained by the conventional
format of the dictionary. Students and teachers are also no longer constrained
in dictionary searching techniques. All applications that required paper or
tape storage could now be located on digital storage medium only a fraction of
the size required before. Of course, dictionaries, requiring relatively large
amounts of physical storage area and mass, could be stored efficiently
digitally. Ideas previously thought unfeasible, such as an electronic
dictionary was suddenly a possibility. However, this opened up a whole new set
of questions to be answered in terms of storage and information. Problems such
as dictionary data storage, multimedia and pictures are all fields that need
research in the context dictionary file construction. Different file structures
have been used create electronic dictionary files. As new storage formats are used,
they open up a whole host of different, new and sometimes unknown methods of
information retrieval and representation. Today, there are many incarnations of
electronic dictionaries, from portable handheld devices to stand-alone PC-based
programs and even World-Wide-Web on-line dictionary systems.
1.2 Aims
Despite the rapid improvement of electronic dictionaries and the
emergence of palm-top bilingual dictionaries, past research into electronic
dictionaries and information retrieval have not focused on the study of the
multilingual dictionary in the electronic medium. This area of study
incorporates research into conventional dictionaries, but there are many issues
that are left unanswered. Little work has been done investigating ways of
representing the dictionary, storing, displaying and understanding user
requirements of the multilingual dictionary. The aim of the research undertaken
in this thesis was to explore the applicability and suitability for using the
JMdict XML-based Japanese-English-German dictionary file as a study target.
The problem to be investigated does not come under one distinct
category. Instead, it is made up of several major aspects, all of which, when
combined, results in an analysis of a complete system, making this research
original. The thesis attempts to be a guide to programmers interested in
creating multilingual-based applications. The problem statement can be
described as follows:
“To investigate and compare old and new
storage techniques, data formats, representation of information and methods of
information retrieval. In addition to these aspects, to investigate the
applicability of these techniques in a multilingual dictionary environment.
Finally, to explore the ways in which a system can be presented to a user and
understand user needs.”
1.3 Thesis overview
The major aspects to be
investigated are as follows: data storage format, internal representations of
the document and the interface with the electronic dictionary. The following
chapter headings mark the discussion of these topics in further detail:
Chapter 2 reviews the past work carried out in the large field
of lexicography, electronic dictionaries and in particular, the area of
information retrieval. With information retrieval, various fields of research
into cross language information retrieval and text retrieval have been
reviewed, in particular, research into query translation and machine
translation.
Chapter 3 discusses the multilingual data available at hand.
It also describes the types of encoding methods available for different
languages. More importantly, Chapter 3 identifies the major (and minor)
operations required from any dictionary and from a multilingual dictionary in
particular. The limitations of multilingual dictionaries and in particular, the
JMdict file, will be examined.
Chapter 4 introduces the concept of XML as a data storage
format. The internal representation of dictionary data is extremely important
as it determines the ease with which data can be retrieved for the user. The merits
and shortcomings of various XML based tools are discussed along with the issue
of object orientation and its suitability in the dictionary setting.
Chapter 5 provides a more in depth look at other techniques
for efficient organization of data for retrieval. The format of the data
storage, ease of data extraction and flexibility in the types of data being
extracted will be investigated.
Chapter 6 provides insight into what the user seeks from an
electronic dictionary. It is important in all software to make sure the user
can operate it successfully and easily. In the same vein, the program must also
be able to provide the information that the user wants. Chapter 6 will identify
what kinds of information different types of user wants and provide possible
solutions for displaying information.
Chapter 7 will complete the discussion on multilingual
dictionaries and provide some suggestions for further work and the direction in
which research is heading in this topic.
Chapter
2
Past
work
Because of the nature of this thesis report, extensive research into both the technological component and language aspects of dictionaries was carried out. Without a full understanding of the whole picture, further investigations cannot occur.
2.1
Dictionaries
Dictionaries and lexicon are terms that are
synonymous with each other; the term ‘lexicography’ is defined as the process
of compiling dictionaries. A bilingual dictionary is usually produced for users
in one source language (L1) to a target language (L2) (for monolingual
dictionaries, L1 and L2 are the same language). There is sometimes also
confusion over the differences between dictionaries and encyclopaedias. Some
believe they are closely linked and are considered interchangeable, but they
are different kinds of reference books with different purposes.2 Dictionaries produced can be either:
- A dictionary of comprehension (allowing the student to understand
L2)
- A dictionary of communication (comprehension and production of L2)
and can consist of some of these components (as outlined by the extensive
entry record of the Online Oxford English Dictionary) [Beryl, 1985]
Multilingual and bilingual dictionaries
Before the construction of a multilingual
dictionary, the creator must consider these issues:
-
The
users of the dictionary
-
The
purpose of the dictionary. There are four ways in which a dictionary can be
aimed towards:
o
Technical
dictionary
o
Learning
dictionary
o
Reference
dictionary
o
Comprehension
dictionary
-
If
the dictionary is used to understand a foreign language or to produce text in a
foreign language
-
Native
language of the user; either the user is a native speaker of the source or the
target.
There are 4 types of multilingual dictionaries
- For speakers of L1 to understand target language text
- For speakers of L1 to produce target language text
- For speakers of L2 to comprehend source language text
- For speakers of L2 to produce source language text
Many experts agree that the
trouble with most dictionaries is that they try to cater for the needs of both
the source and target language together. Some believe that it is impossible to
pay equal attention to both in the same volume. Because of the limitations of
the size of normal paper back dictionaries, the editor usually has to select
the entries to go into the dictionary on the basis of the purpose of the
dictionary [Ali-Kasimi, 1977] [Hartmann, 1983].
There are several criteria to distinguish multilingual dictionary
types, whether they include encyclopaedic information and whether or not they
take account of changes in the language. There are three major types of
multilingual dictionaries:
- Dictionaries for machine translation
- Dictionaries for terminological data banks
- Dictionaries for the human user.
Machine translation requires
detailed grammars of the source and target languages, an interlingual grammar,
a comprehensive bilingual dictionary, and complex computer programming systems
to store, process and retrieve the data. An ordinary dictionary is expected to
provide only the information which the user needs, but the dictionary for
machine translation contains all the grammatical information about both
languages [Al-Kasimi, 1977]. The major focus of this thesis is to investigate
the criterion for dictionaries for human users.
Comparison between bilingual and monolingual
dictionaries
Monolingual and multilingual
dictionaries differ in several respects, according to the users they intend to
serve, the needs they cater for, and the methods of their compilation.
Bilingual and monolingual dictionaries show systematic variations in approach
to wordlist (the stock of vocabulary to be listed). The bilingual dictionary
usually contains a few thousand of the most frequent items to an extensive list
as large as a monolingual dictionary. Furthermore, bilingual dictionaries
contain not one, but 2 discrete wordlists (L1 and L2). Another fundamental
difference between the mono and bilingual dictionaries is that the monolingual
dictionary takes the form of a definition, whilst the bilingual dictionary
attempts to provide an equivalent or a series of equivalents in the target
language [Beryl, 1985]. The layout of the entries and search techniques
may also be different. Some foreign languages can be searched differently
according to phonetics or character construction, or even character codes.
Dictionary
users
There are many different
types of users who use dictionaries. It is very important to understand what
kind of people use the dictionary otherwise there will not have be defined
audience and this will result in a dictionary lacking focus. Users have
different requirements and this will be reflected in the purpose of the
dictionary via the type of entries, number of entries and structure of the
entries. Hartmann [1983] categorises the uses and the users of dictionaries:
Factors
in dictionary use:
Information
Meanings/synonyms Pronunciation/syntax Spelling/etymology Names/facts |
Operations
Finding meanings Finding words Translating |
Situations in dictionary use
Users
Child Pupil/trainee Teacher/critic Scientist/secretary |
Purposes
Extending knowledge of the mother tongue Learning foreign language Playing word games Composing a report Reading/decoding foreign language texts |
Table 1 Uses and users of dictionaries
2.2
Electronic dictionaries
There are many potential
advantages of an electronic dictionary over a conventional dictionary. The
information accessible to a user is now huge. The cost of dictionaries is an
advantage. The cost of storing a dictionary on an electronic medium such as a
disk or CD is far cheaper than the production costs incurred for the printing
and distribution of dictionaries. Furthermore, updates to the dictionary will
no longer mean a complete republishing of new edition; an update patch with the
new data can be distributed very easily. Most conventional dictionaries cannot
contain all the information for all the entries, but instead are selective in
coverage of different aspects such as pronunciation, spelling, etymology and
idioms. There are no longer any problems like this. The amount of data in a dictionary
will now depend on how much a lexicographer is willing to put into the file
[Hartmann, 1983].
Depending
on the type of data format the dictionary is stored in, it is much easier to
insert new entries into the dictionary. For example, introducing additional
attributes to a file such as JMdict is very easy. This leads to the size of a
dictionary. A 20 volume dictionary no longer needs to belong in a state
library; it is possible to have it on your lap, in your laptop. Near unlimited
amounts of free space can mean incorporation of features that were impossible
in print dictionaries. Some larger print dictionaries were able to incorporate
several pictures per page. Pictures
enhance the presentation of the dictionary, but it also serves as a valuable resource
in terms of word-picture recognition. Multimedia is also another feature that
can be incorporated into electronic dictionaries. A user is able to understand
a word and read the phonetics of a word, but now is also able to hear it. The
best way to learn to say a word is by hearing it first. Portability of an
electronic file is another reason for electronic dictionary superiority. A set
of dictionary files can easily fit on portable storage media such as CD. This
means they can be taken anywhere and used whenever needed. No need to go to a
library to find the most comprehensive dictionary. Accessibility is a great
issue. Large dictionaries are not produced in large capacities because of lack
of demand and high costs of production. However, dictionary files, even large
files are easily transmittable over a medium like the internet which means that
people all around the world will be able to take benefit from a most valuable
resource.
One of the most important
advantages that electronic dictionaries are able to offer is its
transparency; it allows access to the
whole of its contents via formalized categories so interesting features of the
vocabulary can be investigated directly. Some dictionaries used to be created
from scratch specifically for beginners and infants. One dictionary is able to
satisfy the needs of all users. How can this be achieved? Electronic
dictionaries provide different ways of presenting data to the user. The format
of entries no longer has to be set like conventional dictionaries. Computerised
dictionaries have the potential to be customized according to the needs of the
user [Hartmann, 1983].
The speed of information
retrieval that electronic dictionaries deliver can be said to lose the memory
retention benefits that manually searching through paper dictionaries provide.
Using a dictionary usually provides the use with the chance to look at other
entries surrounding the word on the same page. This is not to say that an
electronic dictionary cannot do this, provided the correct design principles
are administered.
There are many different types of electronic dictionaries available. To complete the analysis of electronic dictionaries, some of these files should be studied and acknowledged. Breen’s [2000a] EDICT file, KEBI and EDR are studied. The structure and contents of these files are quite different. They are discussed in detail in Appendix A.
2.3
Information retrieval
There
has been limited documented research into dictionary files or electronic
dictionary representation. However, the majority of research related to
electronic dictionaries has been in methods of cross language information
retrieval. The following are the basic information retrieval processes that are
carried out:
Representing
the information needed – Query formulation
Representing
documents – Indexing the text
Comparing
these representations – Retrieval
Evaluating
the retrieved documents , and if necessary, return the query and entry [Croft
et. al]
Various
aspects of these processes may be relevant and useful when investigating
electronic dictionaries. Information retrieval has become an important issue
because there is now emphasis on the efficiency of the retrieval and also the
accuracy and amount of information that can be returned from a search. There have
been several articles published by various researchers who have attempted to
carry out cross-language information retrieval (CLIR).
Of
the issues raised by information retrieval, some of the problems encountered
are relevant for investigations of electronic dictionaries and are listed as
follows:
-
Character encoding -
Documents that are stored in different languages have to be stored, but
subsequently, need to be displayed in a non-ASCII environment. This problem has
been tackled and one solution has been provided by JMdict.
-
Segmentation
of Chinese and Japanese- Asian languages are usually written with separated
lexical elements, so is the case with English languages. Tokens need to be
identified as strings of characters so that searches can be carried out on
them.
Text
retrieval systems, such as the INQUERY system (based on probabilistic retrieval
via Bayesian net framework) [Croft et. al] [Callan], appear to be too complex
and large for a task such as searching through a database of entries. Other attempts
have also been made for multilingual text retrieval on a larger medium such as
the world wide web. Projects such as MULINEX have been set up to develop tools
for cross language retrieval, using retrieval systems such as Fulcrum
SearchServer and SurfBoard [Erbach 1997].
Machine
translation -
Machine translation (MT) is a application to electronic dictionaries but
consistently more complex. Electronic dictionaries set out to translate a
single word, phrase or character into another language, whereas MT really
attempts to translate whole sentences or documents. However, MT uses electronic
dictionary files to carry out the translations. This takes a lot more effort to
achieve than simply looking up a dictionary file. For example, MT performs a
linguistic analysis so that the most suitable Japanese word can be matched to
the English query request [Jones, 1999]. In further studies, Chinese queries
were used also to evaluate the effectiveness of different MT techniques [Kwok,
1997] At the moment, the state of development of MT is such that translation
requires human aid to complete. This is referred to as machine-aided
translation. Some current commercial MT software (such as Logovista E to J)
enables some form of translation but still requires pre and post editing of the
translation for acceptable conversion from English to Japanese [Eichmann et
al., 1998].
Query Translation - As
highlighted by Eichmann et al. [1998], there have been several methods of
handling CLIR. The first method is translating the query, therefore translating
the problem into a monolingual dictionary problem [Ballesteros, 1996] [Davis,
1998]. Another method would be to translate the document. The final approach
would be to automatically establish associations between queries independent of
language differences [Eichmann et al., 1998]. There are also several other
techniques that are being involved such as machine translation systems and
bilingual dictionaries.
Dictionary-based retrieval
can occur by breaking up the query into their root forms, searching the
dictionary file for equivalents for each word and substituting them with the
translated words with the highest precision [Ballesteros, 1996]. Problems with
this type of translation are that an incomplete dictionary will result in inconsistent
results. Furthermore, there may be ambiguities in translation that can also
introduce substantial error [Eichmann et al., 1998]. In other research fronts
in the same area, Fujii [1993] investigated the effects of retrieval using
characters versus word based indexing techniques for text retrieval. It was
found that character based indexing and retrieval was the most efficient out of
these two techniques.
A different method of cross-language
information retrieval can be carried out by using a multilingual thesauri
[Eichmann et al., 1998]. This bears a resemblance to specialised
Japanese-English dictionary file that Breen has created (such as COMPDIC) in
that controlled vocabularies are stored in the thesauri. The medical thesaurus,
called a metathesaurus, is multilingual supporting a range of European
languages. Conventional thesaurus-based retrieval required the queries to be
matched in the thesaurus by their representation. Thus CLIR would follow the
same method of retrieval. From their tests, they discovered that the best
results came from choosing words that contained only query words. Performance
of the metathesaurus based retrieval did not exceed dictionary-based retrieval
[Eichmann et al., 1998]. This evidence indicates that using a thesaurus to
translate can be effective, but a thesaurus is not as flexible in comparison to
a dictionary.
Jones [1999] tested several
different translation methods in one study. Of the translation methods, the
interesting ones were: ‘using a bilingual dictionary to return all the
definitions of the corresponding English query’ and ‘using the bilingual
dictionary to return a single default translation for the matching English
word’. Jones claimed that there was literature that argued that full machine
translation was unsuitable, however he does not back up their claims with any
evidence. Despite these unfounded claims, the results that they produced
indicated that full machine translation produced favourable results compared to
dictionary term lookup. Again Jones failed to disclose any references
describing the method of full machine translation. Hull and Grefensette [1996] mentioned in his paper that the
‘performance of MT systems in the setting of general language translation is
dismal enough to make this option less than entirely satisfactory’. This claim
had also been backed up by Pirkola [1998], Oard and Dorr [1996] and Yamabana et
al. [1996].
Hull
and Grefensette [1996] discussed five different definitions for multilingual
information retrieval (MLIR). Of the definitions that were displayed in the
paper, none of them took into account the use of a pure multilingual dictionary
to carry out MLIR. A multilingual dictionary can play a powerful role in the
development in information retrieval of multiple languages. The definitions
mentioned seemed to revolve around a collection of different dictionaries
either working in parallel or in combination to produce multilingual
translations. Hull’s narrow mindedness could have been a result of his
objectives which were centred around the development of a ‘query translation
module’ that could be easily built on top of an already existing information
retrieval system.
To
further demonstrate the diversity in methods and ideas being exercised, Pirkola
[1998] used a special dictionary and a general dictionary in query translation.
It was found that this technique was highly efficient [Pirkola, 1998].
Several CLIR techniques have been
observed to be a result of avoiding the construction of a specific dictionary
file for research. Indeed, creating a dictionary would be time-consuming,
however, it is believed once a ‘standard’ multilingual dictionary was produced,
it would relieve some of the problems faced by some of the researchers and open
new doors in CLIR.
From the review of the cross
language retrieval papers, it seems that the scope and depth that CLIR presents
is much to broad for most of the aspects to be relevant to multilingual
dictionary research. CLIR attempts to tackle complete indexes of documents in
many different languages. The research that has been uncovered tends to be
focused on either efficient retrieval of documents with the greatest relevance
to a search string query, or the complete translation of particular documents
from one language to another. An example of this would be a Greek speaker who
needs documents on a certain topic. The user would enter the search query for
keywords in the documents. If a Finnish document containing the keywords was
located, the document would then be machine translated into Greek for display
[Hlava et al, 1997]. Techniques and ideas raised by these studies may be
applicable to the current research.
Stemming
The
morphology of a language may mean that words in their plural or past tense may
not be queried effectively by the search engine because they are sometimes
structurally different in construction. The words ‘run, ‘ran’ and ‘running’ are
all related words and it should be reflected in the search by retrieving the
root word ‘run’. This issue is not a usual problem when it comes to searching
through a paper dictionary because whilst the user is flicking through the
pages, they may come across the root word of the query they were looking for
because most of the related words are found quite close together. The stemmer
widely believed to be relatively reliable is the suffix-stripping algorithm
created by Porter [1980]. The algorithm contains a set of rules, or steps where
the query word is compared each of the rules and is subsequently modified if
the suffixes match the comparison. Code to create the Porter suffix stripper is
relatively easy to construct using the guidelines outlined in Porter’s [1980]
paper. By using a stemmer for query modification, it can help to increase the
chance that a matching query can be found.
Spelling
Users
may appreciate options in the application that provide suggestions of possible
words if they have entered a word incorrectly, or if they do not know how to
spell it. Words can be similar in two ways: words that sound alike or
words that are structurally similar to the word. Two techniques could be
implemented to handle misspellings, Soundex and the Levenstein Distance
Algorithm.
The
Levenstein Distance Algorithm (LDA) provides a simple metric for testing the
similarity of words. It therefore can be used as a form of spelling checker
that can be used when entries cannot be found in the dictionary. If two words
which are similar are compared, they can be ‘aligned’ so that sequences of
words in each of them are able to match each other. For example, the words
‘word’ and ‘bird’ are quite similar. They can be aligned in such a way:
B
I R D
| |
W
O R D
This example is rather
simple, because there is no other way to really align the words. However, if
the two words ‘WORD’ and ‘BEARD’ are aligned. It is more difficult because
although there are similar sequences in each of them, there are several other
possibilities to align the rest of the letters. Some of the solutions are:
B E A R D B E A R D B E A R D
| | | | | |
W O R D W . O R D W O . R D
To convert ‘word’ to
‘beard’, insertions, deletions or letter replacements can be carried out. For
example:
B E A R D B E A R D B E A R D B E A
R D
| | | | | | | |
W O R D W O A R D W E A R D B E A
R D
[Word insertion] [Word replacement]
[Word replacement]
The
Levenstein distance of the 2 words is defined as the minimum number of
operations required to transform word 1 to word 2. There are phonetic
comparisons that compare the pronunciation of words to determine similarity.
However, these techniques are very costly and complex. The Levenstein Distance
algorithm assigns a score to each change, deletion or addition required to make
the strings equal. The final distance is the sum of these scores. A threshold
is set up which determines if a string is considered similar or dissimilar.
This algorithm can provide suggestions of words to the user with low Levenstein
distances in the event that the user entered an incorrect query string.
The Soundex code is an
indexing system that translates names into a 4 digit code consisting of 1
letter and 3 numbers. Its most familiar application has been by the US Bureau
of the Census to create an index for individuals using their surnames.
The advantage of Soundex is
its ability to group names by sound rather than the exact spelling. All the
words that have a similar sound are grouped together by having the same Soundex
number.
There several rules when creating Soundex codes:
-
All
Soundex codes have 4 alphanumeric characters
o
1
Letter
o
3
Digits
-
The
letter of the name is the first character of the Soundex code.
-
The
3 digits are defined sequentially from the name using the Soundex Key chart
-
Adjacent
letters in the name belonging to the same Soundex Key code number (such as a
double ‘r’) are assigned a single digit.
|
1 |
b p f w |
|
2 |
c s k g j q x z |
|
3 |
d t |
|
4 |
l |
|
5 |
m n |
|
6 |
r |
|
No code |
a e h i o u y w |
Table 2. Soundex Key table
If users misspell the first
letter of the word, the Soundex system is unable to retrieve the correct
Soundex code. Instead a list of words starting with the same letter will be
retrieved.
Chapter 3
Data, concepts and tasks
The encoding of characters, especially an international character set is important when considering a multilingual dictionary. In the past, different encoding schemes have been used; standardization of codes was non-existent. Studying the options available for encoding a multilingual file is therefore a required point of discussion. Japanese, Chinese and Korean text representation is discussed in detail in Appendix B. The current method of international standardization of encoding called Unicode.
3.1 Unicode
The Unicode Standard is a
superset of all characters in widespread use today. It unifies character sets
from around the world, making multilingual software easier to write,
information systems easier to manage and information exchange around the globe
more accessible. It contains the characters from major international and
national standards as well as prominent industry character sets. For example,
Unicode incorporates the ISO/IEC 6937 and ISO/IEC 8859 families of standards,
the SGML standard ISO/IEC 8879, and bibliographic standards such as ISO 5426.
Important national standards are included within Unicode: ANSI Z39.64, KS C
5601, JIS X 0208, JIS X 0212, GB 2312, and CNS 11643. The primary goal of the
development effort for the Unicode Standard was to remedy serious problems
common to most multilingual computer programs, overloading of character
encoding and also multiple, inconsistent character codes caused by conflicting
national and industry character standards and finally the inadequacy of using 7
and 8 bits (or a maximum of 256 characters) to represent the global character
set.. In Western European software environments, there is often confusion
between the Windows Latin 1 code page 1252 and ISO/IEC 8859-1 [The Unicode
Consortium, 1996].
The Unicode project began in
1988, the inconsistent groups of international character sets affected
publishers of scientific and mathematical software, newspapers, book
publishers, bibliographic information services, and academic researchers. In 1991, the ISO Working Group responsible
for ISO/IEC 10646 (JTC 1/SC 2/WG 2) and the Unicode Consortium decided to
create one universal standard for coding multilingual text. Since then, the ISO
10646 Working Group (SC 2/WG 2) and the Unicode Consortium have worked together
very closely to extend the standard and to keep their respective versions
synchronized.
Although the character codes
are synchronized between Unicode and ISO/IEC 10646, the Unicode Standard
imposes additional constraints on implementations. Unicode 2.1 has the same
character repertoire as ISO/IEC 10646-1:1993 and Unicode 3.0 has the same
character repertoire as ISO/IEC 10646-1:2000. Unicode uses a variable length 16
bit representation called UCS-2. The full 16 bit code space (that is 65000 code
positions) are available to represent characters. For compatibility with other
environments, there are two transformations of Unicode to convert them to 8 or
7 bit environments: UTF 8 (Universal Character Set Transformation
Format) and UTF 7. Furthermore, Unicode stands out from other standards
as it only deals with character codes, leaving the glyph shape and construction
to font vendors [The Unicode Consortium, 1996].
When considering Japanese character representation, there is no longer
any need to inter-convert between the different character set encoding
standards; all Kanji, Katakana and Hiragana characters are supported in
Unicode. Since a standard for Japanese representation is available, it was
logical that the JMdict file be encoded in UTF 8. [The Unicode Consortium,
1996] As for Kanji, the character code for characters that occur in both
Chinese and Korean character sets are the same, further decreasing the number
of character codes required to represent Chinese characters.
Because each Unicode
character is a 16 bit value, it cannot be handled like an ordinary ASCII
character value. Programming in Unicode may be more troublesome if the
programming language cannot handle 16 bit characters easily. C++ is a very
popular programming language however, support for international languages has
not been taken into account when designing this language, so representing Asian
characters using C++ is rather weak. Java has several advantages over C++ when
it comes to international language processing because it has been designed to
handle Unicode as standard input/output. It is the first programming language
to have built in support for Unicode. Clearly, Java is a useful language when
it comes to internationalization and is the recommended programming language to
use when creating a dictionary application. When using the UCS-2 encoding to
map from one locale to another, mapping tables will be required because no code
conversion algorithm exists. This conversion between different codes is carried
out via table-driven conversion.
3.2 Limitations with multilingual dictionaries
A multilingual dictionary is an excellent idea on first thought. It
would be ideal to have a dictionary that could translate English into many
different languages. However, depending on the data format of the dictionary,
there can be limitations when it comes to building the dictionary file. These
limitations can be due to structure of the data file to the clash of various
languages themselves.
Although the use of XML is a
very important step in solving the problem about flexibility and extensibility
of the dictionary file, there are other considerations into the design, content
and lexicographical aspects of the dictionary file. Although not such an
important issue, the actual size of one single dictionary file may become a
problem as more entries are added and multiple languages are included into the
file. Being able to physically store a large file with millions of records,
processing a file tens or even hundreds of megabytes in size may become a
problem if CPU intensive tasks are required to be carried out on the data file.
The issue of handling a multilingual dictionary file by the computer is
discussed further in this report.
One of the most important
limitations when working with multilingual dictionaries is attempting to match
senses and glosses for different languages together. There are words in some
languages that are not defined in another language. There are also some
languages where words do not exist for specific meanings, instead a generalized
term is used. Furthermore, one word in one language may be used to describe
more than one thing in another language. This non-parallelism between languages
is called ‘anisomorphism’ [Landau, 1984]. For bilingual dictionaries such as
the EDR (see Appendix A), there are 190,000 headwords for the English to Japanese
part compared with 230,000 words in the Japanese to English part. The smaller
amount of headwords reflects the fewer headwords required in the generation of
the language. Different languages are bound to have differing types of grammar
so there are simply words in one language that don’t exist in another language.
For example the Aussie word ‘bloke’ may only be able to be defined as ‘guy’ or
‘man’ in another language, and the real meaning cannot be conveyed to the user.
In some languages, there may not be a word to describe a word in another
language. These differences and problems are obstacles that need to be overcome
when combining several dictionaries. [Al-Kasimi, 1977] There is no real
solution to this problem. How this situation is handled can depend on the type
of implementation used to hold the information.
A file like JMdict has
decided to make headwords Japanese with English definitions. The English
definitions could also be used to possibly be headwords too. Usually the
headword defines which language the user is confident with and the target
language is the second language for the user. The language of the headword may
be important in the structure and direction of the multilingual dictionary
file. Furthermore, the additional information included within an entry such as
part of speech and examples needs to be stored in a particular language. A
smaller limitation exists that asks what language should these additional
details be stored in. Entry information such as part of speech and synonym may
be stored in one language. So a possible problem may arise if the information
is in English and a user is wishing to search from German to Japanese. The
information stored in English may require translation to Japanese or German in
order for a person to understand the entry.
This brings the issue to the
next point: the target audience for the multilingual dictionary. The dictionary
may be for English speakers who require translation from English to multiple
languages, or a student who wishes to translate an article. It was mentioned
earlier that bilingual dictionaries could not cater for both L1 and L2 users.
Doing so would mean that effort on creating one dictionary would be cut in half
by trying to create two distinct dictionaries, each for a different target
audience. A possible outcome from this limitation could be to create a
multilingual dictionary file that supports native speakers from only Japan,
Australia and Germany. The dictionary file could be created for users who
specifically wanted to translate between these three languages.
One of the problems with
creating interlingual dictionaries is that of pronunciation. If a user wants to
produce the foreign language from a bilingual dictionary, then they would want
to know the appropriate word and how to pronounce it. For example,
pronunciation of a Chinese character is varied. If the user does not know the
phonetics of pronunciation for PinYin, it is hard for a person to say the
character. Furthermore, there are many dialects of Chinese all around China and
in Asian countries, so inclusion of pronunciation in an entry can be a
difficult task [Ali-Kasimi, 1977].
Although not a major issue concerning the actual multilingual
dictionary file, the difficulty and complexity of linking every definition when
coding the dictionary file, can be a headache to lexicographers. Definitions
not only have to be linked to each entry, but also to a set of identifiers
(such as usage information, parts of speech, and examples) [Landau, 1984].
Multilingual dictionaries increase the complexity of entry input because of the
languages to update. In addition, adding synonyms and other cross-references
can become tedious, especially if the references are in multiple languages that
the dictionary file represents. The process of entry insertion does not
necessarily have to be a problem and could be solved. Depending on the format
of the file, software could be produced, similar to a dictionary search engine,
allowing insertion, deletion and modification of the dictionary file with
little problem.
3.3 Operations, requirements
and tradeoffs
Creating an electronic
dictionary can be considered a difficult task. A dictionary application
consists of many different operations; each operation often constituting an
area of study in computer science. It is therefore important to set out the
operations and basic processes that a dictionary application carries out.
Figure 1 describes the operations executed by typical users of a dictionary
program.
Figure
1 Diagram of user
actions and corresponding programming sections
The features of user options
and possible variants in a dictionary search is discussed later in the paper.
From a software developer’s perspective, there are the three major sections to
the software:
1. Input
of entries for users: program has to be able to read in the
query of the user, in any of the languages specified. The system also needs to
be able to cater for the input of non-English characters into the query space.
2. Retrieval
of entry information: an important aspect, the retrieval
system of the dictionary is the heart of the application. A weak system will
result in fewer users due to lack of efficiency. The retrieval system is
required to search through the dictionary file, find matches in the entries,
compile relevant information and send the data to the application.
3. Output
of entries for users: data is received from the retrieval and
the next major operation is to format the information in for the user to view
the information. This task is a user interface and data representation problem.
The
following is a diagram describing an overview of the possible data paths for
the application:
Figure 2 Diagram describing the
possible data paths dictionary data
The operation of information
retrieval presents various issues, or ‘trade offs’ when choosing a technique to
implement. Usually, when implementing a technique, there are many advantages to
why the technique is used over another technique, however there are also
tradeoffs for using the certain method. Deciding which implementation is used
is generally a weighing up of priorities; important features that must be
optimal and other features which are not as important. The factors and costs
that need to be weighed when choosing implementations are:
-
Efficiency
of the searching system
-
Storage
space required to store the file
-
Storage
space required when processing the file
-
Frequency
of access of entries
-
Frequency
of access of certain entries
-
Amount
of data to be extracted per search
-
Accessibility
to multiple users
These costs will be analysed as different
information retrieval techniques are discussed.
Multilingual text input
An information retrieval system may typically consist of phases for
input, storage, processing, editing, output and transfer. Input is a serious
problem, resulting in a bottleneck. In a non-ASCII context, there is the
problem of text input. More than 500 encoding schemes have been devised alone
for Chinese character input. In many cases, a traditional western keyboard,
which will undoubtedly be the main input source for the multilingual
dictionary, is insufficient for text input. [Lunde, 1999]
Japanese is a language where each kana or compound character can be
represented with a unique phonetic. By understanding this, and creating a
conversion file or table listing the compound characters and Romaji equivalent,
it is possible to type a Romaji query and have it converted immediately into
Kana for the user to see before sending the query to the search engine. This is
a well-established technique and can be applied quite effectively for Japanese
character input.
Figure
3 Excerpts from a
Romaji to Kana conversion file. Pairing together a combination of phonetics
will result in a character, and combining these together will result in a
compound word.
There are two general types of ‘fast’, or configurative input for
characters: those which require memorization of character code numbers and
those which depend on the internalization of key positions directly related to
character codes. An example of this is the input of Chinese characters input
via the decomposition of each Chinese character into radicals or strokes that
are subsequently transformed into letters of numerals on the keyboard. ‘Easy’,
or phonetic methods, seek to exploit the users existing knowledge of Chinese
language and script to minimize the need to acquire new skills. On the
keyboard, we need to specify, not produce the character we want. One form of
specification is by pronunciation, the most widely used in ‘easy’ input [Mair
and Liu, 1991].
There is a general class of input of foreign characters called IM or
IME (Input Method Editors). Microsoft has developed a free IME they have called
Microsoft Global IME 5.02. It has been developed to enhance East Asian
character input. Global IME allows users to input Asian characters without any
special keyboard or equipment. Because Microsoft operating systems are used so
widely, this free input system is fast becoming a ‘de facto’ standard for
character input [IME, 2000].
Figure 4 Microsoft Global IME in
action
A common and well-known example of an IM is an IM for word processors.
The following is a diagram of the Chinese Star IM that can be used to input
Chinese characters into Microsoft Applications.
Figure
5 Chinese Star word
processor Chinese character input system. Involves typing PinYin into a translation
box. A list of characters that correspond to the PinYin are displayed.
Outputting the correct characters is a process of choosing the correct
character
However, some of these techniques are much too complicated and only
simple versions of them may be needed for short text input queries. The
emphasis for inputting foreign characters into a multilingual dictionary is not
speed, because so few letters will be put in at one time, but more of usability
and ease of use for the user. The investigation and implementation of various
input methods for different languages is out of the scope of this project. As
more languages are added to the multilingual dictionary, more and more input
techniques may be required to support the languages. Users could find it
difficult to adapt to all the different kinds of input techniques available.
Multilingual text
output
In the past, there a major obstacle in foreign text processing was the output of the characters. There are several ways of representing the same font information. The different representations came from the different font houses (such as Adobe) creating their own standards. Fonts are either bitmapped or outlined (scalable). Bitmapped fonts represent each character as a rectangular grid of pixels. There are a number of disadvantages to this approach, but the most important one is the difficulty to change the size, shape, and resolution of a bitmapped character without loss of quality because the bitmap is defined at a certain size and resolution. Outline fonts represent each character mathematically as a series of lines and curves. The font must be 'rasterized' into a bitmap. LaserJet .SFP and .SFL files, TeX PK, PXL, and GF files, Macintosh Screen Fonts, and GEM .GFX files are all examples of bitmapped font formats. PostScript Type 1, Type 3, and Type 5 fonts, Nimbus Q fonts and TrueType fonts are all examples of outline font formats.
In
addition to these two types of font archive formats, certain font standards,
there are further issues. Identical formats on different platforms are not
necessarily the same. For example Type 1 fonts on the Macintosh are not
directly usable under MS-DOS or Unix, and
vice-versa. There are just as many different font formats. Two major
font formats are discussed:
PostScript Type 1 Fonts: Postscript Type 1 fonts
(Also called ATM (Adobe Type Manager)
fonts, Type 1, and outline fonts) contains information, in outline form,
that allows a postscript printer, or ATM, to generate fonts of any size.
TrueType Fonts: Truetype fonts are a new font format
developed by Microsoft with Apple. The rendering engine for this font is built
into MS Windows v3.1 and subsequent versions. Like PostScript Type 1 and Type 3
fonts, it is also an outline font format that allows both the screen, and
printers, to scale fonts to display them in any size.
The following is a table
that describes some of the font extensions and their platform usage. Despite
all these difficulties, there are now standard libraries (such as those listed
above) that define the font types and font information, so the problem of
displaying foreign characters is not such a major problem today.
|
Extension |
Usage |
|
* .fon |
An MS-Windows bitmapped font |
|
* .pfa |
Adobe Type 1 Postscript font in ASCII format
(PC/Unix) |
|
* .pfb |
Adobe Type 1 PostScript font in "binary`'
format (PC/Unix) |
|
* .ps |
Any PostScript file (Type 3 font) |
|
* .pxl |
TeX pixel bitmap font file |
|
* .ttf |
MS-Windows True Type font |
Table
3. Font extensions and their platform usage
Chapter 4
Internal representation of the XML document
The ‘heart’ of the application lies in the internal representation of the multilingual dictionary. Failure to create an efficient system will result in a disappointing application. The concepts of XML are explored in this chapter and its role in the JMdict file. In addition, XML specific tools are examined, outlining their properties and also their applicability for use in a multilingual dictionary application.
4.1 XML
XML stands for ‘eXtensible
Markup Language’. It is the standard system for defining the content and format
of an electronic document. HTML tells how the data should look, but XML tells you what it means[Goldfarb, 1998]. The
differences are more distinct than just that. HTML has permanent markup tags,
for example the <bold> or <href> tags. Additional tags cannot be
defined therefore restricting the applications of HTML. From the computer’s
perspective, there is no structure of the information supplied in an HTML file.
It differs from a similar markup language in that XML is designed keeping in
mind that document format should be specific to the type of document that is
being created. This allows XML to be used as a genuine storage method, as
JMdict has shown. It allows the programmer to separate data from display. The
following are some of the strengths of XML:
Extensibility – allows
users to define their own tags (or attributes) to suit the data being
represented.
Structure – allows
nested structures of any depth, hence being suited to dictionary-type entries.
Validation – allows
the document to be validated before use by applications
Another important feature of
XML is that it enables the definition of tags for each individual document. The
formal definition that describes each type of tag is called a document type
definition (DTD). The following is a small sample of what a tag definition
looks like:
<!DOCTYPE
label[
<!ELEMENT label (name, street, city,
state)>
<!ELEMENT name (#PCDATA)>
<!ELEMENT street(#PCDATA)>
<!ELEMENT city (#PCDATA)>
<!ELEMENT state(#PCDATA)>
]><label>
This DTD defines a
label. Each label will contain an element name, street, city and state. By allowing user-defined tags, XML
improves functionality and increases the appeal to developers, enabling them to
create any type of data document [XML.com, 2000].
Elements
have their own attributes that are properties for elements. This is simply the
value for the element that is being presented. An example of this in the JMdict
file is the <gloss g_lang="de"> tag. It is an element of type
gloss g_lang. To define the element, it is given the attribute “de” which
stands for German. By using this unique feature of XML, different languages can
easily be entered into the dictionary file without the need to design
additional tags.
XML has a hierarchical view
of XML documents which is referred to the tree structure of the document. The
structure of the entry: ‘New Year’s sake’ is displayed in the following
diagram.
Figure
6 Tree structure
view of an entry in JMdict
JMdict
Structuring of a document or storage file is an
extremely important factor, especially with one as large as the EDICT file (See
Appendix) containing over 100,000 entries. A plain, unstructured dictionary
file format such as EDICT allows the linguist a great deal of freedom with the
structure of the dictionary. The disadvantage is that the file must be accessed
from top to bottom, from beginning to end. It is also application independent.
This is a good method of storage if the amount of data is relatively small and the
combined delays of searching are not an issue. Even word processor searches for
particular words in a relatively long file can output a result in good time.
The dictionary file has to be more structured to allow for more efficient
access as the data set gradually increases in size, and when extra data
attributes are added. In addition to these problems, the limited structure of
flat file dictionaries like EDICT means that for each entry, the amount of
includable information is restricted as information extraction from the file
would become extremely difficult. XML was chosen as the data format for the new
dictionary file because it allowed for additional attributes and most
importantly extendibility. Grasping the opportunity to utilize the
possibilities of extendibility of the file, German glosses were added to
dictionary entries. Breen called this new file the ‘Japanese Multilingual
Dictionary’ or JMdict. Its superiority over other electronic dictionary files
of this type meant that the JMdict file would be the primary data source, and
the dictionary application would be built around this file.
According to Breen[2000 (b)], the aims of the JMdict
project were as follows:
- “To convert the EDICT file to a new dictionary structure which
overcomes the deficiencies in the current structure. With regard to this
goal, the particular structural and content aspects to be addressed
include, but are not limited to:
- The handling of orthographical variation (for example, in Kanji
usage, Okurigana usage, readings) within the single entry;
- Additional and more appropriately associated tagging of
grammatical and other information;
- Provision for separation of different senses (polysemy) in the
translations;
- Provision for the inclusion of translational equivalents from several
languages;
- Provision for inclusion of examples of the usage of words;
- Provision for cross-references to related entries.
- To publish the dictionary in a standard format which is accessible
by a wide range of software tools; it is proposed that this goal be
addressed by developing the structure so that it can be released as an XML
document, with an associated XML DTD.
- To retain backward compatibility with the original EDICT structure
in order to enable legacy software systems to use later versions of the
EDICT files”
The following is figure of a sample of JMdict as
displayed on Internet Explorer 5 that is capable of displaying Unicode fonts:
Figure 7 Sample of the JMdict
file displayed on a Unicode capable browser
XML is a simplified subset of SGML and is not
really optimized for the Web environment. However, it does mean that it is data
processing-oriented (compared to browser-oriented HTML) and should not be seen
by the end-user. This is not to say that XML cannot be presentable. Data stored
in XML documents still need to be presented to users in an attractive format.
Rules for formatting an XML document is called a stylesheet and can be used to
transform the raw XML data into a HTML document complete with hyperlink markups.
The language used to create the stylesheets is called XSL, or extensible style
language. [Goldfarb, 1998]
4.2 XML information access
XML parsing systems
Because such a structured storage
structure is being used to hold the dictionary entries, more sophisticated
methods of searching and entry handling can be carried out. By using the XML
storage structure, the entries need to be parsed by an XML parser. There are
two major types of implementation of XML parsers available for processing XML
documents:
-
Tree-based APIs
-
Event-based APIs
Firstly,
there is a SAX (Simple API for XML) implementation of XML parsing. This method
is an event-based API that uses a simple technique in which the parser searches
through the XML document in a logical order to find matching entries. On the
other hand, there is a tree-based API called DOM (document object model)
implementation. This is slightly more sophisticated in that it treats the
entries individually as objects. [Laddad, 2000] Both of these APIs were created
to serve the same purpose: to provide access to the information stored in an
XML file. The advantages and disadvantages of using both these methods will be
studied consequently to determine if they can process the JMdict efficiently.
Of particular interest is how the XML document is broken down by the parser.
Additional features that can be implemented in electronic dictionaries such as
cross-referencing will be compared when using these two different XML parsing
techniques
Using Java
When deciding what
programming language to use to implement the multilingual dictionary, several
requirements need to be fulfilled.
-
Ability
to handle Unicode
-
Compatibility
with XML tools, in particular SAX and DOM
-
Ability
to integrate with the internet
-
Efficient
and re-usable code
Java appears to be the best
language to code the application. One of its greatest strengths is that its
internal character is in fact Unicode, immediately reducing the complexity of
attempting to convert between foreign character standards and internal
character representations. In comparison with C or C++, string handling in Java
is far superior. Many XML tools are being developed for Java and XML is
beginning to be synonymous with Java. For example, IBM’s implementation of the
XML parser has been released as XML4J (XML for Java). Standards such as DOM
(and JDOM, see section XX) are being put forward to be included in the next
Java release.
4.3 DOM (Document Object Model)
DOM not only addresses the object model in XML, but also in HTML, as it is also a structured document. Currently, DOM is specified up to Level 1 core. The DOM level 2 specification is still in a working draft form and it is guaranteed to change before it is officially a standard.
DOM represents a document
tree fully held in memory. It is a large API designed to perform almost every
conceivable XML task. It also must have the same API across multiple languages.
Because of those constraints, DOM does not always come naturally to Java
developers who expect typical Java capabilities such as method overloading, the
use of standard Java object types, and simple set and get methods. DOM also
requires lots of processing power and memory, making it untractable for many
lightweight Web applications and programs. However, we will still investigate
the features of DOM and its suitability.
The DOM class heirachy is divided into several
layers.
-
Document – the Document node is the master node – only one of
these can exist for an XML document. It represents the XML document as a whole.
-
NodeList – This type of node is used
to hold a collection of child nodes. It basically allows access to the children
-
Named NodeMap
– Contains
additional functionality in relation to NodeList in that it is able to access
the children by their names
-
Element – Contains an element from
an XML document. This can be thought of as the name of the tag used in an XML
document <gloss>
-
Text – These are used to
represent text contained within the element tags. <gloss>to
conspire</gloss>
-
Attr (attribute) – This node
represents the attributes declared within the scope of an element. An example
of this in XML format is <gloss g_lang=”de”>
-
CDATASection – Similar to the text node,
however, it can contain markup. This allows the user to specify text with XML
control characters such as ‘/’ and ‘>’
-
DocumentType – is a node that represents
the tags used in the Document Type Definition.
-
Entity,
EntityReference and
Notation – Are used to describe nodes
used in the DTD.
The relationship and hierarchy between the different
node types can be visualized on the following page. Various interface methods
can be carried out on these nodes to manipulate and access them. Examples of
various functions are:
-
Child
modifiers
-
Node
creation, grabbing, moving and deletion
-
Element
methods
-
Element
usage such as child iteration
Thus, DOM provides
mechanisms needed to dynamically interact with the elements and content in an
XML document. With DOM, handlers, or hooks, are created to encapsulate
behaviors to associate with elements in the DOM tree. DOM sets out to be able
to model every possible well-formed XML document. Therefore, DOM classes
contain features that many XML applications never use. An electronic dictionary
requires few of these classes. In almost all procedures, operation to be
carried out would be the grabbing of nodes and attributes. The file is not
required to be, or should be, modified by different user session, so interfaces
provided for dynamic modification of XML document structure is not required.
In summary, DOM is really
useful for business applications which require the dynamic manipulation of
elements and content in XML files. These features that are important in other
applications are not a major factor in the context of creating a dictionary
application. The only concern in the usage of JMdict is the efficient access
and searching of the XML file. DOM can provide this, but at a huge cost, which,
unfortunately is not suitable for this application. An initial experiment
carried out using the DOM parser to create the DOM tree on JMdict resulted in a
program crash, the problem being the system was out of memory. With ever
increasing dictionary file, DOM alone does not provide a feasible option.
