-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy patheolas.html
374 lines (372 loc) · 21.2 KB
/
eolas.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<html lang="en">
<head>
<title>aimsigh.com: English</title>
<meta http-equiv="Content-Language" content="en">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="description" content="Detailed description of the aimsigh.com search engine in English">
<meta name="keywords" content="cuardach, cuardaigh, aimsigh, gréasán, inneall cuardaigh, Gaeilge, Éire, search, search engine, Irish, Ireland">
<link rel="stylesheet" href="/aimsigh/aimsigh.css" type="text/css">
<link rel="shortcut icon" href="/aimsigh/favicon.ico" type="image/x-icon">
</head>
<body>
<div class="content">
<p>
First, I apologize to Irish-speaking users of the site for writing this page
in English; it is directed primarily at my colleagues working
on natural language processing and minority languages, particularly
at those who might be interested in creating similar
"linguistically sophisticated" search engines.
Because of this, I won't assume any prior knowledge of
Irish linguistics in what follows.
</p>
<p>
There were two primary motivations for creating this site;
the first was that existing search engines like
<a href="http://www.google.com/">Google</a> and
<a href="http://www.yahoo.com/">Yahoo!</a>,
as powerful as they can be for English search, are unsuitable for
Irish in various ways that
will be discussed in detail below.
The second was a desire on my part to create a single tool
that harnesses most of my
previous work on language technology for Irish.
These earlier projects
include the first Irish spellchecker
(<a href="http://www.gaelspell.com/">GaelSpell</a>),
part-of-speech tagger, morphological analyzer, grammar checker
(<a href="http://borel.slu.edu/gramadoir/"><i>An Gramadóir</i></a>),
electronic thesaurus, and web-crawled corpora
(both <a href="http://borel.slu.edu/crubadan/">monolingual</a> and
<a href="http://borel.slu.edu/corpas/">bilingual</a>).
</p>
<p>
Below I've listed some of the features of <i>aimsigh.com</i>
that are not available with standard general-purpose search
engines (and because of the marginalized position of Irish
are unlikely ever to be so — more on this point below).
</p>
<p>
<b>1. Spelling standardization</b><br>
Irish underwent a major spelling reform in the 1940's
and 1950's, introducing the so-called <i>Caighdeán Oifigiúil</i> (Official Standard):<br><br>
<i>An t-árd-cheannas ar na Fórsaíbh Cosanta is le dligheadh a riaghlóchar an modh ar a n-oibreochar é.</i><br>
<i>An t-ardcheannas ar na Fórsaí Cosanta is le dlí a rialófar an modh ar a n-oibreofar é.</i><br><br>
Most writing in Irish today conforms to the standard,
though certainly not 100%: many writers
still prefer to use spellings and grammatical
constructs that reflect more accurately their
own dialect of Irish. In addition there are quite
a few historical and legal documents on the web that use
pre-standard orthography, most of them produced
and published by
the Irish government (one major source is the site
<a href="http://www.achtanna.ie/">www.achtanna.ie</a>,
which contains the full text of all Acts enacted by the Irish
Parliament since 1922).
So independent of the side one might take in the debate over the
merits of the <i>Caighdeán</i>, these pre-standard and dialect
documents are "out there", and we are faced with the
inescapable engineering challenge of making them easily available
through a search interface.
</p>
<p>
To achieve this, the <i>aimsigh.com</i> engine employs a sophisticated
"Irish standardizer" which amounts to a finite state transducer
encoding the morphological rules of non-standard Irish together
with mappings to standardized forms. These rules are augmented
with a large database of non-standard/standard word pairs that was extracted in
part from a parallel corpus of
English and Irish texts (read more about this here:
<a href="http://borel.slu.edu/pub/ccgb.pdf">Applications of
parallel corpora to the development of monolingual
language technologies</a>).
The end result is that if a user selects the box
<i>Litriú neamhchaighdeánach</i> (Non-standard spelling)
on the main <i>aimsigh.com</i> page, and enters in word like
<i>Gaeilge</i> (Irish language), any documents containing either
<i>Gaeilge</i> or one
of the non-standard spellings <i>Gaolainn, Gaedhilge, Gaedhilg, Gaedhealg, Gaedhealaing, Gaeilic, Gaoidhealg, Gaodhalainn, Gaelainn, Gaeluinn, ...</i>, etc.
will be retrieved.
As an illustration, try a Google search for <i>"truailliú uafásach"</i>
and then try <i>aimsigh.com</i> with the non-standard spelling box
activated:<br><br>
<b><i>"truailliú uafásach"</i></b>: <a href="http://www.google.com/search?q=%22truailli%C3%BA+uaf%C3%A1sach%22">Google</a> <a href="http://borel.slu.edu/cgi-bin/aimsigh.cgi?ionchur=%22truailli%C3%BA+uaf%C3%A1sach%22&feicthe=0&claochlu=NN&neamhchaighdean">aimsigh.com</a><br>
</p>
<p>
A nice side-effect of this feature is that the standardization process
also corrects common spelling errors, so if you can't remember
how to spell <i>ionannas</i> and you search for <i>ionnanas</i>, you will
still retrieve all documents containing the correct spelling.
Conversely a search for <i>Údarás</i> ("Authority", spelled correctly)
will turn up documents containing
misspellings <i>Udarás</i> or <i>Údaras</i> which is probably the
desired behavior, since such misspellings are remarkably common,
even in presumably-edited texts.
</p>
<p>
<b>2. Initial mutations</b><br>
The beginning of a word in Irish can be written in
different ways depending on the grammatical context.
For example, <i>bean</i> (woman) becomes <i>an bhean</i> after
the definite article <i>an</i> and <i>ár mbean</i> after the
possessive pronoun <i>ár</i> (our). Several other possibilities
occur when a word begins with a vowel: <i>athair</i> (father)
can become <i>t-athair, n-athair, d'athair</i>, etc.
In most cases, the presence or absence of one of these
mutations has no real effect on the semantics of the word
in question, somewhat like the presence or absence of an
initial capital on a (non-proper) noun in English.
In other words, someone searching for information on lexicography
(<i>foclóireacht</i>) would surely be just as happy
to retrieve documents containing <i>fhoclóireacht</i> or <i>bhfoclóireacht</i>.
</p>
<p>
This behavior can be achieved by selecting
the button <i>Focail chlaochlaithe</i> (Mutated words) on
the main <i>aimsigh.com</i> page (I generally select this button
for all of my own searches). For example, to find documents concerning
the country of Sudan, one might search for the term <i>Súdáin</i>,
but since this word generally follows the definite article in Irish,
and is therefore prefixed with a "t", it is much more effective
to search with <i>aimsigh.com</i>:
<br><br>
<b><i>Súdáin</i></b>: <a href="http://www.google.com/search?q=S%C3%BAd%C3%A1in">Google</a> <a href="http://borel.slu.edu/cgi-bin/aimsigh.cgi?ionchur=S%C3%BAd%C3%A1in&feicthe=0&claochlu=YN">aimsigh.com</a><br>
</p>
<p>
<b>3. Inflectional morphology</b><br>
Irish morphology is much more complicated than English
morphology, and because of this, it is desirable to perform
"stemmed" searches in many instances. For example,
if one is interested
in Irish language schools it is convenient to be able
to search for the single term <i>gaelscoil</i> and retrieve all documents
containing <i>gaelscoil, gaelscoile</i> (genitive),
or <i>gaelscoileanna</i> (plural) as well as the mutated forms
of these words (<i>ghaelscoil, ngaelscoil</i>, etc., nine words in all).
Verbal morphology is even more complicated,
with a single root word typically producing more than 50
inflected/mutated forms.
</p>
<p>
Stemmed searches can be performed by selecting
the button <i>Focail chlaochlaithe infhillte</i> (Mutated and inflected words).
For example, if you are interested in monetary policy, you would
naturally try to search for <i>airgeadaíocht</i>; while Google returns
only a couple of results for this query, we get several hundred with the
<i>aimsigh.com</i> stemming feature activated:<br><br>
<b><i>airgeadaíocht</i></b>: <a href="http://www.google.com/search?q=airgeada%C3%ADocht">Google</a> <a href="http://borel.slu.edu/cgi-bin/aimsigh.cgi?ionchur=airgeada%C3%ADocht&feicthe=0&claochlu=YY">aimsigh.com</a><br>
</p>
<p>
Stemmed searching has a mixed reputation in information retrieval circles,
though this might be due in part to the fact that most research has
been done on English or other languages with similarly limited
morphological complexity.
The other issue is that a lot of online stemming is done with
"resource-light" approaches like the Porter algorithm;
<i>aimsigh.com</i> instead uses a full lexicon and morphological analysis to
guarantee correct stemming.
It is worth noting also that Irish morphology is not nearly as complicated
as languages like Basque, Swahili, or Hiligaynon,
where a certain amount of stemming
would seem to be absolutely essential.
</p>
<p>
<b>4. Only Irish language documents</b><br>
Irish language documents on the web are drowned in a veritable
sea of English, Spanish, German, etc., and
simple searches with Google or Yahoo! are often fruitless
because of this.
One issue of course is that some Irish words
accidentally coincide with English words: think of <i>bean</i> (woman),
<i>punt</i> (pound), <i>file</i> (poet), or <i>tine</i> (fire).
And English is not the only problem;
if you search for a very Irish-looking word like <i>luach</i> (value)
with a standard search engine, you'll turn up very few
Irish documents because this word
also means "calendar" in Hebrew.
In addition, there are many lexical conflicts with
Scottish Gaelic, which has a somewhat smaller
but not inconsequential web presence; a search
for a word like <i>ceist</i> will yield documents
split roughly half-and-half between Scottish and
Irish Gaelic.
So perhaps, in frustration, you swear off ever using
any words that happen to coincide with anything from
any other written language, and decide to search for
<i>"Bunreacht na hÉireann"</i> (Constitution of Ireland).
As it turns out, the first seven hits on Google point
to English documents!
A related (but obviously less important) issue is the irritation
of having to click through
"Choose your language" splash pages on Irish governmental web sites
(when indeed a choice is offered). You are taken to such a page
for example when you
search for <i>Foras na Gaeilge</i> with Google;
the top hit for <i>aimsigh.com</i> is instead
the Irish language home page.
Try some of these queries for yourself:<br><br>
<b><i>bean</i></b>: <a href="http://www.google.com/search?q=bean">Google</a> <a href="http://borel.slu.edu/cgi-bin/aimsigh.cgi?ionchur=bean&feicthe=0&claochlu=NN">aimsigh.com</a><br>
<b><i>luach</i></b>: <a href="http://www.google.com/search?q=luach">Google</a> <a href="http://borel.slu.edu/cgi-bin/aimsigh.cgi?ionchur=luach&feicthe=0&claochlu=NN">aimsigh.com</a><br>
<b><i>ceist</i></b>: <a href="http://www.google.com/search?q=ceist">Google</a> <a href="http://borel.slu.edu/cgi-bin/aimsigh.cgi?ionchur=ceist&feicthe=0&claochlu=NN">aimsigh.com</a><br>
<b><i>"Bunreacht na hÉireann"</i></b>: <a href="http://www.google.com/search?q=%22Bunreacht+na+h%C3%89ireann%22">Google</a> <a href="http://borel.slu.edu/cgi-bin/aimsigh.cgi?ionchur=%22Bunreacht+na+h%C3%89ireann%22&feicthe=0&claochlu=NN">aimsigh.com</a><br>
<b><i>"Foras na Gaeilge"</i></b>: <a href="http://www.google.com/search?q=%22Foras+na+Gaeilge%22">Google</a> <a href="http://borel.slu.edu/cgi-bin/aimsigh.cgi?ionchur=%22Foras+na+Gaeilge%22&feicthe=0&claochlu=NN">aimsigh.com</a><br>
</p>
<p>
This feature is, of course, not particularly remarkable
in a technological sense; most search
engines offer the ability to restrict results
to particular languages. The problem is that they usually
only offer a selection of the most prominent 30-35 languages
on the web. Now without too much work (and granted a certain amount
of volunteer help from native speakers) I was able
to train statistical language recognizers for the "next" 150
or so languages and run web crawlers
for each of them (see <a href="http://borel.slu.edu/crubadan/">Corpus building for minority languages</a> for more information). So I suspect that the
restriction to 30+ languages on Google's site must be a user interface
decision on their part, so that
Swedish speakers won't have to scan through a list of
200 (or 500 or 1000) languages to find Swedish in a pull-down menu.
</p>
<p>
<b>5. Non-standard representations of <i>síntí fada</i></b><br>
For the non-Irish-speaking readers,
<i>síntí fada</i> are the acute accents that appear on many
vowels in Irish. Back in the day before 8-bit email was
widely available, messages sent to popular email discussion
lists like
<a href="https://listserv.heanet.ie/gaelic-l.html">GAELIC-L</a>
were written with the infamous <i>slaiseanna</i>
to indicate the accents: <tt>u/rsce/alai/</tt> = <i>úrscéalaí</i> (novelist)
or <tt>te/acschomhad</tt> = <i>téacschomhad</i> (text file).
So, as a consequence, the archives of such mailing lists
(which form the single largest source
of Irish language material on the web as of this writing),
are essentially invisible to standard search engines (which
would all index the above words as separate units "u"+"rsce"+"alai"
or "te"+"acschomhad").
In contrast, the <i>aimsigh.com</i> engine automatically detects pages
that use unusual conventions and converts them to a standard format for
indexing. See for yourself:<br><br>
<b><i>acschomhad te ar</i></b>: <a href="http://www.google.com/search?q=acschomhad+te+ar">Google</a> <a href="http://borel.slu.edu/cgi-bin/aimsigh.cgi?ionchur=acschomhad+te+ar&feicthe=0&claochlu=NN">aimsigh.com</a><br>
<b><i>"ar téacschomhad"</i></b>: <a href="http://www.google.com/search?q=%22ar+t%C3%A9acschomhad%22">Google</a> <a href="http://borel.slu.edu/cgi-bin/aimsigh.cgi?ionchur=%22ar+t%C3%A9acschomhad%22&feicthe=0&claochlu=NN">aimsigh.com</a><br>
</p>
<p>
<b>6. Decapitalization according to Irish conventions</b><br>
When certain initial mutations ("t" and "n" before vowels) are
added to uppercase words, the mutating letter is written in
lowercase and without a hyphen: <i>Acht</i> > <i>tAcht</i> (a legislative act),
or <i>Ocht</i> > <i>nOcht</i> (eight).
On the other hand, the lowercase versions
of the same words are written <em>with</em> hyphens: <i>t-acht</i>, <i>n-ocht</i>.
The really bad news is that in these two cases, naïve conversion to lowercase
produces completely different Irish words (<i>tacht</i> is a verb meaning
"choke" and <i>nocht</i> is either a verb or adjective meaning "bare").
So if (for whatever reason)
you enter <i>tacht</i> into Google, the Irish language results that are
retrieved are all incorrect, referring to <i>tAcht</i>:<br><br>
<b><i>tacht</i></b>: <a href="http://www.google.com/search?q=tacht">Google</a> <a href="http://borel.slu.edu/cgi-bin/aimsigh.cgi?ionchur=tacht&feicthe=0&claochlu=NN">aimsigh.com</a><br>
</p>
<p>
<b>7. Automatic translation and augmentation of document titles</b><br>
One of my main interests is in machine translation, and I have
a rudimentary system in place for translating
English text to Irish.
This is used when documents are harvested from the web
to translate boilerplate English titles into Irish;
for example "GAELIC-L Archives - June 2000" would appear in
<i>aimsigh.com</i>
search results as "Cartlann GAELIC-L - Meitheamh 2000".
Or bilingual titles like "TG4 - Irish language television channel - Teilifis Gaeilge" [sic] are shortened and corrected to
"TG4 - Teilifís Ghaeilge". This makes for more effective
searching (since we assume primarily Irish language search terms
will be used) and also a more pleasing Irish-only visual experience
when scanning results.
</p>
<p>
In addition, we augment useless titles with additional
information to help clarify the contents of the document.
Here "useless" means, in the worst case, no title at all, as is the case
for articles in the newspaper <a href="http://www.nuacht.com/"><i>Lá</i></a>,
but also refers to situations in which the title, even if translated, fails
to provide useful clues as to the contents of the document, or fails
to distinguish the document from others on the site; good examples
are pages from the Irish Times <i>Teanga Bheo</i> site,
the majority of which are titled simply
"An Teanga Bheo - The Irish Times weekly Irish language site".
For the <i>Lá</i> articles, we do our best to extract the
title from the body of the HTML document and construct a title from that.
For the <i>Teanga Bheo</i> articles, it is possible to extract the date of
publication from the URL; this is translated to Irish and appended
to the title. Similar tricks are used for a number of other sites.
Here are a couple of sample queries illustrating these points:<br><br>
<b><i>tionscnamh Chomsky</i></b>: <a href="http://www.google.com/search?q=tionscnamh+Chomsky">Google</a> <a href="http://borel.slu.edu/cgi-bin/aimsigh.cgi?ionchur=tionscnamh+Chomsky&feicthe=0&claochlu=NN">aimsigh.com</a><br>
<b><i>ollscriosta Gilligan</i></b>: <a href="http://www.google.com/search?q=ollscriosta+Gilligan">Google</a> <a href="http://borel.slu.edu/cgi-bin/aimsigh.cgi?ionchur=ollscriosta+Gilligan&feicthe=0&claochlu=NN">aimsigh.com</a><br>
</p>
<p>
<b>8. Manually-curated document database</b><br>
Above, we bemoaned the fact that Irish language documents form
only the tiniest fraction of all those available on the web.
On the other hand, once the Irish documents have been separated
from the rest, the small size of the resulting database
(hundreds of thousands of documents vs. billions) is a great
advantage. On the one hand, there seem to be quite a few
Irish documents (thousands) out there that are not indexed at all by Google
(some examples linked below); I suspect that we had better luck in
turning these up by focusing our crawling on a relatively small number
of especially productive (mostly .ie) domains:<br><br>
<b><i>TnaG mioncheisteanna</i></b>: <a href="http://www.google.com/search?q=TnaG+mioncheisteanna">Google</a> <a href="http://borel.slu.edu/cgi-bin/aimsigh.cgi?ionchur=TnaG+mioncheisteanna&feicthe=0&claochlu=NN">aimsigh.com</a><br>
<b><i>Aifreann Cochlainn</i></b>: <a href="http://www.google.com/search?q=Aifreann+Cochlainn">Google</a> <a href="http://borel.slu.edu/cgi-bin/aimsigh.cgi?ionchur=Aifreann+Cochlainn&feicthe=0&claochlu=NN">aimsigh.com</a><br>
<b><i>orgánach huirbeach</i></b>: <a href="http://www.google.com/search?q=org%C3%A1nach+huirbeach">Google</a> <a href="http://borel.slu.edu/cgi-bin/aimsigh.cgi?ionchur=huirbeach+org%C3%A1nach&feicthe=0&claochlu=NN">aimsigh.com</a><br>
</p>
<p>
Conversely, "spam sites" are becoming more
of a problem for the large search engines, and quite a few searches
for Irish language documents will turn up useless sites that
simply repeat verbatim excerpts from various other pages.
For example, if you search for the title "Irish Times weekly Irish language"
on Google, there are 44 unique results returned of which only 5 appear
to be legitimate. Because of the smaller scale of <i>aimsigh.com</i>
it is an easy matter to detect such sites semi-automatically
and then remove them from our indices:<br><br>
<b><i>"Irish Times weekly Irish language"</i></b>: <a href="http://www.google.com/search?q=%22Irish+Times+weekly+Irish+language%22">Google</a> <a href="http://borel.slu.edu/cgi-bin/aimsigh.cgi?ionchur=%22Irish+Times+weekly+Irish+language%22&feicthe=0&claochlu=NN">aimsigh.com</a><br>
</p>
<p>
<b>9. Irish-centric page ranking</b><br>
Even when there are documents available that contain
legitimate Irish text, it sometimes happens that the page ranking algorithms
are skewed in favor of sites that are heavily linked
from non-Irish web pages. A good example is the Open Directory
Project <a href="http://dmoz.org/">dmoz.org</a> which often
gets high page rankings because of its wide popularity (especially
among English and German speakers). Unfortunately almost
no Irish language sites are contained in the directory and so
results from dmoz.org or one of its many mirrors are of little
use to an Irish speaker. This can be illustrated by searching for
something like <i>Eolaíocht</i> with Google; about half of the returned
results are from Open Directory mirror sites:<br><br>
<b><i>Eolaíocht</i></b>: <a href="http://www.google.com/search?q=Eola%C3%ADocht">Google</a> <a href="http://borel.slu.edu/cgi-bin/aimsigh.cgi?ionchur=Eola%C3%ADocht&feicthe=0&claochlu=NN">aimsigh.com</a><br>
</p>
<p>
Another example is the "Foras na Gaeilge" query discussed and linked above;
the default English page is listed as the second search result,
while the default Irish page is nowhere to be found.
When ranking pages, <i>aimsigh.com</i> only considers links
emanating from other Irish language pages; while this system is
still not perfect it seems to improve matters greatly.
</p>
<hr>
<p class="anbheag">
Cóipcheart © 2005 <a href="http://borel.slu.edu/index.html">Kevin P. Scannell</a>. Gach ceart ar cosnamh.<br>
Déan teagmháil linn ag <a href="mailto:[email protected]">[email protected]</a>.
</p>
</div>
<div class="navigation">
<a href="index.html">Cuardach</a><br>
<a href="cabhair.html">Cabhair</a><br>
English<br>
<a href="http://borel.slu.edu/">Forbróir</a><br>
</div>
</body>
</html>