Decomposition of German compound words for effective searching (2)

I have done some little experiments with the decomposition algorithm I suggested in the blog earlier (you find it here). For hyphenation I used the hyphenator of the Apache FOP project. The dictionary I got from this page: http://wiki.services.openoffice.org/wiki/Dictionaries. For fuzzy matching I use Apache Lucene's spellchecker extension.

Some examples generated from the current code:

  • Donaudampfschifffahrtskapitänsstellvertreter -> [donau, dampf, schiff, schifffahrts, schifffahrt, fahrt, kap, kapitäns, kapitän, pi, stell, steller, stelle, stellvertretend, vertreter, treter, treue]
  • Donaudampfschifffahrtselektrizitätenhauptbetriebswerkbauunterbeamtengesellschaft -> [donau, dampf, schiff, schifffahrt, schifffahrts, fahrt, fahr, elektrizität, elektrizitäts, elektro, elektrik, jäten, täte, haupt, betrieb, betriebs, trieb, werk, wer, bau, bauen, un, unter, werbe, bea, beamten, beamte, am, amten, menge, geselle, gesellschaft, schaft, schaf]
  • Rindfleischetikettierungsüberwachungsaufgabenübertragungsgesetz -> [rind, rindfleisch, fleischer, fleisch, fleischerei, scheu, tibet, kette, rings, über, überwachungs, auf, au, aufgab, aufgaben, aufgabe, gabeln, gabe, ben, übertrat, übertragungs, tagungs, gesetz, set]
  • Urinstinkt -> [urin, urinstinkt, in, instinkt]
  • Drahtschere -> [draht, schere, schor]

For only one hour work this looks promising.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

compound german words

Dear Mr. Thomas,

Could you please send me the piece of source code related to this topic. I just saw a short description on your site and I am interested in this solution.

Best regards,
Florin Popa

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Post new comment

  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.
  • You may post PHP code. You should include <?php ?> tags.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Images can be added to this post.

More information about formatting options

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
Image CAPTCHA
Enter the characters shown in the image.