LetterTokenizer (Lucene 2.9.4 API)

Skip navigation links

All Classes

Summary:
Nested |
Field |
Constr |
Method

Detail:
Field |
Constr |
Method

java.lang.Object
- org.apache.lucene.util.AttributeSource
- - org.apache.lucene.analysis.TokenStream
  - - org.apache.lucene.analysis.Tokenizer
    - - org.apache.lucene.analysis.CharTokenizer
      - org.apache.lucene.analysis.LetterTokenizer

Direct Known Subclasses:

ArabicLetterTokenizer, LowerCaseTokenizer
```
public class LetterTokenizer
extends CharTokenizer
```
A LetterTokenizer is a tokenizer that divides text at non-letters. That's to say, it defines tokens as maximal strings of adjacent letters, as defined by java.lang.Character.isLetter() predicate. Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.

Nested Class Summary
- Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
  AttributeSource.AttributeFactory, AttributeSource.State

Field Summary
- Fields inherited from class org.apache.lucene.analysis.Tokenizer
  input

Constructor Summary

Constructors
Constructor and Description
`LetterTokenizer(AttributeSource.AttributeFactory factory, java.io.Reader in)` Construct a new LetterTokenizer using a given `AttributeSource.AttributeFactory`.
`LetterTokenizer(AttributeSource source, java.io.Reader in)` Construct a new LetterTokenizer using a given `AttributeSource`.
`LetterTokenizer(java.io.Reader in)` Construct a new LetterTokenizer.

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`protected boolean`	`isTokenChar(char c)` Collects only characters which satisfy `Character.isLetter(char)`.

Methods inherited from class org.apache.lucene.analysis.CharTokenizer
end, incrementToken, next, next, normalize, reset

Methods inherited from class org.apache.lucene.analysis.Tokenizer
close, correctOffset

Methods inherited from class org.apache.lucene.analysis.TokenStream
getOnlyUseNewAPI, reset, setOnlyUseNewAPI

Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString

Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait

- Constructor Detail
  - LetterTokenizer
```
public LetterTokenizer(java.io.Reader in)
```
    Construct a new LetterTokenizer.
  - LetterTokenizer
```
public LetterTokenizer(AttributeSource source,
                       java.io.Reader in)
```
    Construct a new LetterTokenizer using a given AttributeSource.
  - LetterTokenizer
```
public LetterTokenizer(AttributeSource.AttributeFactory factory,
                       java.io.Reader in)
```
    Construct a new LetterTokenizer using a given AttributeSource.AttributeFactory.
- Method Detail
  - isTokenChar
```
protected boolean isTokenChar(char c)
```
    Collects only characters which satisfy Character.isLetter(char).
    
    Specified by:
    
    isTokenChar in class CharTokenizer

Skip navigation links

All Classes

Summary:
Nested |
Field |
Constr |
Method

Detail:
Field |
Constr |
Method

Copyright © 2000-2016 Apache Software Foundation. All Rights Reserved.