org.apache.lucene.analysis.ar

## Class ArabicLetterTokenizer

- java.lang.Object
- - org.apache.lucene.util.AttributeSource
  - - org.apache.lucene.analysis.TokenStream
    - - org.apache.lucene.analysis.Tokenizer
      - - org.apache.lucene.analysis.CharTokenizer
        - - org.apache.lucene.analysis.LetterTokenizer
          - - org.apache.lucene.analysis.ar.ArabicLetterTokenizer

-     public class ArabicLetterTokenizer
      extends LetterTokenizer

  Tokenizer that breaks text into runs of letters and diacritics.

  The problem with the standard Letter tokenizer is that it fails on diacritics. Handling similar to this is necessary for Indic Scripts, Hebrew, Thaana, etc.

- - ### Nested Class Summary

    - ### Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

      `AttributeSource.AttributeFactory, AttributeSource.State`

  - ### Field Summary

    - ### Fields inherited from class org.apache.lucene.analysis.Tokenizer

      `input`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `ArabicLetterTokenizer(AttributeSource.AttributeFactory factory, java.io.Reader in)`
    `ArabicLetterTokenizer(AttributeSource source, java.io.Reader in)`
    `ArabicLetterTokenizer(java.io.Reader in)`

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `protected boolean` | `isTokenChar(char c)` Allows for Letter category or NonspacingMark category

    - ### Methods inherited from class org.apache.lucene.analysis.CharTokenizer

      `end, incrementToken, next, next, normalize, reset`

    - ### Methods inherited from class org.apache.lucene.analysis.Tokenizer

      `close, correctOffset`

    - ### Methods inherited from class org.apache.lucene.analysis.TokenStream

      `getOnlyUseNewAPI, reset, setOnlyUseNewAPI`

    - ### Methods inherited from class org.apache.lucene.util.AttributeSource

      `addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString`

    - ### Methods inherited from class java.lang.Object

      `clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

- - ### Constructor Detail

    - #### ArabicLetterTokenizer

          public ArabicLetterTokenizer(java.io.Reader in)

    - #### ArabicLetterTokenizer

          public ArabicLetterTokenizer(AttributeSource source,
                                       java.io.Reader in)

    - #### ArabicLetterTokenizer

          public ArabicLetterTokenizer(AttributeSource.AttributeFactory factory,
                                       java.io.Reader in)

  - ### Method Detail

    - #### isTokenChar

          protected boolean isTokenChar(char c)

      Allows for Letter category or NonspacingMark category
      Overrides:
        `isTokenChar` in class `LetterTokenizer`
      See Also:
        `LetterTokenizer.isTokenChar(char)`

Copyright © 2000-2016 Apache Software Foundation. All Rights Reserved.