org.apache.lucene.search.highlight

## Class TokenSources

- java.lang.Object
- - org.apache.lucene.search.highlight.TokenSources

-     public class TokenSources
      extends java.lang.Object

  Hides implementation issues associated with obtaining a TokenStream for use with the higlighter - can obtain from TermFreqVectors with offsets and (optionally) positions or from Analyzer class reparsing the stored content.

- - ### Constructor Summary

    Constructors
    Constructor and Description
    `TokenSources()`

  - ### Method Summary

    All Methods Static Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `static org.apache.lucene.analysis.TokenStream` | `getAnyTokenStream(org.apache.lucene.index.IndexReader reader, int docId, java.lang.String field, org.apache.lucene.analysis.Analyzer analyzer)` A convenience method that tries a number of approaches to getting a token stream.

    `static org.apache.lucene.analysis.TokenStream`

    `getAnyTokenStream(org.apache.lucene.index.IndexReader reader, int docId, java.lang.String field, org.apache.lucene.document.Document doc, org.apache.lucene.analysis.Analyzer analyzer)`
    A convenience method that tries to first get a TermPositionVector for the specified docId, then, falls back to using the passed in `Document` to retrieve the TokenStream.

    `static org.apache.lucene.analysis.TokenStream` | `getTokenStream(org.apache.lucene.document.Document doc, java.lang.String field, org.apache.lucene.analysis.Analyzer analyzer)`
    `static org.apache.lucene.analysis.TokenStream` | `getTokenStream(org.apache.lucene.index.IndexReader reader, int docId, java.lang.String field)`
    `static org.apache.lucene.analysis.TokenStream` | `getTokenStream(org.apache.lucene.index.IndexReader reader, int docId, java.lang.String field, org.apache.lucene.analysis.Analyzer analyzer)`
    `static org.apache.lucene.analysis.TokenStream` | `getTokenStream(java.lang.String field, java.lang.String contents, org.apache.lucene.analysis.Analyzer analyzer)`
    `static org.apache.lucene.analysis.TokenStream` | `getTokenStream(org.apache.lucene.index.TermPositionVector tpv)`
    `static org.apache.lucene.analysis.TokenStream` | `getTokenStream(org.apache.lucene.index.TermPositionVector tpv, boolean tokenPositionsGuaranteedContiguous)` Low level api.

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### TokenSources

          public TokenSources()

  - ### Method Detail

    - #### getAnyTokenStream

          public static org.apache.lucene.analysis.TokenStream getAnyTokenStream(org.apache.lucene.index.IndexReader reader,
                                                                                 int docId,
                                                                                 java.lang.String field,
                                                                                 org.apache.lucene.document.Document doc,
                                                                                 org.apache.lucene.analysis.Analyzer analyzer)
                                                                          throws java.io.IOException

      A convenience method that tries to first get a TermPositionVector for the specified docId, then, falls back to using the passed in `Document` to retrieve the TokenStream. This is useful when you already have the document, but would prefer to use the vector first.
      Parameters:
        `reader` - The `IndexReader` to use to try and get the vector from
        `docId` - The docId to retrieve.
        `field` - The field to retrieve on the document
        `doc` - The document to fall back on
        `analyzer` - The analyzer to use for creating the TokenStream if the vector doesn't exist
      Returns:
        The `TokenStream` for the `Fieldable` on the `Document`
      Throws:
        `java.io.IOException` - if there was an error loading

    - #### getAnyTokenStream

          public static org.apache.lucene.analysis.TokenStream getAnyTokenStream(org.apache.lucene.index.IndexReader reader,
                                                                                 int docId,
                                                                                 java.lang.String field,
                                                                                 org.apache.lucene.analysis.Analyzer analyzer)
                                                                          throws java.io.IOException

      A convenience method that tries a number of approaches to getting a token stream. The cost of finding there are no termVectors in the index is minimal (1000 invocations still registers 0 ms). So this "lazy" (flexible?) approach to coding is probably acceptable
      Parameters:
        `reader` -
        `docId` -
        `field` -
        `analyzer` -
      Returns:
        null if field not stored correctly
      Throws:
        `java.io.IOException`

    - #### getTokenStream

          public static org.apache.lucene.analysis.TokenStream getTokenStream(org.apache.lucene.index.TermPositionVector tpv)

    - #### getTokenStream

          public static org.apache.lucene.analysis.TokenStream getTokenStream(org.apache.lucene.index.TermPositionVector tpv,
                                                                              boolean tokenPositionsGuaranteedContiguous)

      Low level api. Returns a token stream or null if no offset info available in index. This can be used to feed the highlighter with a pre-parsed token stream In my tests the speeds to recreate 1000 token streams using this method are: - with TermVector offset only data stored - 420 milliseconds - with TermVector offset AND position data stored - 271 milliseconds (nb timings for TermVector with position data are based on a tokenizer with contiguous positions - no overlaps or gaps) The cost of not using TermPositionVector to store pre-parsed content and using an analyzer to re-parse the original content: - reanalyzing the original content - 980 milliseconds The re-analyze timings will typically vary depending on - 1) The complexity of the analyzer code (timings above were using a stemmer/lowercaser/stopword combo) 2) The number of other fields (Lucene reads ALL fields off the disk when accessing just one document field - can cost dear!) 3) Use of compression on field storage - could be faster due to compression (less disk IO) or slower (more CPU burn) depending on the content.
      Parameters:
        `tpv` -
        `tokenPositionsGuaranteedContiguous` - true if the token position numbers have no overlaps or gaps. If looking to eek out the last drops of performance, set to true. If in doubt, set to false.

    - #### getTokenStream

          public static org.apache.lucene.analysis.TokenStream getTokenStream(org.apache.lucene.index.IndexReader reader,
                                                                              int docId,
                                                                              java.lang.String field)
                                                                       throws java.io.IOException

      Throws:
        `java.io.IOException`

    - #### getTokenStream

          public static org.apache.lucene.analysis.TokenStream getTokenStream(org.apache.lucene.index.IndexReader reader,
                                                                              int docId,
                                                                              java.lang.String field,
                                                                              org.apache.lucene.analysis.Analyzer analyzer)
                                                                       throws java.io.IOException

      Throws:
        `java.io.IOException`

    - #### getTokenStream

          public static org.apache.lucene.analysis.TokenStream getTokenStream(org.apache.lucene.document.Document doc,
                                                                              java.lang.String field,
                                                                              org.apache.lucene.analysis.Analyzer analyzer)

    - #### getTokenStream

          public static org.apache.lucene.analysis.TokenStream getTokenStream(java.lang.String field,
                                                                              java.lang.String contents,
                                                                              org.apache.lucene.analysis.Analyzer analyzer)

Copyright © 2000-2016 Apache Software Foundation. All Rights Reserved.