Lucene++ - a full-featured, c++ search engine
API Documentation

Public Member Functions | Static Public Member Functions | Protected Attributes
Lucene::TeeSinkTokenFilter Class Reference

This TokenFilter provides the ability to set aside attribute states that have already been analyzed. This is useful in situations where multiple fields share many common analysis steps and then go their separate ways. More...

`#include <TeeSinkTokenFilter.h>`

Inheritance diagram for Lucene::TeeSinkTokenFilter:

## Public Member Functions

| TeeSinkTokenFilter (const TokenStreamPtr &input)
| Instantiates a new TeeSinkTokenFilter. More...
virtual | ~TeeSinkTokenFilter ()
virtual String | getClassName ()
boost::shared_ptr< TeeSinkTokenFilter > | shared_from_this ()
SinkTokenStreamPtr | newSinkTokenStream ()
| Returns a new SinkTokenStream that receives all tokens consumed by this stream. More...
SinkTokenStreamPtr | newSinkTokenStream (const SinkFilterPtr &filter)
| Returns a new SinkTokenStream that receives all tokens consumed by this stream that pass the supplied filter. More...
void | addSinkTokenStream (const SinkTokenStreamPtr &sink)
| Adds a SinkTokenStream created by another TeeSinkTokenFilter to this one. The supplied stream will also receive all consumed tokens. This method can be used to pass tokens from two different tees to one sink. More...
void | consumeAllTokens ()
| TeeSinkTokenFilter passes all tokens to the added sinks when itself is consumed. To be sure, that all tokens from the input stream are passed to the sinks, you can call this methods. This instance is exhausted after this, but all sinks are instant available. More...
virtual bool | incrementToken ()
| Consumers (ie., IndexWriter) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriate Attributes with the attributes of the next token. More...
virtual void | end ()
| Performs end-of-stream operations, if any, and calls then end() on the input TokenStream. NOTE: Be sure to call TokenFilter::end() first when overriding this method. More...
Public Member Functions inherited from Lucene::TokenFilter
virtual | ~TokenFilter ()
boost::shared_ptr< TokenFilter > | shared_from_this ()
virtual void | close ()
| Close the input TokenStream. More...
virtual void | reset ()
| Reset the filter as well as the input TokenStream. More...
Public Member Functions inherited from Lucene::TokenStream
virtual | ~TokenStream ()
boost::shared_ptr< TokenStream > | shared_from_this ()
Public Member Functions inherited from Lucene::AttributeSource
| AttributeSource ()
| An AttributeSource using the default attribute factory DefaultAttributeFactory. More...
| AttributeSource (const AttributeSourcePtr &input)
| An AttributeSource that uses the same attributes as the supplied one. More...
| AttributeSource (const AttributeFactoryPtr &factory)
| An AttributeSource using the supplied AttributeFactory for creating new Attribute instances. More...
virtual | ~AttributeSource ()
boost::shared_ptr< AttributeSource > | shared_from_this ()
AttributeFactoryPtr | getAttributeFactory ()
| returns the used AttributeFactory. More...
template<class ATTR >
boost::shared_ptr< ATTR > | addAttribute ()
| This method first checks if an instance of that class is already in this AttributeSource and returns it. Otherwise a new instance is created, added to this AttributeSource and returned. More...
void | addAttribute (const String &className, const AttributePtr &attrImpl)
| Adds a custom Attribute instance. More...
bool | hasAttributes ()
| Returns true if this AttributeSource has any attributes. More...
template<class ATTR >
bool | hasAttribute ()
| Returns true, if this AttributeSource contains the passed-in Attribute. More...
template<class ATTR >
boost::shared_ptr< ATTR > | getAttribute ()
| Returns the instance of the passed in Attribute contained in this AttributeSource. More...
void | clearAttributes ()
| Resets all Attributes in this AttributeSource by calling AttributeImpl#clear() on each Attribute implementation. More...
AttributeSourceStatePtr | captureState ()
| Captures the state of all Attributes. The return value can be passed to restoreState to restore the state of this or another AttributeSource. More...
void | restoreState (const AttributeSourceStatePtr &state)

Restores this state by copying the values of all attribute implementations that this state contains into the attributes implementations of the targetStream. The targetStream must contain a corresponding instance for each argument contained in this state (eg. it is not possible to restore the state of an AttributeSource containing a TermAttribute into a AttributeSource using a Token instance as implementation). More...

virtual int32_t | hashCode ()
| Return hash code for this object. More...
virtual bool | equals (const LuceneObjectPtr &other)
| Return whether two objects are equal. More...
virtual String | toString ()
| Returns a string representation of the object. More...
AttributeSourcePtr | cloneAttributes ()
| Performs a clone of all AttributeImpl instances returned in a new AttributeSource instance. This method can be used to eg. create another TokenStream with exactly the same attributes (using {}) More...
Collection< AttributePtr > | getAttributes ()
| Return a vector of attributes based on currentState. More...
Public Member Functions inherited from Lucene::LuceneObject
virtual | ~LuceneObject ()
virtual void | initialize ()
| Called directly after instantiation to create objects that depend on this object being fully constructed. More...
virtual LuceneObjectPtr | clone (const LuceneObjectPtr &other=LuceneObjectPtr())
| Return clone of this object. More...
virtual int32_t | compareTo (const LuceneObjectPtr &other)
| Compare two objects. More...
Public Member Functions inherited from Lucene::LuceneSync
virtual | ~LuceneSync ()
virtual SynchronizePtr | getSync ()
| Return this object synchronize lock. More...
virtual LuceneSignalPtr | getSignal ()
| Return this object signal. More...
virtual void | lock (int32_t timeout=0)
| Lock this object using an optional timeout. More...
virtual void | unlock ()
| Unlock this object. More...
virtual bool | holdsLock ()
| Returns true if this object is currently locked by current thread. More...
virtual void | wait (int32_t timeout=0)
| Wait for signal using an optional timeout. More...
virtual void | notifyAll ()
| Notify all threads waiting for signal. More...

## Static Public Member Functions

static String | _getClassName ()
Static Public Member Functions inherited from Lucene::TokenFilter
static String | _getClassName ()
Static Public Member Functions inherited from Lucene::TokenStream
static String | _getClassName ()
Static Public Member Functions inherited from Lucene::AttributeSource
static String | _getClassName ()

## Protected Attributes

Collection< SinkTokenStreamPtr > | sinks
Protected Attributes inherited from Lucene::TokenFilter
TokenStreamPtr | input
| The source of tokens for this filter. More...
Protected Attributes inherited from Lucene::AttributeSource
AttributeFactoryPtr | factory
MapStringAttribute | attributes
AttributeSourceStatePtr | currentState
Protected Attributes inherited from Lucene::LuceneSync
SynchronizePtr | objectLock
LuceneSignalPtr | objectSignal

## Additional Inherited Members

Protected Member Functions inherited from Lucene::TokenFilter
| TokenFilter (const TokenStreamPtr &input)
| Construct a token stream filtering the given input. More...
Protected Member Functions inherited from Lucene::TokenStream
| TokenStream ()
| A TokenStream using the default attribute factory. More...
| TokenStream (const AttributeSourcePtr &input)
| A TokenStream that uses the same attributes as the supplied one. More...
| TokenStream (const AttributeFactoryPtr &factory)
| A TokenStream using the supplied AttributeFactory for creating new Attribute instances. More...
Protected Member Functions inherited from Lucene::AttributeSource
AttributePtr | getAttribute (const String &className)
| The caller must pass in a className value. This method checks if an instance of that class is already in this AttributeSource and returns it. More...
bool | hasAttribute (const String &className)
| Returns true, if this AttributeSource contains the passed-in Attribute. More...
void | computeCurrentState ()
Protected Member Functions inherited from Lucene::LuceneObject
| LuceneObject ()

## Detailed Description

This TokenFilter provides the ability to set aside attribute states that have already been analyzed. This is useful in situations where multiple fields share many common analysis steps and then go their separate ways.

It is also useful for doing things like entity extraction or proper noun analysis as part of the analysis workflow and saving off those tokens for use in another field.

    TeeSinkTokenFilterPtr source1 = newLucene<TeeSinkTokenFilter>(newLucene<WhitespaceTokenizer>(reader1));
    SinkTokenStreamPtr sink1 = source1->newSinkTokenStream();
    SinkTokenStreamPtr sink2 = source1->newSinkTokenStream();

    TeeSinkTokenFilterPtr source2 = newLucene<TeeSinkTokenFilter>(newLucene<WhitespaceTokenizer>(reader2));
    source2->addSinkTokenStream(sink1);
    source2->addSinkTokenStream(sink2);

    TokenStreamPtr final1 = newLucene<LowerCaseFilter>(source1);
    TokenStreamPtr final2 = source2;
    TokenStreamPtr final3 = newLucene<EntityDetect>(sink1);
    TokenStreamPtr final4 = newLucene<URLDetect>(sink2);

    d->add(newLucene<Field>(L"f1", final1));
    d->add(newLucene<Field>(L"f2", final2));
    d->add(newLucene<Field>(L"f3", final3));
    d->add(newLucene<Field>(L"f4", final4));

In this example, sink1 and sink2 will both get tokens from both reader1 and reader2 after whitespace tokenizer and now we can further wrap any of these in extra analysis, and more "sources" can be inserted if desired. It is important, that tees are consumed before sinks (in the above example, the field names must be less the sink's field names). If you are not sure, which stream is consumed first, you can simply add another sink and then pass all tokens to the sinks at once using consumeAllTokens.

This TokenFilter is exhausted after this. In the above example, change the example above to:

    ...
    TokenStreamPtr final1 = newLucene<LowerCaseFilter>(source1->newSinkTokenStream());
    TokenStreamPtr final2 = source2->newSinkTokenStream();
    sink1->consumeAllTokens();
    sink2->consumeAllTokens();
    ...

In this case, the fields can be added in any order, because the sources are not used anymore and all sinks are ready.

Note, the EntityDetect and URLDetect TokenStreams are for the example and do not currently exist in Lucene.

## Constructor & Destructor Documentation

## ◆ TeeSinkTokenFilter()

Lucene::TeeSinkTokenFilter::TeeSinkTokenFilter | ( | const TokenStreamPtr & | input | ) |

Instantiates a new TeeSinkTokenFilter.

## ◆ ~TeeSinkTokenFilter()

virtual Lucene::TeeSinkTokenFilter::~TeeSinkTokenFilter | ( |  | ) |

virtual

## Member Function Documentation

## ◆ _getClassName()

static String Lucene::TeeSinkTokenFilter::_getClassName | ( |  | ) |

inline static

## ◆ addSinkTokenStream()

void Lucene::TeeSinkTokenFilter::addSinkTokenStream | ( | const SinkTokenStreamPtr & | sink | ) |

Adds a SinkTokenStream created by another TeeSinkTokenFilter to this one. The supplied stream will also receive all consumed tokens. This method can be used to pass tokens from two different tees to one sink.

## ◆ consumeAllTokens()

void Lucene::TeeSinkTokenFilter::consumeAllTokens | ( |  | ) |

TeeSinkTokenFilter passes all tokens to the added sinks when itself is consumed. To be sure, that all tokens from the input stream are passed to the sinks, you can call this methods. This instance is exhausted after this, but all sinks are instant available.

## ◆ end()

virtual void Lucene::TeeSinkTokenFilter::end | ( |  | ) |

virtual

Performs end-of-stream operations, if any, and calls then end() on the input TokenStream. NOTE: Be sure to call TokenFilter::end() first when overriding this method.

Reimplemented from Lucene::TokenFilter.

## ◆ getClassName()

virtual String Lucene::TeeSinkTokenFilter::getClassName | ( |  | ) |

inline virtual

Reimplemented from Lucene::TokenFilter.

## ◆ incrementToken()

virtual bool Lucene::TeeSinkTokenFilter::incrementToken | ( |  | ) |

virtual

Consumers (ie., IndexWriter) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriate Attributes with the attributes of the next token.

The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it. If the producer needs to preserve the state for subsequent calls, it can use {} to create a copy of the current attribute state. This method is called for every token of a document, so an efficient implementation is crucial for good performance. To avoid calls to #addAttribute(Class)} and #getAttribute(Class)}, references to all Attribute}s that this stream uses should be retrieved during instantiation. To ensure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in { #incrementToken()}. false for end of stream; true otherwise

Implements Lucene::TokenStream.

## ◆ newSinkTokenStream() [1/2]

SinkTokenStreamPtr Lucene::TeeSinkTokenFilter::newSinkTokenStream | ( |  | ) |

Returns a new SinkTokenStream that receives all tokens consumed by this stream.

## ◆ newSinkTokenStream() [2/2]

SinkTokenStreamPtr Lucene::TeeSinkTokenFilter::newSinkTokenStream | ( | const SinkFilterPtr & | filter | ) |

Returns a new SinkTokenStream that receives all tokens consumed by this stream that pass the supplied filter.

See also
  SinkFilter

## ◆ shared_from_this()

boost::shared_ptr< TeeSinkTokenFilter > Lucene::TeeSinkTokenFilter::shared_from_this | ( |  | ) |

inline

## Field Documentation

## ◆ sinks

Collection<SinkTokenStreamPtr> Lucene::TeeSinkTokenFilter::sinks

protected

The documentation for this class was generated from the following file:

- TeeSinkTokenFilter.h

clucene.sourceforge.net