# Class Tokenizer

java.lang.Object
nu.validator.htmlparser.impl.Tokenizer

All Implemented Interfaces:
  `Locator2, Locator`

Direct Known Subclasses:
  `ErrorReportingTokenizer`

public class Tokenizer extends Object implements Locator, Locator2
An implementation of https://html.spec.whatwg.org/multipage/syntax.html#tokenization This class implements the `Locator` interface. This is not an incidental implementation detail: Users of this class are encouraged to make use of the `Locator` nature. By default, the tokenizer may report data that XML 1.0 bans. The tokenizer can be configured to treat these conditions as fatal or to coerce the infoset to something that XML 1.0 allows.

Version:
  $Id$
Author:
  hsivonen

## Field Summary

Fields
Modifier and Type
Field
Description
`static final int`
`AFTER_ATTRIBUTE_NAME`
`static final int`
`AFTER_ATTRIBUTE_VALUE_QUOTED`
`static final int`
`AFTER_DOCTYPE_NAME`
`static final int`
`AFTER_DOCTYPE_PUBLIC_IDENTIFIER`
`static final int`
`AFTER_DOCTYPE_PUBLIC_KEYWORD`
`static final int`
`AFTER_DOCTYPE_SYSTEM_IDENTIFIER`
`static final int`
`AFTER_DOCTYPE_SYSTEM_KEYWORD`
`protected LocatorImpl`
`ampersandLocation`
`static final int`
`ATTRIBUTE_NAME`
`static final int`
`ATTRIBUTE_VALUE_DOUBLE_QUOTED`
`static final int`
`ATTRIBUTE_VALUE_SINGLE_QUOTED`
`static final int`
`ATTRIBUTE_VALUE_UNQUOTED`
`protected AttributeName`
`attributeName`
The current attribute name.
`static final int`
`BEFORE_ATTRIBUTE_NAME`
`static final int`
`BEFORE_ATTRIBUTE_VALUE`
`static final int`
`BEFORE_DOCTYPE_NAME`
`static final int`
`BEFORE_DOCTYPE_PUBLIC_IDENTIFIER`
`static final int`
`BEFORE_DOCTYPE_SYSTEM_IDENTIFIER`
`static final int`
`BETWEEN_DOCTYPE_PUBLIC_AND_SYSTEM_IDENTIFIERS`
`static final int`
`BOGUS_COMMENT`
`static final int`
`BOGUS_COMMENT_HYPHEN`
`static final int`
`BOGUS_DOCTYPE`
`static final int`
`CDATA_RSQB`
`static final int`
`CDATA_RSQB_RSQB`
`static final int`
`CDATA_SECTION`
`static final int`
`CDATA_START`
`static final int`
`CHARACTER_REFERENCE_HILO_LOOKUP`
`static final int`
`CHARACTER_REFERENCE_TAIL`
`static final int`
`CLOSE_TAG_OPEN`
`static final int`
`COMMENT`
`static final int`
`COMMENT_END`
`static final int`
`COMMENT_END_BANG`
`static final int`
`COMMENT_END_DASH`
`static final int`
`COMMENT_LESSTHAN`
`static final int`
`COMMENT_LESSTHAN_BANG`
`static final int`
`COMMENT_LESSTHAN_BANG_DASH`
`static final int`
`COMMENT_LESSTHAN_BANG_DASH_DASH`
`static final int`
`COMMENT_START`
`static final int`
`COMMENT_START_DASH`
`protected boolean`
`confident`
`static final int`
`CONSUME_CHARACTER_REFERENCE`
`static final int`
`CONSUME_NCR`
`protected int`
`cstart`
`static final int`
`DATA`
`static final int`
`DECIMAL_NRC_LOOP`
`static final int`
`DOCTYPE`
`static final int`
`DOCTYPE_NAME`
`static final int`
`DOCTYPE_PUBLIC_IDENTIFIER_DOUBLE_QUOTED`
`static final int`
`DOCTYPE_PUBLIC_IDENTIFIER_SINGLE_QUOTED`
`static final int`
`DOCTYPE_SYSTEM_IDENTIFIER_DOUBLE_QUOTED`
`static final int`
`DOCTYPE_SYSTEM_IDENTIFIER_SINGLE_QUOTED`
`static final int`
`DOCTYPE_UBLIC`
`static final int`
`DOCTYPE_YSTEM`
`protected EncodingDeclarationHandler`
`encodingDeclarationHandler`
`protected boolean`
`endTag`
`true` if tokenizing an end tag
`protected ElementName`
`endTagExpectation`
The element whose end tag closes the current CDATA or RCDATA element.
`protected ErrorHandler`
`errorHandler`
The error handler.
`static final int`
`HANDLE_NCR_VALUE`
`static final int`
`HANDLE_NCR_VALUE_RECONSUME`
`static final int`
`HEX_NCR_LOOP`
`protected int`
`index`
`protected boolean`
`lastCR`
Whether the previous char read was CR.
`static final int`
`MARKUP_DECLARATION_HYPHEN`
`static final int`
`MARKUP_DECLARATION_OCTYPE`
`static final int`
`MARKUP_DECLARATION_OPEN`
`static final int`
`NON_DATA_END_TAG_NAME`
`static final int`
`PLAINTEXT`
`static final int`
`PROCESSING_INSTRUCTION`
`static final int`
`PROCESSING_INSTRUCTION_QUESTION_MARK`
`static final int`
`RAWTEXT`
`static final int`
`RAWTEXT_RCDATA_LESS_THAN_SIGN`
`static final int`
`RCDATA`
`static final int`
`SCRIPT_DATA`
`static final int`
`SCRIPT_DATA_DOUBLE_ESCAPE_END`
`static final int`
`SCRIPT_DATA_DOUBLE_ESCAPE_START`
`static final int`
`SCRIPT_DATA_DOUBLE_ESCAPED`
`static final int`
`SCRIPT_DATA_DOUBLE_ESCAPED_DASH`
`static final int`
`SCRIPT_DATA_DOUBLE_ESCAPED_DASH_DASH`
`static final int`
`SCRIPT_DATA_DOUBLE_ESCAPED_LESS_THAN_SIGN`
`static final int`
`SCRIPT_DATA_ESCAPE_START`
`static final int`
`SCRIPT_DATA_ESCAPE_START_DASH`
`static final int`
`SCRIPT_DATA_ESCAPED`
`static final int`
`SCRIPT_DATA_ESCAPED_DASH`
`static final int`
`SCRIPT_DATA_ESCAPED_DASH_DASH`
`static final int`
`SCRIPT_DATA_ESCAPED_LESS_THAN_SIGN`
`static final int`
`SCRIPT_DATA_LESS_THAN_SIGN`
`static final int`
`SELF_CLOSING_START_TAG`
`protected int`
`stateSave`
`static final int`
`TAG_NAME`
`static final int`
`TAG_OPEN`
`protected final TokenHandler`
`tokenHandler`
The token handler.
`protected int`
`value`

## Constructor Summary

Constructors
Constructor
Description
`Tokenizer(TokenHandler tokenHandler)`
The constructor.
`Tokenizer(TokenHandler tokenHandler, boolean newAttributesEachTime)`

## Method Summary

Modifier and Type
Method
Description
`void`
`becomeConfident()`
`protected char`
`checkChar(char[] buf, int pos)`
`boolean`
`dropBufferIfLongerThan(int length)`
`void`
`end()`
`void`
`eof()`
`void`
`err(String message)`
Reports a Parse Error.
`protected void`
`errAstralNonCharacter(int ch)`
`protected void`
`errAttributeValueMissing()`
`protected void`
`errBadCharAfterLt(char c)`
`protected void`
`errBadCharBeforeAttributeNameOrNull(char c)`
`protected void`
`errBogusComment()`
`protected void`
`errBogusDoctype()`
`protected void`
`errCharRefLacksSemicolon()`
`protected void`
`errDuplicateAttribute()`
`protected void`
`errEofAfterLt()`
`protected void`
`errEofInAttributeName()`
`protected void`
`errEofInAttributeValue()`
`protected void`
`errEofInComment()`
`protected void`
`errEofInDoctype()`
`protected void`
`errEofInEndTag()`
`protected void`
`errEofInPublicId()`
`protected void`
`errEofInSystemId()`
`protected void`
`errEofInTagName()`
`protected void`
`errEofWithoutGt()`
`protected void`
`errEqualsSignBeforeAttributeName()`
`protected void`
`errExpectedPublicId()`
`protected void`
`errExpectedSystemId()`
`protected void`
`errGarbageAfterLtSlash()`
`protected void`
`errGtInPublicId()`
`protected void`
`errGtInSystemId()`
`protected void`
`errHtml4LtSlashInRcdata(char folded)`
`protected void`
`errLtGt()`
`protected void`
`errLtOrEqualsOrGraveInUnquotedAttributeOrNull(char c)`
`protected void`
`errLtSlashGt()`
`protected void`
`errMissingSpaceBeforeDoctypeName()`
`protected void`
`errNamelessDoctype()`
`protected void`
`errNcrControlChar()`
`protected char`
`errNcrControlChar(char ch)`
`protected void`
`errNcrCr()`
`protected void`
`errNcrInC1Range()`
`protected char`
`errNcrNonCharacter(char ch)`
`protected void`
`errNcrOutOfRange()`
`protected void`
`errNcrSurrogate()`
`protected void`
`errNcrUnassigned()`
`protected void`
`errNcrZero()`
`protected void`
`errNestedComment()`
`protected void`
`errNoDigitsInNCR()`
`protected void`
`errNoNamedCharacterMatch()`
`protected void`
`errNoSpaceBetweenAttributes()`
`protected void`
`errNoSpaceBetweenDoctypePublicKeywordAndQuote()`
`protected void`
`errNoSpaceBetweenDoctypeSystemKeywordAndQuote()`
`protected void`
`errNoSpaceBetweenPublicAndSystemIds()`
`protected void`
`errNotSemicolonTerminated()`
`protected void`
`errPrematureEndOfComment()`
`protected void`
`errProcessingInstruction()`
`protected void`
`errQuoteBeforeAttributeName(char c)`
`protected void`
`errQuoteOrLtInAttributeNameOrNull(char c)`
`protected void`
`errSlashNotFollowedByGt()`
`void`
`errTreeBuilder(String message)`
`protected void`
`errUnescapedAmpersandInterpretedAsCharacterReference()`
`protected void`
`errUnquotedAttributeValOrNull(char c)`
`protected void`
`errWarnLtSlashInRcdata()`
`void`
`fatal(String message)`
Reports an condition that would make the infoset incompatible with XML 1.0 as fatal.
`protected void`
`flushChars(char[] buf, int pos)`
Flushes coalesced character tokens.
`int`
`getCol()`
Returns the col.
`int`
`getColumnNumber()`
`String`
`getEncoding()`
`ErrorHandler`
`getErrorHandler()`
`HashMap`
`getErrorProfile()`
Gets the errorProfile.
`int`
`getLine()`
Returns the line.
`int`
`getLineNumber()`
`String`
`getPublicId()`
`String`
`getSystemId()`
`String`
`getXMLVersion()`
`void`
`initializeWithoutStarting()`
`void`
`initLocation(String newPublicId, String newSystemId)`
`boolean`
`internalEncodingDeclaration(String internalCharset)`
`boolean`
`isInDataState()`
`boolean`
`isMappingLangToXmlLang()`
Returns the mappingLangToXmlLang.
`boolean`
`isNextCharOnNewLine()`
Returns the nextCharOnNewLine.
`boolean`
`isPrevCR()`
`void`
`loadState(Tokenizer other)`
`protected void`
`maybeErrAttributesOnEndTag(HtmlAttributes attrs)`
`protected void`
`maybeErrSlashInEndTag(boolean selfClosing)`
`protected void`
`maybeWarnPrivateUse(char ch)`
`protected void`
`maybeWarnPrivateUseAstral()`
`protected void`
`noteAttributeWithoutValue()`
`protected void`
`noteUnquotedAttributeValue()`
`void`
`notifyAboutMetaBoundary()`
`void`
`requestSuspension()`
`void`
`resetToDataState()`
`void`
`setCommentPolicy(XmlViolationPolicy commentPolicy)`
Sets the commentPolicy.
`void`
`setContentNonXmlCharPolicy(XmlViolationPolicy contentNonXmlCharPolicy)`
Sets the contentNonXmlCharPolicy.
`void`
`setContentSpacePolicy(XmlViolationPolicy contentSpacePolicy)`
Sets the contentSpacePolicy.
`void`
`setEncodingDeclarationHandler(EncodingDeclarationHandler encodingDeclarationHandler)`
Sets the encodingDeclarationHandler.
`void`
`setErrorHandler(ErrorHandler eh)`
Sets the error handler.
`void`
`setInterner(Interner interner)`
`void`
`setKeepBuffer(boolean keepBuffer)`
`void`
`setLineNumber(int line)`
For C++ use only.
`void`
`setMappingLangToXmlLang(boolean mappingLangToXmlLang)`
Sets the mappingLangToXmlLang.
`void`
`setNamePolicy(XmlViolationPolicy namePolicy)`
`void`
`setState(int specialTokenizerState)`
Sets the tokenizer state and the associated element name.
`void`
`setStateAndEndTagExpectation(int specialTokenizerState, String endTagExpectation)`
Sets the tokenizer state and the associated element name.
`void`
`setStateAndEndTagExpectation(int specialTokenizerState, ElementName endTagExpectation)`
Sets the tokenizer state and the associated element name.
`void`
`setTransitionBaseOffset(int offset)`
Sets an offset to be added to the position reported to `TransitionHandler`.
`void`
`setXmlnsPolicy(XmlViolationPolicy xmlnsPolicy)`
Sets the xmlnsPolicy.
`protected void`
`silentCarriageReturn()`
`protected void`
`silentLineFeed()`
`void`
`start()`
`protected void`
`startErrorReporting()`
`protected String`
`strBufToString()`
The buffer as a String.
`boolean`
`tokenizeBuffer(UTF16Buffer buffer)`
`protected int`
`transition(int from, int to, boolean reconsume, int pos)`
`void`
`warn(String message)`
Reports a warning

### Methods inherited from class Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

## Field Details

### DATA

public static final int DATA

See Also:
  - Constant Field Values

### RCDATA

public static final int RCDATA

See Also:
  - Constant Field Values

### SCRIPT_DATA

public static final int SCRIPT_DATA

See Also:
  - Constant Field Values

### RAWTEXT

public static final int RAWTEXT

See Also:
  - Constant Field Values

### SCRIPT_DATA_ESCAPED

public static final int SCRIPT_DATA_ESCAPED

See Also:
  - Constant Field Values

### ATTRIBUTE_VALUE_DOUBLE_QUOTED

public static final int ATTRIBUTE_VALUE_DOUBLE_QUOTED

See Also:
  - Constant Field Values

### ATTRIBUTE_VALUE_SINGLE_QUOTED

public static final int ATTRIBUTE_VALUE_SINGLE_QUOTED

See Also:
  - Constant Field Values

### ATTRIBUTE_VALUE_UNQUOTED

public static final int ATTRIBUTE_VALUE_UNQUOTED

See Also:
  - Constant Field Values

### PLAINTEXT

public static final int PLAINTEXT

See Also:
  - Constant Field Values

### TAG_OPEN

public static final int TAG_OPEN

See Also:
  - Constant Field Values

### CLOSE_TAG_OPEN

public static final int CLOSE_TAG_OPEN

See Also:
  - Constant Field Values

### TAG_NAME

public static final int TAG_NAME

See Also:
  - Constant Field Values

### BEFORE_ATTRIBUTE_NAME

public static final int BEFORE_ATTRIBUTE_NAME

See Also:
  - Constant Field Values

### ATTRIBUTE_NAME

public static final int ATTRIBUTE_NAME

See Also:
  - Constant Field Values

### AFTER_ATTRIBUTE_NAME

public static final int AFTER_ATTRIBUTE_NAME

See Also:
  - Constant Field Values

### BEFORE_ATTRIBUTE_VALUE

public static final int BEFORE_ATTRIBUTE_VALUE

See Also:
  - Constant Field Values

### AFTER_ATTRIBUTE_VALUE_QUOTED

public static final int AFTER_ATTRIBUTE_VALUE_QUOTED

See Also:
  - Constant Field Values

### BOGUS_COMMENT

public static final int BOGUS_COMMENT

See Also:
  - Constant Field Values

### MARKUP_DECLARATION_OPEN

public static final int MARKUP_DECLARATION_OPEN

See Also:
  - Constant Field Values

### DOCTYPE

public static final int DOCTYPE

See Also:
  - Constant Field Values

### BEFORE_DOCTYPE_NAME

public static final int BEFORE_DOCTYPE_NAME

See Also:
  - Constant Field Values

### DOCTYPE_NAME

public static final int DOCTYPE_NAME

See Also:
  - Constant Field Values

### AFTER_DOCTYPE_NAME

public static final int AFTER_DOCTYPE_NAME

See Also:
  - Constant Field Values

### BEFORE_DOCTYPE_PUBLIC_IDENTIFIER

public static final int BEFORE_DOCTYPE_PUBLIC_IDENTIFIER

See Also:
  - Constant Field Values

### DOCTYPE_PUBLIC_IDENTIFIER_DOUBLE_QUOTED

public static final int DOCTYPE_PUBLIC_IDENTIFIER_DOUBLE_QUOTED

See Also:
  - Constant Field Values

### DOCTYPE_PUBLIC_IDENTIFIER_SINGLE_QUOTED

public static final int DOCTYPE_PUBLIC_IDENTIFIER_SINGLE_QUOTED

See Also:
  - Constant Field Values

### AFTER_DOCTYPE_PUBLIC_IDENTIFIER

public static final int AFTER_DOCTYPE_PUBLIC_IDENTIFIER

See Also:
  - Constant Field Values

### BEFORE_DOCTYPE_SYSTEM_IDENTIFIER

public static final int BEFORE_DOCTYPE_SYSTEM_IDENTIFIER

See Also:
  - Constant Field Values

### DOCTYPE_SYSTEM_IDENTIFIER_DOUBLE_QUOTED

public static final int DOCTYPE_SYSTEM_IDENTIFIER_DOUBLE_QUOTED

See Also:
  - Constant Field Values

### DOCTYPE_SYSTEM_IDENTIFIER_SINGLE_QUOTED

public static final int DOCTYPE_SYSTEM_IDENTIFIER_SINGLE_QUOTED

See Also:
  - Constant Field Values

### AFTER_DOCTYPE_SYSTEM_IDENTIFIER

public static final int AFTER_DOCTYPE_SYSTEM_IDENTIFIER

See Also:
  - Constant Field Values

### BOGUS_DOCTYPE

public static final int BOGUS_DOCTYPE

See Also:
  - Constant Field Values

### COMMENT_START

public static final int COMMENT_START

See Also:
  - Constant Field Values

### COMMENT_START_DASH

public static final int COMMENT_START_DASH

See Also:
  - Constant Field Values

### COMMENT

public static final int COMMENT

See Also:
  - Constant Field Values

### COMMENT_END_DASH

public static final int COMMENT_END_DASH

See Also:
  - Constant Field Values

### COMMENT_END

public static final int COMMENT_END

See Also:
  - Constant Field Values

### COMMENT_END_BANG

public static final int COMMENT_END_BANG

See Also:
  - Constant Field Values

### NON_DATA_END_TAG_NAME

public static final int NON_DATA_END_TAG_NAME

See Also:
  - Constant Field Values

### MARKUP_DECLARATION_HYPHEN

public static final int MARKUP_DECLARATION_HYPHEN

See Also:
  - Constant Field Values

### MARKUP_DECLARATION_OCTYPE

public static final int MARKUP_DECLARATION_OCTYPE

See Also:
  - Constant Field Values

### DOCTYPE_UBLIC

public static final int DOCTYPE_UBLIC

See Also:
  - Constant Field Values

### DOCTYPE_YSTEM

public static final int DOCTYPE_YSTEM

See Also:
  - Constant Field Values

### AFTER_DOCTYPE_PUBLIC_KEYWORD

public static final int AFTER_DOCTYPE_PUBLIC_KEYWORD

See Also:
  - Constant Field Values

### BETWEEN_DOCTYPE_PUBLIC_AND_SYSTEM_IDENTIFIERS

public static final int BETWEEN_DOCTYPE_PUBLIC_AND_SYSTEM_IDENTIFIERS

See Also:
  - Constant Field Values

### AFTER_DOCTYPE_SYSTEM_KEYWORD

public static final int AFTER_DOCTYPE_SYSTEM_KEYWORD

See Also:
  - Constant Field Values

### CONSUME_CHARACTER_REFERENCE

public static final int CONSUME_CHARACTER_REFERENCE

See Also:
  - Constant Field Values

### CONSUME_NCR

public static final int CONSUME_NCR

See Also:
  - Constant Field Values

### CHARACTER_REFERENCE_TAIL

public static final int CHARACTER_REFERENCE_TAIL

See Also:
  - Constant Field Values

### HEX_NCR_LOOP

public static final int HEX_NCR_LOOP

See Also:
  - Constant Field Values

### DECIMAL_NRC_LOOP

public static final int DECIMAL_NRC_LOOP

See Also:
  - Constant Field Values

### HANDLE_NCR_VALUE

public static final int HANDLE_NCR_VALUE

See Also:
  - Constant Field Values

### HANDLE_NCR_VALUE_RECONSUME

public static final int HANDLE_NCR_VALUE_RECONSUME

See Also:
  - Constant Field Values

### CHARACTER_REFERENCE_HILO_LOOKUP

public static final int CHARACTER_REFERENCE_HILO_LOOKUP

See Also:
  - Constant Field Values

### SELF_CLOSING_START_TAG

public static final int SELF_CLOSING_START_TAG

See Also:
  - Constant Field Values

### CDATA_START

public static final int CDATA_START

See Also:
  - Constant Field Values

### CDATA_SECTION

public static final int CDATA_SECTION

See Also:
  - Constant Field Values

### CDATA_RSQB

public static final int CDATA_RSQB

See Also:
  - Constant Field Values

### CDATA_RSQB_RSQB

public static final int CDATA_RSQB_RSQB

See Also:
  - Constant Field Values

### SCRIPT_DATA_LESS_THAN_SIGN

public static final int SCRIPT_DATA_LESS_THAN_SIGN

See Also:
  - Constant Field Values

### SCRIPT_DATA_ESCAPE_START

public static final int SCRIPT_DATA_ESCAPE_START

See Also:
  - Constant Field Values

### SCRIPT_DATA_ESCAPE_START_DASH

public static final int SCRIPT_DATA_ESCAPE_START_DASH

See Also:
  - Constant Field Values

### SCRIPT_DATA_ESCAPED_DASH

public static final int SCRIPT_DATA_ESCAPED_DASH

See Also:
  - Constant Field Values

### SCRIPT_DATA_ESCAPED_DASH_DASH

public static final int SCRIPT_DATA_ESCAPED_DASH_DASH

See Also:
  - Constant Field Values

### BOGUS_COMMENT_HYPHEN

public static final int BOGUS_COMMENT_HYPHEN

See Also:
  - Constant Field Values

### RAWTEXT_RCDATA_LESS_THAN_SIGN

public static final int RAWTEXT_RCDATA_LESS_THAN_SIGN

See Also:
  - Constant Field Values

### SCRIPT_DATA_ESCAPED_LESS_THAN_SIGN

public static final int SCRIPT_DATA_ESCAPED_LESS_THAN_SIGN

See Also:
  - Constant Field Values

### SCRIPT_DATA_DOUBLE_ESCAPE_START

public static final int SCRIPT_DATA_DOUBLE_ESCAPE_START

See Also:
  - Constant Field Values

### SCRIPT_DATA_DOUBLE_ESCAPED

public static final int SCRIPT_DATA_DOUBLE_ESCAPED

See Also:
  - Constant Field Values

### SCRIPT_DATA_DOUBLE_ESCAPED_LESS_THAN_SIGN

public static final int SCRIPT_DATA_DOUBLE_ESCAPED_LESS_THAN_SIGN

See Also:
  - Constant Field Values

### SCRIPT_DATA_DOUBLE_ESCAPED_DASH

public static final int SCRIPT_DATA_DOUBLE_ESCAPED_DASH

See Also:
  - Constant Field Values

### SCRIPT_DATA_DOUBLE_ESCAPED_DASH_DASH

public static final int SCRIPT_DATA_DOUBLE_ESCAPED_DASH_DASH

See Also:
  - Constant Field Values

### SCRIPT_DATA_DOUBLE_ESCAPE_END

public static final int SCRIPT_DATA_DOUBLE_ESCAPE_END

See Also:
  - Constant Field Values

### PROCESSING_INSTRUCTION

public static final int PROCESSING_INSTRUCTION

See Also:
  - Constant Field Values

### PROCESSING_INSTRUCTION_QUESTION_MARK

public static final int PROCESSING_INSTRUCTION_QUESTION_MARK

See Also:
  - Constant Field Values

### COMMENT_LESSTHAN

public static final int COMMENT_LESSTHAN

See Also:
  - Constant Field Values

### COMMENT_LESSTHAN_BANG

public static final int COMMENT_LESSTHAN_BANG

See Also:
  - Constant Field Values

### COMMENT_LESSTHAN_BANG_DASH

public static final int COMMENT_LESSTHAN_BANG_DASH

See Also:
  - Constant Field Values

### COMMENT_LESSTHAN_BANG_DASH_DASH

public static final int COMMENT_LESSTHAN_BANG_DASH_DASH

See Also:
  - Constant Field Values

### tokenHandler

protected final TokenHandler tokenHandler
The token handler.

### encodingDeclarationHandler

protected EncodingDeclarationHandler encodingDeclarationHandler

### errorHandler

protected ErrorHandler errorHandler
The error handler.

### lastCR

protected boolean lastCR
Whether the previous char read was CR.

### stateSave

protected int stateSave

### index

protected int index

### value

protected int value

### cstart

protected int cstart

### endTagExpectation

protected ElementName endTagExpectation
The element whose end tag closes the current CDATA or RCDATA element.

### endTag

protected boolean endTag
`true` if tokenizing an end tag

### attributeName

protected AttributeName attributeName
The current attribute name.

### confident

protected boolean confident

### ampersandLocation

protected LocatorImpl ampersandLocation

## Constructor Details

### Tokenizer

public Tokenizer(TokenHandler tokenHandler, boolean newAttributesEachTime)

### Tokenizer

public Tokenizer(TokenHandler tokenHandler)
The constructor.

Parameters:
  `tokenHandler` - the handler for receiving tokens

## Method Details

### setInterner

public void setInterner(Interner interner)

### initLocation

public void initLocation(String newPublicId, String newSystemId)

### setKeepBuffer

public void setKeepBuffer(boolean keepBuffer)

### dropBufferIfLongerThan

public boolean dropBufferIfLongerThan(int length)

### isMappingLangToXmlLang

public boolean isMappingLangToXmlLang()
Returns the mappingLangToXmlLang.

Returns:
  the mappingLangToXmlLang

### setMappingLangToXmlLang

public void setMappingLangToXmlLang(boolean mappingLangToXmlLang)
Sets the mappingLangToXmlLang.

Parameters:
  `mappingLangToXmlLang` - the mappingLangToXmlLang to set

### setErrorHandler

public void setErrorHandler(ErrorHandler eh)
Sets the error handler.

See Also:
  - `XMLReader.setErrorHandler(org.xml.sax.ErrorHandler)`

### getErrorHandler

public ErrorHandler getErrorHandler()

### getErrorProfile

public HashMap getErrorProfile()
Gets the errorProfile.

Parameters:
  `errorProfile` -

### setCommentPolicy

public void setCommentPolicy(XmlViolationPolicy commentPolicy)
Sets the commentPolicy.

Parameters:
  `commentPolicy` - the commentPolicy to set

### setContentNonXmlCharPolicy

public void setContentNonXmlCharPolicy(XmlViolationPolicy contentNonXmlCharPolicy)
Sets the contentNonXmlCharPolicy.

Parameters:
  `contentNonXmlCharPolicy` - the contentNonXmlCharPolicy to set

### setContentSpacePolicy

public void setContentSpacePolicy(XmlViolationPolicy contentSpacePolicy)
Sets the contentSpacePolicy.

Parameters:
  `contentSpacePolicy` - the contentSpacePolicy to set

### setXmlnsPolicy

public void setXmlnsPolicy(XmlViolationPolicy xmlnsPolicy)
Sets the xmlnsPolicy.

Parameters:
  `xmlnsPolicy` - the xmlnsPolicy to set

### setNamePolicy

public void setNamePolicy(XmlViolationPolicy namePolicy)

### setState

public void setState(int specialTokenizerState)
Sets the tokenizer state and the associated element name. This should only ever used to put the tokenizer into one of the states that have a special end tag expectation.

Parameters:
  `specialTokenizerState` - the tokenizer state to set

### setStateAndEndTagExpectation

public void setStateAndEndTagExpectation(int specialTokenizerState, String endTagExpectation)
Sets the tokenizer state and the associated element name. This should only ever used to put the tokenizer into one of the states that have a special end tag expectation. For use from the tokenizer test harness.

Parameters:
  `specialTokenizerState` - the tokenizer state to set
  `endTagExpectation` - the expected end tag for transitioning back to normal

### setStateAndEndTagExpectation

public void setStateAndEndTagExpectation(int specialTokenizerState, ElementName endTagExpectation)
Sets the tokenizer state and the associated element name. This should only ever used to put the tokenizer into one of the states that have a special end tag expectation.

Parameters:
  `specialTokenizerState` - the tokenizer state to set
  `endTagExpectation` - the expected end tag for transitioning back to normal

### setLineNumber

public void setLineNumber(int line)
For C++ use only.

### getLineNumber

public int getLineNumber()

Specified by:
  `getLineNumber` in interface `Locator`
See Also:
  - `Locator.getLineNumber()`

### getColumnNumber

public int getColumnNumber()

Specified by:
  `getColumnNumber` in interface `Locator`
See Also:
  - `Locator.getColumnNumber()`

### getPublicId

public String getPublicId()

Specified by:
  `getPublicId` in interface `Locator`
See Also:
  - `Locator.getPublicId()`

### getSystemId

public String getSystemId()

Specified by:
  `getSystemId` in interface `Locator`
See Also:
  - `Locator.getSystemId()`

### getXMLVersion

public String getXMLVersion()

Specified by:
  `getXMLVersion` in interface `Locator2`
See Also:
  - `Locator2.getXMLVersion()`

### getEncoding

public String getEncoding()

Specified by:
  `getEncoding` in interface `Locator2`
See Also:
  - `Locator2.getXMLVersion()`

### notifyAboutMetaBoundary

public void notifyAboutMetaBoundary()

### strBufToString

protected String strBufToString()
The buffer as a String. Currently only used for error reporting.

C++ memory note: The return value must be released.

Returns:
  the buffer as a string

### flushChars

protected void flushChars(char[] buf, int pos) throws SAXException
Flushes coalesced character tokens.

Parameters:
  `buf` - TODO
  `pos` - TODO
Throws:
  `SAXException`

### fatal

public void fatal(String message) throws SAXException
Reports an condition that would make the infoset incompatible with XML 1.0 as fatal.

Parameters:
  `message` - the message
Throws:
  `SAXException`
  `SAXParseException`

### err

public void err(String message) throws SAXException
Reports a Parse Error.

Parameters:
  `message` - the message
Throws:
  `SAXException`

### errTreeBuilder

public void errTreeBuilder(String message) throws SAXException

Throws:
  `SAXException`

### warn

public void warn(String message) throws SAXException
Reports a warning

Parameters:
  `message` - the message
Throws:
  `SAXException`

### startErrorReporting

protected void startErrorReporting() throws SAXException

Throws:
  `SAXException`

### start

public void start() throws SAXException

Throws:
  `SAXException`

### tokenizeBuffer

public boolean tokenizeBuffer(UTF16Buffer buffer) throws SAXException

Throws:
  `SAXException`

### transition

protected int transition(int from, int to, boolean reconsume, int pos) throws SAXException

Throws:
  `SAXException`

### silentCarriageReturn

protected void silentCarriageReturn()

### silentLineFeed

protected void silentLineFeed()

### eof

public void eof() throws SAXException

Throws:
  `SAXException`

### checkChar

protected char checkChar(char[] buf, int pos) throws SAXException

Throws:
  `SAXException`

### internalEncodingDeclaration

public boolean internalEncodingDeclaration(String internalCharset) throws SAXException

Throws:
  `SAXException`

### end

public void end() throws SAXException

Throws:
  `SAXException`

### requestSuspension

public void requestSuspension()

### becomeConfident

public void becomeConfident()

### isNextCharOnNewLine

public boolean isNextCharOnNewLine()
Returns the nextCharOnNewLine.

Returns:
  the nextCharOnNewLine

### isPrevCR

public boolean isPrevCR()

### getLine

public int getLine()
Returns the line.

Returns:
  the line

### getCol

public int getCol()
Returns the col.

Returns:
  the col

### isInDataState

public boolean isInDataState()

### resetToDataState

public void resetToDataState()

### loadState

public void loadState(Tokenizer other) throws SAXException

Throws:
  `SAXException`

### initializeWithoutStarting

public void initializeWithoutStarting() throws SAXException

Throws:
  `SAXException`

### errGarbageAfterLtSlash

protected void errGarbageAfterLtSlash() throws SAXException

Throws:
  `SAXException`

### errLtSlashGt

protected void errLtSlashGt() throws SAXException

Throws:
  `SAXException`

### errWarnLtSlashInRcdata

protected void errWarnLtSlashInRcdata() throws SAXException

Throws:
  `SAXException`

### errHtml4LtSlashInRcdata

protected void errHtml4LtSlashInRcdata(char folded) throws SAXException

Throws:
  `SAXException`

### errCharRefLacksSemicolon

protected void errCharRefLacksSemicolon() throws SAXException

Throws:
  `SAXException`

### errNoDigitsInNCR

protected void errNoDigitsInNCR() throws SAXException

Throws:
  `SAXException`

### errGtInSystemId

protected void errGtInSystemId() throws SAXException

Throws:
  `SAXException`

### errGtInPublicId

protected void errGtInPublicId() throws SAXException

Throws:
  `SAXException`

### errNamelessDoctype

protected void errNamelessDoctype() throws SAXException

Throws:
  `SAXException`

### errNestedComment

protected void errNestedComment() throws SAXException

Throws:
  `SAXException`

### errPrematureEndOfComment

protected void errPrematureEndOfComment() throws SAXException

Throws:
  `SAXException`

### errBogusComment

protected void errBogusComment() throws SAXException

Throws:
  `SAXException`

### errUnquotedAttributeValOrNull

protected void errUnquotedAttributeValOrNull(char c) throws SAXException

Throws:
  `SAXException`

### errSlashNotFollowedByGt

protected void errSlashNotFollowedByGt() throws SAXException

Throws:
  `SAXException`

### errNoSpaceBetweenAttributes

protected void errNoSpaceBetweenAttributes() throws SAXException

Throws:
  `SAXException`

### errLtOrEqualsOrGraveInUnquotedAttributeOrNull

protected void errLtOrEqualsOrGraveInUnquotedAttributeOrNull(char c) throws SAXException

Throws:
  `SAXException`

### errAttributeValueMissing

protected void errAttributeValueMissing() throws SAXException

Throws:
  `SAXException`

### errBadCharBeforeAttributeNameOrNull

protected void errBadCharBeforeAttributeNameOrNull(char c) throws SAXException

Throws:
  `SAXException`

### errEqualsSignBeforeAttributeName

protected void errEqualsSignBeforeAttributeName() throws SAXException

Throws:
  `SAXException`

### errBadCharAfterLt

protected void errBadCharAfterLt(char c) throws SAXException

Throws:
  `SAXException`

### errLtGt

protected void errLtGt() throws SAXException

Throws:
  `SAXException`

### errProcessingInstruction

protected void errProcessingInstruction() throws SAXException

Throws:
  `SAXException`

### errUnescapedAmpersandInterpretedAsCharacterReference

protected void errUnescapedAmpersandInterpretedAsCharacterReference() throws SAXException

Throws:
  `SAXException`

### errNotSemicolonTerminated

protected void errNotSemicolonTerminated() throws SAXException

Throws:
  `SAXException`

### errNoNamedCharacterMatch

protected void errNoNamedCharacterMatch() throws SAXException

Throws:
  `SAXException`

### errQuoteBeforeAttributeName

protected void errQuoteBeforeAttributeName(char c) throws SAXException

Throws:
  `SAXException`

### errQuoteOrLtInAttributeNameOrNull

protected void errQuoteOrLtInAttributeNameOrNull(char c) throws SAXException

Throws:
  `SAXException`

### errExpectedPublicId

protected void errExpectedPublicId() throws SAXException

Throws:
  `SAXException`

### errBogusDoctype

protected void errBogusDoctype() throws SAXException

Throws:
  `SAXException`

### maybeWarnPrivateUseAstral

protected void maybeWarnPrivateUseAstral() throws SAXException

Throws:
  `SAXException`

### maybeWarnPrivateUse

protected void maybeWarnPrivateUse(char ch) throws SAXException

Throws:
  `SAXException`

### maybeErrAttributesOnEndTag

protected void maybeErrAttributesOnEndTag(HtmlAttributes attrs) throws SAXException

Throws:
  `SAXException`

### maybeErrSlashInEndTag

protected void maybeErrSlashInEndTag(boolean selfClosing) throws SAXException

Throws:
  `SAXException`

### errNcrNonCharacter

protected char errNcrNonCharacter(char ch) throws SAXException

Throws:
  `SAXException`

### errAstralNonCharacter

protected void errAstralNonCharacter(int ch) throws SAXException

Throws:
  `SAXException`

### errNcrSurrogate

protected void errNcrSurrogate() throws SAXException

Throws:
  `SAXException`

### errNcrControlChar

protected char errNcrControlChar(char ch) throws SAXException

Throws:
  `SAXException`

### errNcrCr

protected void errNcrCr() throws SAXException

Throws:
  `SAXException`

### errNcrInC1Range

protected void errNcrInC1Range() throws SAXException

Throws:
  `SAXException`

### errEofInPublicId

protected void errEofInPublicId() throws SAXException

Throws:
  `SAXException`

### errEofInComment

protected void errEofInComment() throws SAXException

Throws:
  `SAXException`

### errEofInDoctype

protected void errEofInDoctype() throws SAXException

Throws:
  `SAXException`

### errEofInAttributeValue

protected void errEofInAttributeValue() throws SAXException

Throws:
  `SAXException`

### errEofInAttributeName

protected void errEofInAttributeName() throws SAXException

Throws:
  `SAXException`

### errEofWithoutGt

protected void errEofWithoutGt() throws SAXException

Throws:
  `SAXException`

### errEofInTagName

protected void errEofInTagName() throws SAXException

Throws:
  `SAXException`

### errEofInEndTag

protected void errEofInEndTag() throws SAXException

Throws:
  `SAXException`

### errEofAfterLt

protected void errEofAfterLt() throws SAXException

Throws:
  `SAXException`

### errNcrOutOfRange

protected void errNcrOutOfRange() throws SAXException

Throws:
  `SAXException`

### errNcrUnassigned

protected void errNcrUnassigned() throws SAXException

Throws:
  `SAXException`

### errDuplicateAttribute

protected void errDuplicateAttribute() throws SAXException

Throws:
  `SAXException`

### errEofInSystemId

protected void errEofInSystemId() throws SAXException

Throws:
  `SAXException`

### errExpectedSystemId

protected void errExpectedSystemId() throws SAXException

Throws:
  `SAXException`

### errMissingSpaceBeforeDoctypeName

protected void errMissingSpaceBeforeDoctypeName() throws SAXException

Throws:
  `SAXException`

### errNcrControlChar

protected void errNcrControlChar() throws SAXException

Throws:
  `SAXException`

### errNcrZero

protected void errNcrZero() throws SAXException

Throws:
  `SAXException`

### errNoSpaceBetweenDoctypeSystemKeywordAndQuote

protected void errNoSpaceBetweenDoctypeSystemKeywordAndQuote() throws SAXException

Throws:
  `SAXException`

### errNoSpaceBetweenPublicAndSystemIds

protected void errNoSpaceBetweenPublicAndSystemIds() throws SAXException

Throws:
  `SAXException`

### errNoSpaceBetweenDoctypePublicKeywordAndQuote

protected void errNoSpaceBetweenDoctypePublicKeywordAndQuote() throws SAXException

Throws:
  `SAXException`

### noteAttributeWithoutValue

protected void noteAttributeWithoutValue() throws SAXException

Throws:
  `SAXException`

### noteUnquotedAttributeValue

protected void noteUnquotedAttributeValue() throws SAXException

Throws:
  `SAXException`

### setEncodingDeclarationHandler

public void setEncodingDeclarationHandler(EncodingDeclarationHandler encodingDeclarationHandler)
Sets the encodingDeclarationHandler.

Parameters:
  `encodingDeclarationHandler` - the encodingDeclarationHandler to set

### setTransitionBaseOffset

public void setTransitionBaseOffset(int offset)
Sets an offset to be added to the position reported to `TransitionHandler`.

Parameters:
  `offset` - the offset