# Class: YARD::DocstringParser

Inherits:
  Object
  - Object
  - YARD::DocstringParser
  show all

Defined in:
  lib/yard/docstring_parser.rb

## Overview

Parses text and creates a Docstring object to represent documentation for a CodeObjects::Base. To create a new docstring, you should initialize the parser and call #parse followed by #to_docstring.

## Subclassing Notes

The DocstringParser can be subclassed and substituted during parsing by setting the YARD::Docstring.default_parser attribute with the name of the subclass. This allows developers to change the way docstrings are parsed, allowing for completely different docstring syntaxes.

#### Examples:

#####

Creating a Docstring with a DocstringParser

    DocstringParser.new.parse("text here").to_docstring

#####

Creating a Custom DocstringParser

    # Parses docstrings backwards!
    class ReverseDocstringParser
      def parse_content(content)
        super(content.reverse)
      end
    end

    # Set the parser as default when parsing
    YARD::Docstring.default_parser = ReverseDocstringParser

See Also:

- #parse_content

Since:

- 0.8.0

## Constant Summary collapse

META_MATCH =

The regular expression to match the tag syntax

Since:

- 0.8.0

      /^@(!)?((?:\w\.?)+)(?:\s+(.*))?$/i

## Instance Attribute Summary collapse

- #directives ⇒ Array<Tags::Directive>

  A list of directives identified by the parser.

- #handler ⇒ Handlers::Base?

  The handler parsing this docstring.

- #library ⇒ Tags::Library

  The tag library being used to identify registered tags in the docstring.

- #object ⇒ CodeObjects::Base?

  The object associated with the docstring being parsed.

- #raw_text ⇒ String

  The complete input string to the parser.

- #reference ⇒ CodeObjects::Base?

  The object referenced by the docstring being parsed.

- #state ⇒ OpenStruct

  Any arbitrary state to be passed between tags during parsing.

- #tags ⇒ Array<Tags::Tag>

  The list of meta-data tags identified by the parser.

- #text ⇒ String

  The parsed text portion of the docstring, with tags removed.

## Creation and Conversion Methods collapse

- #initialize(library = Tags::Library.instance) ⇒ DocstringParser constructor

  Creates a new parser to parse docstring data.

- #to_docstring ⇒ Docstring

  Translates parsed text into a Docstring object.

## Parsing Methods collapse

- #parse(content, object = nil, handler = nil) ⇒ self

  Parses all content and returns itself.

- #parse_content(content) ⇒ Object

  Parses a given block of text.

## Parser Callback Methods collapse

- #post_process ⇒ void

  Call post processing callbacks on parser.

## Tag Manipulation Methods collapse

- .after_parse {|parser| ... } ⇒ void

  Creates a callback that is called after a docstring is successfully parsed.

- .after_parse_callbacks ⇒ Array<Proc>

  The DocstringParser.after_parse callback proc objects.

- #create_directive(tag_name, tag_buf) ⇒ Tags::Directive

  Creates a new directive using the registered #library.

- #create_ref_tag(tag_name, name, object_name) ⇒ Object

  Creates a Tags::RefTag.

- #create_tag(tag_name, tag_buf = '') ⇒ Tags::Tag, Tags::RefTag

  Creates a tag from the tag factory.

- #tag_is_directive?(tag_name) ⇒ Boolean

  Backward compatibility to detect old tags that should be specified as directives in 0.8 and onward.

## Constructor Details

### #initialize(library = Tags::Library.instance) ⇒ `DocstringParser`

Creates a new parser to parse docstring data

Parameters:

- library (`Tags::Library`) (defaults to: `Tags::Library.instance`) —

  a tag library for recognizing tags.

Since:

- 0.8.0

    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90

    # File 'lib/yard/docstring_parser.rb', line 80

    def initialize(library = Tags::Library.instance)
      @text = ""
      @raw_text = ""
      @tags = []
      @directives = []
      @library = library
      @object = nil
      @reference = nil
      @handler = nil
      @state = OpenStruct.new
    end

## Instance Attribute Details

### #directives ⇒ `Array<Tags::Directive>`

Returns a list of directives identified by the parser. This list will not be passed on to the Docstring object.

Returns:

- (`Array<Tags::Directive>`) —

  a list of directives identified by the parser. This list will not be passed on to the Docstring object.

Since:

- 0.8.0

    44
    45
    46

    # File 'lib/yard/docstring_parser.rb', line 44

    def directives
      @directives
    end

### #handler ⇒ `Handlers::Base`?

Returns the handler parsing this docstring. May be nil if this docstring parser is not initialized through.

Returns:

- (`Handlers::Base`, `nil`) —

  the handler parsing this docstring. May be nil if this docstring parser is not initialized through

Since:

- 0.8.0

    65
    66
    67

    # File 'lib/yard/docstring_parser.rb', line 65

    def handler
      @handler
    end

### #library ⇒ `Tags::Library`

Returns the tag library being used to identify registered tags in the docstring.

Returns:

- (`Tags::Library`) —

  the tag library being used to identify registered tags in the docstring.

Since:

- 0.8.0

    69
    70
    71

    # File 'lib/yard/docstring_parser.rb', line 69

    def library
      @library
    end

### #object ⇒ `CodeObjects::Base`?

Returns the object associated with the docstring being parsed. May be nil if the docstring is not attached to any object.

Returns:

- (`CodeObjects::Base`, `nil`) —

  the object associated with the docstring being parsed. May be nil if the docstring is not attached to any object.

Since:

- 0.8.0

    55
    56
    57

    # File 'lib/yard/docstring_parser.rb', line 55

    def object
      @object
    end

### #raw_text ⇒ `String`

Returns the complete input string to the parser.

Returns:

- (`String`) —

  the complete input string to the parser.

Since:

- 0.8.0

    35
    36
    37

    # File 'lib/yard/docstring_parser.rb', line 35

    def raw_text
      @raw_text
    end

### #reference ⇒ `CodeObjects::Base`?

Returns the object referenced by the docstring being parsed. May be nil if the docstring doesn’t refer to any object.

Returns:

- (`CodeObjects::Base`, `nil`) —

  the object referenced by the docstring being parsed. May be nil if the docstring doesn’t refer to any object.

Since:

- 0.8.0

    60
    61
    62

    # File 'lib/yard/docstring_parser.rb', line 60

    def reference
      @reference
    end

### #state ⇒ `OpenStruct`

Returns any arbitrary state to be passed between tags during parsing. Mainly used by directives to coordinate behaviour (so that directives can be aware of other directives used in a docstring).

Returns:

- (`OpenStruct`) —

  any arbitrary state to be passed between tags during parsing. Mainly used by directives to coordinate behaviour (so that directives can be aware of other directives used in a docstring).

Since:

- 0.8.0

    50
    51
    52

    # File 'lib/yard/docstring_parser.rb', line 50

    def state
      @state
    end

### #tags ⇒ `Array<Tags::Tag>`

Returns the list of meta-data tags identified by the parser.

Returns:

- (`Array<Tags::Tag>`) —

  the list of meta-data tags identified by the parser

Since:

- 0.8.0

    39
    40
    41

    # File 'lib/yard/docstring_parser.rb', line 39

    def tags
      @tags
    end

### #text ⇒ `String`

Returns the parsed text portion of the docstring, with tags removed.

Returns:

- (`String`) —

  the parsed text portion of the docstring, with tags removed.

Since:

- 0.8.0

    32
    33
    34

    # File 'lib/yard/docstring_parser.rb', line 32

    def text
      @text
    end

## Class Method Details

### .after_parse {|parser| ... } ⇒ `void`

This method returns an undefined value.

Creates a callback that is called after a docstring is successfully parsed. Use this method to perform sanity checks on a docstring’s tag data, or add any extra tags automatically to a docstring.

Yields:

- (`parser`) —

  a block to be called after a docstring is parsed

Yield Parameters:

- parser (`DocstringParser`) —

  the docstring parser object with all directives and tags created.

Yield Returns:

- (`void`)

Since:

- 0.8.0

    265
    266
    267

    # File 'lib/yard/docstring_parser.rb', line 265

    def self.after_parse(&block)
      after_parse_callbacks << block
    end

### .after_parse_callbacks ⇒ `Array<Proc>`

Returns the after_parse callback proc objects.

Returns:

- (`Array<Proc>`) —

  the after_parse callback proc objects

Since:

- 0.8.0

    270
    271
    272

    # File 'lib/yard/docstring_parser.rb', line 270

    def self.after_parse_callbacks
      @after_parse_callbacks ||= []
    end

## Instance Method Details

### #create_directive(tag_name, tag_buf) ⇒ `Tags::Directive`

Creates a new directive using the registered #library

Returns:

- (`Tags::Directive`) —

  the directive object that is created

Since:

- 0.8.0

    231
    232
    233
    234
    235
    236
    237
    238
    239
    240
    241
    242
    243
    244
    245
    246
    247

    # File 'lib/yard/docstring_parser.rb', line 231

    def create_directive(tag_name, tag_buf)
      if library.has_directive?(tag_name)
        dir = library.directive_create(tag_name, tag_buf, self)
        if dir.is_a?(Tags::Directive)
          @directives << dir
          dir
        end
      else
        log.warn "Unknown directive @!#{tag_name}" +
                 (object ? " in file `#{object.file}` near line #{object.line}" : "")
        nil
      end
    rescue Tags::TagFormatError
      log.warn "Invalid directive format for @!#{tag_name}" +
               (object ? " in file `#{object.file}` near line #{object.line}" : "")
      nil
    end

### #create_ref_tag(tag_name, name, object_name) ⇒ `Object`

Creates a Tags::RefTag

Since:

- 0.8.0

    225
    226
    227

    # File 'lib/yard/docstring_parser.rb', line 225

    def create_ref_tag(tag_name, name, object_name)
      @tags << Tags::RefTagList.new(tag_name, P(object, object_name), name)
    end

### #create_tag(tag_name, tag_buf = '') ⇒ `Tags::Tag`, `Tags::RefTag`

Creates a tag from the tag factory.

To add an already created tag object, append it to #tags.

Parameters:

- tag_name (`String`) —

  the tag name

- tag_buf (`String`) (defaults to: `''`) —

  the text attached to the tag with newlines removed.

Returns:

- (`Tags::Tag`, `Tags::RefTag`) —

  a tag

Since:

- 0.8.0

    208
    209
    210
    211
    212
    213
    214
    215
    216
    217
    218
    219
    220
    221
    222

    # File 'lib/yard/docstring_parser.rb', line 208

    def create_tag(tag_name, tag_buf = '')
      if tag_buf =~ /\A\s*(?:(\S+)\s+)?\(\s*see\s+(\S+)\s*\)\s*\Z/
        return create_ref_tag(tag_name, $1, $2)
      end

      if library.has_tag?(tag_name)
        @tags += [library.tag_create(tag_name, tag_buf)].flatten
      else
        log.warn "Unknown tag @#{tag_name}" +
                 (object ? " in file `#{object.file}` near line #{object.line}" : "")
      end
    rescue Tags::TagFormatError
      log.warn "Invalid tag format for @#{tag_name}" +
               (object ? " in file `#{object.file}` near line #{object.line}" : "")
    end

### #parse(content, object = nil, handler = nil) ⇒ `self`

Parses all content and returns itself.

Parameters:

- content (`String`) —

  the docstring text to parse

- object (`CodeObjects::Base`) (defaults to: `nil`) —

  the object that the docstring is attached to. Will be passed to directives to act on this object.

- handler (`Handlers::Base`, `nil`) (defaults to: `nil`) —

  the handler object that is parsing this object. May be nil if this parser is not being called from a Parser::SourceParser context.

Returns:

- (`self`) —

  the parser object. To get the docstring, call #to_docstring.

See Also:

- #to_docstring

Since:

- 0.8.0

    112
    113
    114
    115
    116
    117
    118
    119
    120
    121

    # File 'lib/yard/docstring_parser.rb', line 112

    def parse(content, object = nil, handler = nil)
      @object = object
      @handler = handler
      @reference, @raw_text = detect_reference(content)
      text = parse_content(@raw_text)
      @text = text.strip
      call_directives_after_parse
      post_process
      self
    end

### #parse_content(content) ⇒ `Object`

Note:

Subclasses can override this method to perform custom parsing of content data.

Parses a given block of text.

Parameters:

- content (`String`) —

  the content to parse

Since:

- 0.8.0

    128
    129
    130
    131
    132
    133
    134
    135
    136
    137
    138
    139
    140
    141
    142
    143
    144
    145
    146
    147
    148
    149
    150
    151
    152
    153
    154
    155
    156
    157
    158
    159
    160
    161
    162
    163
    164
    165
    166
    167
    168
    169
    170
    171
    172
    173
    174
    175
    176
    177
    178
    179
    180
    181
    182
    183
    184
    185
    186

    # File 'lib/yard/docstring_parser.rb', line 128

    def parse_content(content)
      content = content.split(/\r?\n/) if content.is_a?(String)
      return '' if !content || content.empty?
      docstring = String.new("")

      indent = content.first[/^\s*/].length
      last_indent = 0
      orig_indent = 0
      directive = false
      last_line = ""
      tag_name = nil
      tag_buf = []

      (content + ['']).each_with_index do |line, index|
        indent = line[/^\s*/].length
        empty = (line =~ /^\s*$/ ? true : false)
        done = content.size == index

        if tag_name && (((indent < orig_indent && !empty) || done ||
            (indent == 0 && !empty)) || (indent <= last_indent && line =~ META_MATCH))
          buf = tag_buf.join("\n")
          if directive || tag_is_directive?(tag_name)
            directive = create_directive(tag_name, buf)
            if directive
              docstring << parse_content(directive.expanded_text).chomp
            end
          else
            create_tag(tag_name, buf)
          end
          tag_name = nil
          tag_buf = []
          directive = false
          orig_indent = 0
        end

        # Found a meta tag
        if line =~ META_MATCH
          directive = $1
          tag_name = $2
          tag_buf = [($3 || '')]
        elsif tag_name && indent >= orig_indent && !empty
          orig_indent = indent if orig_indent == 0
          # Extra data added to the tag on the next line
          last_empty = last_line =~ /^[ \t]*$/ ? true : false

          tag_buf << '' if last_empty
          tag_buf << line.gsub(/^[ \t]{#{orig_indent}}/, '')
        elsif !tag_name
          # Regular docstring text
          docstring << line
          docstring << "\n"
        end

        last_indent = indent
        last_line = line
      end

      docstring
    end

### #post_process ⇒ `void`

This method returns an undefined value.

Call post processing callbacks on parser. This is called implicitly by parser. Use this when manually configuring a YARD::Docstring object.

Since:

- 0.8.0

    195
    196
    197

    # File 'lib/yard/docstring_parser.rb', line 195

    def post_process
      call_after_parse_callbacks
    end

### #tag_is_directive?(tag_name) ⇒ `Boolean`

Backward compatibility to detect old tags that should be specified as directives in 0.8 and onward.

Returns:

- (`Boolean`)

Since:

- 0.8.0

    251
    252
    253
    254

    # File 'lib/yard/docstring_parser.rb', line 251

    def tag_is_directive?(tag_name)
      list = %w(attribute endgroup group macro method scope visibility)
      list.include?(tag_name)
    end

### #to_docstring ⇒ `Docstring`

Returns translates parsed text into a Docstring object.

Returns:

- (`Docstring`) —

  translates parsed text into a Docstring object.

Since:

- 0.8.0

    94
    95
    96

    # File 'lib/yard/docstring_parser.rb', line 94

    def to_docstring
      Docstring.new!(text, tags, object, raw_text, reference)
    end