# Class: YARD::Parser::SourceParser

Inherits:
  Object
  - Object
  - YARD::Parser::SourceParser
  show all

Defined in:
  lib/yard/parser/source_parser.rb

## Overview

Responsible for parsing a source file into the namespace. Parsing also invokes handlers to process the parsed statements and generate any code objects that may be recognized.

## Custom Parsers

SourceParser allows custom parsers to be registered and called when a certain filetype is recognized. To register a parser and hook it up to a set of file extensions, call SourceParser.register_parser_type

See Also:

- SourceParser.register_parser_type
- Handlers::Base
- CodeObjects::Base

## Constant Summary collapse

SHEBANG_LINE =

      /\A\s*#!\S+/

ENCODING_LINE =

      %r{\A(?:\s*#*!.*\r?\n)?\s*(?:#+|/\*+|//+).*coding\s*[:=]{1,2}\s*([a-z\d_\-]+)}i

FROZEN_STRING_LINE =

      /frozen(-|_)string(-|_)literal:\s+(true|false)/i

DEFAULT_PATH_GLOB =

The default glob of files to be parsed.

Since:

- 0.9.0

      ["{lib,app}/**/*.rb", "ext/**/*.{c,cc,cxx,cpp,rb}"]

ENCODING_BYTE_ORDER_MARKS =

Byte order marks for various encodings

Since:

- 0.7.0

      {
        'utf-8' => String.new("\xEF\xBB\xBF"),
        # Not yet supported
        # 'utf-16be' => "\xFE\xFF",
        # 'utf-16le' => "\xFF\xFE",
        # 'utf-32be' => "\x00\x00\xFF\xFE",
        # 'utf-32le' => "\xFF\xFE",
      }

## Parser Callbacks collapse

- #contents ⇒ String readonly

  The contents of the file to be parsed.

- #file ⇒ String

  The filename being parsed by the parser.

- #globals ⇒ OpenStruct readonly

  An open struct containing arbitrary global state shared between files and handlers.

- #parser_type ⇒ Symbol readonly

  The parser type associated with the parser instance.

## Class Attribute Summary collapse

- .parser_type ⇒ Symbol

  The default parser type (defaults to :ruby).

## Parser Callbacks collapse

- .after_parse_file {|parser| ... } ⇒ Proc

  Registers a callback to be called after an individual file is parsed.

- .after_parse_file_callbacks ⇒ Array<Proc>

  The list of callbacks to be called after parsing a file.

- .after_parse_list {|files, globals| ... } ⇒ Proc

  Registers a callback to be called after a list of files is parsed via SourceParser.parse.

- .after_parse_list_callbacks ⇒ Array<Proc>

  The list of callbacks to be called after parsing a list of files.

- .before_parse_file {|parser| ... } ⇒ Proc

  Registers a callback to be called before an individual file is parsed.

- .before_parse_file_callbacks ⇒ Array<Proc>

  The list of callbacks to be called before parsing a file.

- .before_parse_list {|files, globals| ... } ⇒ Proc

  Registers a callback to be called before a list of files is parsed via SourceParser.parse.

- .before_parse_list_callbacks ⇒ Array<Proc>

  The list of callbacks to be called before parsing a list of files.

- #initialize(parser_type = SourceParser.parser_type, globals = nil) ⇒ SourceParser constructor

  Creates a new parser object for code parsing with a specific parser type.

- #parse(content = __FILE__) ⇒ Object?

  The main parser method.

- #tokenize(content) ⇒ Array

  Tokenizes but does not parse the block of code using the current #parser_type.

## Class Method Summary collapse

- .parse(paths = DEFAULT_PATH_GLOB, excluded = [], level = log.level) ⇒ void

  Parses a path or set of paths.

- .parse_string(content, ptype = parser_type) ⇒ Object

  Parses a string `content`.

- .parser_type_extensions=(value) ⇒ Object
- .parser_type_for_extension(extension) ⇒ Symbol

  Finds a parser type that is registered for the extension.

- .parser_types=(value) ⇒ Object
- .register_parser_type(type, parser_klass, extensions = nil) ⇒ void

  Registers a new parser type.

- .tokenize(content, ptype = parser_type) ⇒ Array

  Tokenizes but does not parse the block of code.

## Constructor Details

### #initialize(parser_type = SourceParser.parser_type, globals = nil) ⇒ `SourceParser`

Returns a new instance of SourceParser.

Creates a new parser object for code parsing with a specific parser type.

Parameters:

- parser_type (`Symbol`) (defaults to: `SourceParser.parser_type`) —

  the parser type to use

- globals (`OpenStruct`) (defaults to: `nil`) —

  global state to be re-used across separate source files

    406
    407
    408
    409
    410
    411

    # File 'lib/yard/parser/source_parser.rb', line 406

    def initialize(parser_type = SourceParser.parser_type, globals1 = nil, globals2 = nil)
      globals = [true, false].include?(globals1) ? globals2 : globals1
      @file = '(stdin)'
      @globals = globals || OpenStruct.new
      self.parser_type = parser_type
    end

## Class Attribute Details

### .parser_type ⇒ `Symbol`

Returns the default parser type (defaults to :ruby).

Returns:

- (`Symbol`) —

  the default parser type (defaults to :ruby)

    85
    86
    87

    # File 'lib/yard/parser/source_parser.rb', line 85

    def parser_type
      @parser_type
    end

## Instance Attribute Details

### #contents ⇒ `String` (readonly)

Returns the contents of the file to be parsed.

Returns:

- (`String`) —

  the contents of the file to be parsed

Since:

- 0.7.0

    399
    400
    401

    # File 'lib/yard/parser/source_parser.rb', line 399

    def contents
      @contents
    end

### #file ⇒ `String`

Returns the filename being parsed by the parser.

Returns:

- (`String`) —

  the filename being parsed by the parser.

    386
    387
    388

    # File 'lib/yard/parser/source_parser.rb', line 386

    def file
      @file
    end

### #globals ⇒ `OpenStruct` (readonly)

Returns an open struct containing arbitrary global state shared between files and handlers.

Returns:

- (`OpenStruct`) —

  an open struct containing arbitrary global state shared between files and handlers.

Since:

- 0.7.0

    395
    396
    397

    # File 'lib/yard/parser/source_parser.rb', line 395

    def globals
      @globals
    end

### #parser_type ⇒ `Symbol`

Returns the parser type associated with the parser instance. This should be set by the constructor.

Returns:

- (`Symbol`) —

  the parser type associated with the parser instance. This should be set by the constructor.

    390
    391
    392

    # File 'lib/yard/parser/source_parser.rb', line 390

    def parser_type
      @parser_type
    end

## Class Method Details

### .after_parse_file {|parser| ... } ⇒ `Proc`

Registers a callback to be called after an individual file is parsed. The block passed to this method will be called on subsequent parse calls.

To register a callback that is called after the entire list of files is processed, see after_parse_list.

#### Examples:

#####

Printing the length of each file after it is parsed

    SourceParser.after_parse_file do |parser|
      puts "#{parser.file} is #{parser.contents.size} characters"
    end
    YARD.parse('lib/**/*.rb')
    # prints:
    "lib/foo.rb is 1240 characters"
    "lib/foo_bar.rb is 248 characters"

Yields:

- (`parser`) —

  the yielded block is called once after each file that is parsed. This might happen many times for a single codebase.

Yield Parameters:

- parser (`SourceParser`) —

  the parser object that parsed the file.

Yield Returns:

- (`void`) —

  the return value for the block is ignored.

Returns:

- (`Proc`) —

  the yielded block

See Also:

- before_parse_file
- after_parse_list

Since:

- 0.7.0

    324
    325
    326

    # File 'lib/yard/parser/source_parser.rb', line 324

    def after_parse_file(&block)
      after_parse_file_callbacks << block
    end

### .after_parse_file_callbacks ⇒ `Array<Proc>`

Returns the list of callbacks to be called after parsing a file. Should only be used for testing.

Returns:

- (`Array<Proc>`) —

  the list of callbacks to be called after parsing a file. Should only be used for testing.

Since:

- 0.7.0

    352
    353
    354

    # File 'lib/yard/parser/source_parser.rb', line 352

    def after_parse_file_callbacks
      @after_parse_file_callbacks ||= []
    end

### .after_parse_list {|files, globals| ... } ⇒ `Proc`

Registers a callback to be called after a list of files is parsed via parse. The block passed to this method will be called on subsequent parse calls.

#### Examples:

#####

Printing results after parsing occurs

    SourceParser.after_parse_list do
      puts "Finished parsing!"
    end
    YARD.parse
    # Prints "Finished parsing!" after parsing files

Yields:

- (`files`, `globals`) —

  the yielded block is called once before parsing all files

Yield Parameters:

- files (`Array<String>`) —

  the list of files that will be parsed.

- globals (`OpenStruct`) —

  a global structure to store arbitrary state for post processing (see Handlers::Processor#globals)

Yield Returns:

- (`void`) —

  the return value for the block is ignored.

Returns:

- (`Proc`) —

  the yielded block

See Also:

- before_parse_list
- before_parse_file

Since:

- 0.7.0

    258
    259
    260

    # File 'lib/yard/parser/source_parser.rb', line 258

    def after_parse_list(&block)
      after_parse_list_callbacks << block
    end

### .after_parse_list_callbacks ⇒ `Array<Proc>`

Returns the list of callbacks to be called after parsing a list of files. Should only be used for testing.

Returns:

- (`Array<Proc>`) —

  the list of callbacks to be called after parsing a list of files. Should only be used for testing.

Since:

- 0.7.0

    338
    339
    340

    # File 'lib/yard/parser/source_parser.rb', line 338

    def after_parse_list_callbacks
      @after_parse_list_callbacks ||= []
    end

### .before_parse_file {|parser| ... } ⇒ `Proc`

Registers a callback to be called before an individual file is parsed. The block passed to this method will be called on subsequent parse calls.

To register a callback that is called before the entire list of files is processed, see before_parse_list.

#### Examples:

#####

Installing a simple callback

    SourceParser.before_parse_file do |parser|
      puts "I'm parsing #{parser.file}"
    end
    YARD.parse('lib/**/*.rb')
    # prints:
    "I'm parsing lib/foo.rb"
    "I'm parsing lib/foo_bar.rb"
    "I'm parsing lib/last_file.rb"

#####

Cancel parsing of any test_*.rb files

    SourceParser.before_parse_file do |parser|
      return false if parser.file =~ /^test_.+\.rb$/
    end

Yields:

- (`parser`) —

  the yielded block is called once before each file that is parsed. This might happen many times for a single codebase.

Yield Parameters:

- parser (`SourceParser`) —

  the parser object that will #parse the file.

Yield Returns:

- (`Boolean`) —

  if the block returns `false`, parsing for the file is cancelled.

Returns:

- (`Proc`) —

  the yielded block

See Also:

- after_parse_file
- before_parse_list

Since:

- 0.7.0

    295
    296
    297

    # File 'lib/yard/parser/source_parser.rb', line 295

    def before_parse_file(&block)
      before_parse_file_callbacks << block
    end

### .before_parse_file_callbacks ⇒ `Array<Proc>`

Returns the list of callbacks to be called before parsing a file. Should only be used for testing.

Returns:

- (`Array<Proc>`) —

  the list of callbacks to be called before parsing a file. Should only be used for testing.

Since:

- 0.7.0

    345
    346
    347

    # File 'lib/yard/parser/source_parser.rb', line 345

    def before_parse_file_callbacks
      @before_parse_file_callbacks ||= []
    end

### .before_parse_list {|files, globals| ... } ⇒ `Proc`

Registers a callback to be called before a list of files is parsed via parse. The block passed to this method will be called on subsequent parse calls.

#### Examples:

#####

Installing a simple callback

    SourceParser.before_parse_list do |files, globals|
      puts "Starting to parse..."
    end
    YARD.parse('lib/**/*.rb')
    # prints "Starting to parse..."

#####

Setting global state

    SourceParser.before_parse_list do |files, globals|
      globals.method_count = 0
    end
    SourceParser.after_parse_list do |files, globals|
      puts "Found #{globals.method_count} methods"
    end
    class MyCountHandler < Handlers::Ruby::Base
      handles :def, :defs
      process { globals.method_count += 1 }
    end
    YARD.parse
    # Prints: "Found 37 methods"

#####

Using a global callback to cancel parsing

    SourceParser.before_parse_list do |files, globals|
      return false if files.include?('foo.rb')
    end

    YARD.parse(['foo.rb', 'bar.rb']) # callback cancels this method
    YARD.parse('bar.rb') # parses normally

Yields:

- (`files`, `globals`) —

  the yielded block is called once before parsing all files

Yield Parameters:

- files (`Array<String>`) —

  the list of files that will be parsed.

- globals (`OpenStruct`) —

  a global structure to store arbitrary state for post processing (see Handlers::Processor#globals)

Yield Returns:

- (`Boolean`) —

  if the block returns `false`, parsing is cancelled.

Returns:

- (`Proc`) —

  the yielded block

See Also:

- after_parse_list
- before_parse_file

Since:

- 0.7.0

    234
    235
    236

    # File 'lib/yard/parser/source_parser.rb', line 234

    def before_parse_list(&block)
      before_parse_list_callbacks << block
    end

### .before_parse_list_callbacks ⇒ `Array<Proc>`

Returns the list of callbacks to be called before parsing a list of files. Should only be used for testing.

Returns:

- (`Array<Proc>`) —

  the list of callbacks to be called before parsing a list of files. Should only be used for testing.

Since:

- 0.7.0

    331
    332
    333

    # File 'lib/yard/parser/source_parser.rb', line 331

    def before_parse_list_callbacks
      @before_parse_list_callbacks ||= []
    end

### .parse(paths = DEFAULT_PATH_GLOB, excluded = [], level = log.level) ⇒ `void`

This method returns an undefined value.

Parses a path or set of paths

Parameters:

- paths (`String`, `Array<String>`) (defaults to: `DEFAULT_PATH_GLOB`) —

  a path, glob, or list of paths to parse

- excluded (`Array<String, Regexp>`) (defaults to: `[]`) —

  a list of excluded path matchers

- level (`Fixnum`) (defaults to: `log.level`) —

  the logger level to use during parsing. See Logger

    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116

    # File 'lib/yard/parser/source_parser.rb', line 99

    def parse(paths = DEFAULT_PATH_GLOB, excluded = [], level = log.level)
      log.debug("Parsing #{paths.inspect} with `#{parser_type}` parser")
      excluded = excluded.map do |path|
        case path
        when Regexp; path
        else Regexp.new(path.to_s, Regexp::IGNORECASE)
        end
      end
      files = [paths].flatten.
        map {|p| File.directory?(p) ? "#{p}/**/*.{rb,c,cc,cxx,cpp}" : p }.
        map {|p| p.include?("*") ? Dir[p].sort_by {|d| [d.length, d] } : p }.flatten.
        reject {|p| !File.file?(p) || excluded.any? {|re| p =~ re } }.
        map {|p| p.encoding == Encoding.default_external ? p : p.dup.force_encoding(Encoding.default_external) }

      log.enter_level(level) do
        parse_in_order(*files.uniq)
      end
    end

### .parse_string(content, ptype = parser_type) ⇒ `Object`

Parses a string `content`

Parameters:

- content (`String`) —

  the block of code to parse

- ptype (`Symbol`) (defaults to: `parser_type`) —

  the parser type to use. See parser_type.

Returns:

- the parser object that was used to parse `content`

    123
    124
    125

    # File 'lib/yard/parser/source_parser.rb', line 123

    def parse_string(content, ptype = parser_type)
      new(ptype).parse(StringIO.new(content))
    end

### .parser_type_extensions=(value) ⇒ `Object`

    164

    # File 'lib/yard/parser/source_parser.rb', line 164

    def parser_type_extensions=(value) @@parser_type_extensions = value end

### .parser_type_for_extension(extension) ⇒ `Symbol`

Finds a parser type that is registered for the extension. If no type is found, the default Ruby type is returned.

Returns:

- (`Symbol`) —

  the parser type to be used for the extension

Since:

- 0.5.6

    171
    172
    173
    174
    175
    176

    # File 'lib/yard/parser/source_parser.rb', line 171

    def parser_type_for_extension(extension)
      type = parser_type_extensions.find do |_t, exts|
        [exts].flatten.any? {|ext| ext === extension }
      end
      validated_parser_type(type ? type.first : :ruby)
    end

### .parser_types=(value) ⇒ `Object`

    158

    # File 'lib/yard/parser/source_parser.rb', line 158

    def parser_types=(value) @@parser_types = value end

### .register_parser_type(type, parser_klass, extensions = nil) ⇒ `void`

This method returns an undefined value.

Registers a new parser type.

#### Examples:

#####

Registering a parser for “java” files

    SourceParser.register_parser_type :java, JavaParser, 'java'

Parameters:

- type (`Symbol`) —

  a symbolic name for the parser type

- parser_klass (`Base`) —

  a class that implements parsing and tokenization

- extensions (`Array<String>`, `String`, `Regexp`) (defaults to: `nil`) —

  a list of extensions or a regex to match against the file extension

See Also:

- Base

    146
    147
    148
    149
    150
    151
    152

    # File 'lib/yard/parser/source_parser.rb', line 146

    def register_parser_type(type, parser_klass, extensions = nil)
      unless Base > parser_klass
        raise ArgumentError, "expecting parser_klass to be a subclass of YARD::Parser::Base"
      end
      parser_type_extensions[type.to_sym] = extensions if extensions
      parser_types[type.to_sym] = parser_klass
    end

### .tokenize(content, ptype = parser_type) ⇒ `Array`

Tokenizes but does not parse the block of code

Parameters:

- content (`String`) —

  the block of code to tokenize

- ptype (`Symbol`) (defaults to: `parser_type`) —

  the parser type to use. See parser_type.

Returns:

- (`Array`) —

  a list of tokens

    132
    133
    134

    # File 'lib/yard/parser/source_parser.rb', line 132

    def tokenize(content, ptype = parser_type)
      new(ptype).tokenize(content)
    end

## Instance Method Details

### #parse(content = __FILE__) ⇒ `Object`?

The main parser method. This should not be called directly. Instead, use the class methods parse and parse_string.

Parameters:

- content (`String`, `#read`, `Object`) (defaults to: `__FILE__`) —

  the source file to parse

Returns:

- (`Object`, `nil`) —

  the parser object used to parse the source

    418
    419
    420
    421
    422
    423
    424
    425
    426
    427
    428
    429
    430
    431
    432
    433
    434
    435
    436
    437
    438
    439
    440
    441
    442
    443
    444
    445
    446
    447
    448
    449
    450
    451
    452
    453
    454
    455
    456

    # File 'lib/yard/parser/source_parser.rb', line 418

    def parse(content = __FILE__)
      case content
      when String
        @file = File.cleanpath(content)
        content = convert_encoding(String.new(File.read_binary(file)))
        checksum = Registry.checksum_for(content)
        return if Registry.checksums[file] == checksum

        if Registry.checksums.key?(file)
          log.info "File '#{file}' was modified, re-processing..."
        end
        Registry.checksums[@file] = checksum
        self.parser_type = parser_type_for_filename(file)
      else
        content = content.read if content.respond_to? :read
      end

      @contents = content
      @parser = parser_class.new(content, file)

      self.class.before_parse_file_callbacks.each do |cb|
        return @parser if cb.call(self) == false
      end

      @parser.parse
      post_process

      self.class.after_parse_file_callbacks.each do |cb|
        cb.call(self)
      end

      @parser
    rescue ArgumentError, NotImplementedError => e
      log.warn("Cannot parse `#{file}': #{e.message}")
      log.backtrace(e, :warn)
    rescue ParserSyntaxError => e
      log.warn(e.message.capitalize)
      log.backtrace(e, :warn)
    end

### #tokenize(content) ⇒ `Array`

Tokenizes but does not parse the block of code using the current #parser_type

Parameters:

- content (`String`) —

  the block of code to tokenize

Returns:

- (`Array`) —

  a list of tokens

    462
    463
    464
    465

    # File 'lib/yard/parser/source_parser.rb', line 462

    def tokenize(content)
      @parser = parser_class.new(content, file)
      @parser.tokenize
    end