### Tokenlist installation

First you have to copy `tokenlist.sl` somewhere on your jed_library_path. Optionally you can also copy `tkl-modes.sl` if you will use the default settings for some modes (SLang, C, python, PHP, HTML, LaTeX). Insert the content of the INITIALIZATION block (from `tokenlist.sl`) or just

      require("tokenlist");

into your jed.rc (or .jedrc) file (or use the "make_ini" and "home-lib" modes from jedmodes.sf.net). Optionally add some keybindings, e.g.:

      setkey("list_routines", "^R");
      setkey("occur", "^O");

The function list_routines is very simple in its default implementation and is conditionally useful for C and SLang mode. To extend its capabilities you have to add some macros to search for routines and extract them into the list-buffer. Extensions are provided for more modes in the file `tkl-modes.sl` which can be loaded in `tokenlist_hook()` which you define in `.jedrc (jed.rc)`:

      define tokenlist_hook()
      {
         if (expand_jedlib_file("tkl-modes"))
           require("tkl-modes");
      }

If you use `require("tokenlist")` to load `tokenlist.sl` then you should put the definition of `tokenlist_hook()` before `require("tokenlist")`.

You can add more definitions to `tkl-modes.sl` but it is better to write a separate file like `mytklist.sl` and change `tokenlist_hook()` to:

      define tokenlist_hook()
      {
         if (expand_jedlib_file("tkl-modes"))
           require("tkl-modes");
         if (expand_jedlib_file("mytklist"))
           require("mytklist");
      }

### Defining your own routine-search

You can write the macros for routine-search directly into `.jedrc` but it is better to add them to a separate file so they will be loaded only when needed. Here is a minimal example for `mytklist.sl`:

    % ----------- mytklist.sl ------
    provide("mytklist");

Let us now define a special X-TEXT mode where the titles on the first level start with '::', on the second with '==' an and on the third level with '--'. In the result list the titles should be indented depending on the level of the title.

For the new mode to work we have to define the function `x_text_list_routines_setup(opt)` where `opt` is a structure that defines the behaviour of `list_routines`. It has the following fields:

Field | Description | Default
--- | --- | ---
`mode` | The name of the mode. |

`list_regexp`

A list of regular expressions to search for. `list_routines` will take every item from the list and search for all the occurences in the active buffer that match the item (regular expression). After an expression is found, some text is extracted from the buffer with a call to `fn_extract`.

An item from the list can be a function reference instead of a regular expression. In this case the function will be called by `list_routines` until it returns 0. The function takes an integer parameter which is the current index and looks like this:

        % Int_Type searc_fn(Int_Type array_index)
        define searc_fn(idx)
        {
           return fsearch("something");
        }

`{"^[a-zA-Z].*("};`

`fn_extract`

A reference to a (private) function that extracts some text at the current point in the active buffer and returns it as a string. It is called after every match that `list_routines` finds. The function takes an integer parameter which is the current index and looks like this:

        % String extract_fn(Int_Type array_index)
        define extract_fn(idx)
        {
           return line_as_string();
        }

`&_list_routines_extract`

The function is defined in tokenlist and returns the current line (line_as_string).

`onlistcreated`

A reference to a (private) function that is called after all the items from `list_regexp` have been processed. The active buffer is the list of matches and it is narrowed to the matches found in the last buffer.

You can use this hook to sort the matched lines or do any other processing of the buffer. There are two helper functions defined in `tokenlist.sl` to sort the lines by line number (`tkl_sort_by_line`) or by content (`tkl_sort_by_value`).

`NULL`

Here is the code for the "X-TEXT" mode:

    % ----------- mytklist.sl ------
    autoload("tkl_sort_by_value", "tokenlist");
    autoload("tkl_sort_by_line", "tokenlist");
    provide("mytklist");

    private define x_text_list_routines_extract(n)
    {
      variable chars  = [":", "=", "-"];
      variable prefix = ["", "..", "...."];
      if (n < 3) {
        bol();
        skip_chars(chars[n]);
        push_mark();
        eol();
        return (prefix[n] + strtrim(bufsubstr()));
      }
      return "";
    }

    define x_text_list_routines_setup(opt)
    {
      opt.list_regexp = {
        '^::',
        '^==',
        '^--'
      };
      opt.fn_extract = &x_text_list_routines_extract;
      opt.onlistcreated = &tkl_sort_by_line;
    }

If have a buffer TestBufer.xtxt with the content like

    ::Heading 1
    ...
    ==Heading 2
    ...
    --Heading 3
    ..
    ==Heading 4

`list_routines` would generate the following list:

    Buffer: TestBufer.xtxt
         3 : Heading 1
        27 : ..Heading 2
        33 : ....Heading 3
        43 : ..Heading 4

You can see that the list is sorted by line number. This is because we set `opt.onlistcreated` to `&tkl_sort_by_line`. If we set it to `NULL` we get the following list:

    Buffer: TestBufer.xtxt
         3 : Heading 1
        27 : ..Heading 2
        43 : ..Heading 4
        33 : ....Heading 3

If we wanted to sort the titles by value, we could use `tkl_sort_by_value`. Unfortunately it would not sort the titles alphabetically because the titles in the buffer are indented.

Mode writers are encouraged to add `_list_routines_setup` definitions to their modes.

### Extending existing modes

Suppose you would like to list all the special blocks like `#<INITIALIZATION>` in SLang mode. You could change the function `slang_list_routines_setup` in `tkl-modes.sl` or copy the contents from `tkl-modes.sl` to `mytklist.sl` and change it there. This would be hard to maintain.

That is why another hook was added to tokenlist:

      Void  tokenlist_routine_setup_hook(opt)

This hook is not defiend by default. You can add it to a file like `mytklist.sl`. In our example we add a new regular expression and modify the extraction function. We store some values from the structure `opt` for later use.

    private variable slang_nregex = NULL;
    private variable slang_fn_extract = NULL;
    private variable slang_onlistcreated = NULL;

    private define my_slang_fn_extract(idx)
    {
       if (idx < slang_nregex && slang_fn_extract != NULL)
          return (@slang_fn_extract)(idx);
       else
          return strtrim(line_as_string());
    }

    define tokenlist_routine_setup_hook(opt)
    {
       if (opt.mode == "slang")
       {
          slang_nregex = length(opt.list_regexp);
          slang_fn_extract = opt.fn_extract;
          slang_onlistcreated = opt.onlistcreated;
          list_append(opt.list_regexp, "^#\\<[a-zA-Z]");
          opt.fn_extract = &my_slang_fn_extract;
       }
    }

`my_slang_fn_extract` will use the original extraction function for the original regular expressions and extract the current line for the new regular expression.

Imagine that you woud like to sometimes list only functions and at other times only variables. You would define a global variable and prepare the members of the parameter `opt` according to the value of the variable:

    % 0 - all; 1 - variables; 2 - macros
    variable slang_routine_mode = 0;

    private define my_slang_extract_variable(n)
    {
      return(strtrim(line_as_string()));
    }

    private define my_slang_extract_macro(n)
    {
      return(strtrim(line_as_string()));
    }

    define tokenlist_routine_setup_hook(opt)
    {
       if (opt.mode == "slang")
       {
          if (slang_routine_mode == 1)
          {
             opt.list_regexp = { "^variable[ \t]" };
             opt.fn_extract = &my_slang_extract_variable;
          }
          if (slang_routine_mode == 2)
          {
             opt.list_regexp = { "^define[ \t]" };
             opt.fn_extract = &my_slang_extract_macro;
          }
       }
    }

### Future

The structure `opt` could have more members so tokenlist could be made extremely configurable:

- what to do when user presses Enter (replacing tkl_goto_token)
- what to do when user presses Space (replacing tkl_display_token)
- what other keybindings do we want in the list
- what to do when the current line is changed in the list (extending the tkl_update_token_hook)
- ...