tool(3tcl)

NAME
tool - A TclOO and coroutine based web server

SYNOPSIS
package require Tcl 8.6

package require httpd ?4.1.1?

package require sha1

package require dicttool

package require oo::meta

package require oo::dialect

package require tool

package require coroutine

package require fileutil

package require fileutil::magic::filetype

package require websocket

package require mime

package require cron

package require uri

package require Markdown

constructor ?port ?port?? ?myaddr ?ipaddr?|all? ?server_string ?string?? ?server_name ?string??

method add_uri pattern dict

method connect sock ip port

method Connect uuid sock ip

method counter which

method CheckTimeout

method dispatch header_dict

method log args

method port_listening

method PrefixNormalize prefix

method start

method stop

method template page

method TemplateSearch page

method Validate_Connection sock ip

method ENSEMBLE::add field element

method ENSEMBLE::dump

method ENSEMBLE::get field

method ENSEMBLE::reset

method ENSEMBLE::remove field element

method ENSEMBLE::replace keyvaluelist

method ENSEMBLE::reset

method ENSEMBLE::set field value

method http_info::netstring

method request::parse string

method reply::output

method close

method HttpHeaders sock ?debug?

method dispatch newsock datastate

method error code ?message? ?errorInfo?

method content

method EncodeStatus status

method FormData

method MimeParse mimetext

method output

method DoOutput

method PostData length

method puts string

method reset

method timeOutCheck

method timestamp

method TransferComplete args

method Url_Decode string

method cgi_info

option path

option prefix

method proxy_info

method scgi_info

DESCRIPTION

This module implements a web server, suitable for embedding in an application. The server is object oriented, and contains all of the fundamentals needed for a full service website.

MINIMAL EXAMPLE
Starting a web service requires starting a class of type httpd::server, and providing that server with one or more URIs to service, and httpd::reply derived classes to generate them.

    tool::define ::reply.hello {
      method content {} {
        my puts "<HTML><HEAD><TITLE>IRM Dispatch Server</TITLE></HEAD><BODY>"
        my puts "<h1>Hello World!</h1>"
        my puts </BODY></HTML>
      }
    }
    ::docserver::server create HTTPD port 8015 myaddr 127.0.0.1
    HTTPD add_uri /* [list mixin reply.hello]

CLASS ::HTTPD::SERVER
This class is the root object of the webserver. It is responsible for opening the socket and providing the initial connection negotiation.

constructor ?port ?port?? ?myaddr ?ipaddr?|all? ?server_string ?string?? ?server_name ?string??
    Build a new server object. ?port? is the port to listen on

method add_uri pattern dict
    Set the hander for a URI pattern. Information given in the dict is stored in the data structure the dispatch method uses. If a field called mixin is given, that class will be mixed into the reply object immediately after construction.

method connect sock ip port
    Reply to an open socket. This method builds a coroutine to manage the remainder of the connection. The coroutine's operations are driven by the Connect method.

method Connect uuid sock ip
    This method reads HTTP headers, and then consults the dispatch method to determine if the request is valid, and/or what kind of reply to generate. Under normal cases, an object of class ::http::reply is created. Fields the server are looking for in particular are: class: A class to use instead of the server's own reply_class mixin: A class to be mixed into the new object after construction. All other fields are passed along to the http_info structure of the reply object. After the class is created and the mixin is mixed in, the server invokes the reply objects dispatch method. This action passes control of the socket to the reply object. The reply object manages the rest of the transaction, including closing the socket.

method counter which
    Increment an internal counter.

method CheckTimeout
    Check open connections for a time out event.

method dispatch header_dict
    Given a key/value list of information, return a data structure describing how the server should reply.

method log args
    Log an event. The input for args is free form. This method is intended to be replaced by the user, and is a noop for a stock http::server object.

method port_listening
    Return the actual port that httpd is listening on.

method PrefixNormalize prefix
    For the stock version, trim trailing /'s and *'s from a prefix. This method can be replaced by the end user to perform any other transformations needed for the application.

method start
    Open the socket listener.

method stop
    Shut off the socket listener, and destroy any pending replies.

method template page
    Return a template for the string page

method TemplateSearch page
    Perform a search for the template that best matches page. This can include local file searches, in-memory structures, or even database lookups. The stock implementation simply looks for files with a .tml or .html extension in the ?doc_root? directory.

method Validate_Connection sock ip
    Given a socket and an ip address, return true if this connection should be terminated, or false if it should be allowed to continue. The stock implementation always returns 0. This is intended for applications to be able to implement black lists and/or provide security based on IP address.

CLASS ::HTTPD::REPLY
A class which shephards a request through the process of generating a reply. The socket associated with the reply is available at all times as the chan variable. The process of generating a reply begins with an httpd::server generating a http::class object, mixing in a set of behaviors and then invoking the reply object's dispatch method. In normal operations the dispatch method:

[1] Invokes the reset method for the object to populate default headers.

[2] Invokes the HttpHeaders method to stream the MIME headers out of the socket

[3] Invokes the request parse method to convert the stream of MIME headers into a dict that can be read via the request method.

[4] Stores the raw stream of MIME headers in the rawrequest variable of the object.

[5] Invokes the content method for the object, generating an call to the error method if an exception is raised.

[6] Invokes the output method for the object

REPLY METHOD ENSEMBLES
The http::reply class and its derivatives maintain several variables as dictionaries internally. Access to these dictionaries is managed through a dedicated ensemble. The ensemble implements most of the same behaviors as the dict command. Each ensemble implements the following methods above, beyond, or modifying standard dicts:

method ENSEMBLE::add field element
    Add element to a list stored in field, but only if it is not already present om the list.

method ENSEMBLE::dump
    Return the current contents of the data structure as a key/value list.

method ENSEMBLE::get field
    Return the value of the field field, or an empty string if it does not exist.

method ENSEMBLE::reset
    Return a key/value list of the default contents for this data structure.

method ENSEMBLE::remove field element
    Remove all instances of element from the list stored in field.

method ENSEMBLE::replace keyvaluelist
    Replace the internal dict with the contents of keyvaluelist

method ENSEMBLE::reset
    Replace the internal dict with the default state.

method ENSEMBLE::set field value
    Set the value of field to value.

REPLY METHOD ENSEMBLE: HTTP_INFO
Manages HTTP headers passed in by the server. Ensemble Methods:

method http_info::netstring
    Return the contents of this data structure as a netstring encoded block.

REPLY METHOD ENSEMBLE: REQUEST
Managed data from MIME headers of the request.

method request::parse string
    Replace the contents of the data structure with information encoded in a MIME formatted block of text (string).

REPLY METHOD ENSEMBLE: REPLY
Manage the headers sent in the reply.

method reply::output
    Return the contents of this data structure as a MIME encoded block appropriate for an HTTP response.

REPLY METHODS

method close
    Terminate the transaction, and close the socket.

method HttpHeaders sock ?debug?
    Stream MIME headers from the socket sock, stopping at an empty line. Returns the stream as a block of text.

method dispatch newsock datastate
    Take over control of the socket newsock, and store that as the chan variable for the object. This method runs through all of the steps of reading HTTP headers, generating content, and closing the connection. (See class writetup).

method error code ?message? ?errorInfo?
    Generate an error message of the specified code, and display the message as the reason for the exception. errorInfo is passed in from calls, but how or if it should be displayed is a prerogative of the developer.

method content
    Generate the content for the reply. This method is intended to be replaced by the mixin. Developers have the option of streaming output to a buffer via the puts method of the reply, or simply populating the reply_body variable of the object. The information returned by the content method is not interpreted in any way. If an exception is thrown (via the error command in Tcl, for example) the caller will auto-generate a 505 {Internal Error} message. A typical implementation of content look like:

    tool::define ::test::content.file {
    	superclass ::httpd::content.file
    	# Return a file
    	# Note: this is using the content.file mixin which looks for the reply_file variable
    	# and will auto-compute the Content-Type
    	method content {} {
    	  my reset
        set doc_root [my http_info get doc_root]
        my variable reply_file
        set reply_file [file join $doc_root index.html]
    	}
    }
    tool::define ::test::content.time {
      # return the current system time
    	method content {} {
    		my variable reply_body
        my reply set Content-Type text/plain
    		set reply_body [clock seconds]
    	}
    }
    tool::define ::test::content.echo {
    	method content {} {
    		my variable reply_body
        my reply set Content-Type [my request get Content-Type]
    		set reply_body [my PostData [my request get Content-Length]]
    	}
    }
    tool::define ::test::content.form_handler {
    	method content {} {
    	  set form [my FormData]
    	  my reply set Content-Type {text/html; charset=UTF-8}
        my puts "<HTML><HEADER><TITLE>My Dynamic Page</TITLE></HEADER>"
        my puts "<BODY>"
        my puts "You Sent<p>"
        my puts "<TABLE>"
        foreach {f v} $form {
          my puts "<TR><TH>$f</TH><TD><verbatim>$v</verbatim></TD>"
        }
        my puts "</TABLE><p>"
        my puts "Send some info:<p>"
        my puts "<FORM action=/[my http_info get REQUEST_PATH] method POST>"
        my puts "<TABLE>"
        foreach field {name rank serial_number} {
          set line "<TR><TH>$field</TH><TD><input name=\"$field\" "
          if {[dict exists $form $field]} {
            append line " value=\"[dict get $form $field]\"""
          }
          append line " /></TD></TR>"
          my puts $line
        }
        my puts "</TABLE>"
        my puts "</BODY></HTML>"
    	}
    }

method EncodeStatus status
    Formulate a standard HTTP status header from he string provided.

method FormData
    For GET requests, converts the QUERY_DATA header into a key/value list. For POST requests, reads the Post data and converts that information to a key/value list for application/x-www-form-urlencoded posts. For multipart posts, it composites all of the MIME headers of the post to a singular key/value list, and provides MIME_* information as computed by the mime package, including the MIME_TOKEN, which can be fed back into the mime package to read out the contents.

method MimeParse mimetext
    Converts a block of mime encoded text to a key/value list. If an exception is encountered, the method will generate its own call to the error method, and immediately invoke the output method to produce an error code and close the connection.

method output
    Schedules a call to DoOutput when chan becomes writeable

method DoOutput
    Generates the the HTTP reply, and streams that reply back across chan.

method PostData length
    Stream length bytes from the chan socket, but only of the request is a POST or PUSH. Returns an empty string otherwise.

method puts string
    Appends the value of string to the end of reply_body, as well as a trailing newline character.

method reset
    Clear the contents of the reply_body variable, and reset all headers in the reply structure back to the defaults for this object.

method timeOutCheck
    Called from the http::server object which spawned this reply. Checks to see if too much time has elapsed while waiting for data or generating a reply, and issues a timeout error to the request if it has, as well as destroy the object and close the chan socket.

method timestamp
    Return the current system time in the format:

    %a, %d %b %Y %T %Z

method TransferComplete args
    Intended to be invoked from chan copy as a callback. This closes every channel fed to it on the command line, and then destroys the object.

        ###
        # Output the body
        ###
        chan configure $sock -translation binary -blocking 0 -buffering full -buffersize 4096
        chan configure $chan -translation binary -blocking 0 -buffering full -buffersize 4096
        if {$length} {
          ###
          # Send any POST/PUT/etc content
          ###
          chan copy $sock $chan -command [namespace code [list my TransferComplete $sock]]
        } else {
          catch {close $sock}
          chan flush $chan
          my destroy
        }

method Url_Decode string
    De-httpizes a string.

CLASS ::HTTPD::CONTENT
The httpd module includes several ready to use implementations of content mixins for common use cases. Options are passed in to the add_uri method of the server.

CLASS ::HTTPD::CONTENT.CGI
An implementation to relay requests to process which will accept post data streamed in vie stdin, and sent a reply streamed to stdout.

method cgi_info
    Mandatory method to be replaced by the end user. If needed, activates the process to proxy, and then returns a list of three values: exec - The arguments to send to exec to fire off the responding process, minus the stdin/stdout redirection.

CLASS ::HTTPD::CONTENT.FILE
An implementation to deliver files from the local file system.

option path
    The root directory on the local file system to be exposed via http.

option prefix
    The prefix of the URI portion to ignore when calculating relative file paths.

CLASS ::HTTPD::CONTENT.PROXY
An implementation to relay requests to another HTTP server, and relay the results back across the request channel.

method proxy_info
    Mandatory method to be replaced by the end user. If needed, activates the process to proxy, and then returns a list of three values: proxyhost - The hostname where the proxy is located proxyport - The port to connect to proxyscript - A pre-amble block of text to send prior to the mirrored request

CLASS ::HTTPD::CONTENT.SCGI
An implementation to relay requests to a server listening on a socket expecting SCGI encoded requests, and relay the results back across the request channel.

method scgi_info
    Mandatory method to be replaced by the end user. If needed, activates the process to proxy, and then returns a list of three values: scgihost - The hostname where the scgi listener is located scgiport - The port to connect to scgiscript - The contents of the SCRIPT_NAME header to be sent

CLASS ::HTTPD::CONTENT.WEBSOCKET
A placeholder for a future implementation to manage requests that can expect to be promoted to a Websocket. Currently it is an empty class.

SCGI SERVER FUNCTIONS
The HTTP module also provides an SCGI server implementation, as well as an HTTP implementation. To use the SCGI functions, create an object of the http::server.scgi class instead of the http::server class.

CLASS ::HTTPD::REPLY.SCGI
An modified http::reply implementation that understands how to deal with netstring encoded headers.

CLASS ::HTTPD::SERVER.SCGI
A modified http::server which is tailored to replying to request according to the SCGI standard instead of the HTTP standard.

AUTHORS
Sean Woods

BUGS, IDEAS, FEEDBACK
This document, and the package it describes, will undoubtedly contain bugs and other problems. Please report such in the category network of the Tcllib Trackers [http://core.tcl.tk/tcllib/reportlist]. Please also report any ideas for enhancements you may have for either package and/or documentation.

When proposing code changes, please provide unified diffs, i.e the output of diff -u.

Note further that attachments are strongly preferred over inlined patches. Attachments can be made by going to the Edit form of the ticket immediately after its creation, and then using the left-most button in the secondary navigation bar.

KEYWORDS
TclOO, WWW, http, httpd, httpserver, services

CATEGORY
Networking

COPYRIGHT
Copyright (c) 2018 Sean Woods <yoda@etoyoc.com>