Acquiring Tclhttpd

The tclhttpd package is housed at core.tcl.tk with the other Tcl related sources.

To acquire this package, you require the fossil source code system, which can be acquired from
https://www.fossil-scm.org/

The sequence for downloading the source is:

Clone the repository to your system
fossil clone http://core.tcl.tk/tclhttpd httpd.fos
Create a folder for the tclhttpd server.
mkdir tclhttpd
cd tclhttpd
Open the repository
fossil open ../httpd.fos

Contents of Tclhttpd

When you install the tclhttpd package you'll see several folders. The critical ones are:

bin The binaries to run tclhttpd - the main one you care about is httpd.tcl.
custom Tcl files that you add to customize a website. These will be sourced when the httpd.tcl is initialized. They are not re-sourced if they are modified while tclhttpd is running.
htdocs *.html and *.tml data files. The .tml files are templates used to generate .html files. If a .tml file is modified while tclhttpd is running the .html file is regenerated. The .tml files can be defined to always be regenerated with the Doc_Dynamic command.
lib Tcl support files for the tclhttpd. These are static and you probably don't want to modify them.

You can make a fully self-contained version of tclhttpd by adding a new folder lib/tcllib to contain the required packages from tcllib. The contents of lib/tcllib would resemble:


-rw-r--r-- 1 clif clif  29035 May 23 14:00 uri.tcl
-rw-r--r-- 1 clif clif  47535 May 23 14:00 smtp.tcl
-rw-r--r-- 1 clif clif  30116 May 23 14:00 ncgi.tcl
-rw-r--r-- 1 clif clif 113508 May 23 14:00 mime.tcl
-rw-r--r-- 1 clif clif  16820 May 23 14:00 md5.tcl
-rw-r--r-- 1 clif clif  39493 May 23 14:00 html.tcl
-rw-r--r-- 1 clif clif  62126 May 23 14:00 fileutil.tcl
-rw-r--r-- 1 clif clif  35207 May 23 14:00 counter.tcl
-rw-r--r-- 1 clif clif  29728 May 23 14:00 cmdline.tcl
-rw-r--r-- 1 clif clif  11562 May 23 14:00 base64.tcl
-rw-rw-r-- 1 clif clif   1173 May 23 14:00 pkgIndex.tcl

doc Documentation for the tclhttpd package.

Initial test

The fastest way to confirm that your system will run is to start the bin/httpd.tcl server and try to connect.


cd bin
tclsh httpd.tcl

If you have everything you need installed on your system, you'll see something like this:


can't find package limit
Running with default file descriptor limit
/debug user "debug" password "ow+2k7uu07az"
User "webmaster" default password "1O^tV'i"
httpd started on port 28015
SSL startup failed: No CA directory "/tmp/tclhttpd/tclhttpd/certs" nor a CA file ""
Could not write to pid file /var/run/tclhttpd/tclhttpd.pid: couldn't open "/var/run/tclhttpd/tclhttpd.pid": no such file or directory

For a simple site or testing, you can ignore these error messages.

If the folder /var/run/tclhttpd exists, the PID of the running tclhttpd daemon will be stored in the file tclhttpd.pid. This can be used to monitor and control an httpd daemon (perhaps to restart, or monitor that the daemon hasn't crashed.)

If any critical libraries are missing, you'll see something like this:


Error in configuration file "/clif/TCL_STUFF/tclhttpd/temp/bin/tclhttpd.rc"
:Error: can't find package fileutil
    while executing
"config::init $Config(config) Config"
    (file "httpd.tcl" line 159)

If you see missing libraries from tcllib, you can download them from the fossil repository.


fossil clone http://core.tcl.tk/tcllib tcllib.fos 
mkdir tcllib 
cd tcllib 
fossil open ../tcllib.fos 
configure ; make install 

Once the httpd server is running, you can view a page by starting a browser and setting the URL to

127.0.0.1:8015

Creating a site

A website consists of a set of related documents. The documents can be staticly created, generated from templates, fully generated as needed or a combination of methods.

Here are the techniques tclhttpd supports for generating documents, in order of increasing complexity and versatility:

Static web pages

You can create .html files in the htdocs folder with a plain text editor, a text processor that supports exporting to html (OpenOffice, LibreOffice, MS Office, etc), or other html editor.

A new page becomes available as soon as the file is placed in the htdocs tree.

Html Templates

Template files are mixtures of html and embedded Tcl scripts. These files can co-exist with static html files in the htdocs file-system. These are similar to the Active Server Pages (ASP) files.

A template file ends in a .tml suffix. It will be processed by the tclhttpd server to create a static .html file which will then be served to a browser.

By default whenever foo.tml is modified, a new foo.html will be generated. The generated foo.html file will be unchanged until the related foo.tml is modified again.

A file that displays varying data (a time/date stamp, a form being filled out, contents of a shopping cart, etc) can be forced to regenerate whenever it's accessed, by including the Doc_Dynamic command in the page body.

    Syntax: Doc_Dynamic

    Cause a page to be regenerated whenever accessed.


<html>
<head></head>
<body>
[Doc_Dynamic]
[html::h2 "Current Time is: [clock format [clock seconds]]]"
...
</body>
</html>

Within the .tml file you can:

Any type of file can be placed in a folder within the htdocs hierarchy. These may be image files, html text, Tcl scripts, raw data, cgi procedures and more.

These are some of the common file types found in the htdocs hierarchy.

.tml
  • One file named .tml per folder
  • May contain support procedures for pages within this folder.
  • May contain private data for pages within this folder.
  • Automatically reloaded if modified.
Note that by default anything defined within the .tml file is placed in global scope. This can cause unexpected behavior when pages are copied from one folder to another an then modified.

Good practice is to put per-site and per-folder information and procedures within a namespace.

*.tml
  • Multiple name.tml files may be placed in a folder, provided that each has a unique name which does not match a file with a .html extension.
  • Each name.tml will generate a corresponding name.html file when accessed.
  • The corresponding .html file will only be generated once unless the [Doc_Direct] command is included in the body of the .tml file
  • May contain static text, html tags and Tcl scripts.
  • Tcl commands are enclosed within square braces
  • Tcl variable are referenced with a $, as in Tcl
  • Tcl scripts should return text to be displayed in the web page.
*.tcl A folder may contain .tcl files which can be loaded from the .tml files via [source name.tcl].

These can contain data definitions or procedures related to the pages within this folder.

By default, a file sourced within a .tml file will be loaded into the global scope. Good practice is to place procedures data within a namespace.

These files will only be loaded when a .tml file is modified, not necessarily when a .tcl file is modified.

Application Direct URL

An Application Direct URL maps a URL to a Tcl procedure. If the page is accessed with query data, the values are presented as key/value pairs in the procedure argument.

An application direct handler can be defined in a .tcl file placed within the custom folder. It will be loaded when the httpd daemon is started.

    Syntax: Direct_Url pageName procName

    Map a URL to a hierarchy of Tcl procedures.

    pageName The name of the URL, starting with a "/"
    procName The name of a procedure to invoke when the URL is requested.

A hierarchy of pages can be implemented by naming procs for the subordinate pages with the slash separator, as the URL would be defined.

The example below demonstrates using an application direct URL to create pages for /direct and /direct/sub.

The command

Direct_Url /direct direct::handler

maps a url like http://127.0.0.1:8015/direct to the direct::handler procedure.

Note that the return value from the procedure is exactly what the server will send to a browser. In this case, the page generated would be

<h1>direct handler args: $args</h1>

Note that this is not a valid HTML page. It lacks the <html>, <head> and <body> tags.

The second procedure, direct::handler/sub will process URLs like http://127.0.0.1:8015/direct/sub. It uses the html package to generate a complete HTML page.

If a page resembling http://127.0.0.1:8015/direct/sub?a=b&c=d is requested, the return would be:


<html><head>
	<title>Direct Demo</title>
	<!-- Generated by Direct_Url -->
</head>
<body bgcolor="white" text="black">
<h1>direct sub handler args: a b c d</h1>
</body></html>


$> cat custom/direct.tcl
Direct_Url /direct direct::handler
set rev [package require html]

namespace eval direct {
  proc handler {args} {
    return "<h1>direct handler args: $args</h1>"
  }
 
  proc handler/sub {args} {
    html::author "Generated by Direct_Url"
    set page [html::head "Direct Demo"]
    append page [html::bodyTag ]
    append page [html::h1 "direct sub handler args: $args"]
    append page [html::closeTag]
    append page [html::closeTag]
    return $page
  }

}

Domain Handler

The Domain Handler is similar to the Application Direct URL in that it maps a set of URLs to Tcl procedures.

Domain Handlers are a lower-level entry point into the httpd server. As such, you get greater control of how URLs are processed, but will need to do more of your own processing.

Unlike the Application Direct URL, it does not parse query options or map subordinate pages directly to procs. Your procedure must handle these details.

A Domain Handler is defined with the URL_PrefixInstall command.

    Syntax:URL_PrefixInstall pageName script

    Map a URL hierarchy to a Tcl procedures.

    pageName The name of the URL, starting with a "/"
    script The script to invoke when the URL (or subordinate url) is requested.

The procedure that is invoked as a domain handler will get two new arguments added to whatever is defined as the procedure to be invoked by tclhttpd.

socket A socket identifier for this connection. This can be used to map the global State array into the local scope.
suffix A subordinate path string for this page.

When the tclhttpd daemon processes a page request it creates a global state array to hold the information about the page request.

This state array is named Httpdsocket where socket is the string passed to the procedure invoked to handle the domain.

The state array can be mapped into the procedure with the upvar command as shown in the next example.

Note that the Test procedure uses the Httpd_ReturnData call to return the string. There are several Httpd_ procedures available to the website developer. They are described in the next section.


Url_PrefixInstall /test Test

proc Test {sock suffix} { 
  upvar #0 Httpd$sock state

  set rtn "<table>"
  foreach {k v} [array get data] {
    append rtn "<tr><td>$k</td><td>$v</td></tr>"
  }
  append rtn "</table>"

  Httpd_ReturnData $sock text/html $rtn
}

Requesting the url http://192.168.1.2:28015/test/foo?a=b&c=e will assign a value like sock8aecd860 to the sock parameter and /foo to the suffix parameter. The query string is placed in the State array's query index.

The following values will be in the Httpdsock8aecd860 array:

uri /test/foo?a=b&c=e
pathlist {} foo
count 0
key connection
state mime
self http clif.cflynt.com 28015
url /test/foo
mime,accept text/html, image/jpeg, image/png, text/*, image/*, */*
ipaddr 192.168.1.2
proto GET
suffix /foo
mime,accept-language en
cancel after#10
mime,accept-encoding x-gzip, x-deflate, gzip, deflate
line GET /test/foo?a=b&c=e HTTP/1.1
mime,host 192.168.1.2:28015
mime,accept-charset utf-8, utf-8;q=0.5, *;q=0.5
left 25
mime,connection Keep-Alive
mimeorder user-agent accept accept-encoding accept-charset accept-language host connection
headerlist User-Agent {Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.4 (like Gecko)} Accept {text/html, image/jpeg, image/png, text/*, image/*, */*} Accept-Encoding {x-gzip, x-deflate, gzip, deflate} Accept-Charset {utf-8, utf-8;q=0.5, *;q=0.5} Accept-Language en Host 192.168.1.2:28015 Connection Keep-Alive
prefix /test
version 1.1
query a=b&c=e
mime,user-agent Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.4 (like Gecko)

CGI support

The tclhttpd daemon can use classic cgi as well as the Application Direct and Domain Handlers.

There are two methods to invoke a CGI application using tclhttpd.

  1. Place a file with a .cgi extension in the htdocs folder.
  2. Place a file with any extension in the htdocs/cgi-bin folder

The default location for CGI scripts is htdocs/cgi-bin This can be changed by changing the Cgi_Directory command in bin/httpdthread.tcl.

The command

Cgi_Directory /cgi-bin /PATH

will make /PATH the new CGI folder. Note that /PATH is an absolute filesystem path, not a path relative to the htdocs folder.

The next example causes the url http://127.0.0.1:8015/cgi-bin/test.tcl to be mapped to the CGI file /tmp/CGI/test.tcl

Cgi_Directory /cgi-bin /tmp/CGI

Generating CGI pages

A file in the CGI folder, or with a .cgi extension, must have proper ownership and execution mode to be a CGI file.

If the execution bit is not set you'll see an error resembling this in your browser when you try to access the file.


Got the error Bad Request
 while trying to obtain /CGI/foo.cgi. 
couldn't execute "/tmp/tclhttpd/tclhttpd/htdocs/CGI/foo.cgi": permission denied

The first line in the CGI output must be a Content-Type definition for the data, followed by a blank line. If this is not the first line, you'll see an error resembling this in your browser when you try to access the file.


An error occurred while loading http://127.0.0.1:28015/CGI/foo.cgi?a=b:
Connection to host 127.0.0.1 is broken.

The Content-Type definition can be generated manually with a line like:


puts "Content-Type: text/html\n"

Or by using the ncgi package that's part of the standard tclhttpd distribution (lib/cgi.tcl):


package require ncgi
puts [ncgi::header]

Useful data structures and procedures

There are a large number of support procedures and libraries in the lib folder.

Some of these have man pages (ncgi, html), but the definitive source for documentation on these is the comments in the source code files.

CGI, Template and Application Direct pages can be generated with high-level data structures and procedures. The Domain Handler procedures are called at a lower level and have separate, low-level data structures and support procedures.

CGI, Templates and Application Direct

Information about the current page request is held in the ::env array. The fields in this array are:

AUTH_TYPEAuthentication protocol (e.g., Basic).
CONTENT_LENGTHThe size of the query data.
CONTENT_TYPEThe type of the query data.
DOCUMENT_ROOTFile system pathname of the document root.
GATEWAY_INTERFACEProtocol version, which is CGI/1.1.
HTTP_ACCEPTThe Accept headers from the request.
HTTP_AUTHORIZATIONThe Authorization challenge from the request.
HTTP_COOKIEThe cookie from the request.
HTTP_FROMThe From: header of the request.
HTTP_REFERERThe Referer indicates the previous page.
HTTP_USER_AGENTAn ID string for the Web browser.
PATH_INFOExtra path information after the template file.
PATH_TRANSLATEDThe extra path information appended to the document root.
QUERY_STRINGThe form query data.
REMOTE_ADDRThe client’s IP address.
REMOTE_USERThe remote user name specified by Basic authentication.
REQUEST_METHODGET, POST, or HEAD.
REQUEST_URIThe complete URL that was requested.
SCRIPT_NAMEThe name of the current file relative to the document root.
SERVER_NAMEThe server name, e.g., www.beedub.com.
SERVER_PORTThe server’s port, e.g., 80.
SERVER_PROTOCOLThe protocol (e.g., http or https).
SERVER_SOFTWAREA software version string for the server.

The simplest way to generate pages with CGI, Templates or Application Direct procedures is to embed simple text within the page definition.

Template and Application Direct procedures should return the page to be displayed, while CGI scripts should use puts to display the page.

This is a simple HTML page that generates a single line.

As a htdocs/test.tml it resembles:


<head><title>Test</title></head 
<body>
<h1>Large Header</h1>
</body></html

The equivalent page in an Application Direct procedure would be:


Direct_Url /direct direct::handler

namespace eval direct {
  proc handler {args} {
    set rtn <html>
    append rtn "<head><title>Test</title></head"
    append rtn "<body>"
    append rtn "<h1>Large Header</h1>"
    append rtn "</body></html"
    return $rtn
  }
}

Finally, the same page can be generated as a CGI script with this code:


#!/bin/sh
#\
exec /opt/ActiveTcl-8.6/bin/tclsh "$0" "$@"

puts "Content-Type: text/html\n"

puts <html>
puts "<head><title>Test</title></head"
puts "<body>"
puts "<h1>Large Header</h1>"
puts "</body></html"

ncgi and html packages

The ncgi package is included with the standard tclhttpd distribution in /lib/cgi.tcl. The html package is part of tcllib.

Both of these packages include a man page that describes the supported functions.

Some functions in the packages have similar names, but perform different functions. For example, ncgi::header generates a CGI header, while http::head generates an HTML page header.

This is a CGI file that displays the contents of the env array.


#!/bin/sh
#\
exec /opt/ActiveTcl-8.6/bin/tclsh "$0" "$@"

package require ncgi
package require html

ncgi::header

html::author "Tcl Productions"
html::description "Generated by Html library"
html::keywords "demo, tclhttpd, sample, cgi"

set page [html::head "CGI Demo"]

append page [html::bodyTag ]

append page [html::h 2 "Contents of env array" {align="center"}]
append page [html::tableFromArray ::env]
append page [html::closeTag]
append page [html::closeTag]

puts $page
exit 0

The equivalent Direct Url procedure is


  proc handler {args} {
    html::author "Tcl Productions"
    html::description "Generated by Html library"
    html::keywords "demo, tclhttpd, sample, cgi"
    
    set page [html::head "Direct Demo"]
    append page [html::h1 "Direct URL handler args: $args"]
    append page [html::bodyTag ]
    append page [html::tableFromArray ::env]
    append page [html::closeTag]
    append page [html::closeTag]
    return $page
  }

Finally, the equivalent Template file is


    [html::author "Tcl Productions"]
    [html::description "Generated by Html library"]
    [html::keywords "demo, tclhttpd, sample, cgi"] 

    [html::head "Template Demo"]

    [html::h1 "Template Page"]

    [html::bodyTag ]
    [html::tableFromArray ::env]
    [html::closeTag]
    [html::closeTag]

Some helper procedures for generating HTTP compliant strings include:

Syntax:Httpd_Date seconds

Generate a date string in HTTP format

seconds Time in seconds.

Syntax:Cookie_Make args

Returns a cookie definition string suitable for use with Httpd_SetCookie

args A set of key/value pairs for the data in the cookie. Expected values are:
  1. -name - Name of the cookie
  2. -value - Value for cookie
  3. -patn - Path restriction
  4. -domain - Domain restriction
  5. -expires - Expiration time

Generating pages with Domain Handlers

There are no man pages for the Domain Handler procedures. The only source of documentation is the comments in lib/httpd.tcl

This section of the document is far from complete but will give you an introduction to the commonly used features.

The Httpdsock array

Instead of the env state array, each Domain Handler connection is assigned a unique state array that can be mapped into the Domain Handler proc with the upvar command.

This array has many fields documented in lib/httpd.tcl. These are the fields commonly used by a web programmer:

self A list of protocol (http or https), name, and port that capture the server-side of the socket address Available with Httpd_Protocol, Httpd_Name, and Httpd_Port API.
uri The complete URL, including proto, servername, and query
proto http or https
url The URL after the server name and before the ?
query The URL after the ?
ipaddr The remote client's IP address
cert Client certificate (The result of tls::status)
host The host specified in the URL, if any (proxy case)
port The port specified in the URL, if any
mime,* HTTP header request lines (e.g., mime,content-type)
count Content-Length
set-cookie List of Set-Cookie headers to stick into the response Use Httpd_SetCookie to append to this.
headers List of http headers to stick into the response Use Httpd_AddHeaders to append to this.
prefix (Set by Url_Dispatch to be the URL domain prefix)
suffix (Set by Url_Dispatch to be the URL domain suffix)
auth_type (Set by the auth.tcl module to "Basic", etc.)
remote_user (Set by the auth.tcl to username from Basic authentication)
session (Set by the auth.tcl to "realm,$username" from Basic auth) You can overwrite this session ID with something more useful

Returning page data

There are several procedures that can be used within a Domain handler to generate and return page data and status.

These are procedures for returning data to a browser.

Syntax: Httpd_ReturnData sock type content ?code?

Return the content to the client browser. Closes this request/reply pair.

sock The socket handle to identify this session.
type The MIME type of data being returned, such as text/html
content The data being returned. This data should be properly formatted for the type. No further processing is done to this data.
code A status code for this return. The default is 200, for a successful transfer.

Syntax:Httpd_ReturnFile sock type path ?offset?

Return a file to the client browser.

sock The socket handle to identify this session.
type The MIME type for the return, text/html, image/gif, etc.
path Path to the file name, relative to docRoot.
offset Bytes from start of file to be skipped. Default is 0.

Syntax:Httpd_SetCookie sock cookie modify

Define a cookie to be used in a reply. This must be called before Httpd_ReturnFile or Httpd_ReturnData

sock The socket handle to identify this session.
cookie A cookie string to return. Can be genrated with the Cookie_Make command.
modify If true, overwrite preexisting cookie that matches. This allows modifying an expiration time. Default is 0.

Syntax:Httpd_Error sock code ?detail?

send the error message, log it, and close the socket.
Note that the Doc module tries to present a more palatable
error display page, but falls back to this if necessary.

sock The socket handle to identify this session.
code Error code to be returned, probably 4xx or 5xx
detail Optional details to include with error return

Suspending and restarting page processing can be done with these commands.

Syntax: Httpd_Suspend sock ?timeout?

Suspend Wire Callback - for async transactions
Use Httpd_Resume once you are back in business
Note: global page array is not preserved over suspend
Disables fileevents and sets up a timer.

sock The socket handle to identify this session.
timeout Timeout period. After this the request is aborted.

Syntax:Httpd_Resume sock ?timeout

Resume processing of a request. Sets up a bit of global state that has been cleared by Httpd_Suspend.

sock The socket handle to identify this session.
timeout Timeout period. After this the request is aborted.

You can require a login with this procedure.

Syntax:Httpd_RequestAuth sock type realm args

Generate the (401) Authorization required reply

sock The socket handle to identify this session.
type The type of authorization. Commonly "Basic". No default.
realm Browsers use this to cache credentials
args Extra key/value pairs needed to generate the request.

Syntax:Httpd_Name sock

Returns the name of the server. This can be used to handle multiple hosts on a single server if the normal virtual host support is insufficient.

sock The socket handle to identify this session.

Page redirection can be done to a remote machine or within the local server.

Syntax:Httpd_Redirect newurl sock

Generate a redirect (302) to another URL possibly on another server.

newurl A new location to redirect to.
sock The socket handle to identify this session.

Syntax:Httpd_RedirectSelf newurl sock

Generate a redirect to another URL on this server.

newurl A new location to redirect to.
sock The socket handle to identify this session.

Configuration: bin/tclhttpd.rc

The default configuration file is bin/tclhttpd.rc. If you need to change where webpages are saved, where custom code is stored, where to search for libraries, this is the file to modify.

The configuration options are documented internally in this file.

All of the configuration options can be set from the command line as well as within this file. The command line options are useful when you are testing a configuration, but you will probably want a customized tclhttpd.rc file if you have modifications you want to remember and document for a production server.

Some of the configuration options you may want to use include:

docRoot Defines the top of the web page hierarchy, where the .tml and .html files will be stored.
Default: ../htdocs
custom The location for custom code to be sourced into the httpd server. This is where Application Direct and Domain handlers are defined.
Default: ../custom
host The fully qualified hostname the server is running on.
Default: the value returned by [info host]
pidfile The pid of the tclhttpd daemon.
Default: /var/run/tclhttpd/tclhttpd.pid
uid, gid User and Group ID that tclhttpd will run as.
Default: 50
port, https_port The port number that tclhttpd will respond on.
Default: 8015, 8016

In a production server, these should be set to port 80 and 443.

threads The max number of threads to use. There is no threading support if this is set to 0
Default: 0

Virtual Hosts

A single server can host multiple domains. For example, both alpha.com and beta.com could be hosted on a server at IP address 123.45.67.890.

The simplest way to accomplish this is to

  1. copy bin/tclhttpd.rc to a new file (perhaps named for the site).
  2. edit the new file and change ../htdocs to a different path (perhaps ../siteName_htdocs).
  3. start the httpd daemon with command line arguments to point to the appropriate configuration files:
    tclsh httpd.tcl -virtual 'alpha.com alpha.rc beta.com beta.rc'

An alternate method that will not require extra command line arguments when starting tclhttpd is to modify the tclhttpd.rc by adding a line resembling:

Config virtual {alpha.com alpha.rc beta.com beta.rc}