HTML (sample) Text (sample)

Forms to E-Mail Using TCL

by J.M. Ivler

We're talking CGI programming, forms design, TCL and security

This article discusses the process of using a CGI (Common Gateway Interface) program in Tcl to send forms data to someone via e-mail. To start with, let's clarify some things. Tcl was chosen because it is easy for someone else to extend, and this program was written to be placed into the public domain. This doesn't mean that the program couldn't have been written in C (this has been done by Thomas Boutell), or in Perl. Tcl was chosen for a number of reasons; the accompanying article explains some of them.

The Tcl program does rely on a C program called un-cgi, written and maintained by Steven Grimm. This program takes the "put" request and unbundles the input. It then creates Unix environment pairs between the value from the form (prefixed with WWW_ ) and the information associated to the value. Uncgi can be obtained from two separate FTP locations on the Internet, listed below.

Why use uncgi? Couldn't this all have been done in Tcl? Yes, uncgi has numerous powerful features that could have been duplicated in the Tcl programs, but why duplicate effort when someone has already done it? It isn't like uncgi was going to be expanded or enhanced in the duplication (as the email.tcl program was from Thomas Boutell's original in C), so it is used as is.

This article is set up in sections. The first provides the problem scope of the requirements for both the V1.0 and the V2.0 code. The next section discusses some of the issues of form design, and how to utilize the utility. A section on installation of the software follows. The next section talks about the actual Tcl and break the code down at a level that will permit even the most novice Tcl programmer to modify it. Finally, the article will close with an example session with the sample form, and the e-mail reply.

What It Does

This process will take the variable fields from a form and use uncgi to create environment variables from them. These will then be processed by the email.tcl procedure. The email.tcl procedure will locate all the appropriate environment variables, extract them from the environment, and then e-mail them to you.

This process requires the following software:

• Tcl 7.3 (or 7.4) from ftp.aud.alcatel.com
• uncgi 1.4 (also available at www.midwinter.com (since this article was written, uncgi has been updated to V1.5 or higher -- the noted changes between V1.4 and V1.5 should not have any effect on email.tcl. V1.4 is still available from the net-quest.com URL).
• email.tcl 2.0. In order to use this software, a form must be created that will be filled out and sent via the system's e-mail (a sample form is shown in the next section). When a user submits this form, the uncgi processor is started and all the values of the form are made into environment variables prefixed by WWW_ . The email.tcl process is then started and the variables are placed into a file to be sent to the e-mail recipient.

Security

Although that sounds simple, there are other concerns that must be addressed when doing this. These concerns include "Do you allow e-mail internally only, or will you allow e-mail out to the Internet?"

Why would Internet e-mail be a concern? Because of the way the Web works. The e-mail sent would be done so from your http "user" name as established in the httpd.conf configuration file. As most sites use the value of "nobody" as the account name, this provides a very nice way to allow anonymous e-mail from within a Web-based environment.

If the form allowed the user to specify the e-mail address of the end user (in the examples, this appears as the hidden field "mailto" in the form), and had a text area entry box (like the one used in the example below for comments), anyone would be able to compose a message to any Internet e-mail address and send it out as user nobody@your.site. Consider someone using that form to e-mail the president@whitehouse.gov. When the Secret Service comes knocking on your door, and takes away your computer equipment, then you will really understand why you might want to disallow this feature.

It cannot be stressed enough that you should consider what you are doing, and the consequences of that action, by setting this up to permit anonymous e-mail on the Internet. In order to ensure that you have the capability to control this feature, options have been provided that permit the Webmaster control over Internet e-mail access. Additionally, a number of options have been preset within the code (you can choose to turn them on if you wish), and other options are provided in comments to the code, with the Tcl code to insert also provided.

Certain additional variables are provided that can allow the e-mail to be "fine-tuned" to meet the needs of the recipient. The most powerful of these determines if the "form value" will be sent along with the response. This is discussed in detail in the next section.

This Tcl code was originally written to replace Thomas Boutell's C program. The C program (version 1) required that there be too much involvement by the Webmaster to ensure that all the files were in the right place, and that there were subdirectories and data files to support each instance of the use of the program. While that provided a great deal of security, it also created a nightmare for the Webmaster if there were a number of different users with different form needs. There has since been a new version of the code produced (available at the Web site listed earlier) that eliminates some of those concerns. The premise of this software was to develop a program that moved control back to the forms designer and away from the Webmaster (with some noted exceptions).

Designing The Form

This article won't discuss the art of forms design. It will provide a style guide for a clean form design, and will then provide samples that follow that style guide throughout the text. We'll help you understand how the e-mail program will process variables used in the forms, and special variables that you can use to have a finer granularity of control in the design of the form response.

The form being used in this section is simple. It asks for a name and then a radio button asks if your navigation at the site was easy or not, using a scale from 1 to 7. Most forms aren't that simple. In fact, most form design is complex, and poorly done. Before developing a form, try to think about the information that you want to obtain, and how you would feel answering it. If you have more than 5 to 10 questions, and these are not check box or radio button questions, then you have lost your audience. While I praise the Wall Street Journal for their active involvement on the Web and their electronic publication, the form that they use in registration is a bit winded. Not in the questions, but in the excessive text.

In developing a form, the following style is considered very clean.

intro text
<hr>
5-10 questions
<hr>
comment block (text area)
<hr>
Name/address and submit button
<hr>
copyright/authorship

Here is the sample form that we will be using to describe the form and its elements [italic text embedded in the code is commentary about the forms properties and elements, and is not part of the form itself]:

<TITLE>Article Sample Form</TITLE>
<H1>Sample Form from the Article</H1>

Minimalist. Most browsers will accept the title as the minimum requirement for an HTML document.

     Thanks for taking the time to respond to this simple questionnaire.
This information will be used to make the system better meet your needs.
<p>
     Please respond to the questions in the next section. Once completed,
enter your name and then press the submit button. Thanks for helping out.

That ends the introduction to the form.

<FORM METHOD="POST" ACTION="/old?u=http%3A%2F%2Fwww.site.com%2Fcgi-bin%2Funcgi%2Femail.tcl&y=1999">
<input type=hidden name=mailto value=ivler>
<input type=hidden name=msub value="Article Response">
<HR>

Here we establish uncgi/email.tcl as the form, and the subject and recipient of the completed form. This is going to be turned into an email to user "ivler" with the subject "Article Response".

Please answer the questions by checking the value under it using this scale. 
<p>
<img src="/old?u=http%3A%2F%2Fwww.webdeveloper.com%2Fgiflib%2Fyellowball1.gif&y=1999">1=strongly disagree <br>
<img src="/old?u=http%3A%2F%2Fwww.webdeveloper.com%2Fgiflib%2Fyellowball1.gif&y=1999">4=neither agree or disagree<br>
<img src="/old?u=http%3A%2F%2Fwww.webdeveloper.com%2Fgiflib%2Fyellowball1.gif&y=1999">7= strongly agree <br>
<img src="/old?u=http%3A%2F%2Fwww.webdeveloper.com%2Fgiflib%2Fgrad_line.gif&y=1999">

Directions for responding to the question(s). giflib, in this example, is the library where standard .GIF images are stored. In this case we are using the yellowball images as bullets, and have a gradated colored line as the break between the instructions and the questions that follow.

<h2>Site Information</h2>
<ol>
<li> Navigation at this site was easy. 
<p>
1 <input name=m1 Type=radio value="1">
2 <input name=m1 Type=radio value="2">
3 <input name=m1 Type=radio value="3">
4 <input name=m1 Type=radio value="4">
5 <input name=m1 Type=radio value="5">
6 <input name=m1 Type=radio value="6">
7 <input name=m1 Type=radio value="7">
<p>
</ol>

A single question. For ease of reading, each number was separated and placed on its own line. When this is displayed, the numbers and selection button will be running from left to right under the question. To run them down the page, end each line with a <br>.

<hr>
<h2> Suggestions and Comments</h2>
<i>Go ahead, make our day!</i>
<TEXTAREA NAME="content" COLS=55 rows=10> 
</TEXTAREA>
<p>

The comments block. While one may be driven to create an area 24 x 80, please don't. 10 rows at a time are more than enough, and 55 columns provides ample room to keep ideas together.

<hr>
<p>
<input name="name"> Name<p>
To submit the answers, press this button: 
<INPUT TYPE="submit" VALUE="Submit Feedback">.<br> 
Use the back key to exit, and the reload option to clear and restart.
</FORM>
<p>
Did you remember to submit?

Name entry and submission button. I tend to try to make sure that the forms user is aware that they can use the back button on their browser to exit to the screen that they just came from. Others choose to place a "home" link or icon on the page. I also prefer to use the reload feature of the user's browser rather than a button to clear the entry. This ensures that the user won't clear the form in error.

<HR> <P>
<address>J. M. Ivler - email: ivler@i-xpress.com<br>
© Copyright 1995, Infobahn Xpress, Los Alamitos, CA.
All Rights Reserved.</address>

Author and copyright. Note that the form is very minimal. The fewer items you provide for people accessing the form to press, the greater possibility that they will press what you want. Place items of higher interest and specificity near the top, open areas near the bottom, and email and name info down at the submit button.

In some cases you may want to include legal information (again, I suggest you look at the Wall Street Journal Money and Investing Update -- Registration Page for an example). This information could have been included on a hyperlinked page (<a href="/old?u=http%3A%2F%2Fwww.webdeveloper.com%2Fcgi-perl%2Fterms-cond&y=1999">Terms and Conditions - Please Read!</a>) with the submit button saying:

<INPUT TYPE="submit"
VALUE="I have read and agree to the
terms and conditions">

That would have shortened the page, and made it a bit more comfortable for a reader.

Once the form has been developed consider the following questions. Do you want to put a special subject line on the mail message to allow an e-mail processor like procmail to pre-process the email? If so, you can use the msub value.

Have It Your Way

How do you want the information returned to you? This has a great deal to do with how the information returned will be processed.

From a form, using this process, e-mail can look two ways. The first is that the message will contain value pairs matching the data with the variable that the data was in. For instance, consider the following HTML from the sample.

<ol>
<li> Navigation at this site was easy. 
<p>
1 <input name=m1 Type=radio value="1">
2 <input name=m1 Type=radio value="2">
3 <input name=m1 Type=radio value="3">
4 <input name=m1 Type=radio value="4">
5 <input name=m1 Type=radio value="5">
6 <input name=m1 Type=radio value="6">
7 <input name=m1 Type=radio value="7">
<p>

In answering this question, a radio button would be set which would set the variable m1 with the value 1 through 7, as selected by the user. In this case the answer, by itself, would be of little use in a mail message. By using value pairs, the e-mail message would contain (if the user selected the radio button 3):

m1 = 3

This would be useful if the data was to be processed into a database or table. For human-readable data, you would want the following HTML:

<ol>
<li> Navigation at this site was easy.
<p>
1 <input name=m1 Type=radio value="Navigation was hard">
2 <input name=m1 Type=radio value="Navigation was difficult">
3 <input name=m1 Type=radio value="Navigation was uncomfortable">
4 <input name=m1 Type=radio value="Navigation was okay">
5 <input name=m1 Type=radio value="Navigation was fine">
6 <input name=m1 Type=radio value="Navigation was good">
7 <input name=m1 Type=radio value="Navigation was great">
<p>

Now when the user answered the form with a 3, the form data could be sent without the value pairs on. In this case the message sent via e-mail would look like:

Navigation was uncomfortable

This would be much harder to process into a database or table, but is much more informative for a person to read.

In order to support both possible methods of response, the code allows the person building the form to use a hidden variable, and keyword, sendvar. This variable is set with either a true or false value. The default is true. To set the default false, you would include the following HTML code:

<input type=hidden name=sendvar value="false">

Remember, that when sendvar is off (set false), all value pairs will not be printed. That means that the name, address, and other sections won't have "titles". Instead of:

name = J.M. Ivler

the email will just have the phrase

J.M. Ivler

which you had better recognize as the "name" value. Consider the following:

<input name="mname"> Your Name<p>
<input name="fname"> Your Friend's Name<p>

J.M. Ivler
Steven Schwartz

or

mname = J.M. Ivler
fname = Steven Schwartz

In the first case we have sendvar set false, in the second we have it set true. As you can see, in a case like this, it might be better to leave the labels on, so it is clear who the friend is, versus who the respondent is.

Uncgi processes each form variable into a environment variable with the format WWW_<name>. The email.tcl processor loads all variables with a WWW_ prefix to the mail message. That includes WWW_mailto, WWW_msub and WWW_sendvar (the three keyword variables that email.tcl uses). The code can easily be modified to have these values removed from the mail; the regexp to do so has been included in this article. Having these displayed provides a useful debugging tool when trying to locate a problem.

Based on how the email.tcl has been installed, you can use this software as an anonymous remailer. If the software has been installed to allow for Internet addressing, then the users could be asked for the address that they wish to send the email to, and the software will send the email to any valid address. While this is not the purpose of the software, it could be used as such. The prior section clearly explained why you would want to ensure that the software is not being used in that way, but it can not be stressed enough that maintaining an anonymous mailer opens a Pandora's box of possible problems, and should be strongly avoided.

Installing the CGI (And Support Tools)

In order to use Tcl as a command language, the Tcl code must be compiled and installed on the system. While Tk is also available, and is highly recommended just for its own usefulness, it is not required in order to use Tcl.

Uncgi must also be installed on your system. As noted, there have been changes in the uncgi software, and the most current release is V1.5. Although the changes haven't been tested with email.tcl, the release notes indicate that the changes will not have an effect on email.tcl V2.0. If you wish to use V1.4 of uncgi, it will be kept available at the ftp.net-quest.com location. Unzip and untar the file, compile using a standard C compiler, and place the executable into the area you have defined as your cgi-bin on your Web server. Further details on the install can be found in the README file included in the distribution of uncgi. To install the email.tcl software at your site, simply FTP the software to your site and place it in the cgi-bin area of your webserver.

If you wish to pre-process the e-mail messages, you may want to obtain a copy of a software program called procmail. This public domain program will act as a filtering processor for your inbound e-mail. It is possible to set the software up in either a sitewide manner, or as a single user. Procmail can be obtained by FTP from gatekeeper.dec.com in /.b/usenet/comp.sources.misc/volume43 (the software is downloadable as 10 parts in compressed (.Z) format). It is highly recommended that you read all the documentation associated with procmail before installing it. Procmail is a very powerful filtering agent for e-mail, and as such should be fully understood before using it. Improper usage could cause e-mail to fail to be delivered as expected.

Code Review

What follows here is a copy of the code with internal editorial comments. Header information and standard comments that are embedded in the code have been stripped from here. In place of those comments are expanded details into how the code functions and what the code is doing. This will permit you to change the code with a full understanding of its functionality. The original version of the code can be found at ftp://ftp.net-quest.com/users/ivler/tcl/email.tcl.

Comments about the code are again printed in italics. In some cases, a sample piece of replacement code has been provided. This code appears as part of the comments section, but can be inserted into the code as described.

#!/usr/local/bin/tclsh

Sometimes referred to the bang-line, this is the line that stipulates the code is running as a Tcl shell program.

set localonly false
set sendvar true

These two variables were added in V2.0. localonly is used to disallow the use of Internet address specifications. This forces the email to be sent only to an address on the local system. Although the default is false, it is recommended that it be set true if you will be allowing any user to create forms that use this procedure. If this is kept false, it will allow the Internet server to be utilized as an anonymous remailer.

The sendvar variable is designed to allow flexibility for the forms designer. Earlier in the article it was discussed when making the choice on sending "value-data pairs" versus sending "data" alone. Setting the value as true will force pair sending.

puts "Content-type: text/HTML\n"
puts ""

These lines are what will be displayed to the user after submission. Error or not, these lines are needed. Note that the first line must be exactly as shown. That includes the blank return that follows.

foreach envvar [lsort [array names env] ] {
 if {[regexp {^WWW_} $envvar]} {
  set varname [join [lrange [split [lindex [split $envvar=] 0] _] 1 end ] _]
  if {$varname != ""} {
   set var($varname) $env(WWW_$varname)
  }
 }
}

In version 1.0, an exec was used to obtain all the environment variables. Then these were run through the regexp one at a time. The exec process added quite a bit of time to the processing. In 2.0, that was replaced by the foreach and if. Now this process will search all the variable names in the array env (which is where Tcl places all environment variables).

As an example, the variable information for the environment variable PATH is stored in the Tcl variable env(PATH). This process will get each array value (ex: PATH) and then check to see if it starts with WWW_ (the prefix assigned to form variables after they have been run through uncgi).

A form variable name translated by uncgi into WWW_name is found by this routine as env(WWW_name). Once found, the data is extracted into the internal variable var(name). This will continue until all environment variables have been processed.

set knt 0
set msub false
set mailto false

Set some initial flags... an internal counter... there is no passed-in mail subject... there is no passed-in mailto address.

if {[info exists env(WWW_sendvar)]} {
 set sendvar $env(WWW_sendvar)
}

Earlier we said that the user had control of the sendvar variable. This is where it gets reset by the user-issued command.

if {[info exists env(WWW_msub)]} {
 set msub true
}

If the form designer wants to send a user-defined subject... this will do it. NOTE: this is a possible security hole in the program. If the user sends a subject with a ";" in it, the UNIX system may assume that the command line is complete and then the email command will complete and the information that follows will be issued as a command on the system (a worst case scenario, msub = "Hello;rm -rf /" ). To avoid this, add the following line to secure the subject line (insert the line after the "set" above):

regsub -all \; env(WWW_msub) "" env(WWW_msub)

The above line will remove all semicolons from the subject line of the email message. The worst case example above would then look like msub = "Hellorm -rf /" -- a totally useless subject phrase.

if {[info exists env(WWW_mailto)]} {
 set mailto true
 set fname [pid].txt
 set f1 [open /tmp/$fname w]
}

In some cases the form designer may not want to send mail, but will want to test the operation of the form. The form designer, by omitting the mailto reference in the form, can force the information to be displayed back to the browser. The /tmp area is used for writing the temporary mail file, and the user's process id is used for the filename.

foreach vname [array names var] {
 incr knt
##

At this point you could insert an if statement to not include WWW_mailto, WWW_msub and WWW_sendvar in the output variables.

if {![regexp "mailto|msub|sendvar" $vname]}  
{remember to close the brace at the end of this code block }
##
 if {$sendvar} {
  set ostring($knt) "$vname = $var($vname)"
 } else {
  set ostring($knt) $var($vname)
 }
 if {$mailto} {
  puts $f1 "$ostring($knt)"
 } else {
  puts "$ostring($knt) 
"
 }
}

The above code block will build the message -- either to the screen or to a temporary file that will be sent via mail. Insertion of the if statement to eliminate variables should be closed by a brace at the end of this code block.

  if {$mailto} {
   puts $f1 "\n\nThis was generated from a form at: $env(PATH_TRANSLATED)"
   close $f1

If we're mailing it, write out the form information at the foot of the message and close the temporary file. You could also use this opportunity to add a disclaimer if you are using this code with Internet mail (anonymous mail) set on. A sample disclaimer might be: "\n\nThis code was generated via a World-Wide Web based form. The site mysite.com takes no responsibility for the opinions expressed in this email. The user nobody at this site is the default name for WWW users. The form used to send this email was: $env(PATH_TRANSLATED)." While that may not save you from the Secret Service visit, it will make it clear that the site had no control over the contents of the message.

  if {$localonly} {
   if {[regexp "@|!|%" $env(WWW_mailto)]} {
    puts "<center><h1>An invalid form of email notation \
           from this server has been used</h1>"
    puts "Please use the back key to exit.</center>"
    exit
   }
  }

This section enforces the rule of internal email only. @ ! and % are used when sending mail off the internal system to a foreign system. This rule eliminates that capability. If running on VAX/VMS add a check for :: as well. Again, it can't be stressed strongly enough that this filtering is preferred.

    if {$msub} {
     exec mailx -s "$env(WWW_msub)" $env(WWW_mailto) < /tmp/$fname
    } else {
     exec mailx -s "from the CGI automailer @ $env(SERVER_NAME)" \
        $env(WWW_mailto) </tmp/$fname
    }
   }

Send the email. Note that a subject is supplied by default if none is provided. Also, this processor is designed to use the mailx email interface from Unix; your mailer may be different, and may require a different syntax.

  exec rm /tmp/$fname
  puts "<center><h1>Mail sent to $env(WWW_mailto)</h1>"
  puts "<h2>Thank you for your participation $var(name)."\
"</h2>Your answers have been processed,please use the back"\
"key to exit the survey.</center> <hr><P> <address>©"\
"Copyright 1995, Infobahn Xpress, Los Alamitos, CA. All"\
"Rights Reserved.</address>" }

Last, but never least, clean up and tell the user what you did.

Sample Session

In this section a form will be developed, and then user input will be described. The final outcome will be sample mail messages that would be received.

This first form is designed to ask what someone is interested in, and request an e-mail address. The user can select an area of interest from the selection list, along with a level of interest indicator. The indicator is a radio button between 1 and 10.

   <title>Sample 1</title>
   <h1>Sample 1</h1>
Thanks for taking the time to respond to this simple ques-
tionnaire. This information will be used to provide you
detailed information on the subject of your selection.
   <p>
   Please respond to the questions in the next section. Once
completed, enter your email and then press the submit
button. Thanks for helping out.
   <FORM METHOD="POST" 
     ACTION="/old?u=http%3A%2F%2Fwww.site.com%2Fcgi-bin%2Funcgi%2Femail.tcl&y=1999">
   <input type=hidden name=mailto value=ivler>
   <input type=hidden name=msub value="Area of interest">
   <HR>                                                                
   <h2>Sports Information Selection</h2>
    <select name=intr>
   <option> Golf
   <option> Tennis
   <option> Boating
   <option> Chess
   <option> TV
   <option> Football
   <option> Soccer
   <option> Sailing
   </select>
   <h2>Interest Level</h2>
   <p> 1 is low, 10 is high.<p>
   1 <input name=m1 Type=radio value="1">
   2 <input name=m1 Type=radio value="2"> 
   3 <input name=m1 Type=radio value="3">
   4 <input name=m1 Type=radio value="4">
   5 <input name=m1 Type=radio value="5">
   6 <input name=m1 Type=radio value="6">
   7 <input name=m1 Type=radio value="7">
   8 <input name=m1 Type=radio value="8">
   9 <input name=m1 Type=radio value="9">
   10 <input name=m1 Type=radio value="10">
   <p>
   <hr>
   <p>
   <input name="name"> Email<p>
   To submit the answers, press this button: <INPUT
TYPE="submit" VALUE="Submit Feedback">.<br> 
   Use the back key to exit, and the reload option to clear
and restart.
   </FORM>
   <p>
Did you remember to submit?
   <HR> <P>
   <address>J. M. Ivler - email: ivler@i-xpress.com<br>
   © Copyright 1995, Infobahn Xpress, Los Alamitos, CA.
All Rights Reserved.</address>

The second form is the same as the first with the following changes:

sendvar has been turned off:
<input type=hidden name=msub value="Area of interest">
<input type=hidden name=sendvar value="false">

The values for the m1 variable have been changed from numerics to phrases. If 5 was selected in the second example it would generate the phrase: "I am not really interested in this topic." and a 10 would generate "I have a great interest in this topic".

In this sample session the user will pick Football as the pulldown selection, and 8 as the interest level. The user's name is ivler@i-xpress.com. The following are the two mail messages that would be generated by the user's selection.

Mail Message 1

To: ivler
From: nobody@site.com
Subject: Area of interest

intr = football
mailto = ivler
msub = Area of interest
m1 = 8
name = ivler@i-xpress.com

This was generated from a form at: /WWW/test/form1

If the change to remove mailto and msub were active, the message would be easy to process into a database of interest/level/individual. That could then be used as a feeder to a program to determine when to send e-mail to an individual on the topic area.

Mail Message 2

To: ivler
From: nobody@site.com
Subject: Area of interest

football
ivler
Area of interest
I have a fair interest in this topic
ivler@i-xpress.com

This was generated from a form at: /WWW/test/form2.

---

J.M. Ivler owns Infobahn Xpress, a WWW Consultancy, and is currently working at Information Management Systems as their Internet Specialist. He can be reached at ivler@i-xpress.com or at http://www.net-quest.com/~ivler/.

---

Reprinted from Web Developer® magazine, Vol. 1 No.1 Winter 1996 (c) 1996 internet.com Corporation. All rights reserved.

---

Web Developer® Site Feedback

---

Web Developer®
Copyright © 1999 internet.com Corporation. All rights reserved.