The overall CGI Development Process

CGI is an acronym for Common Gateway Interface. This interface allows your browser and a server to pass information back and forth dynamically.

Index of this page

Before the First Time
For Each Project

Linux Basics from a Remote Terminal
Static web page editing and forms are discussed in Html Form Editing.
(this page) The Overall CGI Process for an introduction to dynamic web pages.
The cgiTest Index Page links to all the resources on static and dynamic web development

An Example in Operation

As an example go to my adder web page, follow the instructions, enter numbers, and click on the Find Sum button. You get back a page that obviously used your data. This is the idea that you can generalize.

Execution steps behind the scenes.

Your typing in data is handled directly by the browser. It recognizes forms.
An action is stored in the form saying what to do when you press the button (called a submit button by default -- "Find Sum" just made sense in this case as a my label).
In this case and in the cases we consider in this class, the action is given as web resource, giving the location of a cgi script on some server.
When you press the button, the browser sends the data you entered to that web location (in this case adder.cgi in the same folder as the original web page).
The server recognizes the page as an executable script, sees that it is a Python program, and executes it, giving as input the data sent along from the browser form.
The script runs, takes in the data passed from the form, manipulates it into some results, and puts those results into the text of a web page that is the output of the program.
The server captures this output from the program and send in back to your browser as a new page to display.
You see the results in your browser.

A Simple Buildup

Before we get too complicated, consider a couple of even simpler examples. Browsers can handle data in a number of formats. The simplest is plain text. Of course we will generally use the more elaborate html markup. Furthermore, a CGI script does not necessarily require input data. In that case you can call it directly like any other link.

Hellotxt.cgi

The simplest case is a CGI script with no input that just generates plain text. Hellotxt.cgi is an example You can run it directly from your browser, http://cs.luc.edu/~anh/150/cgiTest/hellotxt.cgi, and I have provided a way to see its source. (You cannot generally see a CGI script file from the web since, giving the URL executes it rather than displaying it.) The source is also copied below:

#!/usr/bin/python2

#
Required header that tells the browser how to render the text.
print
"Content-Type: text/plain\n\n"  # here text -- not html

#
Print a simple message to the display window.
print "Hello,
World!\n"

The top line is what tells the server that is a Python program. It says where to find the interpreter for the script. This exact line is always required.
The first print statement is telling the server that receives this output, that the format of the rest of the output will be plain text. This information gets passed back to the browser later. This line should be included exactly as stated IF you only want the output to be plain text (not our usual case).
The rest of the output (in this case just from one print statement) becomes the body of the plain text document you see on your browser screen, verbatim since it is plain text.

Hellohtml.cgi

We can make some variation and display an already determined html page. Test the link hellohtml.cgi or look at its source, and the source is below:

#!/usr/bin/env python2

print "Content-Type: text/html\n\n" # html markup follows

print """
<html>
<Title>Hello in HTML</Title>
<body>
Hello There!
Hi There!
</body>
</html>
"""

There are two noteworthy changes. The first print statement now declares the rest of the output will be html. This is the standard line you will be using. The remaining print statement has the markup for an html page. Note that the enclosing triple quotes work for a multiline string. Other than as a simple example, this CGI script has no utility - just putting the contents of the last print statement in a file for a static web page hello.html is much simpler.

Now.cgi

One more simple step: we can have a cgi script that generates dynamic output by reading the clock from inside of Python: Click on now.cgi. Then hit refresh and see that it changes. This cannot come from a static page. Get the source from the web or below:

#!/usr/bin/env python2

import time
print "Content-Type: text/html\n\n" # html markup follows

timeStr = time.strftime("%c")

htmlFormat = """
<html>
<Title>The Time Now</Title>
<body>
The current Central date and time is: %s
</body>
</html>
"""

print htmlFormat % timeStr # see embedded %s ^ above

This illustrates a couple more ideas: First a library module, time, is imported and used to generate the string for the current date and time. The other slightly new idea is that the triple quoted string is actually a format string. (Note the embedded %s.) Hence the last line of the program embeds the time data string inside the htmlFormat string. This is a handy pattern to follow: Keep all the preset format together and just embed the dynamic part.

Now we are ready for the more complicated topic of processing dynamic input. This input is usually generated inside a form in a web page. If you have not already done so, look at the latter part of Html Form Editing on composing a form in NVu to see that you specify internal names for form elements which have values, generally modified or supplied by the web user, and the this data is forwarded to the server when an action button is clicked.

A Basic Template: the Adder Example

As an example, you can download and open your local copy of adder.html in NVu, and right click on the form fields to see the settings. You can also look and the name attributes in the html source code. You should see that the user enters strings in fields named "x" and "y". The action for the form is adder.cgi. It is time to look at it. It shows the basic framework for processing CGI input into an output page. It can be used as a template.

You can look at the adder.cgi source online or below. New features are commented in the source and discussed below.

#!/usr/bin/python2

import cgi

def main():
    form = cgi.FieldStorage()      # standard cgi script lines to here!

    # use format of next two lines with YOUR names and default data
    xStr = form.getfirst("x", "0") # get the form value associated with form
                                   # name 'x'. Use default "0" if there is none.
    yStr = form.getfirst("y", "0") # similarly for name 'y'

    # do YOUR thing - manipulate the data, and create results
    x = int(xStr)
    y = int(yStr)
    sum = x+y
    sumStr = str(sum)
    # output by embedding into YOUR separate format page YOUR results
    showPage("adderAnswer.html", (xStr, yStr, sumStr) ) # note tuple paramater


def showPage(templateFileName, substitutions): # keep this handy function
    """
    templateFileName is the string name of the html file used as a format
        string.
    substitutions is either a single data item or an explicit tuple
        containing the data to substitute into the format string in the file.
    The function prints the string created from the file's format string and
    the supplied substitutions.
    """

    fin = open(templateFileName); #page could be a canned string
    page = fin.read(); #   in the program but this allows the visual display
    fin.close()         #    to be changed separately from the logic.

    print page % substitutions

#standard lines from here on
try:
    print "Content-type: text/html\n\n"   # say generating html
    main()
except:
    cgi.print_exception()                 # catch and print errors

First the overall structure of the code:

To handle the CGI input we import cgi.
The main body of the code is in a main method, following good programming practice.
After the defintion of main and the supporting function definition showPage comes the standard wrapper code for our html generation (that you can copy exactly). I chose to put the initial print statement here, that tells the server html is being produced. That mean the main method only needs to produce the actual html code. Also keep the try-except block that catches any execution errors in the program and generates possibly helpful trace information that you can see from your browser.

The main function

Only housekeeping is done outside of the main function. The three sections of main read the CGI input, process it, and generate the html output.

Reading input: The first line of main is a standard one (to copy) that sets up an object called form that holds the CGI form data sent by the browser. You access the form data with statements like the next two that have the pattern:
variable = form.getfirst(nameAttrib, default)
If there is a form field with name nameAttrib, its value from the web form is assigned to variable. If no value is given in the form for nameAttrib, variable is set equal to default instead. In this way data associated with every name can transferred from the browser form to your Python program. Here the values associated with names 'x' and 'y' are extracted. I use variable names that remind you that all values from the browser forms are strings.
Process the input: Here you do anything you want to do in Python with the input to generate the data you want to output. In adder.cgi this is very simple. The one complication is that we need to do arithmetic with ints, not strings, so the strings are converted, the addition is done, and I choose to turn the result back into a string again.
Generate html output: All that remains is to neatly format the output in a web page. We already saw in now.cgi how to embed our data into a format string for a web page. The last line of main in adder.cgi uses a general supporting function, showPage, that generalizes the idea in now.cgi. Its parameters are the name of a file containing an html page with embedded fields and a tuple with the actual data to substitute into the format fields (the results of processing the input in the program).

The showPage function:

I found myself always writing almost the same four lines in my main program, so I save the lines in the showPage function. If you are going to do anything vaguely complicated to format your output, it is convenient to generate the page in NVu and save it to a file. As in the format text in now.cgi, the only thing you do special for the web page is put format symbols (like %s) where you want to substitute the data generated dynamically in your program. It is convenient to leave this format string as a file for two reasons: 1) you do not have to go through an extra step to paste it into your source code, and 2) you can modify the file controlling the display of your output without having to change your program.

You can look at the file that gets read as data for adder.cgi, adderAnswer.html. It of course looks almost like the page that is generated as the result when you use the form in adder.html. The difference is that you do not have the data to substitute in, and %s appears where the real data is supposed to go. (Note the %% in adderAnswer.html where I actually want to display a % character. This is the convention for embedding a literal % in any format string.)

That's it! Copy everything but the main function exactly, and put in your code to extract the browser form data, your code the process the data, and supply the call to showPage with the name or an html format file that you generated and with a tuple of the dynamic data you want to embed in the output page.

It is time to try this all yourself, and do the final steps to get everything set up on your server pages. A summary of the whole process follows, with the final new step of getting everything going on your server page. I suggest that the first time through you just make copies of something that already works, like the three files adder.html, adder.cgi and adderAnswer.html.

Outline of the entire process

Be careful if you copy sample files from the web to edit. Html files can be right clicked and directly saved, and opened in NVu. That is the preferred route. If you cut and paste be careful to pay attention to the mode you are using in NVu. If you copy raw html text, be sure to paste it into the source view, not the normal view!

Before The First Time

Make sure you have an account on a server and can use a terminal interface and have set up a web directory, as discussed in Linux Basics.
We will be using dumpcgi.cgi as a testing tool. Paste the source in a local file dumpcgi.cgi.
Upload (Filezilla) dumpcgi.cgi to the public_html directory on the server.
In your terminal program (PuTTY) make sure your current directory is public_html or else run the command
cd ~/public_html
Run the commands
dos2unix dumpcgi.cgi
chmod +x *.cgi
./dumpcgi.cgi # note the period
You should see the output:
Content-type: text/html

<html><title>Raw CGI Data</title><body>
No CGI data received
</body></html>
This means dumpcgi.cgi is ready to test your web forms.

For Each New Project

I use the name myPage for your project, but substitute whatever name you want, consistently, below.

Copy formTemplate.html, rename it to your new page name, like myPage.html.
Edit in NVu or some such: Copy and move around the form fields, edit the text outside the form fields. See Html Form Editing for details on this step and the next two steps.
Set the form field attributes, either in the graphical view tab via popup forms or directly in the source view tab in NVu.
If you are not running on the server, upload your page to the Linux server with an sftp client like Filezilla.
Access and submit your page in a browser, (with the original action dumpcgi.cgi) and see that the fields all have the names and results you want. This depends on you doing the initial steps B-E. above.
In NVu, change so the action listed for your page is the CGI Python script you are writing, for instance, myPage.cgi.
As you gain experience and have more faith that you set up your form right, you can create your page without my template, and initially enter the final cgi script as the action. I only put dumpcgi.cgi into formTemplate.html to help you find form errors.
If you are just going to just calculate and plug results into fixed positions, make a results page, say myPageResults.html. Put a substitution field like %s, %d, or %.2f wherever you want to plug in values. If you want an explicit percent sign, use two.
Write the script myPage.cgi. Use the standard boilerplate starting and ending code in adder.cgi (source), and you can end with similar code to plug your calculated data into the text of myPageResults.html. You just set what data comes in, how to manipulate it, and what results to embed in what web page format.
Check the module syntax. (In Idle use alt-x in the file window.) This is important! Otherwise your later errors can be hard to find!
Upload everything to your public_html directory. For instance upload myPage.html, myPage.cgi, and myPageResults.html
Getting the script to run: Please run all these steps carefully in order. If you miss one you can get a strange looking or silent error later.

Start (or return to) a terminal program like PuTTY in ssh mode. Change the current directory to the directory containing the script (under public_html). Convert the script to unix format and to executable by others and writable only by you with the terminal commands
dos2unix myPage.cgi
chmod 711 myPage.cgi
An alternate to the second command, to quickly make all your cgi scripts in the directory executable is: chmod 711 *.cgi
Run the script from the terminal command line . Note the command starts with './'
./myPage.cgi
Execution errors are OK here, if they are caused by the lack of the cgi form information, but the script should compile and start to run. The error catching code that prints trace information to the browser only works for execution errors, not syntax parsing errors. That is why you want to make sure it compiles on the server, first. Errors you might receive:
- : bad interpreter: No such file or directory # this means you forgot to run dos2unix in step c.
- -bash: ./myPage.cgi: Permission denied # this means you forgot the chmod command in step c.
Make sure everything is in your public_html directory, and enter the URL for your form page in your browser and try it out!

Security Issues

Caution: When someone runs your script from the web, it runs as if you are doing it - with your permissions. Do not take data into your web form and from there into your Python script that could be converted to an arbitrary command or arbitrary file: you could compromise the system and destroy your data. For example a took care in my filedisplay.cgi to test and then only allow the display of files in the web portion of my files that are set as readable by others.

Fine Points

Input from the form

Above I provided the most common way to access the form data from the browser. Here I give a couple more options. As in the adder.cgi example, I assume the cgi data ends up in a variable called form:

form.getfirst(nameAttrib) which gives the first corresponding value for the name attribute specified, or None if there is none.
Example: addr = form.getfirst("addr")
form.getfirst(nameAttrib, default) which gives the first corresponding value for the name attribute specified, or the default value specified if there is none.
Example: addr = form.getfirst("addr", "No Address given!")
form.getlist(nameAttrib) which returns a list of the zero or more values associated with different form elements with the name attribute specified.

Output with variable length lists

If you just have a fixed number of values to calculate and display, it is easiest to write a web page that will serve as a format string. You can access such a file with the showPage function.

You may want to have a loop that generates a variable number of lines or bullet points or some such. In this case you must also repeat the repetitive part of the markup. If you just want to generate data on a variable number of separate lines in html, you just need to separate the lines with " " not a newline. Be careful: a newline character '\n' is no different than any other white space in html. An example that also maintains a dynamic data file is called from namelist.html. The CGI file is namelist.cgi (source).

The cgiTest Index Page
Comp 150 Home