Rapid Prototyping with Tcl/Tk

Creating software is a complex process that embeds the programmer in rules and constraints. Customer needs fight against bugs in the program, and usability fights against production costs.

The current procedure to solve all these problems is object-oriented design and analysis, accomplished by methodically trying to split a problem into suitably small subdomains and providing a predefined path from analysis through design to implementation and testing.

Such a procedure may help for large programs and may be necessary if many programmers are involved. Most of the really good programs (such as those available for Linux, and even Linux itself) are written by only a few people, and more importantly, most of them don’t start out knowing in which direction their software will evolve. Therefore, another paradigm is needed—one that is better suited to the needs of humans.

Here is a statement about rapid prototyping (RP) I found on the home page of Cycad Corporation at http://www.cycad.com/:

While all of the above is certainly true, RP is particularly suited to the creation of software tailored to the customer’s needs. One of the most important aspects in determining these needs is to find out what the customer really wants. Because the program takes its final shape while it is under development, not in advance as would happen with standard structured design, the programmer can respond to new information about customer needs. With RP this process is straightforward and allowed, while normally it indicates an analysis or design error in other methodologies.

Equally important is an easy way to test new ideas. The final product will be better if you have some sort of playground to find out before implementation whether or not your bright new concept is really as bright as you think. This is even more important in our fast changing world where we have to react quickly to new or changing demands.

Using rapid prototyping for software development does not necessarily depend on a language specifically designed for it. It is possible to use compiled languages like C/C++, but best results can be obtained with an easy to learn/easy to use interpreted language. To get the most benefit, turnaround times have to be short and modifications to existing code should be possible without much effort.

One area where rapid prototyping really shines is user-interface design (provided you have the right tools). You can create a dynamic visual model with minimal effort and provide users with a physical representation of key parts before system implementation. You can accommodate new or unexpected user requirements earlier, and modifications are much easier.

Another aspect is quality assurance and control. The same tools used to build the prototype can be adapted to build up a test harness that is useful up to the final implementation of the system. This results in fewer changes after delivery and therefore productivity improves.

One of the current trends in computer languages is the use of portable languages like Java, Perl or Tcl. While these are generally slower than a compiled counterpart they offer a lot of advantages for the developer (portability, simplicity, short turnaround times, etc.). If such a language can integrate compiled code (e.g., by using shared libraries), a developer can first concentrate on creating a working solution and later optimize the code at the critical spots.

Tcl/Tk is especially well-suited to rapid prototyping, as it has solutions to all of the requirements discussed above:

I do not wish to begin a language war here—the above statements are my own opinions. To make the best use of any language you need to know the language very well (Tcl is no exception) and you need the corresponding tools (syntax sensitive editor, debugger, performance analyzer, testing and documentation tools). Finally, you need a lot of good libraries and extensions which give you the power to create big applications with only a few lines of code.

The following example is meant to give you a feeling for what rapid prototyping is like. To keep it short and understandable it does not use any of the features of Tk at all. Although a graphical example would certainly be better suited to show the advantages of RP, the concepts presented can easily be extended to make use of Tk’s extra features. It is one of my actual projects, and I will show it to you in the same way I developed it. As you may guess the customer and the programmer are in fact just one person—me.

Customer: “I need a nifty little library tool to work with values that are configurable by the end user. The values should come from a simple text file, and that’s all there is to it.”

Programmer: “Okay. This one is really easy.”

I sit down and type some lines of Tcl code that give me the following interface:

proc Cfg <

This way I can use the configuration interface for a (hypothetical) text editor in this manner:

set Lines [Cfg "NumLines"]

The format of the file should be kept simple, so I try the following to read values from it:

NumLines 20
WordWrap 1

Reading the file into memory would need another function. As I don’t want to pollute name spaces (true for both Tcl and C), I modify the signature of my procedure to be the following:

proc Cfg <

which gives me the following:

Cfg read "myconfig.cfg"
Cfg get NumLines

Code to implement all of this is written in 15 minutes. I add another 15 minutes to write some test cases so I can always verify the correctness of the whole implementation. Here is one test:

Test c1 "get simple value" {
  Cfg read "test.cfg"
  Cfg get NumLines
} 20

It reads my simple test configuration file and checks if the value for NumLines is really the same as set in the configuration file. It is all finished in 30 minutes. Sure, I have to rewrite it in C, but first I’ll cross check with my customer.

Customer: “Nice work, but you see, if an application gets bigger, we may have name clashes—consider two modules that use a configuration value named Width. One could be the width of a window and the other the width of the text in the editor. What we really need is a module dependent solution.”

Programmer: “Grrr. I should have known—simple things tend to get complicated with time.”

Modifying the interface is straightforward, but what about the file format? Far back in my mind is a little hint named win.ini. I cross check with a real windows installation on my DOS partition—yes, the format would be just right for my problem, and it will already be familiar to a lot of users.

So, I modify my configuration file to look like this:

; my config test file: 24/02/1997
[Editor]
WordWrap=1
NumLines=20
Width=80
[Window]
Width=320
Height=400
; EOF

Reading such a file is far more complicated than the simple approach taken earlier. I pat myself on the shoulder that I have not yet written any C code.

Using some of the extra features of Tclx, an extension to Tcl, the file reader is ready and working in half an hour.

Now, I modify the interface as follows:

Cfg read "myconfig.cfg"
Cfg get Editor NumLines
Cfg get Editor Width
Cfg get Window Width

and I modify all the test cases. Again, I present the whole thing to my customer.

Customer: “Yeah, that’s just what I wanted. I have some real world examples, could you verify that they work correctly?”

Programmer: The customer gives me some real X Window System configuration files—did you think the above were real world examples? Anyway, I run some tests on them and poof, all my nice looking code breaks. A quick look at the files reveals the problems—backslashes, brackets, blanks and entries that span multiple lines. Parsing an X configuration file is apparently more complicated than I thought at first.

I add a lot of new test cases that show all the above failures. They help me to make sure that I have a working piece of code at last, without one change breaking something that worked previously.

Adapting the Tcl code is comparatively easy. It takes some time though, as I have to fiddle around a bit with regular expressions for the parser. As soon as all the tests complete without error, I show my work to the customer.

Customer: “While waiting, I have used your code in a real world program, and it has one big drawback. Consider the following example configuration:

lpr -#1 -Plj5 myfile.txt

This is a print command which should be executed with exec. Some parts are static but depend on the operating system (e.g., lpr versus lp), others (such as the selection of the printer) may be done by the user and still others (such as the file name) come directly from the program. What I need is a flexible substitution scheme for configuration values.”

Programmer: “Ha. Now that’s a really big request. How do I create such a flexible substitution scheme?”

It takes me a day of thinking (actually this was handled by a background process in my brain) to find the solution—Tk uses substitution for its key binding mechanism, so why not use the same ideas here?

Again, I modify the interface, so it can process code such as the following:

Cfg get Printer Command\
   "%c=1" "%f=myfile.txt"

and I add some lines to my configuration file:

[Printer]
Command= lpr "#%c "Plj5 %f

Before returning the final value to the caller my routine substitutes all “%?” sequences with the corresponding values from the argument list. Using Tcl’s regular expressions makes this quite simple, and the only thing that takes some time is writing the tests to check for all these crazy conditions like “%%” and “\%”.

The above example will return

lpr -#1 -Plj5 myfile.txt

just as expected by the customer.

Customer: “Great. This is what I have been looking for for years. There’s just one sub-optimal item left. If the user accidentally deletes his configuration file, the program will no longer work. Isn’t it possible to keep some default values for this case?”

Programmer: “I have already thought of this situation. The code would be more readable if an actual value were present, and it could be tested more easily as I wouldn’t have to write configuration files all the time.”

I modify the interface once more by adding a new subcommand:

Cfg def <module> <name> \
   <default> ?<sub>? ..."

The general syntax of the get and def sub-commands are the same (and both return the same configuration value when called with the same substitution values), but def also sets a default value when none is given.

If a configuration file has been read by Cfg, the values in this file take precedence, otherwise the default is used.

Here is an example:

Cfg def Printer Command \
   "lpr -#%c -Plj5 %f" \
   "%c=1" \
   "%f=myfile.txt"

This command returns:

lpr -#1 -Plj5 myfile.txt

Customer: “I use your code in all of my Tcl applications, and it works like a charm. The only thing I noticed is that the programs are really slow on startup. I suspect this is related to your configuration code. Couldn’t you do anything about that?”

Programmer: “Aaargh.”

Back in the configuration business. First, I check some of the customers applications. No wonder he has slow startup—he uses configurations quite excessively, even for language internationalization. I instrument the applications for profiling and create some snapshots for analysis. The resulting diagrams (see Figure 1 and Figure 2) clearly state the two highest CPU-cycle-eaters: reading configuration files and substitution of configuration values.

Figure 1. Tcl Profiler Bar Chart

Figure 2. Tcl Profiler Graph

The interface is comparatively stable now (the customer has already used it in all his programs, he won’t change it that easily). I start rewriting some of the code in C. As I know the exact bottlenecks from the analysis, the first two C routines replace the configuration reader and the substitution engine.

Now all of the previously written tests come in handy—I can check if the new code works correctly with nearly no effort. Both of the routines are quite a bit of work as they do some very tricky operations and pointers always point to somewhere unexpected. Sure it’s nice to have a debugger for Tcl (see Figure 3), but without a C debugger, the life of a C programmer would be truly difficult.

Figure 3. Tcl Debugger

The new code is worth the price. Speed can be 5 to 10 times that of the original code, and this can mean the difference between a one second startup and a ten second startup.

There is a good deal more needed to make this little example into a full-fledged library, but it should be enough to see the general concepts. Here are the most important points to learn about the advantages of RP:

I did a really big project that integrated CAD (computer aided drafting) and PPC (production planning and control) software, and I successfully used the ideas presented above. (See Figure 4 and Figure 5.) The project took more than a year, and the specification changed more than once. The final solution shed another light on the versatility of Tcl: the user is able to define his own components (window frames, automatic actuators, etc.) using Tcl and its object-oriented extensions. This feature was just an offspring of our rapid prototyping philosophy as we simply provided the customer with our own tools.

Figure 4. PPC Software Screen

Figure 5. A Second PPC Software Screen

Using Tcl for software development is not the ultimate bells and whistles solution. There are drawbacks such as the lack of real data types (anything is a string), the inefficiency of interpreted code (although a compiler is now available), and the fact that eval stays a mystery even to Tcl fanatics. Tcl shines when your program is mainly centered on a graphical user interface. Other tasks may be done better by Perl, C or even FORTRAN. As dynamically loaded, shared libraries are now quite common on several platforms, we can split problems into chunks and solve these chunks with the language that is most appropriate for it. Tcl/Tk may be the glue for the graphical interface in this scenario.

What is Tcl/Tk?

Tcl (Tool command language) is a freely available simple, interpreted language designed to be used as a common extension and customization language for applications. It was designed and implemented by Dr. John Ousterhout in the hope that application designers could spend more of their time on applications and less on scripting languages, and that users could spend less time learning new scripting languages for each new application. Many useful applications, some of them sold commercially, use Tcl as their scripting language.

Tcl is clean and regular, and relatively easy for non-hackers to learn. It is command-oriented, and commands added by applications and users exist on an equal footing with the built-in Tcl commands. Tcl has both simple variables and associative arrays (tables), and all values (including procedure bodies) are represented as strings. Simple customization scripts (such as preference initialization scripts) usually look much like novice users expect: a series of simple commands used to set options.

Tcl is implemented as a C library, which can be embedded in an application. The application can add its own commands to the interpreter (using a clean C interface). It is distributed under a license which allows use for any purpose with no royalties.

The Tk toolkit is a Tcl extension (a group of new Tcl commands) which provides a Tcl interface to the X Window System. Tk is one of the easiest ways to build a graphical interface to an application, and due to the interpreted nature of Tcl, Tk-based interfaces tend to be much more customizable and dynamic than those built with one of the C or C++-based toolkits. Tk implements the Motif look and feel. A number of interesting X applications are implemented entirely in Tk, with no new application-specific commands at all. Tk also provides a mechanism by which one application can send Tcl scripts to other Tk-based applications running on the same display, for easy cooperation between tools.

For more information write to Glenn Vanderburg at glv@utdallas.edu or see news://comp.lang.tcl/tcl-faq.part.01/.

What is Rapid Prototyping

Prototyping is defined in the Dictionary of Computing, 3rd Ed., Oxford University Press, New York, NY, 1991 as follows:

Development of a preliminary version of a software system in order to allow certain aspects of that system to be investigated. Often the primary purpose of a prototype is to obtain feedback from the intended users; the requirements specification for the system can then be updated to reflect this feedback, and so increase confidence in the final system. Additionally (or alternatively) a prototype can be used to investigate particular problem areas, or certain implications of alternative design or implementation decisions.

Rapid Prototyping is a technique where a prototype is developed soon after initial system requirements are gathered from users. This prototype, which mainly concentrates on the system’s interface, is shown to users as soon as it is ready. Users interact with it while developers are present. Developers observe user interactions with the prototype and ask questions so that feedback can be obtained and project requirements further refined. This process is repeated several times, with more and more functionality added to the prototype, until users are satisfied with the system’s behaviour. Oftentimes, prototyping lasts throughout the system development life span.

Users have continual input into the system that they will eventually depend upon in a production setting. They are always involved in design and can help make sure they will get the product they want. In addition, the prototype grows into the final product with little effort and resources lost during design. This is known as evolutionary prototyping, as opposed to using a prototyping tool which cannot ever implement the final system or using a throw-away prototype.