Netsurfing With Linux

Purportedly, this article is about how an “obscure” operating system called Linux was used to launch a happy band of netsurfers into the wild ocean of the Internet. But it is really a rousing tale of adventure and discovery—with Linux playing the part of a trusty ship. That ship enabled us to chart a map of the vast Internet ocean, in the guise of Netsurfer Digest, a free, sponsor-supported e-zine (electronic magazine), serving as a gateway to on-line adventure for netsurfers all over the world.

It all started in the spring of 1994 when I gathered together a small band of netsurfers and, through the proper application of persuasion, hand waving, and free food, convinced them to put together an on-line publishing company. Following venerable startup tradition, we scribbled on numerous coffee house napkins, ate lots of pizza, exchanged tons of e-mail, and gave birth to Netsurfer Communications. Our goal was to publish interesting and user-friendly e-zines using the hot new technology of the World Wide Web.

Being a fairly on-line literate bunch, we decided that our products would be made available exclusively on-line, and that the company would be run as a virtual corporation. We intended to take every advantage of the vast leverage provided by modern communication technology. By existing entirely on-line, we would be able to effortlessly communicate with our consumers, tap a global pool of talented contributors, and keep our overhead to a minimum.

While prototypes of our flagship publication, Netsurfer Digest, were being prepared and the production process was being designed, a number of technical decisions had to be made. One of the most important was the choice of operating system for our production facilities and for our Internet site. Whatever we chose had to meet a number of fairly stringent requirements.

First, we needed a very reliable e-mail platform. Our system had to be able to support mailing lists serving thousands and provide reliable e-mail storage for our internal editorial communications. Second, we needed a reliable World Wide Web and FTP site. Back issues of our e-zine as well as various background information had to be made available to people all over the world, at all hours of the day and night. We also needed the ability to easily change and update this information, sometimes by automated scripts. Finally, it was important that whatever environment we chose supported good development tools. We planned to create a number of custom programs to aid in production and distribution of our e-zines.

It didn’t take a genius to figure out that some flavor of Unix was what we needed. After all, Unix is the native operating system of the Internet and may well be the best development environment ever designed. The only question was which brand of Unix to go with? There were a number of commercial versions available running on expensive workstations or feature-loaded PCs. The key word there was “expensive”. Now, if you’ve been involved with startups, then you know that next to the occasionally scrambled brains of the startup team, the most precious resource is cash. You only spend it on items absolutely essential for keeping the venture going, such as marketing, hardware, and pizza. This was definitely on our minds when the time came to choose our operating system.

It just so happened that at the time I had a copy of Linux available, which I had purchased on CD-ROM from Trans-Ameritech. I had played around with it at home and had also heard good things about it from my friends. It appeared to be a full-featured, relatively robust operating system which might be able to meet our needs through the early stages of the e-zine. The price was right, and what’s more, we did not need some super-expensive machine to run it. We had little to lose by giving Linux a try. If it worked, we had a very inexpensive solution to our requirements. If not, well, we could always go with one of the more expensive commercial operating systems, something we figured we’d have to do anyway as our enterprise grew.

It was clear that Linux was a nice stand-alone operating system, but we needed to find out if Linux could reliably support an Internet site. Early in May of 1994 the big day came. I had already installed Linux version 1.0.9 on our machine, a humble 486DX33 PC with 8 meg of RAM, a 245 meg drive, and an Ethernet card. The first priority was to see if we could hook it up to the Ethernet network at our provider site and, from there, work on getting it on the Internet. So I lugged the machine across the San Francisco Bay to Berkeley and sat down with our netmaster, Bill Woodcock, to install it on his network. I was prepared to spend a few hours fiddling with the system to get it running. I even brought some snacks to munch on as we worked during the afternoon on getting the whole contraption to work.

First, we read the instructions in the Ethernet-HOWTO, which essentially said to make sure that the kernel had been compiled with support for our Ethernet card. No problem; I had already done this. Next we read the NET-2-HOWTO, which told us how to configure the TCP/IP network. This boiled down to either running a utility called ifconfig or changing a few well commented lines in the rc.inet1 and rc.inet2 files. It seemed deceptively simple, and after we made the changes, Bill and I looked at each other skeptically and rebooted the machine.

I’ve spent all my professional life working with complex hardware and software systems, first as a mainframe design engineer, and later, as a software manager. In my long experience, I’ve learned that the first time you test a new piece of software, turn on new hardware, or configure a network, it never works. Never. There is always some fiddling and adjusting, or even bug-fixing, which must be done before the whole thing works vaguely the way it was designed to. That’s just the nature of the beast. Imagine my consternation when the machine came up, recognized the network, and responded to pings from the rest of the world. This just does not happen in the real world. I was, frankly, stunned and amazed. But in a good way.

In short order, we brought up the standard daemons and had telnet, FTP, and e-mail going between our machine and the rest of the network. What I thought would be a long afternoon of debugging and digging through obscure on-line documentation turned into a half-hour job. We took the rest of the afternoon off and went out to get some pizza. I even sprang for extra sauce.

Bill spent the next few days configuring the machine to our liking. He arranged domain name registration, set up secure FTP and WWW software (all freely available on the Net), wrestled with e-mail configuration, and set up the necessary user accounts. We were on the Internet and ready to support beta testing of our first e-zine, Netsurfer Digest.

Throughout the summer of 1994, the Linux box supported our beta effort, which eventually grew to over 1500 users getting copies of our e-zine twice per week. At the same time we obtained another PC to be used for development and production. This was outfitted with X-Windows as a user interface, Perl as the primary development language, and Seyon as the telecommunications program. Work proceeded on developing both our editorial process and on various support tools. So far, Linux was proving to be a very cost-effective choice and quite flexible as a beta test platform. However, our voyage was not entirely smooth sailing.

Early during the Beta test period, we had problems handling the continually-growing mailing lists. Our problems were caused by an interaction between our mailing software (smail at the time), the relatively slow machine, and a slow Internet connection (28.8K modem). In short order, we located another mailer (sendmail) at a Linux FTP site, wrote some Perl scripts, and had a working mail configuration. Linux’s flexibility as a development environment was starting to pay dividends.

Of greater concern to us was stability. One of the things we noticed was that the machine was not as stable as we would have liked. It appeared that, for one reason or another, we had to reboot two or three times per month, on average. Frequently, this was due to random segmentation exceptions, or other obscure errors, which would hang-up our machine for no discernible reason. We could live with these occasional glitches during our beta test period, but clearly, this would not do for a production machine which had to reliably serve thousands of readers. However, since these glitches did not seriously impact our beta testing effort, we could put up with them for a while, given the economical nature of the software in particular. We were a happy and cheap startup.

By the fall of 1994, we were on the eve of a formal launch of Netsurfer Digest. Our mailing lists were growing larger and our Internet site was being accessed more and more often. As part of our effort to prepare for our public debut, one of the things we did was upgrade our copies of Linux to the then current version of 1.1.61. We noticed immediately that the new version appeared much more stable than the earlier 1.0.9. This has, in fact, proven to be true over the long term, with the latter version of Linux taking us smoothly through the early days of our commercial existence.

Steady as She Goes: Hauling the Cargo

Throughout the winter of ’94-’95, Linux performed flawlessly as our Internet and production platform. Our subscription rates, and therefore our weekly mailings, were growing at a furious pace. We reached the point where it took over 24 hours to mail out one issue to our thousands of direct subscribers. The response time on our system did slow during a mailing, but Linux took the load and completed each run.

The accesses to our WWW server also started taking off. It seemed that every week we were adding another thousand hits to our daily access statistics. The real test came in early January, when we created a special Macintosh issue to coincide with Macworld Expo. Overnight, accesses to our poor 486DX33 PC topped 10 thousand per day as Macintosh aficionados overwhelmed us with accesses. Linux met the load with graceful performance degradation, doing just what a good operating system should do. There finally came a point where our faithful system could take no more and the machine hung. We had pushed our system to its limits.

By the end of January 1995, Linux was routinely handling over 16 thousand accesses to our Web site every day, with peak loads reaching almost 20 thousand hits per day. At the same time, we were handling mailing lists numbering over 15 thousand subscribers for two different e-zines. It was mind-boggling that this free system on a relatively puny machine could do all this. We definitely got value for our money.

We realized in late 1994 that there would come a time when we would have to upgrade our systems to handle the loads imposed by our dizzy growth rate. Simple math told us that, at some point, our weekly mailing would impose such a load on the system that people accessing our site would not be able to get through. Beyond a certain point, a 486 PC, or even a Pentium, would run out of horsepower. To handle tens of thousands of subscribers and tens of thousands of daily hits on our Net site, we would need to upgrade the speed of our feed, arrange for professional mirror sites, and get some serious hardware.

Fortunately, this does not mean that we will abandon Linux. All of our production and development still takes place on a Linux PC workstation. Our current machine, now a veteran of a long and eventful year, will remain hooked up to the Internet as a backup resource. Linux has proven itself in a very demanding environment. We have pushed it to its limits and occasionally beyond. We would not be where we are today without this wonderful operating system, and by extension, without all those who made it the great and powerful tool that it is today. We can only hope that by publishing our free e-zines, by making them entertaining and useful, we can return something to that vast Internet community without whom we could not have our dream jobs.