TidBITS#368/03-Mar-97 ===================== Are you a hotshot at using Macs to build full-text search engines for the Web? Enter the first-ever TidBITS Macintosh Search Tool Shootout! Also this week, we bring you part two of Stuart Cheshire's article on latency and bandwidth, plus information on new versions of Internet Explorer and Quicken. Also, our field correspondents report on highlights from Macworld Tokyo, and we call for additional TidBITS translators. Topics: MailBITS/03-Mar-97 TidBITS Macintosh Search Tool Shootout Macworld Tokyo: Of Cameras and Macs Bandwidth and Latency: It's the Latency, Stupid (Part 2) Copyright 1997 TidBITS Electronic Publishing. All rights reserved. Information: Comments: --------------------------------------------------------------- This issue of TidBITS sponsored in part by: * APS Technologies -- 800/443-4199 -- <-------- NEW! Makers of M*Power Mac OS compatibles & premium storage devices. APS price lists: * Northwest Nexus -- 800/539-3505 -- Professional Internet Services. * Power Computing -- 800/375-7693 -- PowerTower Pro 225 MHz - the fastest desktop system ever. Build Your Own Box online! * EarthLink Network -- 800/395-8425 -- Direct Internet access for Mac users. New Personal Start Page, no setup fee for TidBITS readers! * Aladdin Systems -- 408/761-6200 -- Makers of StuffIt Deluxe 4.0, the Mac compression standard, and InstallerMaker 3.1.1, the leading installer for Mac developers. * Small Dog Electronics -- Special deal for TidBITS#368! <--------- NEW! NEW Power Mac 7200/120 - 32 MB RAM, 1.2 GB HD, 8xCD: $1295 More Info: -- 802/496-7171 --------------------------------------------------------------- MailBITS/03-Mar-97 ------------------ **Translators Needed** -- For the last year or so, teams of dedicated volunteer translators have created award-winning translations of TidBITS in Chinese, Dutch, French, German, Japanese, and Spanish. These teams could use additional volunteers to spread the load. Each team works a bit differently, but all could use more volunteers to translate an article every week or so. If you're interested in helping support the Macintosh in your country or language, please contact the appropriate coordinator below. [ACE] Chinese -- Peter Dutch -- Sander Lam French -- Chantal David German -- Walter J. Ferstl Japanese -- Shuichi Odaka Spanish -- Javier Pedreira **Microsoft Internet Explorer 3.0a** -- Although I use a variety of browsers to view and test Web sites, I'm using Microsoft's Internet Explorer with increasing regularity. With the release of version 3.0a (PowerPC-only, alas) last week, Microsoft has resolved problems with deleting cache files, repeatedly reloading some Web pages, Challenge Response Protocol (used when accessing secured pages), and loading Java under MacTCP. Minimum and full install versions are available, ranging in size from 2.1 MB to nearly 8 MB. [JLC] **Steve Becker** writes: Intuit has released an R6 update for Quicken 7 and Quicken 7 Deluxe. The update fixes several bugs (see TidBITS-353_ and TidBITS-359_), and the non-standard ROI (Return On Investment) calculation in the Portfolio window has been replaced by the preferred ROI calculation used in the Investment Performance report. In addition, Q7 users may wish to know that Connectix's Speed Doubler can significantly speed the opening of Register windows, and indexing error warnings can sometimes be avoided by increasing Quicken's memory allocation (an additional 1 MB worked for me). **Get Even Richer** -- If you were intrigued by the Crack A Mac challenge underway in Sweden to break into a Macintosh Web server (see TidBITS-365_) but felt pot wasn't sweet enough, you might be interested to know that several Mac resellers have donated additional funds to raise the jackpot to over $10,000 U.S. Of course, you still have to alter the target server's home page to claim the money. The contest runs through 10-Apr-97. [GD] TidBITS Macintosh Search Tool Shootout -------------------------------------- by Adam C. Engst For some time, we've been lamenting the fact that TidBITS doesn't have a good, full-text, search engine. Years ago, Ephraim Vishniac set up an excellent WAIS source for TidBITS, but that was when Thinking Machines ran the public WAIS server on their Connection Machine. That service eventually went away, and several attempts were made to replace it. The current search engine is run by Sensei Consulting in Australia, and although it's welcome, we often hear of troubles accessing it. In addition, searches return entire issues, rather than articles, so you must also search within the returned issue. A variety of searching tools that run on Macs have appeared over the years, but we've never had the proper combination of time, hardware, and experience to put them through their paces. So, we've come up with a different method for evaluating these pieces of software - we're going to have a search tool shootout! We have a number of goals in mind. First, we want to pull out the best search tools for the Macintosh among the numerous contenders. Second, we want to let the creators of these programs strut their stuff. Third, we want to provide a way for people to search TidBITS easily. **Who Can Participate?** Anyone can participate, although we expect that those who have written search tools will be the most interested, since this will give them a chance to show off in a real-world test that will be useful to thousands of people. If, however, you're a consultant and specialize in setting up Macintosh-based search tools, you're welcome to compete. **What's the Test?** Once everyone who has expressed interest in participating has contacted our Managing Editor Jeff Carlson at , we'll provide access to all back issues of TidBITS, in HTML format. No pansying around here - the competition will use the contents of over 360 issues of TidBITS, about 11 MB of text covering the last seven years. Once everyone has the issues, they can set up their search engines. We don't have anywhere near enough Macs to host this, so contestants will have to provide their own hardware and Internet connection. Technical questions regarding our format or other issues can be directed to me at . **Specification** -- No contest would be complete without rules. All entries: * Must offer full-text search capabilities of all TidBITS issues. * Must be made with and run on a single Macintosh running the Mac OS. * Must be accessible via the Web. * Must automatically integrate new issues every week. * Must return results at an article level (articles all start with

tags). * Must display results using HTML source from TidBITS issues, including hot links. In addition, these bonus items could be included and will improve an entry's chance of winning: * Sorting results by date or relevance * Low cost * Short setup time * Other additional features, such as suggesting alternative sites to search if a search comes up empty **The Time Frame** -- We don't expect contestants to drop everything and start working on this full time - in fact, we'd prefer to hear things in the best entries like "Yeah, I whipped this off while I was waiting for my pizza to arrive." The Macintosh is about ease-of-use, and we hope that it won't be difficult to set up these systems. Here are the dates to watch: * 17-Mar-97: Deadline for entering the contest. * 21-Apr-97: Deadline for completing entries. Judging starts. * 12-May-97: Winner announced. **How Will We Judge?** Implementation details are up to the people participating in the shootout, but we have guidelines that contestants should keep in mind. All of the specifications should be met, although we won't disqualify entries for not meeting all of them (other than the Mac and Web requirements, which aren't negotiable). * Compliance with the specifications * Speed of searching, independent of connection speed * Attractive and usable interface for the search page * Attractive and readable results pages * Cost and setup time * Additional features **The Prizes** -- Obviously, a contest requires prizes, and we'll reward the winning entry (or entries) with the main thing we have - exposure to an estimated 150,000 Macintosh users. We plan to write about the shootout, looking at each entry and concentrating on the best of the crop. Then, assuming everything works out, we'll implement the best solution on our servers for everyone to use, giving that entry full credit and significant exposure. Other contestants can continue to host their searchable archives of TidBITS as a real-world demonstration of what their software can do, and we'll link to those who keep the archive up-to-date with new issues. Macworld Tokyo: Of Cameras and Macs ----------------------------------- by Chuck and Linda Shotton The Tokyo version of Macworld Expo always comes off brighter, perkier, and quite different from the Macworld shows held in Boston and San Francisco. Booths are generally larger, have more staff, and the "booth babe" is a staple of nearly every venue. If one thing stands out, it's the diversity of products. In addition to items seen at the U.S. shows, Macworld Tokyo features an entire hemisphere's worth of products, ideas, and technologies. Our mission was to ferret out products that aren't generally available in the U.S. or aren't widely known by the Mac community in the States. **Digital Cameras** -- Our first quest sent us on the rounds of the digital camera vendors. The roll-out of Apple's new QuickTake 200 camera begs comparison to some new products offered by Japanese companies. We tried to limit ourselves to notable cameras in the $250 to $1,500 range. Fujifilm was demonstrating its new Fujix DS-300 camera. Although one of the more expensive offerings (educated guesses put it around $1,400), it packs a lot of capability into a package the size of a normal SLR 35 mm camera. In addition to RS-232 and NTSC interfaces, this camera boasts a PC Card slot and a SCSI interface. But the big surprise is a whopping 1280 by 1000 high- resolution mode. You can save 8 photos at this resolution in JPEG or TIFF format, 30 in "fine" resolution, 62 in the normal 640 by 480 mode, and 121 photos in "basic" mode, with reduced resolution. This camera takes normal 35 mm lenses, and the CCD will emulate film speeds from ISO 100 to ISO 400. At the other end of the price spectrum was Panasonic's Cool Shot (KXL-600A-N). This pistol-grip camera is about the size of a 3" by 5" index card, less than an inch thick, and fits comfortably in your palm. It avoids the battery-sucking color LCD viewfinders of its competitors, opting for the simple point-and-shoot viewfinder lens found one-button film cameras. The Cool Shot accepts standard Type 2 PC Cards and stores either 24 640 by 480 images or 96 320 by 240 images on a 2 MB card. The major attraction of this camera is its small size and the one-hand operation allowed by its unique form factor. It has an optional external LCD viewer, a docking station for use with a desktop computer, and software for Macs and PCs. Prices range from $400 to $800. New lines of cameras from Ricoh and Sharp also caught our attention. Sharp's new camera was a PC Card with a built-in digital camera. Designed to work with the Zaurus color PDA, the card could be popped from a portable power supply into a laptop where the images could be accessed immediately. Ricoh's DC-2 camera series has the unique ability to capture not only still images, but full-motion video and/or audio soundtracks and annotations. The basic stills-only model (DC-2E) starts around $650, with the 2L and 2V models including video and audio capabilities for about $800 and $950 respectively. **Pioneering Macs** -- Though Apple's new hardware announcements were a big hit, Pioneer was showing a couple of new Macs that would be welcome on my desktop. The Pioneer clones packed serious horsepower in a mini tower package with features that are unavailable in the U.S. right now. The most exciting feature was CHRP (PPCP) compliance, with the MPC-GX2 model running the CHRP version of System 7.6. Powered by a 200 MHz 604e with 32 MB of memory and 512K of L2 cache, the box seemed very responsive. In addition to the usual Macintosh ports, this box sports four PCI slots, one ISA slot, two IDE channels, a 2 GB SCSI hard disk, and the usual set of mouse, serial, and parallel ports found on an Intel PC. Best of all, a DVD-ROM drive tops the tower. The demo was playing a full-screen version of the latest James Bond movie, Goldeneye, while running System 7 applications in the foreground. Most impressive. Retail prices weren't available but prices seemed to start around $3,500. **Read My Mind** -- Other notable hardware included revised versions of the AtMark Pippin boxes and the IBVA brainwave hardware, which had the coolest demo of all, with direct brainwave-to-MIDI output allowing the user to "think" new music. The new software has an open plug-in architecture that allows you to hook the brainwave hardware and software to nearly any Mac application through the addition of scripts and so on. The possibilities seem novel and exciting, though the cost in Japan was around $1,000 for the wireless headset, base station, and associated software. **Englishbonics** -- On the software front, one of the more useful products was an English language tutor called English Now! from Transparent Language, Inc. This product combines, written, spoken, and visual elements into a system that provides a comprehensive language learning environment. Features include the ability to see English text as each word is highlighted and spoken in a variety of synthesized voices, record your own voice and compare it to sonographs of correctly spoken words, and numerous lessons and games involving translating Japanese text to English and vice versa, spoken text into written words, etc. I was very impressed with the completeness of the package. English Now! costs approximately $100 on CD-ROM for both Mac and Windows. **Overall** -- There was more to see than we were able to get to during our two days at the show. Apple's new hardware was nice, but incremental in its innovation. I give a big thumbs up to the Pioneer clones (and Pioneer's side-by-side demo of a 25-inch, flat panel LCD display - a mere two inches thick!) as the cool hardware for the show, followed closely by the IBVA package. Cool software definitely goes to English Now! Though I'm no expert in computer- aided language instruction, it seemed to me that you could succeed in learning English if you worked through its lessons. Bandwidth and Latency: It's the Latency, Stupid (Part 2) -------------------------------------------------------- by Stuart Cheshire [Last week in TidBITS-367_, Stuart examined issues of latency and delay in typical modem-based Internet communications. This week, Stuart offers general observations on how bandwidth can be used more efficiently and how it effects the overall latency of a connection.] Last week, I asked readers to imagine a world where the only network connection you can get to your house is a modem running over a telephone line at 33 Kbps. Now, imagine that this is not enough bandwidth for your needs. You have a problem. **Making Bandwidth is Easy** -- Technically, the solution is simple. You can install two telephone lines and use them in parallel, giving you a total of 66 Kbps. If you need more capacity you can install ten telephone lines, for a total of 330 Kbps. Sure, it's expensive, having ten modems in a pile is inconvenient, and you may have to write networking software to share the data evenly between the ten lines. But if it was sufficiently important, it could be done. People with ISDN lines already do this using a process called BONDING (which is short for "Bandwidth ON Demand INteroperability Group"), which enables them to use two 64 Kbps ISDN channels in parallel for a combined throughput of 128 Kbps. Getting additional bandwidth is possible, even if it's not always economical. However, equally important is that making limited bandwidth go further is easy. **Compression** -- Compression is an easy way to increase bandwidth. You can apply general purpose compression (such as StuffIt) to the data. Even better, you can apply data-specific compression (such as JPEG for still images and MPEG for video), which can provide much higher compression ratios. These compression techniques trade off use of CPU power for lower bandwidth requirements. However, there's no equivalent way to trade off use of extra CPU power to make up for poor latency. All modern modems utilize internal compression algorithms. Unfortunately, having your modem do compression is nowhere near as good as having your computer do it. Your computer has a powerful, expensive, fast CPU, whereas your modem has a feeble, cheap, slow processor. In addition, as we noted last week, a modem must hold on to data until it has a block big enough to compress effectively. This requirement adds latency, and once added, latency can't be eliminated. Also, since the modem doesn't know what kind of data you're sending, it can't use superior data- specific compression algorithms. In fact, since most images and sounds on Web pages are already compressed, a modem's attempts to compress the data a second time adds more latency without any benefit. This is not to say that having a modem do compression never helps. When the host software at the endpoints of the connection is not smart and doesn't compress data appropriately, then the modem's own compression can compensate somewhat and improve throughput. The bottom line is that modem compression only helps dumb software, and it hurts smart software by adding extra delay. **Send Less Data** -- Another way to cope with limited bandwidth is to write programs that take care not to waste bandwidth. For example, to reduce packet size, wherever possible Bolo (my interactive network tank game) uses bytes instead of 16-bit or 32-bit words. For many kinds of interactive software like games, it's not important to carry a lot of data. What's important is that when the little bits of data are delivered, they are delivered quickly. Bolo was originally developed running over serial ports at 4800 bps and could support eight players that way. Over 28.8 Kbps modems it can barely support two players with acceptable response time. Why? A direct-connect serial port at 4800 bps has a latency of 2 ms. A 28.8 Kbps baud modem has a latency of 100 ms, 50 times worse than the 4800 bps serial connection. Software can cope with limited bandwidth by sending less data. If a program doesn't have enough bandwidth to send high-resolution pictures, it could use a lower resolution. If a program doesn't have enough bandwidth to send colour images, it could send black- and-white images, or images with dramatically reduced colour detail (which is what NTSC television does). If there isn't enough bandwidth to send 30 frames per second, the software could send 15 fps, 5 fps, or fewer. These trade-offs aren't pleasant, but they are possible. You can pay for more bandwidth or send less data to stay within your limited available bandwidth. However, if the latency is not good enough to meet your needs you don't have the same option. Running multiple circuits in parallel won't improve latency, and sending less data won't help either. **Caching** -- One of the most effective techniques for improving computer and network performance is caching. If you visit a Web site, your browser can copy the text and images to your hard disk. If you visit the site again, the browser verifies that the stored copies are up-to-date, and - if so - the browser just displays the local copies. Checking the date and time a file was last modified is a tiny request to send across the network - so small that modem throughput makes no difference. Latency is all that matters. Recently, some companies have begun providing CD-ROMs of entire Web sites to speed Web browsing. When browsing these Web sites, all the Web browser does is check the modification date of each file it accesses to verify that CD-ROM copy is up-to-date. It must download from the Web only files that have changed since the CD-ROM was made. Since most large files on a Web site are images, and since images on a Web site change far less frequently than the HTML text files, in most cases little data has to transfer. Once again, because the Web browser is primarily doing small, modification date queries to the Web server, latency determines performance and throughput is virtually irrelevant. **Latency Workarounds** -- ISDN has a latency of about 10 ms. Its throughput may be twice that of a modem, but its latency is ten times better, and that's the key reason why browsing the Web over an ISDN link feels faster than over a modem. One reason standard modems have such poor latency is that they don't know what you're doing with your computer, or why. An external modem is usually connected through a serial port, and all it sees is an unstructured stream of bytes coming down the serial port. Ironically, the much-maligned Apple GeoPort Telecom Adapter may solve this problem. The Apple GeoPort Telecom Adapter connects your computer to a telephone line, but it's not a modem. Instead, all modem functions are performed by software running on the Mac. The main reason for the criticism is that this extra software takes up memory and slows down the Mac, but in theory it _could_ offer an advantage no external modem could match. When you use the GeoPort Telecom Adapter, the modem software is running on the same CPU as your TCP/IP software and your Web browser, so it could know exactly what you are doing. When your Web browser sends a TCP packet, the GeoPort modem software doesn't have to mimic the behaviour of current modems. It could take that packet, encode it, and send it over the telephone line immediately, with almost zero latency. Sending 36 bytes of data, a typical game-sized packet, over an Apple GeoPort Telecom Adapter running at 28.8 Kbps could take as little as 10 ms, making it as fast as ISDN, and ten times faster than the best modem you can buy today. For less than the price of a typical modem, the GeoPort Telecom Adapter could give you Web browsing performance close to that of ISDN. Even better, people who already own Apple GeoPort Telecom Adapters would need only a software upgrade. **Bandwidth Still Matters** -- Having said all this, you should not conclude that I believe bandwidth is unimportant. It is very important, but not in the way most people think. Bandwidth is valuable for its own sake, but also for its effect on overall latency - the important issue is the total end-to-end transmission delay for a data packet. Remember the example in the first part of this article comparing the capacity of a Boeing 747 to a 737? Here's a real world example of the same issue. Many people believe that a private 64 Kbps ISDN connection is as good (or even better) as a 1/160 share of a 10 Mbps Ethernet connection. Telephone companies argue that ISDN is as good as new technologies like cable modems because though cable modems have much higher bandwidth, that bandwidth is shared between lots of users so the average works out the same. This reasoning is flawed, as the following example will show. Say we have a game where the data representing the game's overall state amounts to 40K. We have a game server, and in this simple example, the server transmits the entire game state to a player once every ten seconds. That's 40K every 10 seconds, an average of 4K per second or 32 Kbps. That's only half the capacity of a 64 Kbps ISDN line, and 160 users doing this on an Ethernet network will utilize only half the capacity of the Ethernet. So far so good: both links are running at 50 percent capacity, so the performance should be the same, right? Wrong. On the Ethernet, when the server sends the 40K to a player, the player can receive that data as little as 32 ms later (40K / 10 Mbps). If the game server is not the only machine sending packets on the Ethernet, then there could be contention for the shared medium, but even in that case the average delay before the player receives the data is only 64 ms. On the ISDN line, when the server sends the 40K to a player, the player receives that data five seconds later (40K / 64 Kbps). In both cases the users have the same average bandwidth, but the actual performance is different. In the Ethernet case, the player receives the data almost instantly because of the connection's high capacity. But in the ISDN case, the connection's lower capacity means the information is already 5 seconds old when the player receives it. The problem is that sending a 40K chunk every ten seconds and sending data at a uniform rate of 4K per second are not the same thing. If they were, ISDN, ATM, and other telephone company schemes would be good ideas. Telephone companies assume all communications are like the flow of fluid in a pipe. You just tell them the rate of flow you need, and they tell you how big the pipe has to be. Voice calls work like the flow of fluid in a pipe, but computer data does not. Computer data comes in lumps. A common mistake is to think that sending 60K of data once per minute is exactly the same as sending 1K per second. It's not. A 1K per second connection may be sufficient _capacity_ to carry the amount of data you're sending, but that doesn't mean it will deliver the entire 60K in a timely fashion. It won't. By the time the lump finishes arriving, it will be one minute old. The conclusion here is obvious: the capacity of a connection has a profound affect on its performance. If you're given the choice between a low bandwidth private connection, or a small share of a larger bandwidth connection, take the small share. Again, this is painfully obvious outside the computer world. If a government said it would build either a large shared freeway, or a million, tiny, separate footpaths, one reserved for each citizen, which would you vote for? **What Can You Do?** I've received numerous messages from people who want to know what they can do to spread the word about these latency and bandwidth problems. I've found that calling modem vendors directly is futile, so I recommend that you circulate these two articles to friends who might find them interesting, and most important, send letters to editors of major magazines asking them to include latency times via ping and traceroute when testing modems for review. Perhaps if we can raise awareness about the horrible latency problems that all modems suffer, modem manufacturers will start putting effort into decreasing latency instead of just increasing throughput. [Portions of this article come from Stuart Cheshire's white paper entitled "Latency and the Quest for Interactivity," commissioned by Volpe Welty Asset Management, L.L.C.] $$ Non-profit, non-commercial publications may reprint articles if full credit is given. Others please contact us. We don't guarantee accuracy of articles. Caveat lector. Publication, product, and company names may be registered trademarks of their companies. This file is formatted as setext. For more information send email to . A file will be returned shortly. For information on TidBITS: how to subscribe, where to find back issues, and other useful stuff, send email to: Send comments and editorial submissions to: Issues available at: ftp://ftp.tidbits.com/pub/tidbits/issues/ And: http://www.tidbits.com/tb-issues/ To search back issues with WAIS, use this URL via a Web browser: http://wais.sensei.com.au/macarc/tidbits/searchtidbits.html -------------------------------------------------------------------