TidBITS#588/16-Jul-01 ===================== Survival is on our mind this week, as Geoff Duncan relates how his broadband provider abruptly went dark, and how you can live through a similar event in today's world of uncertain ISPs. Plus, Adam looks at why online grocer Webvan failed to spot its own rotted fruit before ceasing operations. In the news, we note the releases of Panorama 4.0 and Mailsmith 1.1.7, and celebrate Adam's third place rank in the 2001 MDJ Power 25. Topics: MailBITS/16-Jul-01 Where Webvan Went Wrong Surviving Your ISP's Darkest Hour Copyright 2001 TidBITS Electronic Publishing. All rights reserved. Information: Comments: --------------------------------------------------------------- This issue of TidBITS sponsored in part by: * READERS LIKE YOU! You can help support TidBITS via our voluntary <- NEW! contribution program. Our thanks this week to Robert Lombaert, Bertrand Cahuet, and Michael Perry for their generous support! * APS Tech -- 800/395-5871 -- Burn a full CD in less than five minutes with the APS CD-RW 16x10x40 FireWire Plus. FireWire and USB ports let you easily connect to all recent Macs. Order at: * WinStar Northwest Nexus. Visit us at . Internet business solutions throughout the Pacific Northwest. * Small Dog Electronics: PowerLogix Macworld Expo Specials! <-------- NEW! PowerForce G3s! 350 Mhz: $129; 400 MHz: $192; 500 MHz: $289 PowerForce 450 MHz G4 PCI or ZIF: $375! iForce 500 MHz: $412 466 MHz BlueChip: $249! -- 802/496-7171 * Bare Bones Software BBEdit 6.1 -- Built for Mac OS X The award-winning HTML and text editor is now native for Mac OS X. Buy, upgrade, or download the demo at our Web site: It doesn't suck. * Protect Yourself from Internet Intruders! IPNetSentry is the <----- NEW! intelligent, easy-to-use way to deter hackers from your Internet-connected Mac. Rated 5 "locks" by SecureMac.com. Just $35 from Sustainable Softworks --------------------------------------------------------------- MailBITS/16-Jul-01 ------------------ **TidBITS Publisher Ranks Third in MDJ Power 25** -- In the now-annual survey of industry insiders coordinated by Macintosh publication MDJ, TidBITS publisher Adam C. Engst once again placed in the top five most influential figures in the Macintosh industry. This year Adam claimed third place, coming in behind Apple CEO Steve Jobs (no surprise there!), and giving up last year's second place spot to Apple's head of software engineering, Avie Tevanian, the man most directly responsible for Mac OS X. Rounding out the top five were Microsoft chairman Bill Gates in fourth place and Apple's hardware design virtuoso, Jonathan Ive. Congratulations to Adam and the others recognized in the MDJ Power 25! [GD] **ProVUE Ships Panorama 4.0 Database** -- Longtime Macintosh developer ProVUE Development has released Panorama 4.0, the latest version of its RAM-based database application. Panorama's been around since 1988, and has always featured blinding performance because it stores working data in RAM rather than on comparatively slow hard disks. This might sound risky, but Panorama's always been so stable that it rarely poses a problem, and the performance benefits are substantial. Back when high-end Macs had 8 MB of RAM, Panorama could be expensive and resource-intensive; today, RAM is plentiful and cheap, so Panorama's performance benefits are more accessible than ever. Panorama 4.0 has been almost completely re-engineered, and the application is now PowerPC-native (previous versions had featured hand-tweaked 68K assembler code, making it difficult to port to PowerPC) and for the first time a version is available for Windows - all Panorama's files are fully cross- platform and fully backward-compatible. Version 4.0 also sports improved development tools including a formula wizard, a debugging wizard, online references, improved editing tools, timer-based events, and extensions to Panorama's internal programming language. Panorama costs $300 (upgrading from version 3.x costs $70), and requires a PowerPC-based system with Mac OS 7.6 or later, including Mac OS X's Classic environment, and at least 16 MB of RAM (the more the merrier!). [GD] **Mailsmith 1.1.7 Available** -- Bare Bones Software has released Mailsmith 1.1.7, a small update to its email client application. Version 1.1.7 improves support for SMTP AUTH (a protocol which enables validated users to send mail even if they aren't connecting to their server from a trusted address), the capability to change the ports Mailsmith uses for POP and SMTP (useful if you're trying to tunnel via SSH), as well as changes which let Mailsmith function better under Mac OS X's Classic environment. As usual, Bare Bones has made a complete list of changes available online. The update is a 2.7 MB download and free to registered Mailsmith users. [GD] Where Webvan Went Wrong ----------------------- by Adam C. Engst Last week's demise of Webvan came as absolutely no surprise to Tonya and me, since we'd been Webvan customers - for a while - after their acquisition of HomeGrocer a year ago. We'd seen the differences in the way the two companies did business, and while HomeGrocer certainly had an uphill battle to survive, Webvan seemed set on driving the company out of business. Here are a few of the many places they went wrong, particularly in comparison with HomeGrocer, the Seattle-based service that got us turned on to Internet grocery shopping in the first place. **Too Much Money to Burn** -- I've seen differing numbers for the amount of money that Webvan burned through, but it's between $800 million and $1 billion. That's a lot of money, and is indicative, I think, of both the exuberance of the Internet investment community when Webvan started and the belief that any sort of shopping could be done better on the Internet than in person. Significant investment is normally a good thing, but in this case the vast sums acted to Webvan's detriment, not to mention the detriment of the entire industry. Webvan used its money in two basic ways: to build up an expensive infrastructure and to expand rapidly across the country. Webvan spent huge sums on high-tech warehouses that were designed to revolutionize distribution, but they turned out to be mostly a waste of money. The problem is that all the technology was meant to reduce labor costs, and labor is relatively cheap. Worse, Webvan designed the warehouses so they could scale to 8,000 orders per day, but that's a lot of unnecessary expense when you're receiving less than half that many orders. So Webvan would spend something like $35 million on a warehouse, whereas HomeGrocer spent only about $15 million for a much less automated warehouse - you can buy a lot of labor for $20 million. The fancy warehouses didn't even necessarily work better. For instance, Webvan created an automated freezer room that required only a single employee to pick items for customer orders. But since the freezer room was in fact _freezing_, no one could stay in there for more than a few minutes without suffering hypothermia. In contrast, HomeGrocer's low-tech freezer rooms worked fine, since the pickers could quickly run in and out to get the necessary items and stay warm in the process. **Infinite Expansion Creates Infinite Dilution** -- The grandiose expansion plans Webvan executed even in the face of the dot-com bubble bursting were problematic at best. Most significantly, they put pressure on competing Internet grocers in those markets. In an established, profitable business, pressuring competitors in key markets makes sense, but in a situation where everyone is losing massive sums of money in attempts to gain market share, forcing head-to-head competition just makes it all the more likely that everyone will fail. Numerous high-flying Internet grocers such as ShopLink, Streamline, and, most recently, HomeRuns have fallen by the wayside in vain attempts to compete with one another while trying to set themselves apart from the traditional grocery stores. The incredibly complex logistics surrounding Webvan's expansion plans also made it difficult for management to concentrate on the basic business of serving the customer. (In this case, I'll give them the benefit of the doubt and assume they were distracted, not just incompetent, though as you'll see, opinions vary on that count.) We'd had essentially no complaints with HomeGrocer, particularly in terms of customer service, where they always answered their email promptly and were great about providing refunds for the occasional mistake or damaged food. As Webvan took over, our exchanges with customer service gradually became more and more generic, until the last few, which disappeared into the ether. The final straw for us, though, was when produce quality started to suffer. HomeGrocer employees had always done a good job at picking good produce, so you didn't feel as though you were losing anything by letting them pick out your peppers and apples. After we received an entire bag of rotten oranges and were subsequently ignored by customer service, we decided to patronize local grocery stores once again. It's entirely possible that Webvan was buying inferior produce in an attempt to save money at that point, but one of our drivers said that orders were being picked by temporary employees with no incentive to do a good job. **Merge and Die** -- Acquiring HomeGrocer was also a mistake. Though it made sense on the surface, Webvan botched the acquisition almost entirely, failing to merge the organizations in some ways and overriding HomeGrocer's leaner approach in others. One painfully obvious mistake was eliminating HomeGrocer's widely recognized peach logo on the delivery trucks frequenting Seattle's congested freeways. The peach immediately conveyed the idea of delivering fresh food, whereas Webvan's unremarkable, characterless "W" logo indicated, well, nothing. Tonya and I jokingly awarded each other "peach points" for being the first to spot a HomeGrocer truck while driving; after the change, we mostly didn't even notice Webvan trucks. And as we complained to our drivers after the trucks were repainted, they threw away a brand that even toddlers like Tristan recognized. If it looked bad from the outside, it was worse inside. HomeGrocer founder Terry Drayton, who left HomeGrocer a month before the acquisition, has been widely quoted as saying, "All I can say is that I am astonished at how staggeringly incompetent [the Webvan management has] proven to be. In our wildest dreams we never imagined that they would be this bad." After the acquisition, morale among the HomeGrocer employees dropped precipitously, to the point where they were openly disgusted with management changes. In the early days of HomeGrocer, the drivers were excited by what they were doing, and that excitement encouraged customers to have faith in the then-unusual notion of buying groceries online. The difference in attitude after the acquisition was particularly shocking. **Was Survival an Option?** If Webvan had held onto much of its money, spent the remainder wisely, concentrated on its original San Francisco area market, and expanded carefully once it had perfected its model, the company might still be around today. The drive to capture market share that was so prevalent in the exuberant days of Internet commerce makes sense in some fields, but in the grocery field, where margins are razor-thin, it's difficult to see how an unprofitable business model can easily be turned into a money-making one if only there are enough people ordering - 750,000 in Webvan's case. (It's a perfect example of the saying: "We lose money on every sale, but we make up for it in volume.") More interesting is the question of whether HomeGrocer could have survived if the acquisition hadn't happened. The company was still losing money at the time Webvan came knocking, but customer loyalty in Seattle was extremely high, they had great brand recognition, and they hadn't lost sight of the fact that their customer service had to win over people who were utterly accustomed to visiting physical grocery stores every week. Terry Drayton has even talked about bringing HomeGrocer back, and although we're no longer in Seattle to take advantage of it if he does, I'd certainly encourage him to give it a try. Despite the tremendous failures of Webvan and so many other Internet grocers, it strikes me that the lesson is not that Internet grocery shopping can't succeed, but that it requires tremendous care and attention to detail when working out the business model. As indication that it's here to stay, look no further than traditional grocery stores, which are continuing their limited forays into Internet grocery shopping. Albertsons has slowly expanded their coverage for delivery of orders place over the Internet, the Dutch grocery chain Royal Ahold has a controlling stake in Peapod, and Safeway has a significant investment in Texas-based GroceryWorks. GroceryWorks also just received more money from the UK's Tesco supermarket chain, which apparently has done a good job of making its Internet delivery service profitable. In fact, the moral may be that creating a new distribution network and stocking warehouses simply costs too much when much of the infrastructure is already available from existing supermarket chains. That may be bad news for Terry Drayton in any attempts to revitalize HomeGrocer, but I think the communities that can take advantage of Internet grocery shopping will appreciate the services no matter who provides them. Surviving Your ISP's Darkest Hour --------------------------------- by Geoff Duncan Since 1996, I've connected my home network to the Internet via an ISDN line, making me an early adopter of the "dedicated Internet access from home" concept. (Adam & Tonya beat me to the punch: they got their first dedicated frame relay line in 1994.) I've also run my own servers - including Web, email, and DNS - on a small collection of Macs, and I host several domains and services for my clients, including TidBITS. If you've ever searched TidBITS, read an article using a GetBITS URL, read TidBITS using AvantGo on your Palm device, or browsed the TidBITS Talk archive, you've accessed Macs in my office closet six feet from where I sit. **The Lure of Broadband** -- A 128 Kbps ISDN offers just over twice the bandwidth of a 56 Kbps modem, so you might think my connection often bogs down. Over the last five years or so, this hasn't really been true: the line might be saturated for 10 or 15 seconds at a time, but over the space of an hour my systems typically use 5 to 8 percent of the available bandwidth. Nonetheless, I wanted _more_ bandwidth: in addition to my Internet-related work, I'm a session musician. In the last year I've taken some tiny steps into the world of digital audio, and on occasion hundreds of people inexplicably decide to download MP3 files from my servers. More significantly, I sometimes need to transfer hundreds of megabytes of individual audio tracks to and from clients. Those gigantic file sets can take several hours to transfer over the ISDN line, and they _do_ bog down services I host for TidBITS and other clients. However, I had few connectivity options. DSL isn't available in my area, and my landlord won't let me install the extra pairs of wires which would be necessary for frame relay - assuming I could afford frame relay. Similarly, megabit wireless options exist, but are currently too pricey to consider, and I'd apparently have to move to a new apartment to get a proper line of sight - in which case, I might as well move to where I can get cheap DSL. So, this spring, I bit the bullet and ordered high-speed service from a local residential broadband provider who had set up service in my area. I'd been watching the company for almost a year, and even though the ISP market was shrinking, they had been able to secure additional funding and expand their services, plus their customers were generally giving them good marks. The idea was slick: they brought in a high-speed link to "captive" sites like apartments or condominiums without other broadband options, then (in my case) used nifty gizmos from Tut Systems to run Ethernet a few hundred yards over phone lines from their high-speed connection to individual residences. Once installed, the system worked great: I was able to transition my network with a minimum of fuss, and suddenly I was getting up to 2 Mbps of bandwidth, or 16 times the bandwidth of my ISDN line. Those big audio files now took minutes to transfer, and I spent a fair bit of time gloating to all my friends and colleagues. I also gleefully cancelled my ISDN line, removing another major thorn from my side: the local telephone monopoly, Qwest Communications. But like all happiness, my broadband experience was fleeting. After about six weeks (and despite my careful research) the broadband ISP shut down with virtually no warning. My network and everything on it (including my personal email, my client's email, and the services I run for TidBITS and everyone else) dropped off the face of the earth. TidBITS readers might remember a weekend at the beginning of March where many items on the TidBITS home page were inaccessible: that was when my provider went dark, and my blood pressure could be measured on the Richter scale. **Into Thin Air** -- My experience is not unusual in the present economic climate. The market for ISPs in the U.S. has been shrinking for a few years; at first, this decline was mainly due to mergers and acquisitions, as large companies swallowed up smaller providers in an effort to acquire customers, revenues, and facilities. However, the ongoing dot-com bust (and the concomitant bankruptcies) have further thinned the ranks of ISPs at all levels, from the local mom-and-POP providers serving small areas to enormous regional and national services. As the industry consolidates, most people can expect still fewer Internet providers to be available to them, while the number of people with Internet access continues to increase. A surprising number of people have been left scrambling for Internet access, email addresses, and connectivity with little or no warning. For some folks, having their ISP vanish overnight may not be much of an inconvenience, but for others it can be a serious disruption of business, personal communication, and services. Simply moving to a new email address can disrupt your correspondence for weeks or months as people and businesses gradually update nicknames, address books, and other databases: the longer you've used your current email address, the more disruption changing it will cause. Similarly, if your ISP hosts your Web site, you may simply lose the pages, images, scripts, and other data you created for the site, unless you've been scrupulous about your backups. (You _are_ scrupulous about backups, right?) It's worse for other folks. Consumer-oriented broadband technologies like DSL have spawned thousands or even millions of comparatively small networks with full-time Internet connections. Some are run by hobbyists who think it's nifty to be online all the time; others represent small- and medium-sized businesses or organizations who can't afford (or who can't get) dedicated commercial Internet access. When their ISPs vanish, their entire networks can completely disappear from the Internet. Further, their ability to switch to another provider might vanish with the darkened ISP, particularly if the ISP was hosting their domain or provided email service. I'm a somewhat extreme example of this group, since I not only do everything myself (all the way down to DNS), I create and sell custom online services to clients. So when my connection died, _everything_ I do went with it. **Outliving Your ISP** -- The process of rescuing my network from the bit bucket has led me to draw up these suggestions for folks who want to be ready in the unfortunate event their ISP closes up shop, is swallowed whole by an inferior company, or stops offering you service. Some of these details are more specific (and more technical) than others, but they should serve as a starting point for preparedness - and put you miles ahead of where I was when my LEDs went out. For most users, I recommend the following: * Get alternative email addresses. If your ISP vanishes, you'll need email addresses separate from your ISP or (if you run your own network) your own mail server. Web-based email may be helpful, even though it's awkward compared to using a real email program. Numerous companies like Pobox, Bigfoot, Yahoo, HotMail, and Mail.com offer forwarding services and free email accounts. If you've set up an iTools account, Apple's Mac.com email service is an option, although I personally can't connect to it very often. I'd recommend at least two alternative addresses since there's no telling how long email providers will be around. For instance, USA.net recently announced plans to charge users because its advertising revenue wasn't adding up; if your email provider relies on ad revenue, they're probably feeling the same pinch. * Get alternative dial-up access. If you switched to a broadband connection, you may have cancelled your dial-up Internet access, or have dial-up access through the same ISP that's running your broadband connection. Either way, you need to secure modem-based Internet access that's independent of your ISP. You can, of course, pay another provider for dial-up access you're rarely (hopefully never) going to use, but at typical prices of $150 to $300 a year, it might not be economical. One option might be to make arrangements to "borrow" the use of a dial-up account belonging to a friend, business associate, or family member for use in emergencies; obviously, this person would have to be comfortable knowing that you won't snoop through any waiting email, change their password, or do other unexpected things with the account. Another option might be a "group account," where a handful of people split the costs of a backup dial-up account with a reliable provider. This is what I've done with a small group of local folks: we each chip in $20 a year to maintain a dial-up account with a nationwide provider, both as an emergency backup and for occasional travel use. * Research alternative broadband providers in your area. Broadband Internet access will remain spotty in most locales for years to come, and many people only have one broadband option, if they have any options at all. If you have several choices - or if you're able to purchase high-speed access from a phone company but get Internet routing from other ISPs - investigate those options _before_ you have to make a choice on short notice. Check out the pricing and features of other ISPs and broadband options, and keep an eye out for new alternatives in your area: perhaps a neighborhood wireless network is taking shape, or satellite access from a company like Starband is a possibility. In particular, take into account promised installation times, any necessary (or mandatory) hardware, and what sort of service guarantees are offered (if any). Some options won't be cheap or technically viable, but even if you think your current provider is stable (I did!), it can't hurt to check the details every few months. At least if you have to move in a hurry, you'll be able to move in the right direction. For the first two points, make sure you collect (and save!) the information you'll need to use these addresses and/or accounts _without_ first being able to connect to the Internet. The most important items are probably passwords, local dial-up numbers, the names of mail servers, and the IP addresses of domain name servers you would need to use. In addition, have handy any account numbers or other information you might need to get telephone support in an emergency. For folks connecting their local networks to the Internet and/or running their own domains, I also recommend the following: * Be prepared to live with few IP addresses. When I first connected my network to the Internet in 1996, my ISP gave me a block of 64 IP addresses without blinking; these days, it's rare for consumer broadband users to be able to get more than a few static IP addresses, if any at all. The most common way to cram a bunch of machines into a small IP address space is to use a router with Network Address Translation (NAT) which acts as a gateway for machines on a private network hidden behind the router. Although NAT is beyond the scope of this article, stand-alone NAT routers are available from companies like Netopia and Asante. You can also easily leverage an older Macintosh to serve as a NAT router programs like TidBITS sponsor Sustainable Softworks' IPNetRouter or Vicomsoft's Internet Gateway. If you use a Mac, you'll get the best results (and better security) using two (or more) Ethernet cards to separate the private network from machines directly visible to the Internet, so plan to have some extra Ethernet hardware on hand. * Arrange for a backup location or address space. If losing your ISP also means losing your Internet connection, make arrangements ahead of time to relocate your most important services. This may mean hauling a few machines to a friend or colleague with a dedicated Internet connection, or making arrangements with a local hosting company which can co-locate your machines for a short time. Even if you can't finalize deals, knowing your options can be useful, enabling you to get crucial servers back online in hours instead of days. The possibility of bankrupt ISPs isn't the only reason to explore these options: at this very moment I'm providing a temporary home to a colleague's mail server because a kitchen fire forced him out of his home (and away from his Internet connection). * If you manage domains, be sure you can change your domain records without being able to send and receive mail from a particular email address. If your ISP goes dark, you may lose access to your primary email address: if your registrar will only process changes it can verify as having been sent from that address, you may be in a heap of trouble. Depending on the company managing your domain data, you may merely need to know the passwords to Web-based administration tools. In the case of Network Solutions, you may have to negotiate rather cryptic steps to make sure your PGP keys are on file with them and that you can use them successfully (for example, you probably won't be able to use PGP software which integrates with your email program). Be sure to test your ability to make changes: I usually recommend harmless modifications - such as expanding or contracting a word like "drive" or "street" in your contact address - so you know your changes have been successfully processed. **What Did I Do?** Since my provider ran a private network and handled both my Internet routing and my physical connection, their failure was particularly gruesome for me: I couldn't simply pay another ISP to handle Internet service on that provider's high- speed system. So, I had to revert back to an ISDN-based Internet connection - both because I already had the equipment to use ISDN, and because ISDN remained my only viable option. That meant again dealing with the local telephone monopoly; as usual, that proved to be a lengthy, unsatisfying process which is still punctuated by bouts of incompetence, internal miscommunication, billing errors, and flat-out mistakes. It took Qwest four weeks to re-activate my ISDN line - but at least their personnel have been uniformly polite to me this time around, an improvement over some of my past interactions with the company. So, for four weeks, my only Internet connectivity was a modem which, since the Seattle area earthquake last February, typically only achieves 19.2 Kbps connections. I connected to the Internet using borrowed dial-up accounts; my preference was for a local provider, since (unlike EarthLink, MSN, and other major providers) they would let me send and receive email without using their mail servers. What about my servers, which handle services for TidBITS and some of my other clients? About two hours after my provider went dark, five machines were in the back of my car, headed to Adam & Tonya's previous house outside Issaquah, Washington. Adam & Tonya only had a 56K frame relay connection to their home - considerably less bandwidth than even my ISDN line - but they'd established their network back in the days when IP numbers roamed freely in large herds across the undeveloped habitat of the Internet. They had unused static IP numbers coming out their ears: I borrowed five, plugged everything in, and essentially waited two days for Network Solutions to propagate changes to my domain records correctly. When I was finally able to re-establish ISDN service, I moved the servers back home - which was, fortunately, before Adam and Tonya sold their house. **It's Only a No-Brainer If You Think About It** -- If you've gone to the trouble to set yourself up with a broadband Internet connection (let alone the trouble of hitching up a network and running your own servers), many of these points probably seem obvious. After all, you had to cope with these details (and more) just to get up and running. But you were probably thinking in terms of converting _to_ broadband server, rather than what to do should that service _fail_ - and it's easy to let details slide once things are operating correctly. Consider how few people back up their data because their computer seems to be reliable, only to find themselves without their critical documents and information when their system breaks down, or if their computer is damaged or stolen. The same principles apply to Internet connections: it might be working fine now, but a connection can vanish with little or no warning. Forewarned is forearmed. $$ Non-profit, non-commercial publications may reprint articles if full credit is given. Others please contact us. We don't guarantee accuracy of articles. Caveat lector. Publication, product, and company names may be registered trademarks of their companies. This file is formatted as setext. For more information send email to . A file will be returned shortly. For information: how to subscribe, where to find back issues, and more, email . TidBITS ISSN 1090-7017. Send comments and editorial submissions to: Back issues available at: And: Full text searching available at: -------------------------------------------------------------------