Hidden Universes of Information on the Internet
	2. Internet's Information Space

Most of the well-known search engines (e.g. Yahoo, Lycos, AltaVista ) focus primarily on web pages. There is a huge amount of information that is hosted within other kinds of servers/information spaces. Remember, not everyone has learned to develop HTML pages (or has the budget to translate existing online information into HTML)

Telnet - One of the oldest internet applications. Telnet involves a terminal emulation (usually VT100) session where you connect into a host computer and use a text-menu interface. Many libraries (including the Library of Congress) provide a telnet interface to their information systems. The index of Telnet hosts is called Hytelnet. A telnet application is included with windows in the following location: C:\WINDOWS\Telnet.exe. If you don't have terminal emulation software, you can get some through Stroud's site. (After you get the software, you will want to configure your browser to automatically launch your telnet application. In Netscape, select "general preferences" under the "Options Menu"; then click on the "Applications" tab.)

Usenet Newsgroups - This is where Internet users with similar interests exchange messages covering thousands of specific topics (Here is an overview of Newsgroups). Full-text search engines include Dejanews and AltaVista (select newsgroups in the main page pull-down menu) Another great tool to identify which newsgroup cover a specific topic is Tile.net. Tile.net also provides usage statistics about some of the newsgroups. I can't emphasize enough the value of the newsgroups. To learn more about why I think the Newsgroups are probably the Internet's most valuable resource, read my August column in Enterprise reengineering.

Mailing Lists - There are over 90,000 mailing lists where Internet users participate in an active email dialog about their favorite topic. To find a mailing list, try Liszt. To Search the messages in a mailing list, try Reference.com. Reference.com has subscribed to many mailing lists and is developing a searchable archive of all the messages posted to the mailing lists.

FTP File Archives - There are thousands of large file archive sites which only support file retrieval via File Transfer Protocol (FTP). Many public FTP sites will register themselves with the original multi-host index: Archie. If there is a specific filename you are searching for, then Archie is your tool. If you are looking for software that performs a certain function (e.g recipe organizer, Graphics converter, label maker) then look at Shareware.com. Shareware.com is a searchable index of file names and descriptions of over 200,000 files from the major platform-specific FTP archives.

Gopher - A protocol whose menu-hierarchy paradigm (including point and click navigation) became quite popular in 1991-1992, until Mosaic was released and the Web really took off. Gopher continues to be useful for organizations who have a bunch of existing text documents and files they want to share without HTMLing them all. The searchable index of Gopher sites is called Veronica. There is also a good subject tree (like a Yahoo) of gopherspace called Gopher Jewels.

World Wide Web - AltaVista and Hotbot may seem to cover a vast majority of the Internet's web pages, but they do have significant blind spots:

• They obey The Robot Exclusion Standard. The web site being visited by a search engine's spider (robot) the web site can limit which directories and files the search engine is allowed to "harvest" (copy and index) A web site may contain a file called "robots.txt" which spells out where a robot is allowed to go, and even which robots are allowed to copy a site. For example, read these "robot.txt" files located at www.disney.com/robots.txt and www.cisco.com/robots.txt. You can even tell which search engines Yahoo keeps out in their robots.txt file. See Robot Exclusion Standard Revisited for details.
• Some sites require a password. If you have to be a registered user to access a site - then that site's content will be missed by the search engine. For example, the vast majority of the Wall Street Journal is not searchable through the search engines. (although the search engine can help you discover the existence of The Wall Street Journal)
• Many web pages are generated by database queries. For example, at Amazon Books most of the information about 1 million books exists in their database. The web pages describing a particular book, author, publisher, etc are only created in response to User queries. The search engines are very good at discovering pre-defined web pages by following hyperlinks - but they can't interact with web site databases by filling-in an online query form.

Summary:

Go ahead and use the popular search engines to explore the web, but do not rely on them exclusively. To reach the full edges on online information, you must explore other online information spaces, and use search tools within specific sites. With any online tool you use, be sure you understand whether it is a subject tree or a search engine.

Note for my Alumni: Your referrals are always appreciated