ht://Dig Copyright © 1995-1999 The ht://Dig Group
Please see the file COPYING for license information.
Here are some of the major features of ht://Dig. They are in no particular order.
Intranet searching ht://Dig has the ability to search through many servers on a network by acting as a WWW browser. It is free The whole system is released under the GNU General Public License Robot exclusion is supported The Standard for Robot Exclusion is supported by ht://Dig. Boolean expression searching Searches can be arbitrarily complex using boolean expressions. Configurable search results The output of a search can easily be tailored to your needs by means of providing HTML templates. Fuzzy searching Searches can be performed using various configurable algorithms. Currently the following algorithms are supported (in any combination):
- common word endings
Searching of HTML and text files Both HTML documents and plain text files can be searched. Searching of other file types will be supported in future versions. Keywords can be added to HTML documents Any number of keywords can be added to HTML documents which will not show up when the document is viewed. This is used to make a document more like to be found and also to make it appear higher in the list of matches. Email notification of expired documents Special meta information can be added to HTML documents which can be used to notify the maintainer of those documents at a certain time. It is handy to get reminded when to remove the « New » images from a certain page, for example. A Protected server can be indexed ht://Dig can be told to use a specific username and password when it retrieves documents. This can be used to index a server or parts of a server that are protected by a username and password. Searches on subsections of the database It is easy to set up a search which only returns documents whose URL matches a certain pattern. This becomes very useful for people who want to make their own data searchable without having to use a separate search engine or database. Full source code included The search engine comes with full source code. The whole system is released under the terms and conditions of the GNU Public License version 2.0 The depth of the search can be limited Instead of limiting the search to a set of machines, it can also be restricted to documents that are a certain number of « mouse-clicks » away from the start document. Full support for the ISO-Latin-1 character set Both SGML entities like ‘à’ and ISO-Latin-1 characters can be indexed and searched.
Requirements to build ht://Dig
ht://Dig was developed under Unix using C++.
For this reason, you will need a Unix machine, a C compiler and a C++ compiler. (The C compiler is needed to compile some of the GNU libraries)
Unfortunately, I only have access to a couple of different Unix machines. ht://Dig has been tested on these machines:
- Sun Solaris 2.5 SPARC (using gcc/g++ 2.7.2)
- Sun SunOS 4.1.4 SPARC (using gcc/gcc 2.7.0)
- HP/UX A.09.01 (using gcc/g++ 2.6.0)
- IRIX 5.3 (SGI C++ compiler. Don’t know the version)
- Debian Linux 2.0 (using egcs 1.1b)
There are reports of ht://Dig working on a number of other platforms.
If you plan on using g++ to compile ht://Dig, you have to make sure that libstdc++ has been installed. Unfortunately, libstdc++ is a separate package from gcc/g++. You can get libstdc++ from the GNU software archive.
The building relies heavily on the make program. The problem with this is that not all make programs are the same. The requirement for the make program is that it understands the ‘include’ statement as in
The Berkeley 4.4 make program doesn’t use this syntax, instead it wants
.include « somefile »
and hence it cannot be used to build ht://Dig.
If your make program doesn’t understand the right ‘include’ syntax, it is best if you get and install gnumake before you try to compile everything. The alternative is to change all the Makefiles.
Disk space requirements
The search engine will require lots of disk space to store its databases. Unfortunately, there is no exact formula to compute the space requirements. It depends on the number of documents you are going to index but also on the various options you use. To give you an idea of the space requirements, here is what I have deduced from our own database size at San Diego State University.
If you keep around the wordlist database (for update digging instead of initial digging) I found that multiplying the number of documents covered by 12,000 will come pretty close to the space required.
We have about 13,000 documents:
13,000 12,000 x ----------- 156,000,000
or about 150 MB.
Without the wordlist database, the factor drops down to about 7500:
13,000 7,500 x ---------- 97,500,000
or about 93 MB.
Keep in mind that we keep at most 50,000 bytes of each document. This may seen a lot, but most documents aren’t very big and it gives us a big enough chunk to almost always show an excerpt of the matches.
You may find that if you store most of each document, the databases are almost the same size, or even larger than the documents themselves! Remember that if you’re storing a significant portion of each document (say 50,000 bytes as above), you have that requirement, plus the size of the word database and all the additional information about each document (size, URL, date, etc.) required for searching.
Andrew Scherpbier Last modified: Thu May 6 17:06:10 EDT 1999