GoServe 1.20 GoServe -- A Gopher Server for OS/2 2.x """"""""""""""""""""""""""""""""""""""" Copyright (c) International Business Machines Corporation, 1993, 1994. All rights reserved. Introduction """""""""""" 'Gopher' is a client-server protocol designed for text location and retrieval. It was designed at the University of Minnesota, and is widely used; both clients and servers exist for most significant operating systems. The Gopher protocol is described in documents distributed by the University of Minnesota Microcomputer and Workstation Networks Center. 'GoServe' is a Gopher server for OS/2. It is assumed that the reader is familiar with Gopher concepts, and has access to a Gopher client. GoServe allows processing of a variety of Gopher requests, using a simple REXX filter to allow for customization. Multiple servers can be started (using different ports), and an audit trail of requests and actions can be recorded. While running, a Presentation Manager display of GoServe activity is shown. For control, a limit may be placed on the number of concurrent requests that will be handled (hence limiting the load on the server machine). Similarly, timeouts may be set to close connections after a chosen total or inactive time. Certain GoServe operations can be controlled remotely, if desired, using any Gopher client (or a utility OS/2 program or CMD file). Goserve is a 32-bit OS/2 application. It requires TCP/IP for OS/2 to be installed and operational (either on a real network or using the loopback driver). ----------- Mike Cowlishaw, IBM UK Laboratories How GoServe is used """"""""""""""""""" When GoServe is started, it sets up a TCP/IP socket on port 70 (this is the 'well-known' port for Gopher, but can be changed--see below), then waits for requests from clients. When a request from a Gopher client arrives, it is recorded in the audit file (if desired) and then the GoServe filter for the port is called. The GoServe filter is a program written in REXX, which is given details of the client and the selector string or strings. The filter locates the information (usually a file, containing a menu or a document) to be sent to the client, and returns details to GoServe. GoServe then sends the data to the client and closes the TCP/IP connection. Each incoming request is given a new thread for communication, filter processing, and response. GoServe handles all the details of threads and TCP/IP communications, so the installer need only be concerned with the filter (which can be very simple) and the data (menus and documents). Installation """""""""""" Only two files (plus data) are needed to run GoServe: the GOSERVE.EXE program file, and the REXX filter. The filter must have filename GOFILTER, with an extension that is the TCP/IP port number to be used (for example, 'GOFILTER.70' for the filter for the default Gopher port). It is recommended that GOSERVE.EXE and the REXX filter be placed in one directory (for example, 'D:\GoServe'), and data for the server be placed in another (for example, 'D:\GoData70') or its subdirectories. The REXX filter should be in the working directory for GoServe, and any audit file will be written to that directory. The following files are included: goserve.doc -- this document goserve.exe -- the server program gofilter.70s -- sample filter (rename or copy as GOFILTER.70) gosmenu.cmd -- menu maker utility (see below) TCP/IP 1.2.1 (or later) must be installed; the "0.0024" (or later) Corrective Service Diskette (CSD) for TCP/IP is required for TCP/IP version 1.x.x. Over time, other CSDs may become available--if TCP/IP problems are suspected, always try the latest CSD. Filter programs """"""""""""""" Filter programs are written in REXX. Based on the arguments they are passed, they decide on the menu or document to be sent to the client and then return a result that defines that data to the server. Three argument strings are passed to the filter: 1. The source of the request. This has four words [more words could be added at the end in the future], separated by one or more blanks: a) the Internet address (primary host identifier) of the server's machine or cluster (in numeric form) b) the server's port number used for the connection c) the transaction number (a number, incremented for each new request, that starts at 0 when GoServe is started) d) the Internet address of the client (in numeric form). Example: "9.15.11.189 70 101 9.20.25.65" 2. The selector string. If this is an empty string, it is the initial contact from a client, and the filter would normally respond with a main menu. Otherwise, it will (or should) be the selector string from a Gopher menu line previously sent to the client, unchanged. You have complete freedom in setting selector strings in menus, and deciding how the filter should respond depending on the string received. Often, the name of a file is used, and that file is sent to the client. [Note: it is recommended that fully qualified filenames NOT be used for selector strings.] 3. The extended selector string. Some Gopher clients can add additional information to a request, attached to the selector string after a Tab character. GoServe discards the Tab character and presents the remainder as the third argument to the filter. See the Gopher protocol documents for more information on how the extended selector string is used. The filter can specify that GoServe send either a file or a single string to the client, or request certain other GoServe operations. This is done by returning a result string to GoServe. This string takes one of several forms (in which keywords and file specifications may be in lower, upper, or mixed case): 1. FILE [BINARY|TEXT] [ERASE] NAME filespec The file named by 'filespec' will be sent to the client. 'filespec' should normally be a fully qualified name (if it is not, GoServe will look for it in its working directory). It may not include an 'upwards reference' sequence ("..\"), as these could allow clients access to any file on the server machine. The optional keywords may be specified, in any order, and have the following effects: BINARY -- the file is a binary file; after sending, no Gopher terminator line will be sent (and any trailing DOS end-of-file character will not be stripped off). GoServe will close the TCP/IP connection as soon as the complete file has been sent. You should use this option for any files that are not simple text, for example ".ZIP" files, etc. TEXT -- The file is a text file: any trailing DOS end-of-file character will be stripped off, and a Gopher terminator line will be added. This is the default. ERASE -- the file is a temporary file, and should be erased after being sent. [The transaction number can be used for generating a safe name for a temporary file.] 2. STRING string The single string is returned to the client [along with added final carriage return, etc., as demanded by the Gopher protocol]. This can be used to return a simple message to the client. Multiple lines can be sent, if necessary, by embedding a CR-LF sequence to separate lines. A CR-LF is not needed at the end of the string. 3. CONTROL options A GoServe action (such as requesting statistics, resetting counters, or moving the audit file) is initiated. This is used for controlling the GoServe server from a remote client or from another process on the same machine--for details, see the section "Remote control of GoServe". A document is returned, indicating success or failure of the request. Invalid options on CONTROL are not considered a failure of the filter. 4. Null string/no string If a null string, or no string, is returned by the filter, a one-line "no information available" document is sent to the client. Any other response from the filter is not allowed; for safety, any failure of the filter is considered catastrophic and will cause GoServe to end. Here is an example of a 'minimal' filter, GOFILTER.70: /* Sample GoServe filter program */ parse arg source, selector, selector2 /* Get arguments */ parse var source server port transaction who . /* Often useful */ drive='d:' /* Data drive */ dir=drive'\godata'port'\' /* Data root directory (e.g. "d:\godata70\") */ /* This very simple filter just builds the full path name of a file from */ /* the selector and returns that. Note we do not put the full path in */ /* menus, as this could allow clients to access all files on the server */ /* machine, which is rarely desirable. */ if selector='' then selector='mainmenu.'port /* 'startup' case */ if translate(right(selector,4))='.ZIP' /* use Binary for ZIP files */ then data='Binary' else data='Text' return 'File' data 'name' dir''selector Notes: 1. Remember that GoServe can run multiple copies of the filter at once, if more than one client connects at the same time. 2. By default, each copy of a filter may have up to five files open at once. This limit may be increased using the Limits selection dialog (see below). 3. The results of SAY instructions (or tracing) in the filter program may be seen using the PMprintf package. However, REXX interactive tracing is not supported. PMprintf should be available from the same source as GoServe. 4. You can execute OS/2 commands from the filter using the Address CMD instruction, for example: address cmd 'start e foo.bar' 'call d:\foobar\test.cmd argstring' Use caution when calling external programs: any slowness directly affects the response time to the client and its user. 5. On the Internet, you can get some useful sample filters and search engines from the IBM Almaden Research Center Gopher server, at index.almaden.ibm.com. Look in the "OS/2 Gopher Client and Server" menu. Inside IBM, the same samples are available on the IBM TCP/IP network, from n6tfx.almaden.ibm.com; again, look in the "OS/2 Gopher Client and Server" menu. Running GoServe """"""""""""""" GoServe may be started in any of the usual ways, such as from an OS/2 command line or from a Program Reference object. By default, GoServe uses the standard Gopher port (port 70). Once GoServe has started, a PM window should appear. If the window is large enough, it will include at the top a bar display that shows any client activity (the full width of the bar corresponds to the maximum number of clients allowed). A half-height grey bar indicates the peak activity seen since GoServe was started. This may be reset to zero by selecting 'Reset peak indicator' from the 'Options' menu, and will be shown in purple if any errors have been counted. The second line of the window gives the total number of transactions initiated since GoServe started, and the count of errors that have occurred (if more than zero). If room, it shows on the right the instantaneous count of connected clients and the peak number recorded since GoServe started (if more than one). The transaction count may be reset to zero by selecting 'Reset transaction count' from the 'Options' menu). Similarly, the error count and peak count may also be reset to zero. The third line shows the number of bytes, thousands of bytes, or millions of bytes sent and received since GoServe started. These counts may be reset to zero by selecting 'Reset byte counts' from the 'Options' menu. The 'Options' menu also offers 'Reset all counts', which resets all of the counters. The fourth line of the window displays two timestamps; on the left is shown the time when GoServe last entered the 'idle' state (that is, was waiting for clients). On the right is the time when GoServe was started (or when the last 'Reset All' was done). Some or all of these items will be omitted if the window is too small for satisfactory display. You can get more in a small GoServe window by clicking on 'Use menu bar' on the 'Options' menu--this will remove the menu bar from the display. To use the menu after it has been removed, click anywhere on the GoServe window with mouse button 2, in the usual Workplace Shell manner (this works even if the menu bar is visible). Two other menu options are supplied: 'Audit selection' lets you control the auditing of transactions, and 'Limits selection' is used to control the operating limits of the server. These are described later. Parameters on the GoServe command """"""""""""""""""""""""""""""""" Keyword parameters may be used when starting GoServe to control the TCP/IP port used and the behaviour after a serious failure. 1. The PORT parameter The default port number [70] may be changed for special applications by using the PORT parameter when starting GoServe, for example: start goserve port 71 Servers started with different ports are independent; they use different filter programs, and maintain a different collection of settings, window position information, audit file, etc. Note that the port number is used as the required file extension for some files--for port numbers greater than 999 you must therefore be sure that the working directory for GoServe is on a disk that supports extensions of more than three characters (for example, an HPFS disk). 2. The QUIETFAIL parameter Certain "catastrophic" errors (such as memory allocation errors, or an error in the REXX filter) are considered failures by GoServe; that is, they cause a message box to be displayed on the screen--once the message box is acknowledged, GoServe ends. The QUIETFAIL parameter can be used to prevent the message box being displayed: start goserve quietfail In this case, GoServe will end immediately after a failure (after auditing the failure, as usual). GoServe also accepts some experimental keyword parameters. 'TRACE' copies audit information to the PMprintf window (without GoServe timestamps); this does not affect auditing to the audit file (see below). 'DIAG' shows TRACE information together with additional details of GoServe operation (also using PMprintf). This diagnostic information is intended for development use and may change over time. If desired, more than one parameter can be specified; the parameters may be in any order. The limits selection dialog """"""""""""""""""""""""""" The 'Limits selection' dialog lets you change various limits which apply to running GoServe. The limits are grouped into two sets: limits affecting clients, and timeouts. The Clients set has three limits: 1. The maximum number of clients allowed concurrently [default 10]. Additional clients will be refused connection. This count lets you limit the maximum load on your server--especially useful if it is also being used as a personal workstation. 2. The percentage at which the bar chart will show a red warning indicator [default 75%]. This gives a visual indication of heavy use of your server (the peak use is also shown on the bar display). 3. The maximum number of open files allowed, per client filter [default 5]. Increase this if you expect your GoServe filter to have more than five files open at a time. The Timeouts set also has three limits. The first two timeout settings control the time at which any uncompleted response may be terminated automatically by GoServe's "watchdog" thread. The first timeout setting is the 'inactive timeout', in seconds; after at least the specified time without any data being sent or received the connection will be closed [default 60 seconds]. The second is the 'total timeout', in seconds; after at least the specified time the connection will be closed, even if there appears to be activity [default 600 seconds]. This value must not be less than the 'inactive' timeout. For either timeout, a value of zero seconds may be specified to indicate an indefinite time (that is, the timeout check is not made). You should only set both timeouts to zero if the filter is trusted, the data it accesses is always available locally, and all clients used with the server have a timeout mechanism. Finally, a third timeout, the 'start timeout', specifies the time in seconds for which GoServe will wait on startup for TCP/IP to become operational, retrying appropriate TCP/IP calls as necessary [default 600 seconds]. This allows asynchronous starting of TCP/IP and GoServe. Here, a value of 0 seconds means immediate timeout (that is, no retrys are performed, and GoServe will not start if an error is found). No TCP/IP retry is attempted if the requested port is in use (that is, it is likely that another copy of GoServe is already running). The audit mechanism """"""""""""""""""" GoServe includes a built-in audit mechanism that records events in an audit file while Goserve is running. This maintains a record of usage, and also records any errors or failures of the server. By default, all audit information (see details below) except selectors is recorded. Choose the 'Audit selection' menu item (under 'Options') for a dialog that gives you detailed control over the content of the audit file. You should only select auditing of 'Selectors' if the recording of selectors would not compromise confidentiality or privacy. The audit file has a fixed filename (GOAUDIT), with an extension that is the TCP/IP port number used (for example, 'GOAUDIT.70'). It is written in the working directory for GoServe. It may be read but not altered while GoServe is running. Each line in the audit file starts with three words, separated by one or more blanks: the time (hhmmss), the audit type (one character), and the transaction number (a number, incremented for each request accepted, that begins at 0 when GoServe is started). Additional information follows, depending on the audit type: A (Accept) - the numeric TCP/IP address of the client (n.n.n.n) C (Complete) - transaction time (secs), and description of action D (Day) - the current day (yyyymmdd), and description of audit E (Error) - description of an error in processing (e.g., timeout) F (Failure) - description of the failure (terminating error) I (Information) - description of information (statistics, etc.) P (Port) - server identity and port (n.n.n.n nn) S (Selector) - selector received from client The format of any "description" in the above is not strictly defined. Additional audit types may be added later. For example: 151717 D 0 19930512 ----- GoServe 1.00 ----- 151717 P 0 9.20.5.59 70 151730 A 1 9.20.5.59 151730 S 1 151731 C 1 0.55 Sent file "d:\godata70\mainmenu.70" 151733 A 2 9.20.5.59 151733 S 2 about.doc 151734 C 2 0.2 Sent file "d:\godata70\about.doc" 151931 I 2 Bytes sent 99 received 11 151931 I 2 Close The transaction time recorded on 'C' lines is rounded to the nearest 50ms, and measures the total elapsed time from receiving the request to complete delivery of the data. This depends on the speed and activity of the server, client, and the network between the two. It is likely to be rather variable. To look at an 'active' log file, use a program or command that allows shared reading of files. TYPE is one such command; its output may be redirected to a file to take a snapshot copy of the log. For example: type goaudit.70 > snapshot.log In addition, the 'Options' pull down menu has an item 'Move audit to archive'. This closes the audit file, appends it to the GoServe archive file in the current working directory, then erases and re-opens the audit file. The GoServe filter can also initiate this action, by returning "control moveaudit"--see elsewhere for details. The GoServe archive file has a fixed filename (GOARCH), with an extension that is the TCP/IP port number used (for example, 'GOARCH.70'). It is only touched by GoServe during a "Move audit to archive", so is readily copied, edited, or renamed. GOSMENU.CMD """"""""""" The GoServe package includes a simple utility that makes it easier to create 'canonical' Gopher menu files. See the start of GOSMENU.CMD for details. A 'menu template' file, perhaps called MAINMENU.MEN, might look like this: Sample main menu ; rexx.central.edu 70 0 About this Gopher server; about.doc 1 IBM Almaden Gopher server;; almaden.gopher.ibm.com 1 Interesting stuff; interest.70 The first line is a description; the second gives the field separator character (which will become Tab in the generated menu), together with the default server address and port. Subsequent lines are menu lines, as required by the Gopher protocol. Lines whose first character is a blank are ignored (that is, they are not included in the menu file). Running GOSMENU against this file (with a different output filename) like this: gosmenu mainmenu.men mainmenu.70 will create a 'canonical' menu file, with real tab characters instead of semicolons, address and port added where required, and extra blanks removed. The Gopher protocol suggests that the first (description) field be no more than 70 characters, and requires that the entire menu line (including address and port) fit within 255 characters. Notes: 1. The default output filename for GOSMENU.CMD is constructed by replacing the extension of the input filename by '70', so the sample command could have been simply: gosmenu mainmenu.men Or, of course, the file MAINMENU.MEN could have been dropped on a Program Reference object that starts GOSMENU.CMD 2. It is recommended that 'active' menu files be identified by an extension which is the Gopher port for which they are to be used. Remote control of GoServe """"""""""""""""""""""""" GoServe filters (see "Filter programs") can request that GoServe perform certain actions, instead of just returning files or other data. These requests may be used to control GoServe remotely (from another machine or from another process on the same machine). All requests return a document of one or more lines, either showing successful completion of the request or indicating an error. Specifically, if the filter response string begins with the verb "CONTROL", then the remainder (after one or more blanks) may take one of several forms: 1. SAY text This is a simple "loopback" test: the specified text is returned to the sender as a document. 2. STATISTICS Current statistics and settings are returned as a document. These include: * Transaction, error, byte, and client counts * Settings (not including for audit selections) * The local time of certain events (if an event has not occurred, it is shown as the GoServe start time). [The exact format of these statistics could change in the future.] 3. RESET [BYTES] [PEAK] [TRANSACTIONS] [ERRORS] [ALL] The specified GoServe counters and statistics are reset (after auditing, if appropriate). Any or all of the keywords may be given, in any order. Each works in the same way as the corresponding menu pull-down. If ALL is used, a record of the time is kept, and will be shown on the display window and in any statistics requests. 4. MOVEAUDIT The audit file is copied to the archive file (just as though the 'Move audit to archive' menu item had been selected). The filter mechanism provides flexibility in how these requests are accepted and passed to GoServe. For example, to accept control requests (here identified by a selector that starts with '!') only from certain addresses, the last two lines in the sample filter in "Filter programs" could be replaced with: if left(selector,1)<>'!' then do /* this is not a control request */ if selector='' then selector='mainmenu.'port /* 'startup' case */ return 'File name' dir''selector /* normal selector case */ end /* this is a control request */ owners=server 19.112.45.7 /* list of authorized addresses */ if wordpos(who, owners)=0 then return /* ignore unauthorized requests */ return 'control' substr(selector,2) /* remainder of string is request */ A control menu for issuing control requests might include lines like these (prior to processing with GOSMENU.CMD): 0 Show statistics; !statistics 0 Move audit to archive; !moveaudit 0 Test loopback; !say Hello world! 0 Reset all; !reset all 0 Reset peak count; !reset peak 0 Reset byte counts; !reset bytes 0 Reset error count; !reset errors 0 Reset transactions; !reset transactions The filter could be further modified so that only clients with authorized addresses would see a menu with these options.