TRICKS WITH MERGE
or WHAT YOU ALWAYS WANTED TO KNOW ABOUT MERGE BUT WERE AFRAID TO ASK
(and after reading this you'll know why)
This file started out at less than 12K, but more and more was added as new
subjects were brought up, so I decided I'd better list the topics discussed.
All the commands here refer to MERGESYS. For MERGEPRO, always add /s and
/f1 if no other field options are indicated.
REDUCE THE SIZE OF DUPLICATE TRASH FILES (AND USE THE "/1K" SWITCH)
PREPARE UNSORTED "MARKED" TRASH FILES USING THE "/1S" SWITCH
LOG TRICKS (AND HOW TO MAKE ONE LINE DESCRIPTIONS)
REPLACE DESCRIPTIONS ALREADY IN A DIR USING THE "/1M" SWITCH
MERGE ONLY DESCRIPTIONS THAT DON'T MATCH (OPPOSITE OF "/1M")
SWAP DESCRIPTIONS DURING COMPARES
KEEP DUPLICATES BECAUSE THEY ARE REALLY DIFFERENT PROGRAMS
GET A LIST OF FILE NAMES
REMOVE OLDER VERSIONS OF PROGRAMS
TAKE A CD-ROM OFF-LINE (AND HOW TO SORT CD-ROM DIRS WITH PCBFILER)
ADD A NEW CD-ROM BY MERGING IT
REDUCE THE SIZE OF DUPLICATE TRASH FILES (AND USE THE "/1K" SWITCH)
The "tricks" described here refer to files created by CRSDUPE1.BAT, but
these same principles can also be applied to files created with the other
batch files, or for that matter, any time MERGE is used. As mentioned at
the end of CRSDUPE1.BAT, you can shrink CROSSDUP.DES (a sorted trash file of
stripped duplicates) by selectively merging on different keys. For example:
MERGE crossdup.des newdup.des /ns /eo /f1 /f2 /f3
will remove all duplicates with the same name, size, and date (use /ns since
you don't want to do the sequence test and /eo to not halt with an error if
a key field is missing below; CROSSDUP.DES is not sorted by size and date in
ascending order, but the duplicates are still removed because they are next
to each other). NEWDUP.DES will have only a single occurrence (the one that
came first) of any names that had identical sizes and dates. Or you could
use /f1 /f2 to remove all with the same name and size, then /f1 /f3 for the
same name and date. You could even use /f1 /f4 /f5 to remove matches of the
name and the first two words in the description (fields 4 and 5; I would not
recommend /f1 /f4; an even more cautious approach would be /f1 /f4 /f5 /f6).
You can repeat this process as many times as you like on NEWDUP.DES (merge
it on itself after creating it the first time you merge CROSSDUP.DES) to
successively eliminate descriptions that meet whatever criteria you choose.
However, there is no assurance the description that remains is the best
description; it will only be the latest date, or the one from the last DIR
merged if names and dates are the same. A further step you could take with
the descriptions that matched whatever criteria you selected above is to
get all of them together in one file (and at the same time have all the
descriptions that weren't eliminated together; these are the ones that may
be different programs with the same name). Since you are only interested in
keeping the best description, a quick scan with an editor is possible (e.g.,
to see which is the longest and delete the others). First merge NEWDUP.DES
on itself to strip all remaining duplicates to a trash file (they will be
the second or more occurrences that were left behind after the "shrinking"
because they may not really be duplicates). Then "compare" the trash file
to the original CROSSDUP.DES with MERGE's new "/1K" switch, which will only
keep descriptions that match and strip the rest to another trash file (you
need to add /k so all matching descriptions are kept, otherwise the second
or more occurrences would also be stripped to the trash file):
MERGE /o newdup.des /tnodup.des+n
MERGE /k /1k nodup.des crossdup.des diff.des /tsame.des+n
SAME.DES now has all occurrences of duplicates that are probably the same
program (so you can pick out the best description), DIFF.DES has those that
may be different programs, and CROSSDUP.DES isn't changed because a separate
output file was named. You can also use these techniques on the trash files
which were created when the DIRs were merged on themselves and compared to
DIRs 60 and 61 to weed out descriptions that are probable duplicates (the
retained description will have the "happy face"). This may all seem very
tedious, but you can automate it with a batch file (if you use a replaceable
parameter, the same batch file can process any of these trash files; see the
end of LOG TRICKS further below for an example batch file).
PREPARE UNSORTED "MARKED" TRASH FILES USING THE "/1S" SWITCH
The trash files created by CRSDUPE2 and 3.BAT are not sorted, so they would
need to be first if you wanted to do the "compare" as shown above. However,
the relative order of the "happy face" and stripped descriptions may change,
and the "happy face" will be truncated if it is in column 79, but you can
shrink them selectively before sorting (the sequence won't matter when using
/eo /ns, and since the duplicates are paired, the shrinking process still
works). But there is another trick to get around PCBFiler's shortcomings
by first separating the descriptions into two files, sorting each, and then
merging the files back together. Some "happy faces" may be missing, but the
relative order will still be the same (well, actually the order between the
descriptions that the DIR60 compare stripped from DIR61 are reversed because
of the "switcheroo" where the kept description, now in DIR60 but originally
from DIR61, followed DIR60's replaced description in the trash file, but the
important thing for this usage is since the "happy face" may be missing, you
can still identify the kept description because it will always come first).
You can do this because when /1s is used with the trash option, "marked"
descriptions are stripped to the trash file. CRSDUPE3.BAT's trash files
will only have one occurrence of each per duplicate because the DIRs already
had all internal dupes removed and every successive "compare" DIR had any of
its cross dupes from a lower numbered DIR stripped, but CRSDUPE2.BAT's trash
files may have several occurrences, so add /k to keep all "unmarked" dupes
(that's also why HAPPY.DUP is shown merged on itself before merging it back
with CROSSDUP.D??: to get only one occurrence of the kept description).
Although an error in file 1 during compares won't cause a halt, since errors
are still "swapped" and forced out, use /ns to not perform the sequence test
because you don't want to change the order if there's a sequence error (/eo
could also have been used instead, but out of sequence records should be the
only type of error if there are any at all, so why do the test if it's not
necessary and you are going to ignore it anyway). /1s is different from the
other "compares" in that you can merge a file on itself, or even strip the
"marked" occurrences from file 1 and merge it with file 2 at the same time.
MERGE /o /k /ns /1s /thappy.dup+n crossdup.d??
sort both on name ascending
MERGE /o happy.dup
MERGE /o /k happy.dup crossdup.d??
Now CROSSDUP.D?? can be treated just like the "marked" trash files from
CRSDUPE1.BAT (the only difference is you can't safely use /1s again to strip
the marked first occurrences since some "happy faces" may be missing).
LOG TRICKS (AND HOW TO MAKE ONE LINE DESCRIPTIONS)
Besides having descriptions that may be different programs, DIFF.DES above
will still have many that are really the same. In most cases, you can tell
if a description is really a duplicate or a completely different program
that just happens to have the same name by looking at the one line written
to the log when /vl1 is used. The log was rather unwieldy when it came to
extricating a description from a trash file so it could be reinserted into a
DIR, but not anymore. Create a log using the command line below:
MERGE /o /k /vl1 /loneline.des+n diff.des
Since /k keeps duplicates, you haven't changed DIFF.DES, but you have a log
with the entire first line of the duplicated descriptions (/vl1 writes both
the first occurrence and any following duplicates when a file is merged on
itself or when cross file duplicates are encountered during the compare or
merge of two files, while +n writes the log without a header or statistics,
and when used with /vl1, the "type" of error or duplicate is also omitted).
The line and record numbers, along with DIFF.DES's file name, still follow
the description line, but the word "Duplicate" doesn't start the line, and
since there is also no header or stats at the end, it can be "merged". You
can even "chop off" the extra stuff by running CHK4DES on ONELINE.DES, which
will truncate the line at 80 characters, leaving only the description part.
Now you can edit the much smaller log and delete all descriptions that are
probable duplicates while keeping the descriptions that could be different
programs with the same name (you may be surprised at how many there are; you
only need the name once, so you can also delete any other occurrences, or
merge ONELINE.DES on itself when you're done editing). To extract the full
descriptions from DIFF.DES, just merge them using the /1k switch.
MERGE /o /k /1k oneline.des diff.des /tmoresame.des
MERGE /o /k moresame.des same.des
Now DIFF.DES has only the descriptions that may be different programs while
SAME.DES has all the probable duplicates. Instead of creating DIFF.DES and
SAME.DES first by using the "shrinking" techniques above, it may be quicker
to make a one line log file with CROSSDUP.DES, do the shrinking on it, and
then create DIFF and SAME.DES (reverse the order in which you prepare them;
since the one line files are smaller, "merges" won't take as long). The
sample batch file at the end of this section takes that approach.
If you want to make sure you've really got only different programs with the
same name you can use your editor again to check DIFF.DES (it shouldn't be
that big). Since DIFF.DES will be a bunch of duplicate names, you probably
won't want them in the same directory, so you could also use the editor and
save them to different files for merging into specific DIRs. If you are
using this method on the cross duplicates stripped from DIRs 60 and 61, it
won't be as difficult getting the descriptions back into those DIRs because
the trash file you are working on corresponds to the DIR that will receive
the reinserted descriptions (they were stripped from their respective DIRs
because a lower numbered DIR had the same name). However, in this case, the
equivalent to DIFF.DES will also contain the "happy face" first occurrences
that were kept, so after you've verified that these are really different
programs (you could also just delete the "happy face" occurrences with the
editor), merge DIFF.DES's equivalent on itself with the /1s switch.
The cross dupe trash files created with CRSDUPE3.BAT are just like those
from CRSDUPE1.BAT, except they aren't sorted (unless you modified all those
"compares" to be like the ones in CRSDUPE1.BAT for DIRs 60 and 61). If you
followed the "separate/sort/remerge" procedure described above, don't use
/1s here because some "happy faces" are probably missing. Your best bet may
be to use the editor and delete the one that occurs first (it was the one
marked by a "happy face" if there is none). However, since there is only
one occurrence of the kept description and it comes first, you could also
merge the file on itself using the trash option, and the descriptions you
want to put back in the DIR will be in the trash file. CRSDUPE2's trash
files aren't sorted either, but a duplicate description that is really a
different program won't go back into the DIR that corresponds to the trash
file you're working on since it was stripped from a higher numbered DIR (you
would have to decide where it goes as with CROSSDUP.DES above).
You may have noticed that you can use the log like this to generate a one
line description for all your files (of course, some of the one-liners won't
have anything useful because many authors have a penchant for using highbit
characters or some other nonstandard design for the first line of their
descriptions). If you want one line descriptions for anything, copy a DIR
to another file name and merge them with log's +n and /vl1, then merge the
log on itself to get only one occurrence, and voila, one line descriptions
(you'd still need to run CHK4DES to truncate the extra stuff; sorting with
PCBFiler also truncates it, but then you'd have stray "description" parts to
contend with; however, you could strip them using MERGE again with /ea).
Well, that's sort of an obtuse method, but it is dependable. A simpler way
would be just to merge each DIR on itself to a second output file name using
/s1 /ea (for MERGESYS; use /f1 /ea for MERGEPRO). What this should do is
strip all subsequent lines of a description as "key" errors (/s1 sets single
line records and all the lines that start with a space are considered to be
errors). I haven't tried it on every DIR, but it worked on the ones I did.
The commands below could be used in a batch file that shrinks and prepares
descriptions for examination. It makes a one line description first, then
creates the desired description files from it. It may also run a little
faster than working on the full descriptions because the files are smaller.
The trash files from CRSDUPE2 and 3.BAT will need sorting before they can be
used here, otherwise just invoke this file with the trash file name. Since
MERGE defaults to case sensitive, line 5 is a case insensitive check of the
first three words in the description. CHK4DES just truncates ONELINE.DES to
80 characters, and NO_EXIST.FIL is used because CHK4DES requires two file
names to run (it won't even exist unless a description has CHK4DES's default
"description not found" phrase; if you wanted to remove any names that don't
have a description, as well as the ones where the first line is blank, add
++ after the file names for CHK4DES's "no description" phrase).
MERGE /o /k /vl1 /loneline.des+n %1
CHK4DES oneline.des no_exist.fil
MERGE /eo /ns /f1 /f2 oneline.des work.lin
MERGE /o /eo /ns /f1 /f3 work.lin
MERGE /o /eo /ns /f1 /f4 /f5 work.lin
MERGE /o /eo /ns /f1 /f4 /f5 /f6 /c work.lin
MERGE /o /tnodup.lin+n work.lin
MERGE /k /1k nodup.lin oneline.des diff.lin
At this point, you could stop and use an editor to delete all except the
descriptions that look like different programs from DIFF.LIN (there is no
trash file to catch the stripped nonmatching one line descriptions because
these would be probable duplicates and the main reason you'd want to examine
their descriptions is to keep the best one, which you usually can't tell by
looking at just one line). DIFF.LIN is also not merged on itself since it
may not have been checked with an editor yet, and having duplicates doesn't
effect getting the full descriptions out of the cross duplicate file. In
any case, finish with (don't forget to delete the work files and rename DIFF
and SAME.DES with something that corresponds to the processed trash file; if
you haven't looked at DIFF.LIN yet, rename it too):
MERGE /k /1k diff.lin %1 diff.des /tsame.des+n
REPLACE DESCRIPTIONS ALREADY IN A DIR USING THE "/1M" SWITCH
There is a problem with trying to replace a description in a lower numbered
DIR with a better one that was removed from DIR60 or 61, since their trash
files contain all of their stripped descriptions and there is no way to tell
which DIR caused it to be stripped out. Well, MERGE has another new compare
switch, "/1M", which will merge into file 2 all descriptions from file 1
that match those in file 2, and only the ones that match, REPLACING the
descriptions in file 2 with ones from file 1 (if you also used /k to keep
duplicates, you'd get both descriptions into file 2). SO, you don't need to
know which DIR a description came from in order to replace it with a better
one. Just go through the trash files from DIRs 60 and 61 (do this before
sorting with PCBFiler) and delete all the descriptions that you don't want
(the "happy face" will mark whatever description is currently in a DIR).
You could call the file with the descriptions you didn't delete REPLACE.DES,
which you would then sort with PCBFiler on name ascending. Next write a
batch file with 59 lines using the command line below (the "?" = 1 to 59).
If the DIR headings are in place, add /b2??h (they will also be kept):
MERGE /o /1m /td:\temp\replaced.dup+n /ld:\temp\replaced.log /vl1 d:\temp\replace.des d:\pcb\dirc\dir?
The trash and log files are optional (REPLACED.DUP would get descriptions
stripped out of the DIRs). When this batch file has run, all the matching
descriptions that were in the lower numbered DIRs will have been replaced
with the ones from REPLACE.DES (since some of the descriptions you wanted to
replace may have been removed from all the lower numbered DIRs because of
the way CRSDUPE1.BAT makes CROSSDUP.DES, if you didn't put them back in a
DIR but just added them to DIR61, you'll also need to include DIR61).
MERGE ONLY DESCRIPTIONS THAT DON'T MATCH (OPPOSITE OF "/1M")
I was wrong when I said the one compare MERGE couldn't do was merge in only
records that don't match. I forgot about the "/X" switch, and I wrote it!
It's not a compare switch, but it accomplishes the same thing by making file
1 the output file when there are two file names. Suppose you had an older
list of descriptions that you wanted to add to a DIR, but you didn't want
any currently listed names to be replaced because they would be later dates.
Since file 1 always has precedence and keeps a duplicate over file 2, you
can make the DIR file 1 and have it also be the output file (you could also
do it by naming a third file for the output file, but then you'd need to
delete the old DIR and rename the output file with the DIR's name, whereas
/x will accomplish it in one step). Just use the command line below:
MERGE dir? old_dir? /x /o
SWAP DESCRIPTIONS DURING COMPARES
All the CRSDUPE?.BATs pull a fast one when comparing DIR60 to DIR61 and put
DIR61's description, since it is probably the best and latest, into DIR60.
You can also do that with all the other DIR compares, and in most cases the
best will be kept. The only problem here is that the description replaced
in the categorized lower numbered DIR may NOT be the same program, just one
with the same name (it didn't make any difference with DIR60 since it's just
a "misc" category). You can check the trash files after the first time any
of these batch files run and decide whether the number of wrong replacements
would be prohibitive. Plus, since most of the files in DIR61 are "uploaded"
by you, and many are from CD-ROM's you haven't combined with the rest of the
DIRs yet, when they are added, their descriptions will end up in the correct
DIRs. Several methods are shown below for swapping descriptions. The first
technique is similar to that used in CRSDUPE1 and 2.BAT, except the "marked"
kept description from the higher numbered DIR doesn't follow the stripped
one from the lower numbered DIR as is done in the batch files, which also
eliminates a step. If the DIR heading is in place, add /b1?? /b2??h to line
1 and /b2??h to line 3 in all three examples, and in the last add /b1??h to
line 4 and /b1?? /b21 to line 5.
MERGE /o /1 /tkeep_d61.dup+n /lcrossdup.log dir?_low dir61
if not exist keep_d61.dup goto NEXT
MERGE /o /tstripdup.low+n /lcrossdup.log /vl1 keep_d61.dup dir?_low
MERGE /o /1t /tcrossdup.low+n /lcrossdup.log keep_d61.dup stripdup.low
del keep_d61.dup
del stripdup.low
:NEXT
...
The first compare removes any cross file dupes from DIR61, but then they are
merged back into the lower numbered DIR, which strips whatever it had to the
trash file STRIPDUP.LOW. Then KEEP_D61.DUP is compared using /1t to the
stripped duplicates so that the kept descriptions are marked and added to
the continuing trash file, CROSSDUP.LOW, along with the stripped ones (which
came from DIR?_LOW; in this arrangement, the rest of the descriptions are
ones that DIR?_LOW stripped from a higher numbered DIR). The next method is
like that in CRSDUPE3.BAT and uses the /1m switch, first replacing the lower
numbered DIR's descriptions with any from DIR61 that match (the "compare"
direction is reversed here, and the lower numbered DIR's stripped duplicates
then remove from DIR61 the descriptions that replaced them).
MERGE /o /1m /tstripdup.low+n /lcrossdup.log /vl1 dir61 dir?_low
if not exist stripdup.low goto NEXT
MERGE /o /1 /tkept_d61.dup+n /lcrossdup.log stripdup.low dir61
MERGE /o /1t /tcrossdup.d61+n /lcrossdup.log kept_d61.dup stripdup.low
del kept_d61.dup
del stripdup.low
CROSSDUP.D61 in this example normally has the descriptions that were removed
from DIR61 by the lower numbered DIRs, but if all the DIR?_LOW compares were
done as shown above, it would have the descriptions from lower numbered DIRs
that DIR61 replaced. The last method makes sure the kept description in the
lower numbered DIR has the latest date (ties will keep the ones from DIR61).
MERGE /o /1 /tstripd61.dup+n /lcrossdup.log /vl1 dir?_low dir61
if not exist stripd61.dup goto NEXT
MERGE /o /k /f1 /f3ddt /lcrossdup.log stripd61.dup dir?_low
MERGE /o /tstripped.dup+n /lcrossdup.log /vl1 dir?_low
MERGE /o /1t /tcrossdup.???+n /lcrossdup.log dir?_low stripped.dup
del stripd61.dup
del stripped.dup
There is an equivalent using the /1m switch, but it requires an extra step
(a "sort" pass with /ef to get the latest date first; you can't just compare
with /1m /f1 /f3ddt because only exact matches are merged, and if there were
any, they would be name and date duplicates).
KEEP DUPLICATES BECAUSE THEY ARE REALLY DIFFERENT PROGRAMS
If you keep different programs with the same name, besides needing a method
where a caller can specify which one to download, another problem is every
time you add a new CD-ROM, the duplicated descriptions that you reinserting
into the DIRs when the last CD-ROM was added will be stripped again and you
would have to do them all over. I don't know of an easy way to get around
this, but I do have two suggestions. You could make a new DIR, e.g., 62,
where all you have is descriptions that are duplicated somewhere else and
the rest of the DIRs would only have a single occurrence of the same name
between them. This may also make it easier for callers to download them as
you could have a special "door" or something to access these files.
Another idea is to have a "master" list of all dupes, plus a separate list
for each DIR that has a name duplicated somewhere else. After you add a new
CD-ROM, you could "compare" the master duplicate list to every trash file of
stripped dupes to remove their names and anything else that may have been
added with the same name (use /1k /k as was done in the examples above):
MERGE /1k /k master.dup dirdup?? temp.dup /tnew??.dup+n
MERGE /o /k temp.dup old.dup
MERGE temp.dup /ttmpstrip.dup
MERGE /k tmpstrip.dup oldstrip.dup
DIRDUP?? would be a sorted trash file of stripped dupes (the crossdupe trash
files created by CRSDUPE2 and 3.BAT should be prepared with the method shown
in PREPARE UNSORTED TRASH FILES above). NEW??.DUP has only new duplicates
that were caused by this CD-ROM and can be manipulated with the techniques
previously discussed. OLD.DUP is a continuing file which, after every trash
file is processed, will contain all the dupes already on-line plus any new
ones from this CD-ROM that also have the same name. When TEMP.DUP is merged
on itself, the dupes that were stripped from the DIR will be in TMPSTRIP.DUP
(the kept first occurrence is still in TEMP.DUP), which is then merged with
/k to keep duplicates into OLDSTRIP.DUP, another continuing file. After all
the DIRs are compared, merge OLDSTRIP.DUP on itself using the trash option.
Any names from the CD-ROM that match a duplicate file currently on-line will
be in the trash file (as well as any descriptions you might have that are
already duplicated more than twice; you'd just have to keep a separate list
of them if there are any, but it certainly wouldn't be a long one):
MERGE oldstrip.dup /tadded.dup
If ADDED.DUP doesn't exist (or the only descriptions in it are the same as
the list of names already occurring more than twice), all that's left to do
is merge the separate lists of duplicates back into their respective DIRs.
However, if it has other descriptions, compare it with /1k /k to OLD.DUP so
you can see what was added (there may be cases where you have two different
programs with the same name, the CD-ROM replaced one of them in a DIR, then
stripped the other during the crossdupe compares, so that now both of the
previous descriptions are stripped, or a CD-ROM replaced a DIR description,
then the CD-ROM was stripped by a compare, etc.; you just won't know until
you check). When you examine the "compared" OLD.DUP, the sequence in which
it was originally created is important. If you first added the individual
DIRs above, then the cross duplicate trash files, the very first occurrence
will be the one that is kept (the descriptions in OLD.DUP will be paired and
the first of each pair is the one that was retained, but a kept description
from one DIR can then be stripped during the cross duplicate compares). If
you made it in the reverse order, the first of the last pair is the one that
is currently in a DIR somewhere. You may even have to check the original
DIRDUP?? trash files to decide what belongs where. This sounds like a lot
of trouble, and about the only good thing is it shouldn't happen very often!
GET A LIST OF FILE NAMES
To get a list of every duplicate file name, first merge all the DIR trash
files together (there may be 61 of them, DIR1 to 61.DUP):
MERGE dir1.dup dir2.dup dir.dup
MERGE /o dir3.dup dir.dup
...
DIR.DUP will only have the "marked" first occurrence descriptions that were
originally kept, but that isn't important here since you're just after the
file names. Now merge it with DIR61_60.DUP that was created previously:
MERGE dir.dup dir61_60.dup moredupe
Next, copy MOREDUPE to another file (e.g., DUPE2) and merge them together
with the trash file parameters +k+n:
MERGE moredupe dupe2 /tmoredupe.nam+k+n.
Since all names will be duplicated, the trash file will be another list of
just file names, this time with the duplicates obtained from merging the
DIRs on themselves and "comparing" them to DIR60 and DIR61. Finally, merge
CROSSDUP.NAM and MOREDUPE.NAM (or whatever you called the trash file above)
and you will have the name of every file that had a duplicate description,
which may be useful for checking that there is only one occurrence of the
file on your hard drive or between your hard drive and CD-ROMs (just make
sure the file you keep matches the DIR description, otherwise someone could
download SERIAL.ZIP, expecting a programming aid but instead end up with a
program that checks serial ports). You can use this same technique to make
a list of just file names for all the files you have on-line by copying the
master file list to another name and merging them with trash's +k+n.
REMOVE OLDER VERSIONS OF PROGRAMS
Suppose an author asks you to remove older versions of his or her programs
from your DIR descriptions and provides you with a list of them. How do you
go about it now? Let MERGE do it for you. Put each file name on a line by
itself and save them to a text file (all should be in uppercase and in name
ascending order, just like the names would appear in the DIR descriptions).
Then write a 61 line batch file (which could be used anytime you need to
remove certain descriptions) to compare the list to each DIR. If the list
of names is in REMOVE.LST, the batch file might look like the example shown
below. The /b2??h assumes the DIR headings are still in place, and the DIRs
need to be sorted in name ascending order, which the batch file can also do
by running PCBFiler from the command line (see BATNOTES). The trash file is
optional but you may want to check that these are really the descriptions
meant to be removed and not some other programs with the same names (you can
also use a separate trash file for each DIR so if any of the descriptions
need to be replaced, you know where they go, and a trash file won't exist if
the DIR being compared had none of the names in the list). The /ef is used
in case you accidentally (like I did when testing these commands) get a name
out of sequence in REMOVE.LST. Although file 1 errors won't cause a "halt
on error", out of sequence records can still be "swapped" so their order is
changed, but /ef can't fix an out of sequence record that needs to be moved
forward more than one position, so you may want to first merge REMOVE.LST on
itself using /ef to correct any sequence errors (see BATNOTES for more).
MERGE /o /b2??h /1 /ef /td:\temp\removed.des+n d:\temp\remove.lst d:\pcb\dirc\dir1
MERGE /o /b2??h /1 /ef /td:\temp\removed.des+n d:\temp\remove.lst d:\pcb\dirc\dir2
MERGE /o /b2??h /1 /ef /td:\temp\removed.des+n d:\temp\remove.lst d:\pcb\dirc\dir3
...
TAKE A CD-ROM OFF-LINE (AND HOW TO SORT CD-ROM DIRS WITH PCBFILER)
You can easily remove a CD-ROM's listings from your DIRs when you take it
off-line, providing it's sorted the same as the bbs DIRs (which may be a big
catch). The major purpose of BATNOTES is to keep from separately sorting a
CD-ROM's DIR listings, but it can be done without making a special DIR.LST,
although that is a possibility (you wouldn't need one for every CD-ROM, just
have a special CD-ROM.LST and copy it over your regular DIR.LST, next copy
the DIRs from whatever CD-ROM you are taking off-line to the directory named
in CD-ROM.LST, renaming them appropriately as they are copied, and then run
PCBFiler from the command line). The method shown below temporarily renames
your bbs DIRs and copies to their directory the CD-ROM DIR. If they are not
already called DIR1, etc., you can rename them while they are copied, e.g.,
copy cd-rom\dirpath\list* d:\pcb\dirc\dir*. Of course, this doesn't work if
they have some unusual naming convention and you would need a separate copy
line for each one (hint: you could use "merge" batch files, if you write
them so they call another batch file that has the actual merge commands, to
do the copying since all the DIRs would be appropriately paired, then just
replace the merge commands batch file with another one that does a "copy").
ren d:\pcb\dirc\dir* d:\pcb\dirc\tmp*
copy cd-rom\dirpath\* d:\pcb\dirc
PCBFILER /sort
Since you already know what CD-ROM DIRs were combined with the bbs DIRs from
when they were added, "compare" each CD-ROM DIR to the appropriate bbs DIR
(now called TMP). By using several fields for the key (along with /eo /ns),
you can be fairly certain you'll remove only the listings from this CD-ROM
(in case a CD-ROM added later replaced a name from this one, BUT if this and
another CD-ROM have identical descriptions for the same file, you'll remove
one that you meant to keep). Then delete the CD-ROM DIRs and rename yours.
Don't forget to add /b2??h if the DIR headings are in place, and you'll also
need /b1?? if the CD-ROM's DIRs don't start on line 1 (if both sets of DIRs
start on a line other than 1 and it's the same, you can just use /b1??, but
if you want to keep the DIR heading, you would still need to add /b2??h).
MERGE /o /eo /ns /1 /f1 /f3ddt /f2 /f4 /f5 /f6 d:\pcb\dirc\dir? d:\pcb\dirc\tmp?
MERGE /o /eo /ns /1 /f1 /f3ddt /f2 /f4 /f5 /f6 ...
...
del d:\pcb\dirc\dir*
ren d:\pcb\dirc\tmp* d:\pcb\dirc\dir*
If your bbs DIRs are in date descending order, put /f3ddt before the other
fields (how they are sorted doesn't matter in this case, only that both are
sorted the same, so you can also sort the CD-ROM in date descending order).
To help promote the success of this method, always place a CD-ROM's DIR
before an existing bbs DIR when adding a new CD-ROM (see below). Then if
the new CD-ROM has a program name that duplicates one already on-line, the
new CD-ROM's description will replace the one currently in use. However,
what if 10 CD-ROMs all have identical descriptions for the same file, as
could easily happen if the CD-ROMs used the FILE_ID.DIZ, and you took one of
those CD-ROMs off-line: you'd loose that listing, and I can't think of any
way to keep it. Still, I don't see how else you can take a CD-ROM off-line
without comparing out its description. You can't just add back to the bbs
DIRs all the CD-ROMs minus the one going off-line since its descriptions
would still be in the bbs DIRs. What you'd need to have is a separate list
of bbs files that doesn't contain any of the CD-ROM files for each DIR, then
add to them the CD-ROMs staying on-line each time you remove one.
Still, adding back 40 or so CD-ROMs every time you remove one is going to be
time consuming. Rather than doing that and having to keep separate bbs file
lists, it may be simpler to compare out the descriptions using the technique
below. It uses the same method as above, except it saves each DIR's removed
descriptions in a trash file that corresponds to the DIR. Then, the on-line
CD-ROMs are compared to the trash files and any matches are removed to other
trash files, which will be merged back into the original DIRs. If you are
set up to "merge" CD-ROM DIRs (see further below and COMBINE.BAT), the merge
batch files can do the work (when you write them, let them call a separate
batch file that has the actual merge commands, which can be easily replaced
with whatever commands are needed at the time, or just add extra replaceable
parameters to the command line so you can feed the appropriate commands when
necessary). Ok, so you've got a WHIZBANG.BAT, the merge batch file for a
CD-ROM you're taking off-line, that looks like (the file name sequence may
look backwards, but that's only because it's probably easier to edit names
when they are on the end; MERGEROM.BAT places the parameters correctly; the
second parameter will become the first file name and vice versa):
call MERGEROM.BAT dir1 e:\whizpath\whiz5
call MERGEROM.BAT dir2 e:\whizpath\whiz11
call MERGEROM.BAT dir3 e:\whizpath\whiz1
...
It doesn't really matter what technique the current MERGEROM.BAT implements
for adding CD-ROMs because it is going to be temporarily replaced with the
line below (the nice part about calling batch files). This also assumes all
DIRs are sorted by name ascending. If not, you can use one of the methods
described to sort them when needed, although that will add some time. The
line below is actually the REALLY short example shown in COMBINE.BAT for
merging the CD-ROMs, it just has the needed commands added to the end, which
could also be done by passing replaceable parameters. Since anything that's
removed will be added back if another CD-ROM has it, only three key fields
are used, which may speed things up slightly, but the files created should
still be small, which is important too (/eo isn't needed for three fields).
As discussed in COMBINE.BAT, if any of the CD-ROM DIRs have headings, it's
better to pass /b1?? as a parameter from the appropriate merge batch file:
MERGE /o /b1?? /b2??h %2 d:\pcb\dirc\%1 /1 /td:\temp\%1.off+n /f1 /f3ddt /f2 /ns
If the CD-ROM DIRs haven't had their errors fixed, you could add /e, but I
can't say how successful this will be if unknown types of errors are present
(file 1 errors during a compare won't halt MERGE, but they could interfere;
with /e, hopefully things like stray description parts will be disregarded).
Another possibility is to put the line below first, which strips any errors
before the compare (similar to when the bbs DIRs are merged on themselves)
and can also remove a CD-ROM DIR heading so /b1?? isn't needed afterwards:
MERGE /o /b1?? /ea /f1 /f3ddt %2
Now, instead of calling WHIZBANG.BAT with COMBINE.BAT as when you merged it
into the bbs DIRs, call it with REMOVE.BAT, which first sets everything up
and could also do any sorting (don't forget the bbs DIRs), for example:
ren d:\pcb\dirc\dir* d:\pcb\dirc\tmp*
copy cd-rom\dirpath\* d:\pcb\dirc
PCBFILER /sort
copy d:\pcb\dirc\dir* cd-rom\dirpath
del d:\pcb\dirc\dir*
ren d:\pcb\dirc\tmp* d:\pcb\dirc\dir*
If CD-ROM\DIRPATH is the actual CD-ROM, you obviously can't copy the sorted
DIRs back, but you can cheat and make a directory with the same path on some
hard disk, then use the ASSIGN or SUBST command (first copy the DIRs to your
hard drive) to change the CD-ROM drive's letter to the hard drive's so that
all further requests for the CD-ROM DIRs get faked out and are sent to your
hard drive, and you can do all this in the batch file (also do during normal
maintenance down-time; if you try it while callers can connect, no telling
what might happen). The example below assumes there are already different
versions of MERGEROM.BAT and copies MERGEOFF.BAT over the existing one.
copy d:\pcb\dirc\mergeoff.bat d:\pcb\dirc\mergerom.bat
call WHIZBANG.BAT
Now you need another MERGEROM.BAT with the lines shown below that compares
these trash files to the rest of the CD-ROMs when their "merge" batch files
are called (MERGEON.BAT also keeps the created DIR??.ON trash files sorted
so that descriptions can be merged back in, otherwise as each new CD-ROM is
compared, any stripped descriptions would be appended, and by overwriting
the DIR??.OFF file each time, as matching descriptions are removed, it gets
smaller and subsequent compares will take less time). This also uses the
new Compare "E" parameter, which deletes an input/output file if its size
would be zero, so if it turns out that all the descriptions for a particular
DIR will be returned, the OFF file won't exist and any further "merges" will
just halt with a "file not found" error (see the discussion on FIXDIR.BAT in
COMBINE.BAT for more information on the "E" parameter):
MERGE /o %2 /b1?? d:\pcb\dirc\%1.off /b21 /1e /td:\temp\temp.on+n /f1 /f3ddt /f2 /ns
if not exist d:\temp\temp.on goto END
MERGE /o /k d:\temp\temp.on d:\temp\%1.on
del d:\temp\temp.on
:END
As with MERGEOFF.BAT earlier, if the CD-ROM DIRs could have errors, either
add /e to the first line above or put the extra line to strip errors first.
Any descriptions in DIR??.ON will result from another CD-ROM having it too
and stripping it from the files WHIZBANG created (do any sorting as needed).
copy d:\pcb\dirc\mergeon.bat d:\pcb\dirc\mergerom.bat
call STILLON1.BAT
call STILLON2.BAT
call STILLON3.BAT
...
del d:\pcb\dirc\dir*.off
copy d:\pcb\dirc\mergeadd.bat d:\pcb\dirc\mergerom.bat
MERGEADD.BAT has the commands needed to merge CD-ROM and bbs DIRs, although
for this purpose, it could just have the command line below, which probably
is faster (if you're keeping duplicates because they are really different
programs and have any in the same DIR, add /f1 /f3ddt /f2 to the line):
MERGE /o %2 d:\pcb\dirc\%1 /b2??h
Anything that needs to be returned to the bbs DIRs are in properly sorted
files that are ready to be merged back. REMOVE.BAT can do it in the next 61
lines, or you can have another batch file it calls do the work, just as with
the merge batch files. In any case, the lines would look like:
call MERGEROM.BAT dir1 d:\pcb\dirc\dir1.on
call MERGEROM.BAT dir2 d:\pcb\dirc\dir2.on
call MERGEROM.BAT dir3 d:\pcb\dirc\dir3.on
...
Finally, REMOVE.BAT deletes DIR*.ON and returns everything to normal. Ok,
so maybe it turns out that you can't easily take a CD-ROM off-line, but like
all the other things MERGE can do, once the batch file is written you are
home free. You can use the same REMOVE.BAT any time you remove a CD-ROM by
editing the names (most of the time there would only be two changes). Just
in case all these comments have obliterated what is going on, a repeat of
the commands in REMOVE.BAT follows (add any extra ones where needed, such as
for sorting, ASSIGNing, SUBSTituting, etc.):
copy d:\pcb\dirc\mergeoff.bat d:\pcb\dirc\mergerom.bat
call WHIZBANG.BAT
copy d:\pcb\dirc\mergeon.bat d:\pcb\dirc\mergerom.bat
call STILLON1.BAT
call STILLON2.BAT
call STILLON3.BAT
...
del d:\pcb\dirc\dir*.off
copy d:\pcb\dirc\mergeadd.bat d:\pcb\dirc\mergerom.bat
call MERGEROM.BAT dir1 d:\pcb\dirc\dir1.on
call MERGEROM.BAT dir2 d:\pcb\dirc\dir2.on
call MERGEROM.BAT dir3 d:\pcb\dirc\dir3.on
...
del d:\pcb\dirc\dir*.on
ADD A NEW CD-ROM BY MERGING IT
You can use a sort method similar to the above for adding CD-ROMs by first
sorting your existing DIRs on name ascending, renaming them, copying the
CD-ROM's and sorting them, then merging the two (you could use /ef and /ea
together or separately with a "sort" loop, although that shouldn't be needed
with only one occurrence of a name within a DIR). If you are adding more
than one CD-ROM, the first line below would be replaced with a "merge" line
for each CD-ROM and you would probably want a sort loop (see DIRDUPE.BAT,
COMBINE.BAT, and BATNOTES for more information). Since these commands are
the same for each DIR except the file names, you can save some typing by
placing them in a separate batch file and using replaceable parameters, then
call the batch file with the appropriate file names. Add /b1?? to the first
line if the CD-ROM DIR has a heading, and if your DIR headings are in place,
add /b2??h to the first line, /b1??h to the fourth, and /b1?? /b21 to the
sixth (if you remove your DIR heading in the first line by not using the "h"
parameter, then don't add any /b's to the rest of the lines).
MERGE /o /k /ef /ea /f1 /f3ddt /tdir1.err+n /ldirerror.log /vl dir1 tmp1
del dir1
ren tmp1 dir1
MERGE /o /ttemp.dup+n /ldir_dupe.log /vl1 dir1
if not exist temp.dup goto DIR2
MERGE /o /1te /tdir1.dup+n /ltemp_dup.log dir1 temp.dup
:DIR2
MERGE /o /k /ef /ea /f1 /f3ddt /tdir2.err+n /ldirerror.log /vl dir2 tmp2
...
