This file contains information on MERGE's new /EF switch, which can be used to
fix out of sequence descriptions.  With /EF, you can automate the whole process
of adding new CD-ROM DIRs to your bbs.  Although it is quicker to merge each
bbs DIR with its corresponding CD-ROM DIR, you would need to check the CD-ROM
DIRs first to make sure they were sorted correctly.  This way, MERGE does it.
Also included at the end is a method to create new file lists every day.  The
commands shown are for MERGESYS; for MERGEPRO, add /s to the command line.
Since this is intended to automate everything, I assume the directory headings
will still be in place and the "merge" won't be starting on line 1, so also add
"begin on line" (/b1??h) to the commands.  If you want to remove the existing
heading from the new DIR because you'll be adding a new one, you can omit the
"h" parameter, but it will have to be on a separate command line as shown below
before calling SORT.BAT, then the rest of the commands can remain as they are:

MERGE /o /k /e /ld:\temp\heading.log /b1?? d:\pcb\dirc\dir1


1. Use your "copy" batch files to append CD-ROM descriptions to the bbs DIRs
   (add /b to the end of each "copy" line to not write ^Z to the appended DIR).

call WHIZBANG.BAT

2. Run PCBFiler from the command line.  Use whatever switches you need for a
   PCBOARD.DAT file you've prepared to sort your uploads DIR on name ascending.
   I think you'll also need a second DIR.LST so PCBFILER sorts the individual
   DIRs on name ascending.  In that case, just add a line to "copy" the special
   ASCEND.LST (or whatever you call it) over the existing DIR.LST (it will also
   need a backup), then copy your backup DESCEND.LST back to DIR.LST when this
   is ready to resort on date descending.  Don't worry about getting the latest
   date of a duplicate first; MERGE will take care of that in the next step.

PCBFILER /sort ...

3. Now run MERGE on every DIR (repeat the steps below for each DIR).  You can
   use the same "error" trash file every time or name one for each DIR, like
   DIR1.ERR for DIR1.  After everything has run, check the trash file(s) to
   see what the errors were and possibly remerge any descriptions that were
   stripped because a date wasn't a real date or some other key error occurred.
   If there are any descriptions that were stripped for key errors, since all
   the DIRs will have their own and cross dupes removed whenever you look at
   the error trash file, before remerging, you should do a "compare" of each
   DIR to the repaired descriptions so you can see if some other DIR already
   has the same description (you don't want to just merge them in; also use /1t
   with a trash file so a copy of the current description is written to trash
   and/or /vl1 with a log).  I'm using a separate batch file to fix sequence
   errors because it cuts down on the number of lines and simplifies things
   (SORT.BAT is described after these steps).  You could also make a separate
   STRIP.BAT instead of the second line and call it with the file name (and
   trash file name for different error files) if the errorlevel is 1.  An easy
   way for STRIP.BAT to have a trash file matching the DIR name would be to use
   the same replaceable parameter for each:  /td:\temp\%1.err+n d:\pcb\dirc\%1

call SORT.BAT dir1
if errorlevel==1 MERGE /o /k /ea /f1 /f3ddt /td:\temp\dir1.err+n /ld:\temp\direrror.log /vl d:\pcb\dirc\dir1
call SORT.BAT dir2
if errorlevel==1 MERGE /o /k /ea /f1 /f3ddt /td:\temp\dir2.err+n /ld:\temp\direrror.log /vl d:\pcb\dirc\dir2
etc.

NOTE:  Remember our conversation about the files with alphabetic dates, e.g.,
04-Sep-92, which MERGE can handle and PCBFiler can't.  PCBFiler placed them at
the very end of the DIR after sorting on name ascending (Why???  The file names
were ok.  Oh well, that's PCBFiler!), and SORT.BAT ran for a LONG time because
MERGE was sorting them into the correct sequence.  Since /ef fixes in one pass
two dupes with only their dates out of sequence, I recommend that you replace
both the call to SORT.BAT and the "if errorlevel==1" line with the line below
(you could also use it in a batch file; remove or change /b1?? as needed; you
won't need a separate line here to strip the heading; use /b1??h to keep it):

MERGE /o /k /ef /ea /f1 /f3ddt /b1?? /td:\temp\dir1.err+n /ld:\temp\direrror.log /vl d:\pcb\dirc\dir1

4. Now run DIRDUPE.BAT, your preferred CRSDUPE?.BAT, and PCBFiler again to sort
   the finished DIRs in date descending order, and go to bed.  When you get up,
   everything will be done (except for checking the error files)!

call DIRDUPE.BAT
call CRSDUPE3.BAT
PCBFILER /sort ...

Here is what SORT.BAT should look like.  With no halt on error (/e), merge
exits with:  0 if there were no errors of any type (or if the only errors were
blank/empty keys and they were stripped because /eb or /ea was used), 1 if
there are no out of sequence errors but some other type of error(s) occurred,
and 2 if there are out of sequence errors (there may also be other types).  By
testing for code 2 after each run, the merge repeats if there are still out of
sequence records (sooner or later there won't be, but once your DIRs are fixed,
there shouldn't be more than two duplicates per file name if there are any, and
each DIR would be sorted after the first time; you could probably just replace
the call to SORT.BAT with the "MERGE ..." line below).  This shows how you can
write a batch file for MERGE to "sort" if more than one pass might be required
to correct out of sequence records (like three dupes whose dates happened to be
in ascending order).  It only works with /ef, which also does not count as an
error out of sequence records that are fixed.  When you use /ea, all errors are
stripped, so the exit code doesn't matter (you know the merged file won't have
any errors, but you might also be stripping fixable out of sequence records).
/eo doesn't change the order of records, and /e can't move records that occur
too early toward the back of the DIR, so they would just stay out of sequence.

:SORT
MERGE /o /k /ef /f1 /f3ddt /ld:\temp\sort.log d:\pcb\dirc\%1
if errorlevel==2 goto SORT

I should mention one other thing about testing for "errorlevel"; you may want
to consider the possibility that an unexpected exit code could show up, such as
11 for file 1 not found (you might make a typo somewhere).  After the commands
above, you could add the line:  if errorlevel 3 (do something).  Then if the
exit code is 3 or greater, the "do something" could "beep" the speaker, "pause"
the batch file, or execute some "exception" program.  This is also why a log is
used (assuming you look at it!):  to tell you if something didn't work right.
In this case, you don't need for it to be "verbose"; you're only interested in
seeing that everything is ok.  You could probably get by with just log's "+n"
parameter for no heading or stats, which would keep it small and still give you
the information you needed; if nothing went wrong, the log wouldn't even exist.
Don't forget to add /b1??h if the DIR heading is in place.  If you are removing
the headings, instead of adding a line to the main batch file before calls to
SORT.BAT, you could do it here outside the "loop" by adding as the first line:

MERGE /o /k /e /ld:\temp\sort.log /b1?? d:\pcb\dirc\%1

However, this wastes a "merge", so you might prefer the batch file below:

MERGE /o /k /ef /f1 /f3ddt /ld:\temp\sort.log /b1?? d:\pcb\dirc\%1
if not errorlevel==2 goto END
:SORT
MERGE /o /k /ef /f1 /f3ddt /ld:\temp\sort.log d:\pcb\dirc\%1
if errorlevel==2 goto SORT
:END


COMBINE DIRDUPE AND THE /EF COMMAND
DIRDUPE.BAT was written before the /ef switch was available, otherwise it would
have included the steps shown here.  You could easily combine these steps with
DIRDUPE's.  In fact, a new DIRDUPE could be the "control" batch file with just
67 lines and call other batch files to do the work.  I suggest the setup below.
You could do more than one CD-ROM at a time, but then you will have to contend
with the possibility of umpteen copies of some file name instead of maybe just
two.  However, most of those multiple dupes would be confined to the individual
DIR duplicate trash files, and some would also probably end up in the DIR error
trash files since only one "sort" pass is done (if there were three or more and
all their dates were out of sequence, at least one would go to the error file,
and it could be the one with the latest date; you could write FIXDIR.BAT with a
sort "loop", but then you run the risk of repeated loops to sort any alphabetic
dates, plus you'd never know they were there since MERGE fixed their sequence,
although a good "hint" would be a 1000 sort log entries for a particular DIR):

call WHIZBANG.BAT
copy d:\pcb\dirc\ascend.lst d:\pcb\dirc\dir.lst
PCBFILER /sort ... (name ascending)
call FIXDIR.BAT dir1
call FIXDIR.BAT dir2
...
call CRSDUPE?.BAT
copy d:\pcb\dirc\descend.lst d:\pcb\dirc\dir.lst
PCBFILER /sort ... (date descending)

FIXDIR.BAT would look like (change or remove /b1??h as necessary; if you don't
have a heading or strip it in line 1, remove all the /b's from lines 2 and 4):

MERGE /o /b1??h /k /ef /ea /f1 /f3ddt /td:\temp\%1.err+n /ld:\temp\direrror.log /vl d:\pcb\dirc\%1
MERGE /o /b1??h /td:\temp\temp.dup+n /ld:\temp\dir_dupe.log /vl1 d:\pcb\dirc\%1
if not exist d:\temp\temp.dup goto END
MERGE /o /b1?? /b21 /1t /td:\temp\%1.dup+n /ld:\temp\temp_dup.log d:\pcb\dirc\%1 d:\temp\temp.dup
del d:\temp\temp.dup
:END

If you want a sort loop (e.g., you know there won't be any date errors because
they were fixed previously; if all DIRs, including the CD-ROM's, have had their
errors corrected, you can omit the last line, but you may want to leave it just
in case; it won't run unless there are errors), replace the top line above with
the lines shown below after adjusting them for the DIR headings.  If they are
in place and you want to keep them, remove the first two lines and the sixth
line, and only change the /b's for the starting line numbers.  If there are no
headings, remove the same lines and all the /b's from the other lines.  If the
DIRs have headings but you want them stripped, keep the first two lines and the
sixth line but remove the /b's from the rest of the lines.

MERGE /o /b1?? /k /ef /f1 /f3ddt /ld:\temp\sort.log d:\pcb\dirc\%1
if not errorlevel==2 goto NOSORT
:SORT
MERGE /o /b1??h /k /ef /f1 /f3ddt /ld:\temp\sort.log d:\pcb\dirc\%1
if errorlevel==2 goto SORT
:NOSORT
if errorlevel==1 MERGE /o /b1??h /k /ea /f1 /f3ddt /td:\temp\%1.err+n /ld:\temp\direrror.log /vl d:\pcb\dirc\%1


DAILY FILE LIST UPDATE
And what about having an updated "master" list every day (you could also use
this method to update your INFOFILE with the individual DIRs)?  If you've got
your "index" set up, nobody can upload a name that already exists, but since
you're using the ADDTODAY/CHK4DES system to "upload" your own files, it would
be possible to get duplicate names unless you do a "compare" of your "master"
list to the "addfiles" list so any duplicates are stripped before being added.
Assuming you are keeping out duplicates, write a batch file that sorts your
uploads DIR on name ascending each night and then merges it with the master
(some of the descriptions will already be in the master, but it doesn't make
any difference; you're only replacing them with the same description).  If you
are keeping duplicate names that are different programs, it gets a little more
complicated (you can't just "keep duplicates" or you will duplicate the names
in the master that are already there from DIR61), but I've got two suggestions.
If you're also allowing callers to upload file names that already exist, then
take a look at the second method (add "begin on line" where appropriate below).

1. After DIR61 is sorted, temporarily rename it something else.

ren d:\pcb\dirc\dir61 d:\pcb\dirc\tempname

2. Compare the master to it (this strips all file names already in the master,
   but since you're using a third output file, NEWDIR61, the original won't be
   changed, and NEWDIR61 may not even exist).  Use whatever paths are correct.

MERGE /1 d:\pcb\dirc\master d:\pcb\dirc\tempname d:\pcb\dirc\newdir61

3. Next run ADDTODAY to "upload" descriptions to DIR61, which creates it again,
   then sort it with PCBFiler and use SORT.BAT to insure dates are in the right
   order.  To be safe, add a line to strip any errors like was done above.

ADDTODAY addlist c:\pcb\dirc\dir61 ...
PCBFILER /sort;uploads ...
call SORT.BAT dir61
if errorlevel==1 MERGE /o /k /ea /f1 /f3ddt /td:\temp\error.d61+n /ld:\temp\d61error.log /vl d:\pcb\dirc\dir61

4. Now merge DIR61 with NEWDIR61 and keep duplicates, then merge the master.

MERGE /o /k /f1 /f3ddt d:\pcb\dirc\dir61 d:\pcb\dirc\newdir61
MERGE /o /k /f1 /f3ddt d:\pcb\dirc\newdir61 d:\pcb\dirc\master

5. Finally, merge DIR61 with the original to add ADDTODAY's files, delete the
   work files, and rename the original back to DIR61.

MERGE /o /k /f1 /f3ddt d:\pcb\dirc\dir61 d:\pcb\dirc\tempname
del d:\pcb\dirc\newdir61
del d:\pcb\dirc\dir61
ren d:\pcb\dirc\tempname d:\pcb\dirc\dir61

All that's left is to resort DIR61 on date descending, and zip the master.  The
above is probably the safest way.  However, there are others, and another is
below (you may come up with a better idea of your own).  Even if you're keeping
duplicate names because they are really different programs, I highly doubt that
two of them would also have the same date and size.  In that case, you could
merge DIR61 using "/f1 /f3ddt /f2rj8 /ns" after running ADDTODAY and sorting on
name ascending (still use SORT.BAT first to make sure the dates are in the
right sequence).  Even though the sizes aren't sorted in ascending order, you
won't get a sequence error when /ns is used, and the dates will still merge in
descending order for files with the same name.  If by chance any files have the
same name and date, the smallest will come first.  The "rj8" will right justify
the size in a field of 8 spaces, big enough for a 99 meg file!  That's also the
largest size file you can have in a DIR description and still use MERGE without
getting a key error (the name and size run together if it's any bigger).  The
"right justify" isn't really needed, but I wanted to show how to merge numbers.
The existing listings in DIR61 will just replace themselves in the master.

ADDTODAY addlist c:\pcb\dirc\dir61 ...
PCBFILER /sort;uploads ...
call SORT.BAT dir61
if errorlevel==1 MERGE /o /k /ea /f1 /f3ddt /td:\temp\error.d61+n /ld:\temp\d61error.log /vl d:\pcb\dirc\dir61
MERGE /ns /o /f1 /f3ddt /f2rj8 d:\pcb\dirc\dir61 d:\pcb\dirc\master

If you want to be even more cautious, use additional fields and add /eo (it is
needed so that if any fields are missing, you don't halt with an error and the
order of the records is not changed because of MERGE's error swap/force out):

MERGE /ns /eo /o /f1 /f3ddt /f2 /f4 /f5 /f6 d:\pcb\dirc\dir61 d:\pcb\dirc\master