N4GN.com DX Log Search
How does it Work?

Lots of people have asked how my e-mail log searches work. Well, here's my attempt to answer that question!

First, I assign the e-mail addresses. This is where it really comes in handy to have my own domain name (N4GN.com). I currently have e-mail log searches for several DXpeditions, but we'll use TXØDX as an example here.

<tx0dx-log@n4gn.com> is the address where log search requests should be sent.

<search-result@n4gn.com> is the address from which the log search reply message is (apparently) sent back to the requestor.

<tx0dx-help@n4gn.com> is the address where people can send angry hate mail if one of their "100% solid QSOs" is not found in the log search for some reason. More enlightened DXers also use this address to provide helpful information that might allow someone to manually look at the original logs and possibly even recover a "lost" QSO due to a handwriting problem, data entry error, etc.

I'll refer to these as the "log", "result" and "help" addresses. If you have a dedicated domain name, the easiest thing to do is set up these addresses in your aliases file. Just pipe the log address to the appropriate script (more on this later) and redirect the help address to the unlucky person who has to deal with all the "not in log" problems. Or you might wish to sort those e-mails into a separate directory in your own mailbox if you were the one left holding the short stick.

Since some misguided DXers will try to send e-mail to the result address, you should at least be sure it doesn't go somewhere that will cause a mail loop. I prefer to send it to /dev/null, and I make it clear in the result message that any replies sent to that address are deleted automatically.

The next step is to format the log data and put it in place. For this task, I let DX4WIN do all the heavy lifting. I have used DX4WIN for many years, and I'm thoroughly impressed with its capabilities. As an active DXpeditioner, I find the ability to store logs for multiple operations in the same database to be very handy. I just assign each operation a different group number and print that call sign on the QSL labels.

Another great feature of DX4WIN is the ability to write custom import and export filters. Once I have all the log data in DX4WIN, I use a simple custom export filter to create the log search data file, which is a plain text file that looks something like this (call sign, space, band, mode):

G4BUE 160C
ON5NT 160C
TF3IRA 160C
EA8AH 80S
W4DR 80C
N4OKX 20S
K4WW 15R
W4DR 12C
WD4NGB 10S
9V1YC 10C

If you want to include other information (e.g., the operator who made that QSO in a multi-operator DXpedition, or the year of the QSO for a long-term DX log), you can simply add it to each line. Note that you should NEVER include date and time information in this file! Sadly, there are some unscrupulous DXers who would misuse this information to try to obtain QSLs for QSOs they didn't make. Including date/time information in your log search might even jeopardize the DXCC accreditation of your operation.

Name the data file whatever you want (e.g., tx0dxlog.txt) and upload it to the server.

Now comes the core task of processing the incoming request e-mail, doing the actual log search, and composing the reply message containing the search results. For these functions, I use a combination of procmail, formail and Perl. If you'd like to learn about these programs, there are several good sites available. Two good places to start are www.procmail.org and www.perldoc.com. I make no claims to being a software guy, and I'm certainly NOT an expert on Perl, formail nor procmail. For these reasons, not to mention the security concerns, I have chosen not to make my scripts public. But the basic process is described here.

I first pass the incoming request e-mail through a procmail run command file (tx0dx.rc in this example). I test for mail loops and bounced messages, then I grab the "from" address, subject header and first 100 characters of the message body:

     FROMADDR=`$FORMAIL -rt -xTo:`
     BODY=`sed -e '1,/^$/ d' | tr '\n' ' ' | dd bs=1 count=100`
     :0
     * ^Subject: *['"]?\/.*
     {
         SUBJ=$MATCH
     }

Then I use formail to format a reply message with the appropriate "to" address, "from" address, subject line, etc. The subject and message body grabbed above (which should contain at least one call sign if the requestor followed directions) are passed to a Perl script (tx0dxsearch, in this case) that generates the body of the reply message. The outgoing "result" message is then piped to sendmail, which sends it on it's merry way to the requesting DXer.

This is all accomplished with the following procmail run command recipe:

       :0 chi
       | ($FORMAIL -rt \
       -I"From: N4GN.com Log Search Robot <search-result@n4gn.com>" \
       -I"Subject: TX0DX log search" \
       -A"X-Loop: tx0dx-log";\
       $SEARCHDIR/tx0dxsearch $FROMADDR $SUBJ $BODY) | $SENDMAIL -oi -t

Note the addition of the "X-Loop:" header element. This is a good way of marking a message in order to avoid mail loops.

The tx0dxsearch Perl script first converts the text from the subject and body to capital letters, then checks for any text matching the regular expression for a valid amateur call sign. Certain invalid "calls" like C6A, VP2V or the DXpedition call itself are thrown out. It amazes me how many people try to search for the DXpedition call sign in the DXpedition log! Others apparently think a human actually processes their request, so they include extraneous instructions like "Please search the FW8ZZ log for my call sign."

If no valid call is found in the subject or the body, the log search robot even checks the requestor's e-mail address to see if it can find a call sign there. If no call sign can be found, the log search robot gives up and sends a reply message explaining that fact.

If at least one valid search call sign is found, the top part of the normal reply message text is generated. This is a good place to thank the sponsors and include any important information about the DXpedition.

Next, the actual log search is performed. This part is simpler than most people realize. The Perl script simply looks through the log data file and prints any line that contains the full call sign (including any variant of the call containing a slash "/"). Care is taken such that a search for K4AB does not also print QSOs for K4ABC. I also include a counter to indicate the total number of QSOs found. If no QSOs are found, the "sorry" message is printed.

The remainder of the message text is simply printed at the bottom. This is a good place to include QSL information, details about the "help" address to use if any QSOs appear to be missing, the URL for the DXpedition web site, etc.

Finally, the web form simply takes all the required input from the user (DX log to search, calls to be searched for and e-mail address where the reply should be sent) and pipes that information to the same Perl script used for requests received via e-mail. The Perl script processes the search from that point just as if it had been received via e-mail.

So I hope that answers your questions about the N4GN.com e-mail log search. If not, feel free to drop me a line, and I'll try to provide more details.