I have done a number of extensions to Sendmail which suited my needs. Alot of them involved editing the sendmail.cf file directly. I won't supply these modifications because I do not have them in M4 format. But the things I have in a standard format, I will supply.
The concept of this script was originally taken from the POP before SMTP page by John Levine, Scott Hazen Mueller, and Neil Harkins. I had a personal distaste for the 'tail -f' method of watching logfiles, because it breaks when the files are rotated. So I decided to write a different script which used a FIFO from syslog instead. This script has since been extended by Harlann Stenn.Original popauther.pl
Harlann Stenn's popauther.pl
The software on this page has not been updated in quite some time, and in general I recommend against this approach these days (see information about SMTP AUTH at the bottom of this page). As of 2004, William W. Kimball Jr. has decided this script indeed does have some merit, and has picked up the reigns extending the functionality massively. His work was specifically for use with Postfix, and may or may not carry over to Sendmail. You can check his additions from his page about popauth3.
The script's use is explained within the the file. If you have any questions about it's implementation you can contact me at firstname.lastname@example.org, and I will likely endevour to talk you out of using this script. The M4 rules which allow the use of this script can be found at the following pages.Using check_* in sendmail 8.8 by Claus Aßmann
Extended spam filter: check_local 3.15 for sendmail 8.9 by Jan Krüger
Extended spam filter: check_local 4.0 for sendmail 8.10 by Jan Krüger
I have personally used both Claus' 8.8.x and Jan's 8.9.x rulesets. Both are quite excellent and extend Sendmail in a variety of ways. These rulesets are of particular usefullness when you have to deal with a large percentage of "Roaming" users who need to relay mail through your servers.
A note on Sendmail 8.10.x, each new major version of Sendmail includes more features to prevent spam and to control relaying of mail. Sendmail 8.10.x introduces the SMTP AUTH. This is a far better approach to limiting relaying and supporting relaying. Though admittedly there are many possible hangups to implementation. I am currently in discussions with Sendmail, Inc. about some of these and will update this when I get more information. Here are some links with more information on SMTP AUTH.Sendmail.net's article on SMTP AUTH
Claus Aßmann's SMTP AUTH page
Cyrus SASL Library page