I will share with you some experiences from implementing a good SMTP engine to avoid being spammed, being categorized as spam etc. This article is product independent therefore if you run Exchange, Windows SMTP, ScanMail, Notes or whatever, this is something good to consider.
This guide specifically focuses on larger organizations, but of course, large/small is just a definition based on the number of users but you may apply them if you so wish.
1. Always put a dedicated SMTP box in a DMZ or internal network. SMTP is relatively a simple and fast protocol so it doesn't really require a super end machine. Have redundancy if needed by simply doing an internal DNS MX with weights without the need of expensive load balancing hardware. Or, if you run Windows, use NLB.
2. Put Anti-malware and Antispam scanning on both the SMTP gateway and your internal email server. Disable NDRs if possible. Do not cross scan between your host based antimalware engine and your protocol based antimalware engine. This can slow down it's performance tremendeously.
3. Register all your email servers that eventually send emails out with a reverse lookup DNS. Do not send emails out directly from internal email servers unless they are equally protected like your SMTP engine or using these tips.
4. All SMTP talkers (this could be your email server or even your client if they send emails out directly using the SMTP protocol) should use RFC1918 addresses not Live IPs for all internal hosts. This because all SMTP chatter will stamp addresses in their headers and if you use a non RFC1918 number, there's a chance the emails will get checked by your recipient's host email server and perform a reverse lookup and of course, you won't be the owner of that Live IP and you can be categorized as spam.
5. Place email send and receive limits. The last thing you need is an email server retrieving 100MB of attachments. There's always secure file shares for that.
6. Verify internal users sending email using your SMTP with your LDAP or similar. Some form of authentication is a good start. As for external users sending you emails, you cant authenticate ..too bad. If possible disable NDR for non-existent recipients. The NDR should be generated by the sender's email server, not yours!
7. Do not run SMTP engines/filtering on your firewall!. Firewalls do not need this unnecessary burden because your SMTP engine is suppose to clean up messages.
8. Do not use your internal mail server as your secondary MX, you are better of not having a secondary MX. Internal mail servers are where all the "juices" are, so if you get whacked, your data is at risk. Spammers have known this trick and sometimes send emails to secondary MX servers in hopes that there's no antispam/antimalware engine installed on them.
9. Create a SPF record on your DNS. Sender Privacy Framework is very easy to setup and can be implemented in mere minutes.
10. Disable relay on all email servers, both internal, external etc.
11. Test all your SMTP server's implementation by running auditing tools such as Nessus - SMTP/product TCP/IP based auditing, www.dnsreport.com - DNS configuration test, abuse.net/relay.html- relay test,
Any of those terms you don't know or want more info, you can write me or simply Google it up.