[unisog] IPS

Gary Flynn flynngn at jmu.edu
Wed Feb 9 22:31:41 GMT 2005

Hunt,Keith A wrote:
>>2. You could drop sessions associated with those packets.
>>    e.g. Drop an SMTP session if an attempt is made to
>>         transfer malicious content (generally not a good
>>         idea by the way but useful as an example)
> I have been wondering about the implications of dropping an SMTP session
> like this. Could you expound on why you think this is not a good idea?

In general, because I think it is better handled in a mail
server, SMTP application proxy firewall, or something else
that has more fully developed email (as opposed to simply
SMTP) understanding and capabilities than a network state
engine. Some possible problems include:

1. There is no notification to the recipient that a message
    to them was tossed. In some cases, such as when the IPS
    accurately identifies a virus, that is an advantage.
    However, in other cases its troublesome. For example, if
    you're letting your IPS enforce attachment policy by
    blocking executable attachments.

2. I'm not sure if the following scenario is a real problem
    or not but it is something that I worry about when
    thinking about controlling SMTP at the network layer.
    If I'm a mail server with ten messages to transmit to
    you and the first message in the queue is interpreted
    as malicious by your IPS, the IPS will drop the
    connection while trying to send the first message.
    What happens to the nine innocent messages in the
    queue behind the malicious one? Would the server try
    other messages bound to the same destination or
    recognize they're all bound for the same destination
    so wait and try all of them later? Would all servers
    act the same way?

An actual SMTP gateway, such as a mail server or
proxy firewall, can receive messages and deal with
them locally one by one. It can also add functionality
such as attachment quarantine areas, user notification,
message rewriting, and, in general, be more intelligent
about handling the higher concepts in e-mail than a
network disconnect...stateful or otherwise.

That said, in some circumstances, the IPS comes in handy
for SMTP handling. Some examples:

a) I remember a MyDoom outbreak where we were getting
    thousands of incoming infected messages per hour which
    drove the mail server CPU up pretty high trying to scan
    all those attachments. Temporarily, the IPS could be
    used to take the bulk of the load off the mail server
    (assuming it had more CPU horsepower to spare than the
    mail servers). I tested this and it worked rather well
    once I properly handled the connections.

    When handling SMTP problems in an IPS you MUST send RST
    packets to both sides as well as any internal dropping
    that you do inside the IPS. Otherwise, in periods of
    high activity or possibly, as Valdis pointed out, when
    hosts retry quickly, your mail server may be starved
    for resources due to the number of connections left
    open. I learned that the hard way.

b) Certain patterns may be easier to parse in the IPS. Our
    Mirapoint mail server filters support only simple
    character matches with wildcards. Our IPS supports
    regular expressions. So our IPS does some parsing for
    messages meant to exploit Outlook that couldn't be
    done in the mail server.

c) Similar to (b), the IPS may have anomaly filtering
    capabilities not available to the mail server. I've
    seen waves of SPAM and Phishing messages that are
    flagged by our IPS as having unique characteristics
    that violate or stretch various (ahem) standards.
    When I can find such a characteristic that is
    unique to SPAM or phishing, it is tempting to use
    the IPS to control it. Then again, there are some
    characteristics that 999 out of 1000 times indicate
    something unwanted and then one silly application
    that just has to do something its own way. I
    may have to ignore the anomalies in the IPS and let
    the mail server dump 999 messages in the Junkmail
    folder rather than block that single legitimate
    message. Ugh. Especially when its phishing.

    SMTP/MIME anomalies seem to be a way of life on the
    Internet. By that I mean that there are a lot of (ahem)
    interesting interpretations of (ahem) standards.
    Something more complete than a state engine may do
    something like adjust SPAMASSASSIN weights based on
    certain message characteristics like missing or
    duplicate MIME borders or illegal base64 encoding.

Gary Flynn
Security Engineer
James Madison University

More information about the unisog mailing list