Opinion: Should web clients be required to have DNS PTR records?

[D. Stussy]

I've noted alot of activity coming from IP addresses that don't have PTR
records in DNS which appear to be potentially malicious in nature.
Conversely, I've noted that the majority (but not all) requests which
appear to be user requests do have valid (yet generic) DNS hostnames, and
similarly with the well-known web robots and spiders. Therefore, I am
considering the denial of requests where the client IP has no [reverse]
hostname.

Implementing the check is easy (e.g. for Apache):

RewriteCond %{REMOTE_ADDR}#%{REMOTE_HOST} ^([^#]*)#\1$ [NC]
RewriteRule . - [E=noptr:1]

...Where environment variable "noptr" is set to 1 when the PTR record is
missing (i.e. remote host name is the same as the remote address,
indicating no name available), and left undefined a name exists.

With e-mail, the equivalent policy has been popular for some time as a
spam-cutting measure. However, my question for the group is for opinions
regarding this proposed policy. Comments?

[Tim Watts]

Post by D. Stussy
I've noted alot of activity coming from IP addresses that don't have PTR
records in DNS which appear to be potentially malicious in nature.
Conversely, I've noted that the majority (but not all) requests which
appear to be user requests do have valid (yet generic) DNS hostnames, and
similarly with the well-known web robots and spiders. Therefore, I am
considering the denial of requests where the client IP has no [reverse]
hostname.
RewriteCond %{REMOTE_ADDR}#%{REMOTE_HOST} ^([^#]*)#\1$ [NC]
RewriteRule . - [E=noptr:1]
...Where environment variable "noptr" is set to 1 when the PTR record is
missing (i.e. remote host name is the same as the remote address,
indicating no name available), and left undefined a name exists.
With e-mail, the equivalent policy has been popular for some time as a
spam-cutting measure. However, my question for the group is for opinions
regarding this proposed policy. Comments?

Lots' of half arsed ISPs (including mobile ISPs) fail to set PTR records.
It's reasonable to expect a well configured DNS for a mail server, but
rather less so for a random internet client.

So personally I would not do that.

[Eli the Bearded]

In comp.infosystems.www.misc,

Post by D. Stussy
Conversely, I've noted that the majority (but not all) requests which
appear to be user requests do have valid (yet generic) DNS hostnames, and
similarly with the well-known web robots and spiders. Therefore, I am
considering the denial of requests where the client IP has no [reverse]
hostname.

If it's your server, you can do what you want. I think that's foolish.
It will not stop malicious traffic to your servers. It will stop some
legitimate traffic. It will make your server very slow: DNS is not instant,
and a non-responsive name server will block your apache for a noticable
amount of time.

There is reason the default httpd.conf file has had this for a *long* time:

# HostnameLookups: Log the names of clients or just their IP numbers
# e.g. www.apache.org (on) or 204.62.129.132 (off)
# The default is off because it'd be overall better for the net if people
# had to knowingly turn this feature on.

HostnameLookups off

If you do turn it on, be aware that the default logging for apache will
log only the looked up name and not the IP address when lookups have been
done. Since multiple IP addresses can map to a single name this will result
in a loss of information in your log. And since reverse lookups might not
match forward lookups (a standard spam check) you can log misleading
information in your log.

Elijah
------
fixing the LogFormat left as an exercise for the reader

[D. Stussy]

Post by Eli the Bearded
In comp.infosystems.www.misc,

Post by D. Stussy
Conversely, I've noted that the majority (but not all) requests which
appear to be user requests do have valid (yet generic) DNS hostnames, and
similarly with the well-known web robots and spiders. Therefore, I am
considering the denial of requests where the client IP has no [reverse]
hostname.

If it's your server, you can do what you want. I think that's foolish.
It will not stop malicious traffic to your servers. It will stop some
legitimate traffic. It will make your server very slow: DNS is not instant,
and a non-responsive name server will block your apache for a noticable
amount of time.
# HostnameLookups: Log the names of clients or just their IP numbers
# e.g. www.apache.org (on) or 204.62.129.132 (off)
# The default is off because it'd be overall better for the net if people
# had to knowingly turn this feature on.
HostnameLookups off
If you do turn it on, be aware that the default logging for apache will
log only the looked up name and not the IP address when lookups have been
done. Since multiple IP addresses can map to a single name this will result
in a loss of information in your log. And since reverse lookups might not
match forward lookups (a standard spam check) you can log misleading
information in your log.

1) I already have host name lookups enabled for other reasons. I do not
see a signficant slowdown in that all non-cgi queries get satisfied within
1 second.

2) I don't use the default log format. My logs contain both the client
host name and address fields.

[crankypuss]

Post by D. Stussy
1) I already have host name lookups enabled for other reasons. I do not
see a signficant slowdown in that all non-cgi queries get satisfied within
1 second.

That sounds pretty slow to me.

[Eli the Bearded]

In comp.infosystems.www.misc,

Post by D. Stussy

Post by Eli the Bearded
If it's your server, you can do what you want. I think that's foolish.

My opinion has not changed.

Post by D. Stussy
1) I already have host name lookups enabled for other reasons. I do not
see a signficant slowdown in that all non-cgi queries get satisfied within
1 second.

Either you are not very precise in measuring the time-to-serve, or you
are a lot more forgiving of slowness than I am. Possibly both.

As an experiment, take all the IP addresses from a day of your logs
and on a different network, just do lookups on all of them and time
how long it takes. That's how much you are slowing down your webserver
every day.

I haven't done that in years (now I just do off-line geo-location on
the IPs) but it used to be about two hours a day for my servers. It
will vary a lot with the sorts of traffic you get. A lot from one site?
You get a nice caching benefit. Much of it from slow third-world ISPs?
You take a big hit on the distant DNS.

Post by D. Stussy
2) I don't use the default log format. My logs contain both the client
host name and address fields.

No problem for you on that front, then.

Elijah
------
thinks D. Stussy has made up his mind and just wants others to follow

[D. Stussy]

Post by Eli the Bearded
In comp.infosystems.www.misc,

Post by D. Stussy

Post by Eli the Bearded
If it's your server, you can do what you want. I think that's foolish.

My opinion has not changed.

Post by D. Stussy
1) I already have host name lookups enabled for other reasons. I do not
see a signficant slowdown in that all non-cgi queries get satisfied within
1 second.

Either you are not very precise in measuring the time-to-serve, or you
are a lot more forgiving of slowness than I am. Possibly both.

You're correct: I'm not being precise. Many queries are answered in less
than 1 millisecond. As long as an end-user sees a response within a second
of their query, that's what is important.

Post by Eli the Bearded
As an experiment, take all the IP addresses from a day of your logs
and on a different network, just do lookups on all of them and time
how long it takes. That's how much you are slowing down your webserver
every day.

As long as the response time to the client is reasonable, I don't care
about the DNS delay. Also, most ISPs are setting generic hostnames for
their dynamic IPs so they will be unaffected (unlike in the SMTP service
where such is checked for and rejected).

Post by Eli the Bearded
I haven't done that in years (now I just do off-line geo-location on
the IPs) but it used to be about two hours a day for my servers. It
will vary a lot with the sorts of traffic you get. A lot from one site?
You get a nice caching benefit. Much of it from slow third-world ISPs?
You take a big hit on the distant DNS.

Post by D. Stussy
2) I don't use the default log format. My logs contain both the client
host name and address fields.

No problem for you on that front, then.
thinks D. Stussy has made up his mind and just wants others to follow

Actually, no. To date, all I implemented was to set an environment
variable and add it to the information logged so I can see which queries
would be affected.

[Just Another UBE Reporter]

Post by D. Stussy
I've noted alot of activity coming from IP addresses that don't have
PTR records in DNS which appear to be potentially malicious in nature.
Conversely, I've noted that the majority (but not all) requests which
appear to be user requests do have valid (yet generic) DNS hostnames,
and similarly with the well-known web robots and spiders. Therefore,
I am considering the denial of requests where the client IP has no
[reverse] hostname.

It all depends on what/how much legitimate traffic you are willing to
lose.

$HOBBY_DOMAIN has been viewed by a number of machines at local school
districts where the end user terminal IP addresses do not resolve to a
host name. While your proposal would likely not work for me, YMMV.

[D. Stussy]

Post by D. Stussy
I've noted alot of activity coming from IP addresses that don't have PTR
records in DNS which appear to be potentially malicious in nature....

Although this is a personal choice, and I considered the comments against a
policy of denying HTTP server connections based on a lack of a PTR record,
I find that such a denial may be proper based on the habits observed.

Here is what I noted:
1) About half of any such requests get rejected for other reasons which
result in either a 4xx or 5xx code.
2) With what remains, about two-thirds appear to be spiders which are
banned by "/robots.txt" anyway.
3) This leaves about one-sixth of all such requests. Most of these are
hotlinking to image files, which I also block.

I did run the various IP addresses through whois to determine if any ISPs'
dyanmic assignments were being used. I did not find much there. Most ISPs
seem to have dummy names for their dynamic assignments (i.e. PTR records
exist). Therefore, the conditions arising in the comments arguing against
this policy did not seem to materialize in my sample of requests monitored.

A handful of the reverse names came back with PTR records that contained
"unassigned" or "reserved" in the dummy host name (e.g.
"reserved.voxility.net" - 93.114.40.173). Those are still suspicious,
especially since they were detected as other improper requests that should
be denied by policy (like an off-site hotlink to an image file from someone
else's site).