[pfSense] Annoying Comcast Issue When Changing Hardware

[Aaron C. de Bruyn]

Spent about an hour beating my head against the wall with this issue,
hopefully this will save others some time.

We had a stand-alone pfSense router.
We just purchased two machines from ixsystems and were preparing them to be
a failover pair of pfSense routers and then decommission the smaller older
box.

While we were installing the new servers, the HDD in the old firewall died.

We figured we would just get the two new boxes up.

Plugged them into the Comcast modem and configured everything.

Comcast assigned us a /28 a while back and we were using a handful of IPs
to access various internal services over HTTPS.

The /28 looked roughly like:
.1 - router1
.2 - router2
.3 - exchange (CARP)
.4 - remote (CARP)
.5 - VPN (CARP)
.6 - spamfilter (physical machine)
...etc

After everything was configured, I had someone test remotely that they
could access the interface for router1 and router2 remotely.

I then went home to finish up a few config details remotely.

When I got home, I found I could access router1 and router2 as well as the
physical spam filter, but I couldn't access any of the HTTPS services on
the CARP IPs.

I checked my NAT rules about 100 times, looked through firewall logs, and
found nothing.

Finally I connected in to the spam filter (linux box) and ran 'openssl
s_client -connect exchange.example.tld:4433' and noticed it worked
perfectly from a machine on the same WAN segment. ...but not remotely.

I called Comcast and had them remotely reboot the modem. Everything
immediately came up and started working perfectly.

Hopefully this will save someone time. Reboot the brain-damaged Netgear
CPE after swapping hardware around.

-A

[Ryan Coleman]

I’m not running CARP but I am doing many things like yours on my Comcast Business account…

I’ve never had that happen - and I think my modem only reboots when I lose power (it’s on the UPS but not on battery - by design).

Which modem did they install? I suspect it’s a firmware “feature” of that modem.

Spent about an hour beating my head against the wall with this issue, hopefully this will save others some time.
We had a stand-alone pfSense router.
We just purchased two machines from ixsystems and were preparing them to be a failover pair of pfSense routers and then decommission the smaller older box.
While we were installing the new servers, the HDD in the old firewall died.
We figured we would just get the two new boxes up.
Plugged them into the Comcast modem and configured everything.
Comcast assigned us a /28 a while back and we were using a handful of IPs to access various internal services over HTTPS.
.1 - router1
.2 - router2
.3 - exchange (CARP)
.4 - remote (CARP)
.5 - VPN (CARP)
.6 - spamfilter (physical machine)
...etc
After everything was configured, I had someone test remotely that they could access the interface for router1 and router2 remotely.
I then went home to finish up a few config details remotely.
When I got home, I found I could access router1 and router2 as well as the physical spam filter, but I couldn't access any of the HTTPS services on the CARP IPs.
I checked my NAT rules about 100 times, looked through firewall logs, and found nothing.
Finally I connected in to the spam filter (linux box) and ran 'openssl s_client -connect exchange.example.tld:4433' and noticed it worked perfectly from a machine on the same WAN segment. ...but not remotely.
I called Comcast and had them remotely reboot the modem. Everything immediately came up and started working perfectly.
Hopefully this will save someone time. Reboot the brain-damaged Netgear CPE after swapping hardware around.
-A
_______________________________________________
List mailing list
https://lists.pfsense.org/mailman/listinfo/list

[compdoc]

Post by Aaron C. de Bruyn
I called Comcast and had them remotely reboot the modem.

Whenever I connect a different network card to my home Comcast modem, I have
to power cycle the modem for it come up. I think it keys off the MAC address
of the old card, and won't accept the new one until then. I get a new IP
address each time I test firewall builds. Not exactly the same situation,
but something like.

[Aaron C. de Bruyn]

Yeah--I figured it was related to the MAC address.

It'd be nice to know why the Comcast equipment does that--I've never run in
to it with other providers.

-A

Post by compdoc

Post by Aaron C. de Bruyn
I called Comcast and had them remotely reboot the modem.

Whenever I connect a different network card to my home Comcast modem, I have
to power cycle the modem for it come up. I think it keys off the MAC address
of the old card, and won't accept the new one until then. I get a new IP
address each time I test firewall builds. Not exactly the same situation,
but something like.
_______________________________________________
List mailing list
https://lists.pfsense.org/mailman/listinfo/list

[Ryan Coleman]

You may want to make sure the DHCP server is disabled on the modem completely. I’ve noticed that caused issues in the past for me.
The default user/pass is cusadmin/highspeed on those modems.

Post by Aaron C. de Bruyn
Yeah--I figured it was related to the MAC address.
It'd be nice to know why the Comcast equipment does that--I've never run in to it with other providers.
-A

Post by Aaron C. de Bruyn
I called Comcast and had them remotely reboot the modem.

Whenever I connect a different network card to my home Comcast modem, I have
to power cycle the modem for it come up. I think it keys off the MAC address
of the old card, and won't accept the new one until then. I get a new IP
address each time I test firewall builds. Not exactly the same situation,
but something like.
_______________________________________________
List mailing list
https://lists.pfsense.org/mailman/listinfo/list
_______________________________________________
List mailing list
https://lists.pfsense.org/mailman/listinfo/list

[compdoc]

Post by Ryan Coleman
You may want to make sure the DHCP server is disabled on the modem

completely.



It's a cable modem that I guess is in bridge mode, and they don't let me
mess with settings. Anyway, I think the DHCP server is in their headend
somewhere.



I'm just glad it's not like the old days when Comcast wouldn't let you
switch network cards without contacting them.

[Aaron C. de Bruyn]

Yeah--I had gone over all the 'usual' stuff. DHCP disabled, firewall
settings disabled, Smart Packet Detection disabled.

-A

Post by Ryan Coleman
You may want to make sure the DHCP server is disabled on the modem
completely. I’ve noticed that caused issues in the past for me.
The default user/pass is cusadmin/highspeed on those modems.
Yeah--I figured it was related to the MAC address.
It'd be nice to know why the Comcast equipment does that--I've never run
in to it with other providers.
-A

Post by compdoc

Post by Aaron C. de Bruyn
I called Comcast and had them remotely reboot the modem.

Whenever I connect a different network card to my home Comcast modem, I have
to power cycle the modem for it come up. I think it keys off the MAC address
of the old card, and won't accept the new one until then. I get a new IP
address each time I test firewall builds. Not exactly the same situation,
but something like.
_______________________________________________
List mailing list
https://lists.pfsense.org/mailman/listinfo/list

_______________________________________________
List mailing list
https://lists.pfsense.org/mailman/listinfo/list
_______________________________________________
List mailing list
https://lists.pfsense.org/mailman/listinfo/list

[Aaron C. de Bruyn]

It happens occasionally with their older SMC modems, but it seems to happen
frequently with the Netgear modems.

If you don't reboot the modem, it usually picks up on the changes within
5-15 minutes. Sometimes longer.

-A

I’m not running CARP but I am doing many things like yours on my Comcast
Business account

I’ve never had that happen - and I think my modem only reboots when I lose
power (it’s on the UPS but not on battery - by design).
Which modem did they install? I suspect it’s a firmware “feature” of that
modem.

Post by Aaron C. de Bruyn
Spent about an hour beating my head against the wall with this issue,

hopefully this will save others some time.

Post by Aaron C. de Bruyn
We had a stand-alone pfSense router.
We just purchased two machines from ixsystems and were preparing them to

be a failover pair of pfSense routers and then decommission the smaller
older box.

Post by Aaron C. de Bruyn
While we were installing the new servers, the HDD in the old firewall

died.

Post by Aaron C. de Bruyn
We figured we would just get the two new boxes up.
Plugged them into the Comcast modem and configured everything.
Comcast assigned us a /28 a while back and we were using a handful of

IPs to access various internal services over HTTPS.

Post by Aaron C. de Bruyn
.1 - router1
.2 - router2
.3 - exchange (CARP)
.4 - remote (CARP)
.5 - VPN (CARP)
.6 - spamfilter (physical machine)
...etc
After everything was configured, I had someone test remotely that they

could access the interface for router1 and router2 remotely.

Post by Aaron C. de Bruyn
I then went home to finish up a few config details remotely.
When I got home, I found I could access router1 and router2 as well as

the physical spam filter, but I couldn't access any of the HTTPS services
on the CARP IPs.

Post by Aaron C. de Bruyn
I checked my NAT rules about 100 times, looked through firewall logs,

and found nothing.

Post by Aaron C. de Bruyn
Finally I connected in to the spam filter (linux box) and ran 'openssl

s_client -connect exchange.example.tld:4433' and noticed it worked
perfectly from a machine on the same WAN segment. ...but not remotely.

Post by Aaron C. de Bruyn
I called Comcast and had them remotely reboot the modem. Everything

immediately came up and started working perfectly.

Post by Aaron C. de Bruyn
Hopefully this will save someone time. Reboot the brain-damaged Netgear

CPE after swapping hardware around.

Post by Aaron C. de Bruyn
-A
_______________________________________________
List mailing list
https://lists.pfsense.org/mailman/listinfo/list

_______________________________________________
List mailing list
https://lists.pfsense.org/mailman/listinfo/list

[Christoph Hanle]

Post by Aaron C. de Bruyn
We figured we would just get the two new boxes up.

[...]

Post by Aaron C. de Bruyn
I called Comcast and had them remotely reboot the modem. Everything
immediately came up and started working perfectly.

Hi Aaron,
this is no unexpected behavior.
Arp table on the router or modem has to be cleared and a new one has to
be build up.
But don't worry: you are not the first one and you will not be the last
one who will spend some time due to this feature (-:

bye
Christoph

[Moshe Katz]

Post by Aaron C. de Bruyn
Spent about an hour beating my head against the wall with this issue,
hopefully this will save others some time.
We had a stand-alone pfSense router.
We just purchased two machines from ixsystems and were preparing them to
be a failover pair of pfSense routers and then decommission the smaller
older box.
While we were installing the new servers, the HDD in the old firewall died.
We figured we would just get the two new boxes up.
Plugged them into the Comcast modem and configured everything.
Comcast assigned us a /28 a while back and we were using a handful of IPs
to access various internal services over HTTPS.
.1 - router1
.2 - router2
.3 - exchange (CARP)
.4 - remote (CARP)
.5 - VPN (CARP)
.6 - spamfilter (physical machine)
...etc
After everything was configured, I had someone test remotely that they
could access the interface for router1 and router2 remotely.
I then went home to finish up a few config details remotely.
When I got home, I found I could access router1 and router2 as well as the
physical spam filter, but I couldn't access any of the HTTPS services on
the CARP IPs.
I checked my NAT rules about 100 times, looked through firewall logs, and
found nothing.
Finally I connected in to the spam filter (linux box) and ran 'openssl
s_client -connect exchange.example.tld:4433' and noticed it worked
perfectly from a machine on the same WAN segment. ...but not remotely.
I called Comcast and had them remotely reboot the modem. Everything
immediately came up and started working perfectly.
Hopefully this will save someone time. Reboot the brain-damaged Netgear
CPE after swapping hardware around.
-A

Hi Aaron,

Most cable modems I have worked with in the US (on Comcast, Optimum, and
RCN) all do ARP caching, so you need to reboot them when you change the
connected device (or you need to clone the old device's MAC address).

Actually though, working with DSL is worse. Verizon DSL does ARP caching
in the Central Office for up to four hours. I have found that replacing
equipment hooked up to Verison DSL, it is best to already be on the phone
with Verizon support to have them manually clear the cache. At least
rebooting the cable modem is something you can do yourself.

Moshe

--
Moshe Katz
-- ***@ymkatz.net
-- [phone]

[Aaron C. de Bruyn]

Good to know.

Slightly OT, but why would they have ARP cache timeouts of four hours?
What benefit do you get with such high cache times as opposed to the
obvious support calls you will get when equipment is swapped around?

-A

Post by Christoph Hanle

Post by Aaron C. de Bruyn
Spent about an hour beating my head against the wall with this issue,
hopefully this will save others some time.
We had a stand-alone pfSense router.
We just purchased two machines from ixsystems and were preparing them to
be a failover pair of pfSense routers and then decommission the smaller
older box.
While we were installing the new servers, the HDD in the old firewall died.
We figured we would just get the two new boxes up.
Plugged them into the Comcast modem and configured everything.
Comcast assigned us a /28 a while back and we were using a handful of IPs
to access various internal services over HTTPS.
.1 - router1
.2 - router2
.3 - exchange (CARP)
.4 - remote (CARP)
.5 - VPN (CARP)
.6 - spamfilter (physical machine)
...etc
After everything was configured, I had someone test remotely that they
could access the interface for router1 and router2 remotely.
I then went home to finish up a few config details remotely.
When I got home, I found I could access router1 and router2 as well as
the physical spam filter, but I couldn't access any of the HTTPS services
on the CARP IPs.
I checked my NAT rules about 100 times, looked through firewall logs, and
found nothing.
Finally I connected in to the spam filter (linux box) and ran 'openssl
s_client -connect exchange.example.tld:4433' and noticed it worked
perfectly from a machine on the same WAN segment. ...but not remotely.
I called Comcast and had them remotely reboot the modem. Everything
immediately came up and started working perfectly.
Hopefully this will save someone time. Reboot the brain-damaged Netgear
CPE after swapping hardware around.
-A

Hi Aaron,
Most cable modems I have worked with in the US (on Comcast, Optimum, and
RCN) all do ARP caching, so you need to reboot them when you change the
connected device (or you need to clone the old device's MAC address).
Actually though, working with DSL is worse. Verizon DSL does ARP caching
in the Central Office for up to four hours. I have found that replacing
equipment hooked up to Verison DSL, it is best to already be on the phone
with Verizon support to have them manually clear the cache. At least
rebooting the cable modem is something you can do yourself.
Moshe
--
Moshe Katz
-- [phone]
_______________________________________________
List mailing list
https://lists.pfsense.org/mailman/listinfo/list

[Alexandre Paradis]

I have the same behavior with Videotron here in canada.

a power off for 2-3 minutes does the trick everytime.

Post by Aaron C. de Bruyn
Good to know.
Slightly OT, but why would they have ARP cache timeouts of four hours?
What benefit do you get with such high cache times as opposed to the
obvious support calls you will get when equipment is swapped around?
-A

Post by Christoph Hanle

Post by Aaron C. de Bruyn
Spent about an hour beating my head against the wall with this issue,
hopefully this will save others some time.
We had a stand-alone pfSense router.
We just purchased two machines from ixsystems and were preparing them to
be a failover pair of pfSense routers and then decommission the smaller
older box.
While we were installing the new servers, the HDD in the old firewall died.
We figured we would just get the two new boxes up.
Plugged them into the Comcast modem and configured everything.
Comcast assigned us a /28 a while back and we were using a handful of
IPs to access various internal services over HTTPS.
.1 - router1
.2 - router2
.3 - exchange (CARP)
.4 - remote (CARP)
.5 - VPN (CARP)
.6 - spamfilter (physical machine)
...etc
After everything was configured, I had someone test remotely that they
could access the interface for router1 and router2 remotely.
I then went home to finish up a few config details remotely.
When I got home, I found I could access router1 and router2 as well as
the physical spam filter, but I couldn't access any of the HTTPS services
on the CARP IPs.
I checked my NAT rules about 100 times, looked through firewall logs,
and found nothing.
Finally I connected in to the spam filter (linux box) and ran 'openssl
s_client -connect exchange.example.tld:4433' and noticed it worked
perfectly from a machine on the same WAN segment. ...but not remotely.
I called Comcast and had them remotely reboot the modem. Everything
immediately came up and started working perfectly.
Hopefully this will save someone time. Reboot the brain-damaged Netgear
CPE after swapping hardware around.
-A

Hi Aaron,
Most cable modems I have worked with in the US (on Comcast, Optimum, and
RCN) all do ARP caching, so you need to reboot them when you change the
connected device (or you need to clone the old device's MAC address).
Actually though, working with DSL is worse. Verizon DSL does ARP caching
in the Central Office for up to four hours. I have found that replacing
equipment hooked up to Verison DSL, it is best to already be on the phone
with Verizon support to have them manually clear the cache. At least
rebooting the cable modem is something you can do yourself.
Moshe
--
Moshe Katz
-- [phone]
_______________________________________________
List mailing list
https://lists.pfsense.org/mailman/listinfo/list

_______________________________________________
List mailing list
https://lists.pfsense.org/mailman/listinfo/list

[Moshe Katz]

Post by Aaron C. de Bruyn
Good to know.
Slightly OT, but why would they have ARP cache timeouts of four hours?
What benefit do you get with such high cache times as opposed to the
obvious support calls you will get when equipment is swapped around?
-A

Post by Christoph Hanle
Hi Aaron,
Most cable modems I have worked with in the US (on Comcast, Optimum, and
RCN) all do ARP caching, so you need to reboot them when you change the
connected device (or you need to clone the old device's MAC address).
Actually though, working with DSL is worse. Verizon DSL does ARP caching
in the Central Office for up to four hours. I have found that replacing
equipment hooked up to Verison DSL, it is best to already be on the phone
with Verizon support to have them manually clear the cache. At least
rebooting the cable modem is something you can do yourself.
Moshe
--
Moshe Katz
-- [phone]

I have no idea why they set it up that way. I do know that they do not
have that problem on Verizon FIOS. If you have a static IP on a FIOS
connection, then you can plug in and unplug whatever you want and it works
just fine as long as you set the IP address properly.

Also OT, if you are ever working with a FIOS connection that has a dynamic
IP address, you must *manually* release the DHCP-assigned address from the
old device before you put in the new device. Verizon's DHCP servers also
do MAC caching and will not respond to a new MAC address on your connection
unless the old device has explicitly given up the address. In these cases,
you either need to wait until the address expires (usually less than 24
hours), clone the old MAC address, or call support and have them end the
old lease.

Moshe

--
Moshe Katz
-- ***@ymkatz.net
-- [phone]

[Chris Buechler]

Slightly OT, but why would they have ARP cache timeouts of four hours? What
benefit do you get with such high cache times as opposed to the obvious
support calls you will get when equipment is swapped around?

That's Cisco's default and others aren't too far from that generally.
I believe that's something that hasn't changed since originally
implemented decades ago. Originally, it was likely because networks
were slow and not switched, so you didn't want to chew up a lot of
bandwidth just handling ARP. As with many cases along those lines, it
got entrenched and once a vendor sets a specific default, they tend to
not want to change it. That's largely educated guessing, as I'm not
completely sure the reasoning, just that it's been like that more or
less forever.

Yes, with modern networks, in a lot of cases it's really not sensible
to hang onto your ARP cache for hours.

A number of cable modems are worse than 4 hours. I can think of a
handful of times over the last 7 years or so, with the most recent
being a couple months ago, where a support customer got in touch with
us after trying to move some IPs and messing with it for multiple days
and couldn't make it work. Packet capture on WAN for the affected IPs,
check the destination MAC, see something other than the firewall. Ask
"What's this X MAC?" "The old box we unplugged last week." Power cycle
cable modem, all is well.

[Aaron C. de Bruyn]

Interesting. Thanks Chris.

-A

Post by Chris Buechler

Post by Aaron C. de Bruyn
Slightly OT, but why would they have ARP cache timeouts of four hours?

What

Post by Aaron C. de Bruyn
benefit do you get with such high cache times as opposed to the obvious
support calls you will get when equipment is swapped around?

That's Cisco's default and others aren't too far from that generally.
I believe that's something that hasn't changed since originally
implemented decades ago. Originally, it was likely because networks
were slow and not switched, so you didn't want to chew up a lot of
bandwidth just handling ARP. As with many cases along those lines, it
got entrenched and once a vendor sets a specific default, they tend to
not want to change it. That's largely educated guessing, as I'm not
completely sure the reasoning, just that it's been like that more or
less forever.
Yes, with modern networks, in a lot of cases it's really not sensible
to hang onto your ARP cache for hours.
A number of cable modems are worse than 4 hours. I can think of a
handful of times over the last 7 years or so, with the most recent
being a couple months ago, where a support customer got in touch with
us after trying to move some IPs and messing with it for multiple days
and couldn't make it work. Packet capture on WAN for the affected IPs,
check the destination MAC, see something other than the firewall. Ask
"What's this X MAC?" "The old box we unplugged last week." Power cycle
cable modem, all is well.
_______________________________________________
List mailing list
https://lists.pfsense.org/mailman/listinfo/list