[pfSense] Using to interfaces to access two separate switches that are in the same subnet.

[j***@millican.us]

Hello,
Here is my layout:
Two pfSense boxes with Carp for HA on WAN
LAN1 and LAN2 on both pfSense configured with CARP Addresses for HA
One HP V191016G switch connected to LAN1 of each pfSense box
One HP V191016G switch connected to LAN2 of each pfSense box
Currently LAN1 is assigned an address in the 192.168.100.0/24 subnet and
LAN2 is assigned an address in the 192.168.101.0/24 subnet so that I can
have two separate networks with Identical VMs in each network load
balanced for HA of the VMs. While this scenario has been working
relatively well there have been some issues that have come up when using
the VM provisioning controllers.

So, I would like to be able to have, for example, one VM assigned the IP
of 192.168.100.100 connected to switch one and another VM assigned
192.168.100.101 connected to switch two. These would then be be load
balanced but built in such a way that all VMs with odd numbered IPs are
physically connected to switch one and VMs with even numbered IP are all
on switch two. This way if a port fails anywhere, traffic will still
flow to at least one of the VMs. Yes, if I loose one of the VMs under
heavy load the response time may suffer but slow response is better than
no response. As a side note I really do not want to get involved with
Spanning tree if I can avoid it and based on my limited exposure doesn't
sound like an option here anyway.

I am thinking I should bridge the LAN1 and LAN2 interfaces, this way
they can both be in the same subnet and route traffic through the
appropriate switch. I have seen a number of Google posts though that
say this may cause an L2 loop which I obviously need to avoid and
believe I would since there would still only be one path to each VM,
either through switch one or through switch two which are NOT up-linked
to each other.
I originally though I might set up the LAN1 and LAN2 interfaces of each
pfSense box with LAGG but I do not want the interfaces to load balance
the traffic and based on the information found here
http://doc.pfsense.org/index.php/LAGG_Interfaces LAGG seems to always
use some sort of load balance or fail over on the interfaces which means
traffic pointed to 192.168.100.100 could conceivably sometimes leave on
the first interface of the bridge and other times the second interface.

Could I be missing a much simpler and more elegant solution?

Thank You,
JohnM

[j***@millican.us]

Post by j***@millican.us
Hello,
Two pfSense boxes with Carp for HA on WAN
LAN1 and LAN2 on both pfSense configured with CARP Addresses for HA
One HP V191016G switch connected to LAN1 of each pfSense box
One HP V191016G switch connected to LAN2 of each pfSense box
Currently LAN1 is assigned an address in the 192.168.100.0/24 subnet
and LAN2 is assigned an address in the 192.168.101.0/24 subnet so that
I can have two separate networks with Identical VMs in each network
load balanced for HA of the VMs. While this scenario has been working
relatively well there have been some issues that have come up when
using the VM provisioning controllers.
So, I would like to be able to have, for example, one VM assigned the
IP of 192.168.100.100 connected to switch one and another VM assigned
192.168.100.101 connected to switch two. These would then be be load
balanced but built in such a way that all VMs with odd numbered IPs
are physically connected to switch one and VMs with even numbered IP
are all on switch two. This way if a port fails anywhere, traffic
will still flow to at least one of the VMs. Yes, if I loose one of the
VMs under heavy load the response time may suffer but slow response is
better than no response. As a side note I really do not want to get
involved with Spanning tree if I can avoid it and based on my limited
exposure doesn't sound like an option here anyway.
I am thinking I should bridge the LAN1 and LAN2 interfaces, this way
they can both be in the same subnet and route traffic through the
appropriate switch. I have seen a number of Google posts though that
say this may cause an L2 loop which I obviously need to avoid and
believe I would since there would still only be one path to each VM,
either through switch one or through switch two which are NOT
up-linked to each other.
I originally though I might set up the LAN1 and LAN2 interfaces of
each pfSense box with LAGG but I do not want the interfaces to load
balance the traffic and based on the information found here
http://doc.pfsense.org/index.php/LAGG_Interfaces LAGG seems to always
use some sort of load balance or fail over on the interfaces which
means traffic pointed to 192.168.100.100 could conceivably sometimes
leave on the first interface of the bridge and other times the second
interface.
Could I be missing a much simpler and more elegant solution?
Thank You,
JohnM

So just to better define a more specific set of related questions:
1) Does CARP currently work with bridged interfaces in pfSense. I have
seen older posts that say no it does not due to FreeBSD/pf functionality
and newer posts that say the underlying issue has been patched on FreeBSD.

2) I am assuming that if CARP with a bridged interface is possible I
would remove the IP from both of the physical interfaces and assign it
to the bridge interface, correct?



Thank You,
JohnM

[Jim Pingle]

Post by j***@millican.us
1) Does CARP currently work with bridged interfaces in pfSense. I have
seen older posts that say no it does not due to FreeBSD/pf functionality
and newer posts that say the underlying issue has been patched on FreeBSD.

It can, but just because it can, doesn't mean it should. It should be
avoided at all costs.

Post by j***@millican.us
2) I am assuming that if CARP with a bridged interface is possible I
would remove the IP from both of the physical interfaces and assign it
to the bridge interface, correct?

Yes, but be prepared for lots of other messes with CARP+Bridging that
aren't really pfSense-related, such as your switches shutting down ports
with STP because of Layer 2 loops.

Things can be nudged to behave, to a point, but it's best to avoid it
and route. Do yourself a favor and keep it routed and find a better way
to do what you're after that won't be a giant mess of L2 issues.

The kind of redundancy you're looking for is best solved with LAGG/LACP
mixed with VLANs for port and switch redundancy, though that does
require stackable switches or using a failover mode on LAGG instead of LACP.

So each pfSense box has one port plugged into each switch, using LAGG,
and tagged VLANs manage getting LAN1/LAN2 to pfSense rather than doing
one port per network. Then you have redundancy if a switch dies, or if a
NIC dies.

Using the same technique on your hypervisors and you don't need to worry
about failing between switches there, either.

Jim

[j***@millican.us]

Post by Jim Pingle

Post by j***@millican.us
1) Does CARP currently work with bridged interfaces in pfSense. I have
seen older posts that say no it does not due to FreeBSD/pf functionality
and newer posts that say the underlying issue has been patched on FreeBSD.

It can, but just because it can, doesn't mean it should. It should be
avoided at all costs.

Post by j***@millican.us
2) I am assuming that if CARP with a bridged interface is possible I
would remove the IP from both of the physical interfaces and assign it
to the bridge interface, correct?

Yes, but be prepared for lots of other messes with CARP+Bridging that
aren't really pfSense-related, such as your switches shutting down ports
with STP because of Layer 2 loops.
Things can be nudged to behave, to a point, but it's best to avoid it
and route. Do yourself a favor and keep it routed and find a better way
to do what you're after that won't be a giant mess of L2 issues.
The kind of redundancy you're looking for is best solved with LAGG/LACP
mixed with VLANs for port and switch redundancy, though that does
require stackable switches or using a failover mode on LAGG instead of LACP.
So each pfSense box has one port plugged into each switch, using LAGG,
and tagged VLANs manage getting LAN1/LAN2 to pfSense rather than doing
one port per network. Then you have redundancy if a switch dies, or if a
NIC dies.
Using the same technique on your hypervisors and you don't need to worry
about failing between switches there, either.
Jim
_______________________________________________

Thank you, based on this info I will look into using LAGG/LACP on the
pfSense boxes and the hosts.
Thanks again for the help,
John

[Adam Thompson]

Post by j***@millican.us
1) Does CARP currently work with bridged interfaces in pfSense. I
have seen older posts that say no it does not due to FreeBSD/pf
functionality and newer posts that say the underlying issue has been
patched on FreeBSD.
2) I am assuming that if CARP with a bridged interface is possible I
would remove the IP from both of the physical interfaces and assign
it to the bridge interface, correct?

I agree with the other posters: even if CARP *works* in a bridged scenario, don't do it. For your own sanity, and the sanity of whoever else has to troubleshoot the network, just don't do it.

Quoting from Wikipedia, "The Common Address Redundancy Protocol or CARP is a protocol which allows multiple hosts on the same local network to share a set of IP addresses." You're trying to do redundant bridging (L2) with a redundant routing (L3) protocol. Am I missing something here?

Remember that a pfSense box in bridge mode is functionally a two-port switch, with all the spanning-tree implications that go along with that.
You're asking to have two switches in parallel - only one of which can ever be active at one time - and to also have, for reasons unknown, a redundant routing address on the boxes.

The question is, can pfSense be made to force an STP recomputation upon CARP promotion/demotion, and will it stop bridging on the CARP slave node? I'm not sure, although I haven't seen anything to indicate it can. If this doesn't happen automagically, does pf continue filtering correctly on the slave that is still processing the L2 traffic? Again, I can't tell if it does or not.

Then, the next question is what do you hope to gain from CARP? Synchronization of the firewall rules?

Keep in mind that what you appear to be asking for is a sufficiently obscure use case that no-one will be able to help you when it breaks :-)

Some commercial products do support this out-of-the-box (FortiGate & PaloAlto both come to mind); I'm sure if you paid BSD Perimeter they'd make it work for you just as well.

-Adam Thompson
***@athompso.net

[j***@millican.us]

Post by Adam Thompson

Post by j***@millican.us
1) Does CARP currently work with bridged interfaces in pfSense. I
have seen older posts that say no it does not due to FreeBSD/pf
functionality and newer posts that say the underlying issue has been
patched on FreeBSD.
2) I am assuming that if CARP with a bridged interface is possible I
would remove the IP from both of the physical interfaces and assign
it to the bridge interface, correct?

I agree with the other posters: even if CARP *works* in a bridged scenario, don't do it. For your own sanity, and the sanity of whoever else has to troubleshoot the network, just don't do it.
Quoting from Wikipedia, "The Common Address Redundancy Protocol or CARP is a protocol which allows multiple hosts on the same local network to share a set of IP addresses." You're trying to do redundant bridging (L2) with a redundant routing (L3) protocol. Am I missing something here?
Remember that a pfSense box in bridge mode is functionally a two-port switch, with all the spanning-tree implications that go along with that.
You're asking to have two switches in parallel - only one of which can ever be active at one time - and to also have, for reasons unknown, a redundant routing address on the boxes.
The question is, can pfSense be made to force an STP recomputation upon CARP promotion/demotion, and will it stop bridging on the CARP slave node? I'm not sure, although I haven't seen anything to indicate it can. If this doesn't happen automagically, does pf continue filtering correctly on the slave that is still processing the L2 traffic? Again, I can't tell if it does or not.
Then, the next question is what do you hope to gain from CARP? Synchronization of the firewall rules?
Keep in mind that what you appear to be asking for is a sufficiently obscure use case that no-one will be able to help you when it breaks :-)
Some commercial products do support this out-of-the-box (FortiGate & PaloAlto both come to mind); I'm sure if you paid BSD Perimeter they'd make it work for you just as well.
-Adam Thompson

The purpose behind this was to have two NIC's on the same network but in
such a way that each IP in the network beyond the bridged interfaces
would only be available on one of the two switches thereby avoiding the
L2 loop scenario since there would be one and only one path to each
address, with no spanning tree needed. For example a VM with the IP of
192.168.100.101 physically connected only to switch one and it's mirror
with an IP of 192.168.100.102 physically connected only to switch two.
The switches would not be stacked. The bad side of this was that it
would mean that if a switch or a port went down that VM would be lost
but, since it would have a mirror VM there would still be availability
via load balancing in pfSense.

Having said all that if I can get the interfaces into LAGG with LACP
that would not only effectively double my bandwidth, since LACP does a
load balance between the interfaces, but give me a scenario with no
network single point of failure even if the VMs are not mirrored, and I
can still use load balancing between the mirrors since most will be
mirrored anyway so as to not have the VM itself not be the single point
if failure.

All in all LAGG/LACP is a much better solution even if there were not
"issues" with Bridging+CARP.

I am having trouble getting pfSense to see the LAN1 and LAN2 interfaces
in the LAGG configuration -> Parent Interface selection box. Will be
goggling that tonight.

I do have another question, if I use the LAGG/LACP will that be
available to use CARP to allow for fail over should the primary pfSense
box die for some reason? I am just trying to do to much here? Paranoia
runs deep and Murphy and I seem to run into each other all to
frequently, so I always try to plan for the worst case and then hope it
never happens.
Thank You for all the help,
JohnM

[j***@millican.us]

Post by j***@millican.us

Post by Adam Thompson

Post by j***@millican.us
1) Does CARP currently work with bridged interfaces in pfSense. I
have seen older posts that say no it does not due to FreeBSD/pf
functionality and newer posts that say the underlying issue has been
patched on FreeBSD.
2) I am assuming that if CARP with a bridged interface is possible I
would remove the IP from both of the physical interfaces and assign
it to the bridge interface, correct?

I agree with the other posters: even if CARP *works* in a bridged
scenario, don't do it. For your own sanity, and the sanity of
whoever else has to troubleshoot the network, just don't do it.
Quoting from Wikipedia, "The Common Address Redundancy Protocol or
CARP is a protocol which allows multiple hosts on the same local
network to share a set of IP addresses." You're trying to do
redundant bridging (L2) with a redundant routing (L3) protocol. Am I
missing something here?
Remember that a pfSense box in bridge mode is functionally a two-port
switch, with all the spanning-tree implications that go along with that.
You're asking to have two switches in parallel - only one of which
can ever be active at one time - and to also have, for reasons
unknown, a redundant routing address on the boxes.
The question is, can pfSense be made to force an STP recomputation
upon CARP promotion/demotion, and will it stop bridging on the CARP
slave node? I'm not sure, although I haven't seen anything to
indicate it can. If this doesn't happen automagically, does pf
continue filtering correctly on the slave that is still processing
the L2 traffic? Again, I can't tell if it does or not.
Then, the next question is what do you hope to gain from CARP?
Synchronization of the firewall rules?
Keep in mind that what you appear to be asking for is a sufficiently
obscure use case that no-one will be able to help you when it breaks :-)
Some commercial products do support this out-of-the-box (FortiGate &
PaloAlto both come to mind); I'm sure if you paid BSD Perimeter
they'd make it work for you just as well.
-Adam Thompson

The purpose behind this was to have two NIC's on the same network but
in such a way that each IP in the network beyond the bridged
interfaces would only be available on one of the two switches thereby
avoiding the L2 loop scenario since there would be one and only one
path to each address, with no spanning tree needed. For example a VM
with the IP of 192.168.100.101 physically connected only to switch one
and it's mirror with an IP of 192.168.100.102 physically connected
only to switch two. The switches would not be stacked. The bad side
of this was that it would mean that if a switch or a port went down
that VM would be lost but, since it would have a mirror VM there would
still be availability via load balancing in pfSense.
Having said all that if I can get the interfaces into LAGG with LACP
that would not only effectively double my bandwidth, since LACP does a
load balance between the interfaces, but give me a scenario with no
network single point of failure even if the VMs are not mirrored, and
I can still use load balancing between the mirrors since most will be
mirrored anyway so as to not have the VM itself not be the single
point if failure.
All in all LAGG/LACP is a much better solution even if there were not
"issues" with Bridging+CARP.
I am having trouble getting pfSense to see the LAN1 and LAN2
interfaces in the LAGG configuration -> Parent Interface selection
box. Will be goggling that tonight.
I do have another question, if I use the LAGG/LACP will that be
available to use CARP to allow for fail over should the primary
pfSense box die for some reason? I am just trying to do to much
here? Paranoia runs deep and Murphy and I seem to run into each other
all to frequently, so I always try to plan for the worst case and then
hope it never happens.
Thank You for all the help,
JohnM

To answer my question I did some quick Google ing and it appears that
yes CARP and LAGG can be used together. If i am missing something a
gentle 2x4 upside the head is always appreciated though.
I am still looking for how to get the LAN1 and LAN2 to show in the
parent interface selection box.
JohnM

[j***@millican.us]

Post by j***@millican.us

Post by j***@millican.us

Post by Adam Thompson

Post by j***@millican.us
1) Does CARP currently work with bridged interfaces in pfSense. I
have seen older posts that say no it does not due to FreeBSD/pf
functionality and newer posts that say the underlying issue has been
patched on FreeBSD.
2) I am assuming that if CARP with a bridged interface is possible I
would remove the IP from both of the physical interfaces and assign
it to the bridge interface, correct?

I agree with the other posters: even if CARP *works* in a bridged
scenario, don't do it. For your own sanity, and the sanity of
whoever else has to troubleshoot the network, just don't do it.
Quoting from Wikipedia, "The Common Address Redundancy Protocol or
CARP is a protocol which allows multiple hosts on the same local
network to share a set of IP addresses." You're trying to do
redundant bridging (L2) with a redundant routing (L3) protocol. Am
I missing something here?
Remember that a pfSense box in bridge mode is functionally a
two-port switch, with all the spanning-tree implications that go
along with that.
You're asking to have two switches in parallel - only one of which
can ever be active at one time - and to also have, for reasons
unknown, a redundant routing address on the boxes.
The question is, can pfSense be made to force an STP recomputation
upon CARP promotion/demotion, and will it stop bridging on the CARP
slave node? I'm not sure, although I haven't seen anything to
indicate it can. If this doesn't happen automagically, does pf
continue filtering correctly on the slave that is still processing
the L2 traffic? Again, I can't tell if it does or not.
Then, the next question is what do you hope to gain from CARP?
Synchronization of the firewall rules?
Keep in mind that what you appear to be asking for is a sufficiently
obscure use case that no-one will be able to help you when it breaks :-)
Some commercial products do support this out-of-the-box (FortiGate &
PaloAlto both come to mind); I'm sure if you paid BSD Perimeter
they'd make it work for you just as well.
-Adam Thompson

The purpose behind this was to have two NIC's on the same network but
in such a way that each IP in the network beyond the bridged
interfaces would only be available on one of the two switches thereby
avoiding the L2 loop scenario since there would be one and only one
path to each address, with no spanning tree needed. For example a VM
with the IP of 192.168.100.101 physically connected only to switch
one and it's mirror with an IP of 192.168.100.102 physically
connected only to switch two. The switches would not be stacked. The
bad side of this was that it would mean that if a switch or a port
went down that VM would be lost but, since it would have a mirror VM
there would still be availability via load balancing in pfSense.
Having said all that if I can get the interfaces into LAGG with LACP
that would not only effectively double my bandwidth, since LACP does
a load balance between the interfaces, but give me a scenario with no
network single point of failure even if the VMs are not mirrored, and
I can still use load balancing between the mirrors since most will be
mirrored anyway so as to not have the VM itself not be the single
point if failure.
All in all LAGG/LACP is a much better solution even if there were not
"issues" with Bridging+CARP.
I am having trouble getting pfSense to see the LAN1 and LAN2
interfaces in the LAGG configuration -> Parent Interface selection
box. Will be goggling that tonight.
I do have another question, if I use the LAGG/LACP will that be
available to use CARP to allow for fail over should the primary
pfSense box die for some reason? I am just trying to do to much
here? Paranoia runs deep and Murphy and I seem to run into each
other all to frequently, so I always try to plan for the worst case
and then hope it never happens.
Thank You for all the help,
JohnM

To answer my question I did some quick Google ing and it appears that
yes CARP and LAGG can be used together. If i am missing something a
gentle 2x4 upside the head is always appreciated though.
I am still looking for how to get the LAN1 and LAN2 to show in the
parent interface selection box.
JohnM

Answering my own question again in case someone else has the same issue
down the road someday. I had to go to assign Interfaces-> (assign) and
delete the interfaces(em3 and em4 in my case) and then they were
available in the LAGG interfaces -> parent interface selection box.
JohnM

[Adam Thompson]

but in such a way that each IP in the network beyond the bridged
interfaces would only be available on one of the two switches
thereby avoiding the
L2 loop scenario since there would be one and only one path to each
address, with no spanning tree needed. For example a VM with the IP of
192.168.100.101 physically connected only to switch one and it's
mirror with an IP of 192.168.100.102 physically connected only to
switch two.

I'm sorry to be rude, but that paragraph means you don't understand L2 switching well enough to do this. I'm not sure *I* understand L2 switching well enough to implement what you want, and I've been doing it almost every day for a decade!

I'm heading off to a meeting right now, but I'll try to outline a "better" way, given your constraints, later tonight.

-Adam

[j***@millican.us]

Post by Adam Thompson

but in such a way that each IP in the network beyond the bridged
interfaces would only be available on one of the two switches
thereby avoiding the
L2 loop scenario since there would be one and only one path to each
address, with no spanning tree needed. For example a VM with the IP of
192.168.100.101 physically connected only to switch one and it's
mirror with an IP of 192.168.100.102 physically connected only to
switch two.

I'm sorry to be rude, but that paragraph means you don't understand L2 switching well enough to do this. I'm not sure *I* understand L2 switching well enough to implement what you want, and I've been doing it almost every day for a decade!
I'm heading off to a meeting right now, but I'll try to outline a "better" way, given your constraints, later tonight.
-Adam

Constructive correction is never considered rude by me. Looking forward
to any information you feel like giving and/or have time to provide. I
do believe I have found how to do this with a LAGG interface group
based on an earlier post from Jim Pingle and do realize it is a better
way than my initial proposal. I haven't implemented and tested as yet
but I feel I have enough information to do so in the morning.
Thanks again,
JohnM

[Adam Thompson]

I've re-read your emails a few times, and I'm still not quite 100% certain what you hoped to accomplish with the two switches. Based on what I understand of it, however, I suggest two options:

1) VLAN+LAG ("tagged and trunked")
-higher complexity
-more difficult to troubleshoot
-possibly higher bandwidth (not guaranteed)
-fewer switch ports consumed (as many subnets as you want, only four Ethernet ports in total)
-requires cross-stack LAG for full redundancy
-more 'elegant' solution

2) multiple Ethernet ports
-simpler to setup and manage
-limited to 1Gbit/sec of traffic
-many switch ports get used up (every new subnet requires two more Ethernet ports)
-no cross-stack LAG involved


Your switches do not support "stacking", so there's no way to stack them, and there's no possibility of cross-stack LAG. This means that, at a minimum, you should have each pfSense box plugged into a separate switch. Stacking means something very different from simply interconnecting two (or more) switches; it implies that the switches now share a common control plane and your two switches would behave as one large 32-port switch. (Well, yours won't, but other models would.)

In any layer-2 environment, all switches handling traffic on the same IP subnet must be interconnected, and that means you must deal with Spanning Tree. With only two switches, STP isn't difficult, and HP's implementation of it is sane; you do want to ensure something other than the original, basic, STP (802.1d) is enable however. I think HP ships with 802.1W (usually called RSTP) enabled by default.

When creating a 2-member LAG, you don't actually get 2Gbit/sec connections; what you get is the ability to handle more traffic between more pairs of hosts. Per the 802.1ad (LACP) specification, any single conversation between a pair of hosts must be limited to only use one of the LAG member links - so no single conversation can ever exceed the originally-available speed. Typically, LACP implementations use a hashing algorithm based on the source and destination MAC addresses to decide which link to send the traffic across. What you gain is redundancy, and the capacity to handle more conversations between more pairs of hosts. Incidentally, this technique works much better between switches than between hosts and switches. You should always use LAGs to interconnect your switches, if you have enough ports to do so! (STP works fine over LAGs, don't worry about that.)

If you're using VMware, forget about using "teamed" connections or LAG of any kind - use failover NICs instead. VMware has some excellent (albeit disappointing) documents on how to architect a vSphere cluster for maximum redundancy.

All in all, I would recommend using three independent Ethernet ports per pfSense box instead of using LAGs: 1 for mgmt., 1 for WAN, 1 for service LAN. (It might make sense to use the LAN port as your management port, if that's the IP subnet you want to manage pfSense from; then you're back down to two ports per pfSense box.) Plug all two/three links into the same switch, into ports configured to the appropriate VLANs. This might not protect, in the worst case, against the failure of a single Ethernet port or patch cable, but will dramatically simplify troubleshooting when something doesn't work. The problem scenario with LAGs is when something has gone wrong with either LACP or the VLAN tagging, but you need both LACP and VLAN tagging to work correctly in order to reach the pfSense box in order to diagnose the problem: catch-22.
Of course, if you need the pfSense boxes to route between more than two to three VLANs, then you probably should go back to tagged-and-trunked in order to conserve Ethernet ports. Keep a separate Management port configured to work around the troubleshooting catch-22 mentioned above.

If your pfSense boxes are virtualized, there's a noticeable performance penalty to using the LAG+VLAN technique. Also, virtual Ethernet ports don't cost you anything anyway.

Feel free to poke holes in this and ask questions!

-Adam Thompson

-----Original Message-----
Sent: Tuesday, June 18, 2013 4:59 PM
Subject: Re: [pfSense] Using to interfaces to access two separate
switches that are in the same subnet.

Post by Adam Thompson

Post by j***@millican.us
So just to better define a more specific set of related
1) Does CARP currently work with bridged interfaces in pfSense.

I

Post by Adam Thompson

Post by j***@millican.us
have seen older posts that say no it does not due to FreeBSD/pf
functionality and newer posts that say the underlying issue has

been

Post by Adam Thompson

Post by j***@millican.us
patched on FreeBSD.
2) I am assuming that if CARP with a bridged interface is

possible I

Post by Adam Thompson

Post by j***@millican.us
would remove the IP from both of the physical interfaces and

assign

Post by Adam Thompson

Post by j***@millican.us
it to the bridge interface, correct?

I agree with the other posters: even if CARP *works* in a bridged

scenario, don't do it. For your own sanity, and the sanity of
whoever else has to troubleshoot the network, just don't do it.

Post by Adam Thompson
Quoting from Wikipedia, "The Common Address Redundancy Protocol or

CARP is a protocol which allows multiple hosts on the same local
network to share a set of IP addresses." You're trying to do
redundant bridging (L2) with a redundant routing (L3) protocol. Am
I missing something here?

Post by Adam Thompson
Remember that a pfSense box in bridge mode is functionally a two-

port switch, with all the spanning-tree implications that go along
with that.

Post by Adam Thompson
You're asking to have two switches in parallel - only one of which

can ever be active at one time - and to also have, for reasons
unknown, a redundant routing address on the boxes.

Post by Adam Thompson
The question is, can pfSense be made to force an STP recomputation

upon CARP promotion/demotion, and will it stop bridging on the CARP
slave node? I'm not sure, although I haven't seen anything to
indicate it can. If this doesn't happen automagically, does pf
continue filtering correctly on the slave that is still processing
the L2 traffic? Again, I can't tell if it does or not.

Post by Adam Thompson
Then, the next question is what do you hope to gain from CARP?

Synchronization of the firewall rules?

Post by Adam Thompson
Keep in mind that what you appear to be asking for is a

sufficiently

Post by Adam Thompson
obscure use case that no-one will be able to help you when it

breaks

Post by Adam Thompson
:-)
Some commercial products do support this out-of-the-box (FortiGate

& PaloAlto both come to mind); I'm sure if you paid BSD Perimeter
they'd make it work for you just as well.

Post by Adam Thompson
-Adam Thompson

The purpose behind this was to have two NIC's on the same network
but in such a way that each IP in the network beyond the bridged
interfaces would only be available on one of the two switches
thereby avoiding the
L2 loop scenario since there would be one and only one path to each
address, with no spanning tree needed. For example a VM with the IP of
192.168.100.101 physically connected only to switch one and it's
mirror with an IP of 192.168.100.102 physically connected only to
switch two.
The switches would not be stacked. The bad side of this was that it
would mean that if a switch or a port went down that VM would be
lost but, since it would have a mirror VM there would still be
availability via load balancing in pfSense.
Having said all that if I can get the interfaces into LAGG with LACP
that would not only effectively double my bandwidth, since LACP does
a load balance between the interfaces, but give me a scenario with
no network single point of failure even if the VMs are not mirrored,
and I can still use load balancing between the mirrors since most
will be mirrored anyway so as to not have the VM itself not be the
single point if failure.
All in all LAGG/LACP is a much better solution even if there were
not "issues" with Bridging+CARP.
I am having trouble getting pfSense to see the LAN1 and LAN2
interfaces in the LAGG configuration -> Parent Interface selection
box. Will be goggling that tonight.
I do have another question, if I use the LAGG/LACP will that be
available to use CARP to allow for fail over should the primary
pfSense box die for some reason? I am just trying to do to much
here? Paranoia runs deep and Murphy and I seem to run into each
other all to frequently, so I always try to plan for the worst case
and then hope it never happens.
Thank You for all the help,
JohnM
_______________________________________________
List mailing list
http://lists.pfsense.org/mailman/listinfo/list

[j***@millican.us]

Post by Adam Thompson
1) VLAN+LAG ("tagged and trunked")
-higher complexity
-more difficult to troubleshoot
-possibly higher bandwidth (not guaranteed)
-fewer switch ports consumed (as many subnets as you want, only four Ethernet ports in total)
-requires cross-stack LAG for full redundancy
-more 'elegant' solution
2) multiple Ethernet ports
-simpler to setup and manage
-limited to 1Gbit/sec of traffic
-many switch ports get used up (every new subnet requires two more Ethernet ports)
-no cross-stack LAG involved
Your switches do not support "stacking", so there's no way to stack them, and there's no possibility of cross-stack LAG. This means that, at a minimum, you should have each pfSense box plugged into a separate switch. Stacking means something very different from simply interconnecting two (or more) switches; it implies that the switches now share a common control plane and your two switches would behave as one large 32-port switch. (Well, yours won't, but other models would.)
In any layer-2 environment, all switches handling traffic on the same IP subnet must be interconnected, and that means you must deal with Spanning Tree. With only two switches, STP isn't difficult, and HP's implementation of it is sane; you do want to ensure something other than the original, basic, STP (802.1d) is enable however. I think HP ships with 802.1W (usually called RSTP) enabled by default.
When creating a 2-member LAG, you don't actually get 2Gbit/sec connections; what you get is the ability to handle more traffic between more pairs of hosts. Per the 802.1ad (LACP) specification, any single conversation between a pair of hosts must be limited to only use one of the LAG member links - so no single conversation can ever exceed the originally-available speed. Typically, LACP implementations use a hashing algorithm based on the source and destination MAC addresses to decide which link to send the traffic across. What you gain is redundancy, and the capacity to handle more conversations between more pairs of hosts. Incidentally, this technique works much better between switches than between hosts and switches. You should always use LAGs to interconnect your switches, if you have enough ports to do so! (STP works fine over LAGs, don't worry about that.)
If you're using VMware, forget about using "teamed" connections or LAG of any kind - use failover NICs instead. VMware has some excellent (albeit disappointing) documents on how to architect a vSphere cluster for maximum redundancy.
All in all, I would recommend using three independent Ethernet ports per pfSense box instead of using LAGs: 1 for mgmt., 1 for WAN, 1 for service LAN. (It might make sense to use the LAN port as your management port, if that's the IP subnet you want to manage pfSense from; then you're back down to two ports per pfSense box.) Plug all two/three links into the same switch, into ports configured to the appropriate VLANs. This might not protect, in the worst case, against the failure of a single Ethernet port or patch cable, but will dramatically simplify troubleshooting when something doesn't work. The problem scenario with LAGs is when something has gone wrong with either LACP or the VLAN tagging, but you need both LACP and VLAN tagging to work correctly in order to reach the pfSense box in order to diagnose the problem: catch-22.
Of course, if you need the pfSense boxes to route between more than two to three VLANs, then you probably should go back to tagged-and-trunked in order to conserve Ethernet ports. Keep a separate Management port configured to work around the troubleshooting catch-22 mentioned above.
If your pfSense boxes are virtualized, there's a noticeable performance penalty to using the LAG+VLAN technique. Also, virtual Ethernet ports don't cost you anything anyway.
Feel free to poke holes in this and ask questions!
-Adam Thompson

Adam,
Thank You for all your help, I do appreciate it. I am running
cloudstack using KVM for all hypervisors for a semi private cloud and
the pfSense boxes are not virtualized.
What I ended up doing (and it may be all wrong but it is working great
so far) is to use LAGG with round robin. So two NICs on each pfSense
box, the two HP switches, and two NICs on each Linux host bonded with
balance-rr as the mode. On each of the hosts and the pfSense boxes, one
NIC is in SW1 and the other is in SW2(with no interconnect of any type
between the switches). All hosts are on the same 192.168.100.0/24
network. I have done some VERY limited testing for port failure and it
seems to work as expected such that if I unplug an Ethernet cable
anywhere, all traffic continues to flow with little or no packet loss.
I will be testing with more volume over the weekend to our application
and will try the same failure simulation (unplug the wire) and see what
happens. Unless I am missing something, which as we know is very
possible, I may well have found my answer. I believe this is a bit more
reasonable, do to the fact that it is working, than my earlier "not
going to work" idea, do you agree? When I get some more hardware budget
I will be adding NICs and switches and separating out the traffic but
that is just not possible yet. I just had to ensure that I could set up
a mirrored set of VMs, load balance between them with pfSense, and not
have a "single point of failure" in the network or in the hardware.
Any one thing fails and we may be slower than normal but we will still
be running.
Thanks again,
JohnM

<snip>