[pfSense] Routing between interfaces

[Matthew Pounsett]

I've been employing a terrible hack where I had an IP alias on my LAN
interface in order to allow me to use multiple subnets on the same physical
network. I'm trying to correct that, but i'm running into problems with
routing between interfaces.

My network looks like this:

DSL service (pppoe) ---| |-- LAN1 (10.0.1.0/24)
|-- pfsense --|
Cable service (em3) ---| |-- LAN2 (216.235.10.32/28)

There's one more RFC1918 LANs than shown, but I'm trying to keep this
explanation simple and clear.

I have Gateway Groups and NAT rules that result in the following:

1) Anything sourced from the routable block goes out the DSL service
interface without being NAT'd, unless the DSL is down in which case it gets
NAT'd out the cable interface
2) Anything sourced from an RFC1918 address is NAT'd and load balanced out
the two interfaces

pfsense has built what looks like a sane routing table on the firewall:
# netstat -rn -f inet
Routing tables

Internet:
Destination Gateway Flags Netif Expire
default 216.235.0.20 UGS pppoe0
10.0.1.0/24 link#10 U em0_vlan
10.0.1.1 link#10 UHS lo0
10.0.6.0/24 link#2 U em1
10.0.6.1 link#2 UHS lo0
127.0.0.1 link#8 UH lo0
135.23.141.64/27 link#4 U em3
135.23.141.77 link#4 UHS lo0
216.235.0.20 link#11 UH pppoe0
216.235.8.92 link#11 UHS lo0
216.235.10.32/28 link#9 U em0_vlan
216.235.10.33 link#9 UHS lo0


The weird thing is, if I do a traceroute from one of the routable addresses
to an RFC1918 address on LAN1, I get responses from routers outside my
network on the DSL service, implying that pfsense isn't doing interface
forwarding internally.

% ifconfig en0
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=b<RXCSUM,TXCSUM,VLAN_HWTAGGING>
ether 00:25:00:f3:86:4f
inet6 fe80::225:ff:fef3:864f%en0 prefixlen 64 scopeid 0x4
inet 216.235.10.37 netmask 0xfffffff0 broadcast 216.235.10.47
nd6 options=1<PERFORMNUD>
media: autoselect (1000baseT <full-duplex,flow-control>)
status: active
% netstat -rn | grep default
default 216.235.10.33 UGSc 30 599 en0
% traceroute 10.0.1.43
traceroute to 10.0.1.43 (10.0.1.43), 64 hops max, 52 byte packets
1 agg2.tor.egate.net (216.235.0.24) 5.173 ms 4.967 ms 4.698 ms
2 vl501.ge-0-0-0.bdr2.tor.egate.net (216.235.0.133) 4.725 ms 4.998 ms
5.363 ms
3 ge-1-1-0.407.bb4.yyz1.neutraldata.net (204.16.202.170) 6.239 ms !N
5.170 ms !N 5.815 ms !N

And from a host in the RFC1918 space I get nothing at all useful, which is
probably not surprising.

% netstat -rn -f inet
Routing tables

Internet:
Destination Gateway Flags Refs Use Netif Expire
default 216.235.10.33 UGS 0 583 em0.69
10.0.1.0/24 link#7 U 0 60 em0.42
10.0.1.2 link#7 UHS 0 0 lo0
10.0.6.0/24 link#2 U 0 62 em1
10.0.6.3 link#2 UHS 0 0 lo0
127.0.0.1 link#6 UH 0 0 lo0
216.235.10.32/28 link#8 U 0 1299 em0.69
216.235.10.34 link#8 UHS 0 0 lo0

% ifconfig em0.42
em0.42: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=103<RXCSUM,TXCSUM,TSO4>
ether 00:80:2a:e8:37:89
inet 10.0.1.2 netmask 0xffffff00 broadcast 10.0.1.255
inet6 fe80::280:2aff:fee8:3789%em0.42 prefixlen 64 scopeid 0x7
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 42 parent interface: em0
% traceroute -i em0.42 216.235.10.37
traceroute to 216.235.10.37 (216.235.10.37), 64 hops max, 52 byte packets
1 * * *
2 * * *
3 * *^C


Any ideas what I've missed? Things I should try to troubleshoot further?
Thanks in advance,
Matt

[Karl Fife]

I presume your ISP gave you a tunnel network and a public /28, and
you're trying to use the IP's in the /28. Until recently, you had been
binding the tunnel network interfaces directly to your 'wan'.

You should probably be running a second router. The rationale is trust
levels. The first router can function as a DMZ four your hosts that are
on public interfaces. Because the trust levels between your public
subnet and your LANS are significantly different, you don't want a
configuration that opens your LAN to the world if you make a even the
most minor configuration error. Your second router can function as a
firewall for your user-LANs, and the firwall's WAN will be a LAN IP in
your public /30. You can still do multi-WAN on your firewall. Our DMZ
is served by a 6-interface Lanner FW-7541D which allows me different
policies for different physical interfaces.

The way you have drawn your sketch makes me think you conceptualize the
two ISP interfaces as being 'outside', and your 'lans' are 'inside'.
That idea is wrong, and will get you into trouble. To deliberately use
the language of that wrong concept, any interface allowed to talk to the
world (any) is completely and utterly "inside" (including your public
/28). This is true unless you carefully and iteratively block egress TO
any/every LAN,LAN2,Tunnel,VPN etc.. It is error-prone and fussy
especially as you add interfaces/VLANS/VPN's. You can simplify things
somewhat with floating rules and interface group rules, but the ease
with which it does something you don't expect is counter-intuitive.

Good luck.

Post by Matthew Pounsett
vlan: 42 parent interface: em0

[Matthew Pounsett]

I presume your ISP gave you a tunnel network and a public /28, and you're
trying to use the IP's in the /28. Until recently, you had been binding
the tunnel network interfaces directly to your 'wan'.

No, no tunnel involved.

You should probably be running a second router. The rationale is trust
levels. The first router can function as a DMZ four your hosts that are on
public interfaces. Because the trust levels between your public subnet and
your LANS are significantly different, you don't want a configuration that
opens your LAN to the world if you make a even the most minor configuration
error. Your second router can function as a firewall for your user-LANs,
and the firwall's WAN will be a LAN IP in your public /30. You can still
do multi-WAN on your firewall. Our DMZ is served by a 6-interface Lanner
FW-7541D which allows me different policies for different physical
interfaces.
The way you have drawn your sketch makes me think you conceptualize the
two ISP interfaces as being 'outside', and your 'lans' are 'inside'. That
idea is wrong, and will get you into trouble. To deliberately use the
language of that wrong concept, any interface allowed to talk to the world
(any) is completely and utterly "inside" (including your public /28). This
is true unless you carefully and iteratively block egress TO any/every
LAN,LAN2,Tunnel,VPN etc.. It is error-prone and fussy especially as you
add interfaces/VLANS/VPN's. You can simplify things somewhat with
floating rules and interface group rules, but the ease with which it does
something you don't expect is counter-intuitive.

While I appreciate that you're trying to be helpful, you haven't actually
addressed the core problem, which is that pfsense doesn't seem to be
routing between interfaces. The problem holds true regardless of the
addresses involved.

I see that you're in the "NAT is security" camp, which is unfortunately a
misinformed way to approach network security. NAT provides no security in
and of itself.. what you're benefitting from there is the implicit IP ACL.
ACLs themselves aren't difficult unless you don't have a clear idea of
what's running on your network, and then I agree it's easy to miss things.
You clearly haven't operated an IPv6 network where there are no NATs, and
everything is a routable address. You're going to have to get past that
mental block in the next few years if you're operating anything other than
a residential network.

Good luck.

[Karl Fife]

I'm in the needless complexity is insecurity camp. Your other
speculations are baseless.

Post by Matthew Pounsett
I see that you're in the "NAT is security" camp, which is unfortunately a
misinformed way to approach network security.

[PiBa]

Make sure that 'internal' traffic is not pushed out over the
gatewaygroup to the WAN interfaces.
So create pass rules above the pbr>gatewaygroup rules, to allow internal
trafffic to just take the regular routes.

Post by Matthew Pounsett
I've been employing a terrible hack where I had an IP alias on my LAN
interface in order to allow me to use multiple subnets on the same physical
network. I'm trying to correct that, but i'm running into problems with
routing between interfaces.
DSL service (pppoe) ---| |-- LAN1 (10.0.1.0/24)
|-- pfsense --|
Cable service (em3) ---| |-- LAN2 (216.235.10.32/28)
There's one more RFC1918 LANs than shown, but I'm trying to keep this
explanation simple and clear.
1) Anything sourced from the routable block goes out the DSL service
interface without being NAT'd, unless the DSL is down in which case it gets
NAT'd out the cable interface
2) Anything sourced from an RFC1918 address is NAT'd and load balanced out
the two interfaces
# netstat -rn -f inet
Routing tables
Destination Gateway Flags Netif Expire
default 216.235.0.20 UGS pppoe0
10.0.1.0/24 link#10 U em0_vlan
10.0.1.1 link#10 UHS lo0
10.0.6.0/24 link#2 U em1
10.0.6.1 link#2 UHS lo0
127.0.0.1 link#8 UH lo0
135.23.141.64/27 link#4 U em3
135.23.141.77 link#4 UHS lo0
216.235.0.20 link#11 UH pppoe0
216.235.8.92 link#11 UHS lo0
216.235.10.32/28 link#9 U em0_vlan
216.235.10.33 link#9 UHS lo0
The weird thing is, if I do a traceroute from one of the routable addresses
to an RFC1918 address on LAN1, I get responses from routers outside my
network on the DSL service, implying that pfsense isn't doing interface
forwarding internally.
% ifconfig en0
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=b<RXCSUM,TXCSUM,VLAN_HWTAGGING>
ether 00:25:00:f3:86:4f
inet6 fe80::225:ff:fef3:864f%en0 prefixlen 64 scopeid 0x4
inet 216.235.10.37 netmask 0xfffffff0 broadcast 216.235.10.47
nd6 options=1<PERFORMNUD>
media: autoselect (1000baseT <full-duplex,flow-control>)
status: active
% netstat -rn | grep default
default 216.235.10.33 UGSc 30 599 en0
% traceroute 10.0.1.43
traceroute to 10.0.1.43 (10.0.1.43), 64 hops max, 52 byte packets
1 agg2.tor.egate.net (216.235.0.24) 5.173 ms 4.967 ms 4.698 ms
2 vl501.ge-0-0-0.bdr2.tor.egate.net (216.235.0.133) 4.725 ms 4.998 ms
5.363 ms
3 ge-1-1-0.407.bb4.yyz1.neutraldata.net (204.16.202.170) 6.239 ms !N
5.170 ms !N 5.815 ms !N
And from a host in the RFC1918 space I get nothing at all useful, which is
probably not surprising.
% netstat -rn -f inet
Routing tables
Destination Gateway Flags Refs Use Netif Expire
default 216.235.10.33 UGS 0 583 em0.69
10.0.1.0/24 link#7 U 0 60 em0.42
10.0.1.2 link#7 UHS 0 0 lo0
10.0.6.0/24 link#2 U 0 62 em1
10.0.6.3 link#2 UHS 0 0 lo0
127.0.0.1 link#6 UH 0 0 lo0
216.235.10.32/28 link#8 U 0 1299 em0.69
216.235.10.34 link#8 UHS 0 0 lo0
% ifconfig em0.42
em0.42: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=103<RXCSUM,TXCSUM,TSO4>
ether 00:80:2a:e8:37:89
inet 10.0.1.2 netmask 0xffffff00 broadcast 10.0.1.255
inet6 fe80::280:2aff:fee8:3789%em0.42 prefixlen 64 scopeid 0x7
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 42 parent interface: em0
% traceroute -i em0.42 216.235.10.37
traceroute to 216.235.10.37 (216.235.10.37), 64 hops max, 52 byte packets
1 * * *
2 * * *
3 * *^C
Any ideas what I've missed? Things I should try to troubleshoot further?
Thanks in advance,
Matt
_______________________________________________
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

[Matthew Pounsett]

Make sure that 'internal' traffic is not pushed out over the gatewaygroup
to the WAN interfaces.
So create pass rules above the pbr>gatewaygroup rules, to allow internal
trafffic to just take the regular routes.

Ahh.. that sounds like a likely cause of my trouble. Thanks.

Admittedly after only looking for about two minutes, I don't immediately
see how to implement your solution, though. Gateway groups and firewall
rules are managed in separate places in the UI, so it's not clear to me how
to get firewall rules "above" the gateway group rules. I'll be out most of
the day but I can play with that some more tonight and tomorrow to see if
anything pops out at me.

Thanks for your help,
Matt

[PiBa]

Post by Matthew Pounsett

Make sure that 'internal' traffic is not pushed out over the gatewaygroup
to the WAN interfaces.
So create pass rules above the pbr>gatewaygroup rules, to allow internal
trafffic to just take the regular routes.

Ahh.. that sounds like a likely cause of my trouble. Thanks.
Admittedly after only looking for about two minutes, I don't immediately
see how to implement your solution, though. Gateway groups and firewall
rules are managed in separate places in the UI, so it's not clear to me how
to get firewall rules "above" the gateway group rules. I'll be out most of
the day but I can play with that some more tonight and tomorrow to see if
anything pops out at me.
Thanks for your help,
Matt
_______________________________________________
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Alright so your using floating rules for the pbr part i guess :).
In that case you could try and filter those match rules to only affect
traffic that has a destination on 'the internet'.
Create a alias that has local networks like 192.168/16 172.16/12 10/8
and your routed block. Then only apply the floating rules that push
traffic out a gateway group to traffic that does NOT have that alias as
a destination.

Regards,
PiBa-NL