Discussion:
[pfSense] pfSense as a datacentre router (was: dual ISP BGP)
Chris Bagnall
2013-05-28 07:33:48 UTC
Permalink
Greetings list,

Following the recent thread entitled 'dual ISP BGP', I am curious as to
how ready people using the OpenBGP package feel it is for use as a
datacentre router managing several full BGP feeds and IXPs/private peers).

One of our clients has traditionally used Quagga for this task, but net
config is in Linux's traditional conf files, which makes even relatively
simple things like adding VLANs etc. very difficult for less technical
staff. A nice friendly interface like pfSense would make that much easier.

And is the OpenBGP package v6 ready and compatible with 2.1?

TIA.

Kind regards,

Chris
--
This email is made from 100% recycled electrons
Zach Underwood
2013-05-28 10:20:40 UTC
Permalink
If you have a full internet routing table to will have to mod the status
page. If you dont mod the page it will never load.
Post by Chris Bagnall
Greetings list,
Following the recent thread entitled 'dual ISP BGP', I am curious as to
how ready people using the OpenBGP package feel it is for use as a
datacentre router managing several full BGP feeds and IXPs/private peers).
One of our clients has traditionally used Quagga for this task, but net
config is in Linux's traditional conf files, which makes even relatively
simple things like adding VLANs etc. very difficult for less technical
staff. A nice friendly interface like pfSense would make that much easier.
And is the OpenBGP package v6 ready and compatible with 2.1?
TIA.
Kind regards,
Chris
--
This email is made from 100% recycled electrons
______________________________**_________________
List mailing list
http://lists.pfsense.org/**mailman/listinfo/list<http://lists.pfsense.org/mailman/listinfo/list>
--
Zach Underwood (RHCE,RHCSA,RHCT)
My website <http://zachunderwood.me>
My photes <http://zunder1990.openphoto.me>
Mark Tinka
2013-05-28 15:32:11 UTC
Permalink
Post by Chris Bagnall
One of our clients has traditionally used Quagga for this
task, but net config is in Linux's traditional conf
files, which makes even relatively simple things like
adding VLANs etc. very difficult for less technical
staff. A nice friendly interface like pfSense would make
that much easier.
I suppose the biggest issue I have with UNIX-based routers
is the poor IS-IS support.

pfSense does make sense (in theory) as a router. I haven't
deployed it beyond a corporate LAN perimetre (still talking
to a regular router upstream).

Mark.
Adam Thompson
2013-05-28 15:41:11 UTC
Permalink
Post by Mark Tinka
I suppose the biggest issue I have with UNIX-based routers
is the poor IS-IS support.
pfSense does make sense (in theory) as a router. I haven't
deployed it beyond a corporate LAN perimetre (still talking
to a regular router upstream).
I do agree - in a large, heterogenous, complex-topology network, IS-IS
proved to be a winner both for its reliability and the simplicity of
configuration.

I suspect one of the main reasons we don't see UNIX IS-IS
implementations is the hard requirement for an ISO address. There are
ways of faking that, however, which is what I suspect Quagga does.

In a green-field implementation (or anywhere there's a clearly-defined
border), RIPv2 is actually a very usable protocol. Most people hear
"RIP" and think of the old RIPv1 protocol, which had major issues, but
RIPv2 is pretty clean and simple to deploy in production. It's not a
multi-protocol routing protocol, however, so in a complex network it
might still not be suitable. It does have some attributes of its legacy
predecessor, but they're mostly ignorable on modern hardware. Don't
deploy it on 64kbps links, though...

Having said that, I'd still rather deploy RIPv2 than OSPF, given a
choice.

-Adam Thompson
***@athompso.net
Glenn Kelley
2013-05-28 16:27:40 UTC
Permalink
Out of interest - why RIPv2 vs OSPF ?

I guess if link-state, service types and load balancing is not important it may make sense -
but figured I would ask

We have migrated for our BGP implementations to use Mikrotik at the suggestion of a key player here - and could not be happier.
Sadly our multi-honed BGP implementation had tons of issues.

Internally however - I still think pfSense Rocks !
We have a large number of pfsense running virtualized in ProxMox for clients - most who elected to use PFSense vs. vyatta after we showed them the ease of use.
Ermal Luçi
2013-05-28 16:30:03 UTC
Permalink
Post by Glenn Kelley
Out of interest - why RIPv2 vs OSPF ?
I guess if link-state, service types and load balancing is not important
it may make sense -
but figured I would ask
We have migrated for our BGP implementations to use Mikrotik at the
suggestion of a key player here - and could not be happier.
Sadly our multi-honed BGP implementation had tons of issues.
It would be helpful to know what mikrotik gave you better!?
Which issues you faced on multihomed BGP?

So they can be taken care in the future if any.
Post by Glenn Kelley
Internally however - I still think pfSense Rocks !
We have a large number of pfsense running virtualized in ProxMox for
clients - most who elected to use PFSense vs. vyatta after we showed them
the ease of use.
_______________________________________________
List mailing list
http://lists.pfsense.org/mailman/listinfo/list
Glenn Kelley
2013-05-28 16:40:07 UTC
Permalink
In short

BGP would just stop working - and the system would bail.
This happened on multiple hardware implementations - Dell 2950, Dell R300, HP Equipment as well.

I have asked Chris (who gave up his July 4th Holiday and came in Person to debug - I absolutely love that from an Open Source Guy !!! ) and he even recently still has no clue as to why.

We currently are pushing over 5GBPS through pfSense internally - but when we add BGP it just takes a crap
This is on multiple links out to the web - some fiber - some wireless -
but as a mixed Wireless ISP and Datacenter - not much we can do except to use something that works.

Since this is a PF List feel free to hit me off list - as I believe in this project and don't want this to detract from it.
Post by Glenn Kelley
Out of interest - why RIPv2 vs OSPF ?
I guess if link-state, service types and load balancing is not important it may make sense -
but figured I would ask
We have migrated for our BGP implementations to use Mikrotik at the suggestion of a key player here - and could not be happier.
Sadly our multi-honed BGP implementation had tons of issues.
It would be helpful to know what mikrotik gave you better!?
Which issues you faced on multihomed BGP?
So they can be taken care in the future if any.
Internally however - I still think pfSense Rocks !
We have a large number of pfsense running virtualized in ProxMox for clients - most who elected to use PFSense vs. vyatta after we showed them the ease of use.
_______________________________________________
List mailing list
http://lists.pfsense.org/mailman/listinfo/list
_______________________________________________
List mailing list
http://lists.pfsense.org/mailman/listinfo/list
Eugen Leitl
2013-05-29 08:39:35 UTC
Permalink
Post by Glenn Kelley
In short
BGP would just stop working - and the system would bail.
This happened on multiple hardware implementations - Dell 2950, Dell R300, HP Equipment as well.
I have asked Chris (who gave up his July 4th Holiday and came in Person to debug - I absolutely love that from an Open Source Guy !!! ) and he even recently still has no clue as to why.
We currently are pushing over 5GBPS through pfSense internally - but when we add BGP it just takes a crap
Which hardware are you using? If you're pushing 5 GBit/s you
might be running into hardware limitations. There was a thread
about it on nanog a week or two ago.
Post by Glenn Kelley
This is on multiple links out to the web - some fiber - some wireless -
but as a mixed Wireless ISP and Datacenter - not much we can do except to use something that works.
Since this is a PF List feel free to hit me off list - as I believe in this project and don't want this to detract from it.
Chris Bagnall
2013-05-29 09:05:40 UTC
Permalink
Post by Eugen Leitl
Which hardware are you using? If you're pushing 5 GBit/s you
might be running into hardware limitations. There was a thread
about it on nanog a week or two ago.
I'm quite impressed Mikrotik hardware is able to sustain 5Gbps with full
BGP tables from multiple transits to be honest.

Is anyone using pfSense's BGP package with 2.1 and v6 support? Given our
usage in this case is a great deal less than 5Gbps, I'm seriously
considering giving it a try - it would certainly make management a lot
easier, and mean they wouldn't need to call me every time they want to
change a VLAN config :-)

But v6 is a requirement. I couldn't in all good conscience deploy
something that isn't v6 capable these days.

Kind regards,

Chris
--
This email is made from 100% recycled electrons
Seth Mos
2013-05-29 09:13:39 UTC
Permalink
Post by Chris Bagnall
Post by Eugen Leitl
Which hardware are you using? If you're pushing 5 GBit/s you
might be running into hardware limitations. There was a thread
about it on nanog a week or two ago.
I'm quite impressed Mikrotik hardware is able to sustain 5Gbps with full
BGP tables from multiple transits to be honest.
Is anyone using pfSense's BGP package with 2.1 and v6 support? Given our
usage in this case is a great deal less than 5Gbps, I'm seriously
considering giving it a try - it would certainly make management a lot
easier, and mean they wouldn't need to call me every time they want to
change a VLAN config :-)
But v6 is a requirement. I couldn't in all good conscience deploy
something that isn't v6 capable these days.
It works for me.

Keep "Listen on IP" blank (a v4 here causes issue), fill in "Router IP"
Add both networks v4 or v6 in the networks rows.

Configure the IPv4 and IPv6 neighbors.
In the neighbor config, configure the remote neighbor v6 address and the
add a row for "Remote AS" with the remote AS.
Add a row "Local Address", fill in the local v6 address here.

It's been connected dual stack here for about 7 weeks now, but it's a
test box not passing traffic.

Cheers,

Seth
Mark Tinka
2013-05-29 09:13:46 UTC
Permalink
Post by Eugen Leitl
Which hardware are you using? If you're pushing 5 GBit/s
you might be running into hardware limitations. There
was a thread about it on nanog a week or two ago.
In truth, if you're picking up 5Gbps through pfSense, that's
pretty good.

You'd have to fork quite a bit of cash to get that on a
Cisco or Juniper router :-).

Mark.

Adam Thompson
2013-05-28 20:06:48 UTC
Permalink
Post by Glenn Kelley
Out of interest - why RIPv2 vs OSPF ?
Simplicity of configuration and troubleshooting. It's definitely not
the best protocol for complex environments, but there are places where
OSPF is simply overkill - I've seen two OSPF deployments in my career
where some of OSPF's advanced features were needed, and maybe another
half dozen where some of those features were helpful.
In contrast, almost every single OSPF deployment I've seen or worked on
has had issues because someone set up an overly complex system that they
then tripped over later on.
Post by Glenn Kelley
I guess if link-state, service types and load balancing is not
important it may make sense -
but figured I would ask
That's a perfect example. With modern systems, "link state" is
becoming useless - at the large ISP I worked for recently, the
overwhelming majority of routed links ran into switches or radios or
some local ethernet device that did not accurately reflect link state.
Integrating BFD into the link-state logic (à la JunOS) only works if
both ends support BFD, and not much of my gear did.
Service types were irrelevant there.
Load balancing was actually a bug, not a feature, in that network.
We wound up using IS-IS for the big microwave ring because of its
ability to respond to outages and converge extremely quickly. If I
hadn't had IS-IS-capable equipment at each hop, I would have used RIPv2
with the timers set fast. OSPF interaction with the rest of our OSPF
network would have been detrimental.

Just like I often use an Intel server running pfSense instead of buying
a Cisco CRS-1 for every client, not every network needs OSPF. The
aforementioned microwave ring, although spanning ~60 L3 sites (not sure
how many L2 sites) across just over 1000km, and supporting more than
half my customer base, was logically self-contained with a single border
router.
Configuring (and troubleshooting!) a multi-area, multi-vendor OSPF
network on 100+ devices was kind of like running a Windows 95 PC: you
*don't* want to be hand-editing the registry at 3am when customers are
screaming at you.
IS-IS (or RIPv2) on 60-odd devices is like running a Mac... if it
breaks at 3am, you just go home and buy a new one tomorrow ;-).
Facetiousness aside, I've only ever had to troubleshoot IS-IS once, and
RIP (both v1 & v2) almost never. I think that's due to the simplicity:
there's nothing to go wrong!
Post by Glenn Kelley
We have migrated for our BGP implementations to use Mikrotik at the
suggestion of a key player here - and could not be happier.
Sadly our multi-honed BGP implementation had tons of issues.
Interesting... I've had exactly the opposite experience. Mikrotik
routers are, IMHO, difficult to scale up, more difficult to configure
than they need to be, and have been surprisingly (as in,
biting-me-in-the-ass-surprisingly) feature-incomplete in surprisingly
critical places. I agree they do have a place, and they're not
inherently bad products... I've just had poor luck with them.
Post by Glenn Kelley
Internally however - I still think pfSense Rocks !
We have a large number of pfsense running virtualized in ProxMox for
clients - most who elected to use PFSense vs. vyatta after we showed
them the ease of use.
Some anecdotal evidence suggests Vyatta can scale to significantly
higher aggregate throughput than pfSense. I don't know of any
head-to-head comparisons on identical hardware, though, nor have I run
direct comparisons myself. Note that I have pfSense in production, but
not Vyatta - again, I don't need those features. (Or, rather, when I
do, I just go and buy a Cisco ASR or Juniper MX.)

-Adam
Chris Bagnall
2013-05-28 20:30:36 UTC
Permalink
Post by Adam Thompson
Interesting... I've had exactly the opposite experience.
If the Mikrotik forums are to be trusted, there are certainly quite a
few people who have run into problems running full tables on even their
high end Mikrotik platforms.

Despite Quagga's 'quirks', in the 5 years or so we've been using it, I
don't think we've had even one outage that wasn't hardware related.
Touch wood, long may that continue.

Perhaps the answer is to continue using Quagga at the edge to manage the
BGP tables, and use pfSense to handle VLANs etc. - but that's
maintaining twice as much infrastructure for a very marginal benefit.

Kind regards,

Chris
--
This email is made from 100% recycled electrons
Mark Tinka
2013-05-28 21:38:07 UTC
Permalink
Post by Adam Thompson
Post by Glenn Kelley
Out of interest - why RIPv2 vs OSPF ?
Simplicity of configuration and troubleshooting. It's
definitely not the best protocol for complex
environments, but there are places where OSPF is simply
overkill - I've seen two OSPF deployments in my career
where some of OSPF's advanced features were needed, and
maybe another half dozen where some of those features
were helpful. In contrast, almost every single OSPF
deployment I've seen or worked on has had issues because
someone set up an overly complex system that they then
tripped over later on.
In fairness, I've heard of deployments where RIP has been
implemented for TOR scenarios.

Would I do it, probably not. But I can see a use-case.

Mark.
Mark Tinka
2013-05-28 21:34:06 UTC
Permalink
Post by Adam Thompson
I do agree - in a large, heterogenous, complex-topology
network, IS-IS proved to be a winner both for its
reliability and the simplicity of configuration.
In many ways, I find it simpler than OSPF. But let's not
start a war :-).
Post by Adam Thompson
I suspect one of the main reasons we don't see UNIX IS-IS
implementations is the hard requirement for an ISO
address. There are ways of faking that, however, which
is what I suspect Quagga does.
Implementation of HMAC IIH and LSP authentication in Quagga
was also very poor. In a network where we ran Anycast DNS
with the name servers, it was easier to run OSPF on the
servers and carefully redisribute into IS-IS within the
core.
Post by Adam Thompson
In a green-field implementation (or anywhere there's a
clearly-defined border), RIPv2 is actually a very usable
protocol. Most people hear "RIP" and think of the old
RIPv1 protocol, which had major issues, but RIPv2 is
pretty clean and simple to deploy in production. It's
not a multi-protocol routing protocol, however, so in a
complex network it might still not be suitable. It does
have some attributes of its legacy predecessor, but
they're mostly ignorable on modern hardware. Don't
deploy it on 64kbps links, though...
Having said that, I'd still rather deploy RIPv2 than
OSPF, given a choice.
I generally stay away from RIP.

Link state routing protocols tend to be a lot more superior
in their fundamental design than their distance vector
cousins.

Mark.
Glenn Kelley
2013-05-29 03:37:17 UTC
Permalink
I did not want a war either - figured I might learn something :-)
Truth is - I think many do what they feel is best.

I can see how OSPF can become a nightmare to handle if it is not planned well - but I also find the folks who I end up stepping in when they are stuck have tons of other issues.

I help operate a WISP network - and when I first got here they had a network all in one broadcast domain.
You could be on one end of the network some 65+ miles away and have someone have the ip next to yours.
Worse yet - many were in a bridge.

Chris helped me learn a ton - and thus perhaps the reason I am personally loyal to pFsense in so many ways.
I found he was willing to take the time to not only fix what I broke so many times but to also educate me along the way.

Anyhow - I, since we are running a WISP think the OSPF method is best for our case - however I can for sure see the scenario you have described as well.
Post by Mark Tinka
servers, it was easier to run OSPF on the
servers and carefully redisribute into IS-IS within the
core.
Mark Tinka
2013-05-29 07:54:12 UTC
Permalink
Post by Glenn Kelley
I can see how OSPF can become a nightmare to handle if it
is not planned well - but I also find the folks who I
end up stepping in when they are stuck have tons of
other issues.
In the workshops I and a close friend give, we normally say
that OSPF teaches good network design because of its
rigidity with regard to requiring that all areas reach back
to Area 0.

So when we start teaching the IS-IS modules, it turns out to
be very simple because with IS-IS, you don't really need
areas (even though you can introduce hierarchy with multiple
levels), and since there is no requirement for IS-IS routers
to connect back to a so-called "backbone area", the protocol
lends itself well to networks that are stringy (rather than
star-based) in nature.

So I would teach OSPF first, so folk understand the nuances
of what it means to scale up OSPF, and then teach IS-IS on
the back of that which completely de-mystifies it.

Cheers,

Mark.
Loading...