[pfSense] 10GBASE-T hardware

[Paul Mather]

A 10GBASE-T port became available to us in our server rack. The rack currently has a 20-node Hadoop cluster, each node having dual Intel i350 1000BASE-T NICs. The Hadoop nodes connect to an old HP 2910al-48G 48-port GbE switch that, in turn, connects to an old Dell R310 server running pfSense that serves as the WAN gateway for the cluster.

It appears that the choice (not ours) of RJ45 for the 10 GbE provided for us in the rack will necessitate some equipment changes if we are to utilise the 10 GbE connection. Having done some investigation, I've decided the following changes are likely needed, and I would like to solicit from the list comment regarding any obvious blunders in the plan below:

1) I need a 10 GbE uplink capability from my switch to the pfSense gateway and also 10GBASE-T WAN connectivity from my pfSense gateway to the 10GBASE-T port in the rack.

2) The 10 GbE expansion options for the HP 2910al-48G are limited and I couldn't actually find any 10GBASE-T solutions (IIRC). If I went for 10 GbE SFP+ in the HP 2910al-48G that would mean I would also need 10 GbE SFP+ capability in my pfSense gateway---likely meaning I would need two 10 GbE NICs (one SFP+ and one 10GBASE-T), which means...

3) It is probably cheaper (alas, we are on a budget) to buy a new switch to replace the HP 2910al-48G that includes 10GBASE-T uplink capability. That would let me just have a single 10 GbE card for the pfSense gateway. I think the Netgear GS752TX 52-port switch would be a good candidate as it includes two 10GBASE-T ports in addition to the 48 1000BASE-T ports.

4) I am considering a Chelsio NIC for the 10GBASE-T WAN/LAN connections because I keep hearing these are the best-supported 10 GbE cards under FreeBSD. I'd get a Chelsio T420-BT but these seem to be discontinued in favour of the Chelsio T520-BT. Are there any better choices I should be considering? Intel X540-T2??


So, as I said earlier, are there any glaring problems in the above plan? (Does it seem sensible?) Or, alternatively, is there a much better solution that I've overlooked entirely? Constructive criticism/input is appreciated.

Thanks in advance.

Cheers,

Paul.

[Moshe Katz]

According to the specs that I found on HP's website, your HP switch does
not support 10Gb, only 1Gb on its mini-GBIC ports. You will definitely need
a new switch to take advantage of 10Gb.

If you do get a switch that supports 10GBase-T, you should definitely
consider the Intel X540. The vast majority of reports that I have seen say
that it works great. (There was one report I found on a forum claiming
performance issues, but others on the same thread said it worked fine for
them.)

There are also many dual-port SFP+ cards out there (such as the Intel X520)
that are not too expensive and support lots of different types of SFP+
connectors. Although Intel does not make a 10GBase-T SFP+ itself, there are
third parties that make it. You would use one of those to connect to the
10GbE feed into the rack and then a regular fiber SFP (or the option listed
below) to connect to the switch.

To connect the pfSense to the switch, I would probably use a Direct-Attach
cable (DAC) instead of fiber or Ethernet. Approved Optics
<http://approvedoptics.com/> is a company that makes many OEM network
connectors under contract and they also make their own versions of them at
significantly reduced prices. Their DAC Finder
<http://approvedoptics.com/dac-finder/> tool lets you order a cable that
has SFP+ ends for different manufacturers (for example, an Intel end for
your pfSense and an HP end for your switch). There's no need to worry about
fiber or CAT7A Ethernet cables; just plug the cable in (taking care to make
sure it is oriented correctly) and that's it.

Since you have a limited budget, I really recommend going the
direct-attached route. They are so much cheaper and more resilient than
fiber, and switches with SFP+ slots are often much cheaper than switches
with 10GbE. For example, you can get a Uniquiti EdgeSwitch with 48 Gb ports
and 2 SFP+ ports for just around $400. These are the switches I have used
in many of our limited-budget installations in the past (including in a
University setting like yours seems to be from your email address) and they
perform well. (Note that Approved Optics does not have official Ubiquiti
cables, but many on the Ubiquiti forums report that it works with Cisco and
other brand cables as long as they are 2 meters or shorter. In a single
rack, that should not be an issue.)


Moshe

--
Moshe Katz
-- ***@ymkatz.net
-- [phone]

Post by Paul Mather
A 10GBASE-T port became available to us in our server rack. The rack
currently has a 20-node Hadoop cluster, each node having dual Intel i350
1000BASE-T NICs. The Hadoop nodes connect to an old HP 2910al-48G 48-port
GbE switch that, in turn, connects to an old Dell R310 server running
pfSense that serves as the WAN gateway for the cluster.
It appears that the choice (not ours) of RJ45 for the 10 GbE provided for
us in the rack will necessitate some equipment changes if we are to utilise
the 10 GbE connection. Having done some investigation, I've decided the
following changes are likely needed, and I would like to solicit from the
1) I need a 10 GbE uplink capability from my switch to the pfSense gateway
and also 10GBASE-T WAN connectivity from my pfSense gateway to the
10GBASE-T port in the rack.
2) The 10 GbE expansion options for the HP 2910al-48G are limited and I
couldn't actually find any 10GBASE-T solutions (IIRC). If I went for 10
GbE SFP+ in the HP 2910al-48G that would mean I would also need 10 GbE SFP+
capability in my pfSense gateway---likely meaning I would need two 10 GbE
NICs (one SFP+ and one 10GBASE-T), which means...
3) It is probably cheaper (alas, we are on a budget) to buy a new switch
to replace the HP 2910al-48G that includes 10GBASE-T uplink capability.
That would let me just have a single 10 GbE card for the pfSense gateway.
I think the Netgear GS752TX 52-port switch would be a good candidate as it
includes two 10GBASE-T ports in addition to the 48 1000BASE-T ports.
4) I am considering a Chelsio NIC for the 10GBASE-T WAN/LAN connections
because I keep hearing these are the best-supported 10 GbE cards under
FreeBSD. I'd get a Chelsio T420-BT but these seem to be discontinued in
favour of the Chelsio T520-BT. Are there any better choices I should be
considering? Intel X540-T2??
So, as I said earlier, are there any glaring problems in the above plan?
(Does it seem sensible?) Or, alternatively, is there a much better
solution that I've overlooked entirely? Constructive criticism/input is
appreciated.
Thanks in advance.
Cheers,
Paul.
_______________________________________________
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

[Moshe Katz]

Note: that should say CAT*6*A, not *7*. Other than that, no changes.

--
Moshe Katz
-- ***@ymkatz.net
-- [phone]

Post by Moshe Katz
According to the specs that I found on HP's website, your HP switch does
not support 10Gb, only 1Gb on its mini-GBIC ports. You will definitely need
a new switch to take advantage of 10Gb.
If you do get a switch that supports 10GBase-T, you should definitely
consider the Intel X540. The vast majority of reports that I have seen say
that it works great. (There was one report I found on a forum claiming
performance issues, but others on the same thread said it worked fine for
them.)
There are also many dual-port SFP+ cards out there (such as the Intel
X520) that are not too expensive and support lots of different types of
SFP+ connectors. Although Intel does not make a 10GBase-T SFP+ itself,
there are third parties that make it. You would use one of those to connect
to the 10GbE feed into the rack and then a regular fiber SFP (or the option
listed below) to connect to the switch.
To connect the pfSense to the switch, I would probably use a Direct-Attach
cable (DAC) instead of fiber or Ethernet. Approved Optics
<http://approvedoptics.com/> is a company that makes many OEM network
connectors under contract and they also make their own versions of them at
significantly reduced prices. Their DAC Finder
<http://approvedoptics.com/dac-finder/> tool lets you order a cable that
has SFP+ ends for different manufacturers (for example, an Intel end for
your pfSense and an HP end for your switch). There's no need to worry about
fiber or CAT7A Ethernet cables; just plug the cable in (taking care to make
sure it is oriented correctly) and that's it.
Since you have a limited budget, I really recommend going the
direct-attached route. They are so much cheaper and more resilient than
fiber, and switches with SFP+ slots are often much cheaper than switches
with 10GbE. For example, you can get a Uniquiti EdgeSwitch with 48 Gb ports
and 2 SFP+ ports for just around $400. These are the switches I have used
in many of our limited-budget installations in the past (including in a
University setting like yours seems to be from your email address) and they
perform well. (Note that Approved Optics does not have official Ubiquiti
cables, but many on the Ubiquiti forums report that it works with Cisco and
other brand cables as long as they are 2 meters or shorter. In a single
rack, that should not be an issue.)
Moshe
--
Moshe Katz
-- [phone] <(301)%20867-3732>

Post by Paul Mather
A 10GBASE-T port became available to us in our server rack. The rack
currently has a 20-node Hadoop cluster, each node having dual Intel i350
1000BASE-T NICs. The Hadoop nodes connect to an old HP 2910al-48G 48-port
GbE switch that, in turn, connects to an old Dell R310 server running
pfSense that serves as the WAN gateway for the cluster.
It appears that the choice (not ours) of RJ45 for the 10 GbE provided for
us in the rack will necessitate some equipment changes if we are to utilise
the 10 GbE connection. Having done some investigation, I've decided the
following changes are likely needed, and I would like to solicit from the
1) I need a 10 GbE uplink capability from my switch to the pfSense
gateway and also 10GBASE-T WAN connectivity from my pfSense gateway to the
10GBASE-T port in the rack.
2) The 10 GbE expansion options for the HP 2910al-48G are limited and I
couldn't actually find any 10GBASE-T solutions (IIRC). If I went for 10
GbE SFP+ in the HP 2910al-48G that would mean I would also need 10 GbE SFP+
capability in my pfSense gateway---likely meaning I would need two 10 GbE
NICs (one SFP+ and one 10GBASE-T), which means...
3) It is probably cheaper (alas, we are on a budget) to buy a new switch
to replace the HP 2910al-48G that includes 10GBASE-T uplink capability.
That would let me just have a single 10 GbE card for the pfSense gateway.
I think the Netgear GS752TX 52-port switch would be a good candidate as it
includes two 10GBASE-T ports in addition to the 48 1000BASE-T ports.
4) I am considering a Chelsio NIC for the 10GBASE-T WAN/LAN connections
because I keep hearing these are the best-supported 10 GbE cards under
FreeBSD. I'd get a Chelsio T420-BT but these seem to be discontinued in
favour of the Chelsio T520-BT. Are there any better choices I should be
considering? Intel X540-T2??
So, as I said earlier, are there any glaring problems in the above plan?
(Does it seem sensible?) Or, alternatively, is there a much better
solution that I've overlooked entirely? Constructive criticism/input is
appreciated.
Thanks in advance.
Cheers,
Paul.
_______________________________________________
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

[Yehuda Katz]

I agree with everything my brother said except recommending the Uniquiti
EdgeSwitch.
We have seen a few instances of the EdgeSwitch locking up without any
apparent reason (once we traced it to a thermal issue, but we couldn't find
a cause for the others).
The EdgeSwitch also only has a 1 year warranty while the Netgear you
mentioned has a Lifetime Warranty (for whatever that is worth).
At (insert university name here) we were happily standardizing on Brocade
ICX switches until we hit major OSPF firmware bugs. Dell N and S series are
good, but also more expensive than that Netgear.

- Y

Post by Moshe Katz
According to the specs that I found on HP's website, your HP switch does
not support 10Gb, only 1Gb on its mini-GBIC ports. You will definitely need
a new switch to take advantage of 10Gb.
If you do get a switch that supports 10GBase-T, you should definitely
consider the Intel X540. The vast majority of reports that I have seen say
that it works great. (There was one report I found on a forum claiming
performance issues, but others on the same thread said it worked fine for
them.)
There are also many dual-port SFP+ cards out there (such as the Intel X520)
that are not too expensive and support lots of different types of SFP+
connectors. Although Intel does not make a 10GBase-T SFP+ itself, there are
third parties that make it. You would use one of those to connect to the
10GbE feed into the rack and then a regular fiber SFP (or the option listed
below) to connect to the switch.
To connect the pfSense to the switch, I would probably use a Direct-Attach
cable (DAC) instead of fiber or Ethernet. Approved Optics
<http://approvedoptics.com/> is a company that makes many OEM network
connectors under contract and they also make their own versions of them at
significantly reduced prices. Their DAC Finder
<http://approvedoptics.com/dac-finder/> tool lets you order a cable that
has SFP+ ends for different manufacturers (for example, an Intel end for
your pfSense and an HP end for your switch). There's no need to worry about
fiber or CAT7A Ethernet cables; just plug the cable in (taking care to make
sure it is oriented correctly) and that's it.
Since you have a limited budget, I really recommend going the
direct-attached route. They are so much cheaper and more resilient than
fiber, and switches with SFP+ slots are often much cheaper than switches
with 10GbE. For example, you can get a Uniquiti EdgeSwitch with 48 Gb ports
and 2 SFP+ ports for just around $400. These are the switches I have used
in many of our limited-budget installations in the past (including in a
University setting like yours seems to be from your email address) and they
perform well. (Note that Approved Optics does not have official Ubiquiti
cables, but many on the Ubiquiti forums report that it works with Cisco and
other brand cables as long as they are 2 meters or shorter. In a single
rack, that should not be an issue.)
Moshe
--
Moshe Katz
-- [phone]

Post by Paul Mather
A 10GBASE-T port became available to us in our server rack. The rack
currently has a 20-node Hadoop cluster, each node having dual Intel i350
1000BASE-T NICs. The Hadoop nodes connect to an old HP 2910al-48G

48-port

Post by Paul Mather
GbE switch that, in turn, connects to an old Dell R310 server running
pfSense that serves as the WAN gateway for the cluster.
It appears that the choice (not ours) of RJ45 for the 10 GbE provided for
us in the rack will necessitate some equipment changes if we are to

utilise

Post by Paul Mather
the 10 GbE connection. Having done some investigation, I've decided the
following changes are likely needed, and I would like to solicit from the
1) I need a 10 GbE uplink capability from my switch to the pfSense

gateway

Post by Paul Mather
and also 10GBASE-T WAN connectivity from my pfSense gateway to the
10GBASE-T port in the rack.
2) The 10 GbE expansion options for the HP 2910al-48G are limited and I
couldn't actually find any 10GBASE-T solutions (IIRC). If I went for 10
GbE SFP+ in the HP 2910al-48G that would mean I would also need 10 GbE

SFP+

Post by Paul Mather
capability in my pfSense gateway---likely meaning I would need two 10 GbE
NICs (one SFP+ and one 10GBASE-T), which means...
3) It is probably cheaper (alas, we are on a budget) to buy a new switch
to replace the HP 2910al-48G that includes 10GBASE-T uplink capability.
That would let me just have a single 10 GbE card for the pfSense gateway.
I think the Netgear GS752TX 52-port switch would be a good candidate as

it

Post by Paul Mather
includes two 10GBASE-T ports in addition to the 48 1000BASE-T ports.
4) I am considering a Chelsio NIC for the 10GBASE-T WAN/LAN connections
because I keep hearing these are the best-supported 10 GbE cards under
FreeBSD. I'd get a Chelsio T420-BT but these seem to be discontinued in
favour of the Chelsio T520-BT. Are there any better choices I should be
considering? Intel X540-T2??
So, as I said earlier, are there any glaring problems in the above plan?
(Does it seem sensible?) Or, alternatively, is there a much better
solution that I've overlooked entirely? Constructive criticism/input is
appreciated.
Thanks in advance.
Cheers,
Paul.
_______________________________________________
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

_______________________________________________
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

[Paul Mather]

Post by Yehuda Katz
I agree with everything my brother said except recommending the Uniquiti
EdgeSwitch.
We have seen a few instances of the EdgeSwitch locking up without any
apparent reason (once we traced it to a thermal issue, but we couldn't find
a cause for the others).
The EdgeSwitch also only has a 1 year warranty while the Netgear you
mentioned has a Lifetime Warranty (for whatever that is worth).
At (insert university name here) we were happily standardizing on Brocade
ICX switches until we hit major OSPF firmware bugs. Dell N and S series are
good, but also more expensive than that Netgear.

Thank you for the information. Actually, having done some more searching, our budget could probably also stretch to getting a Cisco SG350X-48 switch instead of the Netgear. Like the Netgear, it apparently features 48 1000Base-T ports plus two 10GBASE-T/SFP+ combo ports + 2 10GbE SFP+ ports. So, port-wise, the same as the Netgear, but likely better firmware-wise/support. (I have experience with the firmware of the Cisco SG350-28 model and really like its feature set.)

Cheers,

Paul.

[Paul Mather]

On Mar 27, 2018, at 8:10 PM, Moshe Katz <***@ymkatz.net> wrote:

Many thanks for the information and advice. It is much appreciated.

Post by Moshe Katz
According to the specs that I found on HP's website, your HP switch does
not support 10Gb, only 1Gb on its mini-GBIC ports. You will definitely need
a new switch to take advantage of 10Gb.

It's true that the mini-GBIC ports support only 1Gb, but that HP switch also can accommodate two(?) option modules at the rear of the switch that can be used to provide 10 Gb connectivity. According to the "HP ProCurve Switch - What modules are available for the 2910al?" page at the HP Support site (https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c02620659 <https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c02620659>) you can get dual-port 10-GbE CX4 and 10-GbE SFP+ option modules. Well, at least I suppose you could, as I'm not sure how widely available they are, and, this being an old switch, it may be that buying option modules from vendors with which $WORK are willing to purchase may result in them being prohibitively expensive due to them being legacy/discontinued equipment. In my experience, those tend to command a premium price (except when buying via eBay). (The SFP+ option module---J9008A---appears to cost $700+ on a quick search.)

Post by Moshe Katz
If you do get a switch that supports 10GBase-T, you should definitely
consider the Intel X540. The vast majority of reports that I have seen say
that it works great. (There was one report I found on a forum claiming
performance issues, but others on the same thread said it worked fine for
them.)

Thanks, that's very good to know.

Post by Moshe Katz
There are also many dual-port SFP+ cards out there (such as the Intel X520)
that are not too expensive and support lots of different types of SFP+
connectors. Although Intel does not make a 10GBase-T SFP+ itself, there are
third parties that make it. You would use one of those to connect to the
10GbE feed into the rack and then a regular fiber SFP (or the option listed
below) to connect to the switch.

See below for queries/concerns about obtaining a 10GBase-T SFP+ transceiver.

Post by Moshe Katz
To connect the pfSense to the switch, I would probably use a Direct-Attach
cable (DAC) instead of fiber or Ethernet. Approved Optics
<http://approvedoptics.com/> is a company that makes many OEM network
connectors under contract and they also make their own versions of them at
significantly reduced prices. Their DAC Finder
<http://approvedoptics.com/dac-finder/> tool lets you order a cable that
has SFP+ ends for different manufacturers (for example, an Intel end for
your pfSense and an HP end for your switch). There's no need to worry about
fiber or CAT7A Ethernet cables; just plug the cable in (taking care to make
sure it is oriented correctly) and that's it.

Again, many thanks for the Approved Optics link. That will be very useful.

I don't have any practical SFP+ experience, so maybe you or someone else can verify whether I am understanding this correctly: the Direct-Attach cable basically encapsulates a transceiver at each end with an appropriate cable connecting them, all in one unit?

Post by Moshe Katz
Since you have a limited budget, I really recommend going the
direct-attached route. They are so much cheaper and more resilient than
fiber, and switches with SFP+ slots are often much cheaper than switches
with 10GbE. For example, you can get a Uniquiti EdgeSwitch with 48 Gb ports
and 2 SFP+ ports for just around $400. These are the switches I have used
in many of our limited-budget installations in the past (including in a
University setting like yours seems to be from your email address) and they
perform well. (Note that Approved Optics does not have official Ubiquiti
cables, but many on the Ubiquiti forums report that it works with Cisco and
other brand cables as long as they are 2 meters or shorter. In a single
rack, that should not be an issue.)

My main issue with going the SFP+ route is that my rack uplink port is still 10GBASE-T and so I'd need to find a 10GBASE-T transceiver for the pfSense 10 GbE NIC and these seem difficult to find or they are 3rd party or they are expensive themselves (e.g., $200--$300+). I've also heard there are thermal issues with those transceivers as there's not much opportunity to build in the requisite heat sinks that 10GBASE-T appears to need. (I've noticed 10GBASE-T NICs have pretty hefty heatsinks on them.) Besides that, I've not been able to find a 10GBASE-T transceiver for Chelsio NICs and only 3rd party ones for Intel---e.g., by some company called 10Gtek.

Does anyone have any advice/experience to share regarding 10GBASE-T transceivers?

Thanks again for the info.

Cheers,

Paul.

[Moshe Katz]

Post by Paul Mather
Many thanks for the information and advice. It is much appreciated.

Post by Moshe Katz
According to the specs that I found on HP's website, your HP switch does
not support 10Gb, only 1Gb on its mini-GBIC ports. You will definitely

need

Post by Moshe Katz
a new switch to take advantage of 10Gb.

It's true that the mini-GBIC ports support only 1Gb, but that HP switch
also can accommodate two(?) option modules at the rear of the switch that
can be used to provide 10 Gb connectivity. According to the "HP ProCurve
Switch - What modules are available for the 2910al?" page at the HP Support
site (https://support.hpe.com/hpsc/doc/public/display?docId=emr_
na-c02620659 <https://support.hpe.com/hpsc/doc/public/display?docId=emr_
na-c02620659>) you can get dual-port 10-GbE CX4 and 10-GbE SFP+ option
modules. Well, at least I suppose you could, as I'm not sure how widely
available they are, and, this being an old switch, it may be that buying
option modules from vendors with which $WORK are willing to purchase may
result in them being prohibitively expensive due to them being
legacy/discontinued equipment. In my experience, those tend to command a
premium price (except when buying via eBay). (The SFP+ option
module---J9008A---appears to cost $700+ on a quick search.)

I missed that in the documentation, but something tells me you probably
don't want to do that anyway - with hardware that old that uses a
proprietary expansion card that costs only a few hundred dollars less than
the whole new Cisco switch you referenced in your reply to my brother, that
just doesn't make sense. (That'e especially true because based on your
email address you may be able to get a significant educational discount on
Cisco.)

Post by Paul Mather
If you do get a switch that supports 10GBase-T, you should definitely

Post by Moshe Katz
consider the Intel X540. The vast majority of reports that I have seen

say

Post by Moshe Katz
that it works great. (There was one report I found on a forum claiming
performance issues, but others on the same thread said it worked fine for
them.)

Thanks, that's very good to know.

Post by Moshe Katz
There are also many dual-port SFP+ cards out there (such as the Intel

X520)

Post by Moshe Katz
that are not too expensive and support lots of different types of SFP+
connectors. Although Intel does not make a 10GBase-T SFP+ itself, there

are

Post by Moshe Katz
third parties that make it. You would use one of those to connect to the
10GbE feed into the rack and then a regular fiber SFP (or the option

listed

Post by Moshe Katz
below) to connect to the switch.

See below for queries/concerns about obtaining a 10GBase-T SFP+ transceiver.

Post by Moshe Katz
To connect the pfSense to the switch, I would probably use a

Direct-Attach

Post by Moshe Katz
cable (DAC) instead of fiber or Ethernet. Approved Optics
<http://approvedoptics.com/> is a company that makes many OEM network
connectors under contract and they also make their own versions of them

at

Post by Moshe Katz
significantly reduced prices. Their DAC Finder
<http://approvedoptics.com/dac-finder/> tool lets you order a cable that
has SFP+ ends for different manufacturers (for example, an Intel end for
your pfSense and an HP end for your switch). There's no need to worry

about

Post by Moshe Katz
fiber or CAT7A Ethernet cables; just plug the cable in (taking care to

make

Post by Moshe Katz
sure it is oriented correctly) and that's it.

Again, many thanks for the Approved Optics link. That will be very useful.
I don't have any practical SFP+ experience, so maybe you or someone else
can verify whether I am understanding this correctly: the Direct-Attach
cable basically encapsulates a transceiver at each end with an appropriate
cable connecting them, all in one unit?

Yes, that us correct. Direct-Attach Cables are usually "twin-ax" cables,
similar to "coax" used for TV and Cable Internet, but with two internal
connectors. The cable is permanently connected to two SFP+ transceivers,
which can be either "Active" or "Passive", meaning that they either have
processing hardware inside the ends or that they pass the signal through
with no changes respectively. Some manufacturers refer to this cable as
"10GSFP+Cu" and others refer to it as "10GBASE-CR".

The major benefit of DA over 10GBase-T is that the Base-T transceiver
latency can be 15-25 times higher than the passive DAC (because the passive
DAC does no processing, while the Base-T needs to process the signal). DAC
also draws almost no measurable power, but the Base-T transceiver hardware
usually consumes 4-8 additional watts. (In your case, with only one runs,
the additional power draw is likely not going to even be noticed, but it
still bears mentioning.)

All that said about the benefits of Direct-Attach, see below for the other
side of the coin.

Post by Paul Mather
Since you have a limited budget, I really recommend going the

Post by Moshe Katz
direct-attached route. They are so much cheaper and more resilient than
fiber, and switches with SFP+ slots are often much cheaper than switches
with 10GbE. For example, you can get a Uniquiti EdgeSwitch with 48 Gb

ports

Post by Moshe Katz
and 2 SFP+ ports for just around $400. These are the switches I have used
in many of our limited-budget installations in the past (including in a
University setting like yours seems to be from your email address) and

they

Post by Moshe Katz
perform well. (Note that Approved Optics does not have official Ubiquiti
cables, but many on the Ubiquiti forums report that it works with Cisco

and

Post by Moshe Katz
other brand cables as long as they are 2 meters or shorter. In a single
rack, that should not be an issue.)

My main issue with going the SFP+ route is that my rack uplink port is
still 10GBASE-T and so I'd need to find a 10GBASE-T transceiver for the
pfSense 10 GbE NIC and these seem difficult to find or they are 3rd party
or they are expensive themselves (e.g., $200--$300+). I've also heard
there are thermal issues with those transceivers as there's not much
opportunity to build in the requisite heat sinks that 10GBASE-T appears to
need. (I've noticed 10GBASE-T NICs have pretty hefty heatsinks on them.)
Besides that, I've not been able to find a 10GBASE-T transceiver for
Chelsio NICs and only 3rd party ones for Intel---e.g., by some company
called 10Gtek.
Does anyone have any advice/experience to share regarding 10GBASE-T transceivers?

I don't personally have any of the Intel-compatible 10GBASE-T transceivers
at the moment, but I have seen reports online that the 10Gtek ones are
reliable. (In theory, any SFP+ module that conforms to the official specs
should inter-operate with any other, but, as they say, "that's a nice
theory".)
As far as the heat distribution, that really should be picked up and
handled by the network card and the server's cooling system.


However, based on your response to my brother's email about being able to
budget the Cisco switch with 10GBASE-T, I suggest that you probably
couldn't go wrong with that simply because it's the solution with fewest
number of parts. I would still consider the Intel card over the Chelsio
card if you're really trying to work within a small budget, but you should
go with whatever you think is best for you.

Post by Paul Mather
Thanks again for the info.
Cheers,
Paul.

--
Moshe Katz
-- ***@ymkatz.net
-- [phone]

[Paul Mather]

Post by Moshe Katz

Post by Paul Mather
Many thanks for the information and advice. It is much appreciated.

[[...]]

Post by Moshe Katz

Post by Paul Mather
My main issue with going the SFP+ route is that my rack uplink port is
still 10GBASE-T and so I'd need to find a 10GBASE-T transceiver for the
pfSense 10 GbE NIC and these seem difficult to find or they are 3rd party
or they are expensive themselves (e.g., $200--$300+). I've also heard
there are thermal issues with those transceivers as there's not much
opportunity to build in the requisite heat sinks that 10GBASE-T appears to
need. (I've noticed 10GBASE-T NICs have pretty hefty heatsinks on them.)
Besides that, I've not been able to find a 10GBASE-T transceiver for
Chelsio NICs and only 3rd party ones for Intel---e.g., by some company
called 10Gtek.
Does anyone have any advice/experience to share regarding 10GBASE-T transceivers?

I don't personally have any of the Intel-compatible 10GBASE-T transceivers
at the moment, but I have seen reports online that the 10Gtek ones are
reliable. (In theory, any SFP+ module that conforms to the official specs
should inter-operate with any other, but, as they say, "that's a nice
theory".)
As far as the heat distribution, that really should be picked up and
handled by the network card and the server's cooling system.
However, based on your response to my brother's email about being able to
budget the Cisco switch with 10GBASE-T, I suggest that you probably
couldn't go wrong with that simply because it's the solution with fewest
number of parts. I would still consider the Intel card over the Chelsio
card if you're really trying to work within a small budget, but you should
go with whatever you think is best for you.

Many thanks for the followup information and advice. (I'm finding that 10 GbE networking and above is something of an alphabet soup, so thanks for the cabling explanation.)

I've decided to go with my original solution of using a replacement switch with 10GBASE-T uplink ports. You were exactly right in that when I got a quote for the Cisco SG350X-48 switch it was a more attractive choice than the Netgear once the educational discount was applied.

Thanks again for all the help.

Cheers,

Paul.