[pfSense] Best practice for SSD installs

[Chris Bagnall]

Greetings list,

I've used pfSense embedded for many years on ALIX boards.

However, given the difficulty of getting those boards with 4 NICs, or
more than 256MB RAM, I've recently been exprimenting with an Atom-based
motherboard instead. Unfortunately, but unsurprisingly, the Atom board
in question doesn't have a Compact Flash slot, so my usual approach of
flashing the pfSense embedded image to a card isn't an option here.

The board supports CFast (which is hideously expensive in the UK) and/or
a standard 2.5" SATA device. Given a standard 2.5" 32GB SSD is
considerably less expensive than even a 4GB CFast card, I suspect I'll
be using SSDs for future installs.

Which brings me to the question: the last time I performed a pfSense
'full' install (i.e. not embedded) was several years, and many versions
ago. What's the best practice when using an SSD? Use the CD-based
installer to do a 'full' install, or continue to use the embedded
NanoBSD image?

One other thing I thought I might try is using an USB flash device. I
notice from the snapshots there's an image available for these devices,
but I can't seem to find much by the way of documentation online about
the benefits/pitfalls of this approach.

As an aside, there are several options on the "Advanced" tab relating to
NIC performance options:
- Disable hardware checksum offload
- Disable hardware TCP segmentation offload
- Disable hardware large receive offload
Has anyone done any tests / is there a list maintained anywhere with
details of which NICs are "problematic" with these, and hence should be
disabled? The motherboard I'm using is a mix of Intel and Realtek
gigabit NICs (em and re respectively).

Any suggestions gratefully received.

Kind regards,

Chris

[Jim Thompson]

Post by Chris Bagnall
Greetings list,
I've used pfSense embedded for many years on ALIX boards.
However, given the difficulty of getting those boards with 4 NICs, or more than 256MB RAM, I've recently been exprimenting with an Atom-based motherboard instead. Unfortunately, but unsurprisingly, the Atom board in question doesn't have a Compact Flash slot, so my usual approach of flashing the pfSense embedded image to a card isn't an option here.

"Difficulty"? Is this some kind of Brit understatement? "Impossible" is a more accurate description of the situation. :-)

Post by Chris Bagnall
The board supports CFast (which is hideously expensive in the UK) and/or a standard 2.5" SATA device. Given a standard 2.5" 32GB SSD is considerably less expensive than even a 4GB CFast card, I suspect I'll be using SSDs for future installs.
Which brings me to the question: the last time I performed a pfSense 'full' install (i.e. not embedded) was several years, and many versions ago. What's the best practice when using an SSD? Use the CD-based installer to do a 'full' install, or continue to use the embedded NanoBSD image?

I can tell you what we do at Netgate (and since Chris sits in the office next door, it's in strong alignment with "best practices"):

Load the CD-based installer on an SSD.
If you use a USB DOM, you'll want to use the 'embedded' image.

Post by Chris Bagnall
One other thing I thought I might try is using an USB flash device. I notice from the snapshots there's an image available for these devices, but I can't seem to find much by the way of documentation online about the benefits/pitfalls of this approach.

That image is an 'installer' image.

Post by Chris Bagnall
- Disable hardware checksum offload
- Disable hardware TCP segmentation offload
- Disable hardware large receive offload
Has anyone done any tests / is there a list maintained anywhere with details of which NICs are "problematic" with these, and hence should be disabled? The motherboard I'm using is a mix of Intel and Realtek gigabit NICs (em and re respectively).

The Realtek NICs might not work in 2.0 series releases. 2.1RC is likely a better option. (We've recently patched the Netgate image for an up-n-coming board to support the Realtek NICs, but you might not want to go through that process.)

jim

[Chris Bagnall]

Thanks for the response.

Post by Jim Thompson
"Difficulty"? Is this some kind of Brit understatement? "Impossible" is a more accurate description of the situation. :-)

I've seen other AMD Geode boards with 4 NICs, but not with >256MB RAM,
and we've been seeing issues with <=256MB and 2.1.

Post by Jim Thompson
Load the CD-based installer on an SSD.
If you use a USB DOM, you'll want to use the 'embedded' image.

But the full install for an SSD? Or is it better to stick with embedded
on those too?

Post by Jim Thompson

Post by Chris Bagnall
One other thing I thought I might try is using an USB flash device. I notice from the snapshots there's an image available for these devices, but I can't seem to find much by the way of documentation online about the benefits/pitfalls of this approach.

That image is an 'installer' image.

Is it possible to 'install' pfSense to a bootable USB flash device at
all? Strikes me as a wonderfully elegant solution for updates: just ship
a new stick to the remote site and tell someone to plug it in and reboot :-)

Post by Jim Thompson
The Realtek NICs might not work in 2.0 series releases. 2.1RC is likely a better option.

Running 2.1 anyway - v6 support very much required :-)

FWIW, I've tested one of these boards this evening just using a spare
2.5" SATA spinning disk I had knocking around here, and both the Realtek
and Intel NICs seem to be working in 2.1. I've not put any load through
them yet, so I can't attest to performance.

Given most of these systems are going to be handling very low throughput
(<100Mbps WAN links), is it safer to just disable all the offloading
options to be on the safe side?

Kind regards,

Chris

[Jim Thompson]

Post by Chris Bagnall
Thanks for the response.

Post by Jim Thompson
"Difficulty"? Is this some kind of Brit understatement? "Impossible" is a more accurate description of the situation. :-)

I've seen other AMD Geode boards with 4 NICs, but not with >256MB RAM, and we've been seeing issues with <=256MB and 2.1.

Post by Jim Thompson
Load the CD-based installer on an SSD.
If you use a USB DOM, you'll want to use the 'embedded' image.

But the full install for an SSD? Or is it better to stick with embedded on those too?

full install, yes.
embedded is all about reducing writes to the CF.

Post by Chris Bagnall

Post by Jim Thompson

Post by Chris Bagnall
One other thing I thought I might try is using an USB flash device. I notice from the snapshots there's an image available for these devices, but I can't seem to find much by the way of documentation online about the benefits/pitfalls of this approach.

That image is an 'installer' image.

Is it possible to 'install' pfSense to a bootable USB flash device at all? Strikes me as a wonderfully elegant solution for updates: just ship a new stick to the remote site and tell someone to plug it in and reboot :-)

until it falls out.

Post by Chris Bagnall

Post by Jim Thompson
The Realtek NICs might not work in 2.0 series releases. 2.1RC is likely a better option.

Running 2.1 anyway - v6 support very much required :-)
FWIW, I've tested one of these boards this evening just using a spare 2.5" SATA spinning disk I had knocking around here, and both the Realtek and Intel NICs seem to be working in 2.1. I've not put any load through them yet, so I can't attest to performance.
Given most of these systems are going to be handling very low throughput (<100Mbps WAN links), is it safer to just disable all the offloading options to be on the safe side?

That's what the rest of the list will advise. They'll all claim that these hardware features "don't work". Nevermind that they work on other platforms. This gets spun into fokelore on the list.

The OpenBSD guys were just discussing how they *made* them work at BSDcan though.
http://www.bsdcan.org/2013/schedule/events/372.en.html

So there is hope that FreeBSD will study same and implement fixes.

Jim

[Michael Schuh]

Post by Chris Bagnall
Thanks for the response.
"Difficulty"? Is this some kind of Brit understatement? "Impossible"
is a more accurate description of the situation. :-)
I've seen other AMD Geode boards with 4 NICs, but not with >256MB RAM, and
we've been seeing issues with <=256MB and 2.1.
Load the CD-based installer on an SSD.
If you use a USB DOM, you'll want to use the 'embedded' image.
But the full install for an SSD? Or is it better to stick with embedded on those too?
full install, yes.
embedded is all about reducing writes to the CF.
One other thing I thought I might try is using an USB flash device. I
notice from the snapshots there's an image available for these devices, but
I can't seem to find much by the way of documentation online about the
benefits/pitfalls of this approach.
That image is an 'installer' image.
Is it possible to 'install' pfSense to a bootable USB flash device at all?
Strikes me as a wonderfully elegant solution for updates: just ship a new
stick to the remote site and tell someone to plug it in and reboot :-)
until it falls out.

Than use the embedded at the beginning on an USB-Stick: No need for
shipping to upgrade.
prolongs the life of the stick too and brings the ability to switch back to
the old version if something will not work after an upgrade.

Post by Chris Bagnall
The Realtek NICs might not work in 2.0 series releases. 2.1RC is likely a better option.
Running 2.1 anyway - v6 support very much required :-)
FWIW, I've tested one of these boards this evening just using a spare 2.5"
SATA spinning disk I had knocking around here, and both the Realtek and
Intel NICs seem to be working in 2.1. I've not put any load through them
yet, so I can't attest to performance.
Given most of these systems are going to be handling very low throughput
(<100Mbps WAN links), is it safer to just disable all the offloading
options to be on the safe side?
That's what the rest of the list will advise. They'll all claim that
these hardware features "don't work". Nevermind that they work on other
platforms. This gets spun into fokelore on the list.
The OpenBSD guys were just discussing how they *made* them work at BSDcan though.
http://www.bsdcan.org/2013/schedule/events/372.en.html
So there is hope that FreeBSD will study same and implement fixes.
Jim
_______________________________________________
List mailing list
http://lists.pfsense.org/mailman/listinfo/list

[Eugen Leitl]

Post by Chris Bagnall
Which brings me to the question: the last time I performed a pfSense
'full' install (i.e. not embedded) was several years, and many
versions ago. What's the best practice when using an SSD? Use the
CD-based installer to do a 'full' install, or continue to use the
embedded NanoBSD image?

Modern SSDs are at least as reliable as HDs. I've used SSDs
with pfSense for years (including IDE DoMs) with full install
and never had a failure yet.

Post by Chris Bagnall
As an aside, there are several options on the "Advanced" tab
- Disable hardware checksum offload
- Disable hardware TCP segmentation offload
- Disable hardware large receive offload
Has anyone done any tests / is there a list maintained anywhere with
details of which NICs are "problematic" with these, and hence should
be disabled? The motherboard I'm using is a mix of Intel and Realtek
gigabit NICs (em and re respectively).

I've used Supermicro Atoms with 2 Intel NICs onboard and
with a dual-port Intel NIC added. I would be also interested in
suggested list of settings for Intel NICs.

[Aaron C. de Bruyn]

Just a note of personal experience. I've deployed ~20 pfSense firewalls
that had SSDs (both cheap and rated 'good' from Newegg) over the past 2
years. I am not convinced SSDs are more reliable. Nearly every one has
had an SSD die or become corrupt. We switched them all to USB sticks and
haven't had any more issues. Plus it's easier for us to ship a replacement
USB stick to the client and have them plug it in than to have them pop open
the case and replace the drive.

Maybe we've just had bad luck with SSDs, but I'm not convinced they are
ready.

-A

Post by Eugen Leitl

Post by Chris Bagnall
Which brings me to the question: the last time I performed a pfSense
'full' install (i.e. not embedded) was several years, and many
versions ago. What's the best practice when using an SSD? Use the
CD-based installer to do a 'full' install, or continue to use the
embedded NanoBSD image?

Modern SSDs are at least as reliable as HDs. I've used SSDs
with pfSense for years (including IDE DoMs) with full install
and never had a failure yet.

Post by Chris Bagnall
As an aside, there are several options on the "Advanced" tab
- Disable hardware checksum offload
- Disable hardware TCP segmentation offload
- Disable hardware large receive offload
Has anyone done any tests / is there a list maintained anywhere with
details of which NICs are "problematic" with these, and hence should
be disabled? The motherboard I'm using is a mix of Intel and Realtek
gigabit NICs (em and re respectively).

I've used Supermicro Atoms with 2 Intel NICs onboard and
with a dual-port Intel NIC added. I would be also interested in
suggested list of settings for Intel NICs.
_______________________________________________
List mailing list
http://lists.pfsense.org/mailman/listinfo/list

[Mehma Sarja]

I've also had bad luck with SSDs on a Supermicro Atom. If you have access
to the hardware, SSD is not a bad option.

Post by Aaron C. de Bruyn
Just a note of personal experience. I've deployed ~20 pfSense firewalls
that had SSDs (both cheap and rated 'good' from Newegg) over the past 2
years. I am not convinced SSDs are more reliable. Nearly every one has
had an SSD die or become corrupt. We switched them all to USB sticks and
haven't had any more issues. Plus it's easier for us to ship a replacement
USB stick to the client and have them plug it in than to have them pop open
the case and replace the drive.
Maybe we've just had bad luck with SSDs, but I'm not convinced they are
ready.
-A

Post by Eugen Leitl

Post by Chris Bagnall
Which brings me to the question: the last time I performed a pfSense
'full' install (i.e. not embedded) was several years, and many
versions ago. What's the best practice when using an SSD? Use the
CD-based installer to do a 'full' install, or continue to use the
embedded NanoBSD image?

Modern SSDs are at least as reliable as HDs. I've used SSDs
with pfSense for years (including IDE DoMs) with full install
and never had a failure yet.

Post by Chris Bagnall
As an aside, there are several options on the "Advanced" tab
- Disable hardware checksum offload
- Disable hardware TCP segmentation offload
- Disable hardware large receive offload
Has anyone done any tests / is there a list maintained anywhere with
details of which NICs are "problematic" with these, and hence should
be disabled? The motherboard I'm using is a mix of Intel and Realtek
gigabit NICs (em and re respectively).

I've used Supermicro Atoms with 2 Intel NICs onboard and
with a dual-port Intel NIC added. I would be also interested in
suggested list of settings for Intel NICs.
_______________________________________________
List mailing list
http://lists.pfsense.org/mailman/listinfo/list

_______________________________________________
List mailing list
http://lists.pfsense.org/mailman/listinfo/list

[Espen F. Johansen]

If you buy Intel SSDs you should be able to have a worry free time. After
running them since the first time I was able to get my hands on one. Never
had a single problem with 30+ drives. Remember SSDs behave better with
quality PSUs.

Espen F. Johansen


Sent with AquaMail for Android
http://www.aqua-mail.com

Post by Aaron C. de Bruyn
Just a note of personal experience. I've deployed ~20 pfSense firewalls
that had SSDs (both cheap and rated 'good' from Newegg) over the past 2
years. I am not convinced SSDs are more reliable. Nearly every one has
had an SSD die or become corrupt. We switched them all to USB sticks and
haven't had any more issues. Plus it's easier for us to ship a replacement
USB stick to the client and have them plug it in than to have them pop open
the case and replace the drive.
Maybe we've just had bad luck with SSDs, but I'm not convinced they are
ready.
-A

Post by Eugen Leitl

Post by Chris Bagnall
Which brings me to the question: the last time I performed a pfSense
'full' install (i.e. not embedded) was several years, and many
versions ago. What's the best practice when using an SSD? Use the
CD-based installer to do a 'full' install, or continue to use the
embedded NanoBSD image?

Modern SSDs are at least as reliable as HDs. I've used SSDs
with pfSense for years (including IDE DoMs) with full install
and never had a failure yet.

Post by Chris Bagnall
As an aside, there are several options on the "Advanced" tab
- Disable hardware checksum offload
- Disable hardware TCP segmentation offload
- Disable hardware large receive offload
Has anyone done any tests / is there a list maintained anywhere with
details of which NICs are "problematic" with these, and hence should
be disabled? The motherboard I'm using is a mix of Intel and Realtek
gigabit NICs (em and re respectively).

I've used Supermicro Atoms with 2 Intel NICs onboard and
with a dual-port Intel NIC added. I would be also interested in
suggested list of settings for Intel NICs.
_______________________________________________
List mailing list
http://lists.pfsense.org/mailman/listinfo/list

[Michael Schuh]

i would recommend to read page 12, if i should get asked :-)
(not only but in that context)

http://phk.freebsd.dk/pubs/nanobsd.pdf

i wouldn't only rely on the manufacturer but on the chip type; just saying

= = = http://michael-schuh.net/ = = =
Projektmanagement - IT-Consulting - Professional Services IT
Rev. P.D. Michael
Schuh<http://dudeism.com/ordcertificate?ordname=Michael+Schuh&orddate=05/20/2012>
*Ordained Dudeist Priest <http://dudeism.com/>*
Postfach 10 21 52
66021 Saarbrücken
phone: [phone]
@: m i c h a e l . s c h u h @ g m a i l . c o m

= = = Ust-ID: DE251072318 = = =

Post by Espen F. Johansen
If you buy Intel SSDs you should be able to have a worry free time.
After running them since the first time I was able to get my hands on one.
Never had a single problem with 30+ drives. Remember SSDs behave better
with quality PSUs.
Espen F. Johansen
Sent with AquaMail for Android
http://www.aqua-mail.com
Just a note of personal experience. I've deployed ~20 pfSense firewalls
that had SSDs (both cheap and rated 'good' from Newegg) over the past 2
years. I am not convinced SSDs are more reliable. Nearly every one has
had an SSD die or become corrupt. We switched them all to USB sticks and
haven't had any more issues. Plus it's easier for us to ship a replacement
USB stick to the client and have them plug it in than to have them pop open
the case and replace the drive.
Maybe we've just had bad luck with SSDs, but I'm not convinced they are
ready.
-A

Post by Eugen Leitl

Post by Chris Bagnall
Which brings me to the question: the last time I performed a pfSense
'full' install (i.e. not embedded) was several years, and many
versions ago. What's the best practice when using an SSD? Use the
CD-based installer to do a 'full' install, or continue to use the
embedded NanoBSD image?

Modern SSDs are at least as reliable as HDs. I've used SSDs
with pfSense for years (including IDE DoMs) with full install
and never had a failure yet.

Post by Chris Bagnall
As an aside, there are several options on the "Advanced" tab
- Disable hardware checksum offload
- Disable hardware TCP segmentation offload
- Disable hardware large receive offload
Has anyone done any tests / is there a list maintained anywhere with
details of which NICs are "problematic" with these, and hence should
be disabled? The motherboard I'm using is a mix of Intel and Realtek
gigabit NICs (em and re respectively).

I've used Supermicro Atoms with 2 Intel NICs onboard and
with a dual-port Intel NIC added. I would be also interested in
suggested list of settings for Intel NICs.
_______________________________________________
List mailing list
http://lists.pfsense.org/mailman/listinfo/list

_______________________________________________
List mailing list
http://lists.pfsense.org/mailman/listinfo/list

[Jim Thompson]

Post by Michael Schuh
i wouldn't only rely on the manufacturer but on the chip type; just saying

If by 'chip' you mean 'controller', I agree.

If by 'chip' you mean the actual flash (memory), then… you're likely mistaken. Intel and Micron are the same thing. (Micron is a second source for Intel flash.)

Other manufacturers (Samsung, etc) also make quality flash parts. I suppose there could be some seconds coming out of China, but if you buy the bottom of the price curve, you deserve what you get. Many people who complain about SSD reliability have either mis-used the technology, (e.g. write amplification rears it's ugly head) or have purchased the cheapest SSD they can find, and then complain when the the part fails.

The upthread advice about Intel SSDs is sound. Now that the Sandforce controller debacle is over, Crucial (who are really a rebrand of Micron (see above)) and Samsung also make good, reliable SSDs.

As a none-too-subtle hint: there are reasons why Netgate has, to date, not shipped SSD (or SSD-like) technology in our pfSense-powered appliances. It's not that we didn't know how, but rather the difference between "product" and "technology demonstration". If you're only concerned with making one, or a dozen, for your own use, the effects of your decision are limited. When you're making 1,000s of units per year, the weight of the decisions caries real monetary consequences.

Also note that phk was discussing flash parts a lot more like 'Compact Flash' or USB flash than SSDs in that document, while this thread has been about using SSDs. Apples != Oranges (Just sayin').

Jim

[Michael Schuh]

Post by Michael Schuh
i wouldn't only rely on the manufacturer but on the chip type; just saying
If by 'chip' you mean 'controller', I agree.
If by 'chip' you mean the actual flash (memory), then… you're likely
mistaken. Intel and Micron are the same thing. (Micron is a second
source for Intel flash.)

i mean SLC and MLC Flash-Memory-Chips; regardless which manufacturer. in
first place.

p.e.
Intel actually sells MLC instead of SLC ( iirc they had a series with SLC
but they are to expensive, not sure if they sell those further )
Intel SSD (actual series afaik MLC) compensate the different endurance with
more memory-chips and the controller software that round-robins
the writings over the entire disk except a reserved space for dying cells.

And yes there are manufacturers with much cleaner production and higer
quality of the memory-chips.

Post by Michael Schuh
Other manufacturers (Samsung, etc) also make quality flash parts. I
suppose there could be some seconds coming out of China, but if you buy the
bottom of the price curve, you deserve what you get. Many people who
complain about SSD reliability have either mis-used the technology, (e.g.
write amplification rears it's ugly head) or have purchased the cheapest
SSD they can find, and then complain when the the part fails.
The upthread advice about Intel SSDs is sound. Now that the Sandforce
controller debacle is over, Crucial (who are really a rebrand of Micron
(see above)) and Samsung also make good, reliable SSDs.
As a none-too-subtle hint: there are reasons why Netgate has, to date, not
shipped SSD (or SSD-like) technology in our pfSense-powered appliances.
It's not that we didn't know how, but rather the difference between
"product" and "technology demonstration". If you're only concerned with
making one, or a dozen, for your own use, the effects of your decision are
limited. When you're making 1,000s of units per year, the weight of the
decisions caries real monetary consequences.
Also note that phk was discussing flash parts a lot more like 'Compact
Flash' or USB flash than SSDs in that document, while this thread has been
about using SSDs. Apples != Oranges (Just sayin').
Jim
_______________________________________________
List mailing list
http://lists.pfsense.org/mailman/listinfo/list

[Jim Thompson]

Intel actually sells MLC instead of SLC ( iirc they had a series with SLC but they are to expensive, not sure if they sell those further )

They do. As you note, they are more expensive per bit than MLC.

Intel SSD (actual series afaik MLC) compensate the different endurance with more memory-chips and the controller software that round-robins
the writings over the entire disk except a reserved space for dying cells.

Same as it ever was. Wear-leveling.

And yes there are manufacturers with much cleaner production and higer quality of the memory-chips.

Did I not say, "Intel, Crucial/Micron, Samsung"?

Jim

[Michael Schuh]

Post by Michael Schuh
Intel actually sells MLC instead of SLC ( iirc they had a series with SLC
but they are to expensive, not sure if they sell those further )
They do. As you note, they are more expensive per bit than MLC.

The last thing i heard of, was that they now use HET MLC instead of SLC. So
all actual series uses those newer MLC-Chips
http://www.tomshardware.de/ssd-710-enterprise-x25-e-MLC-HET,testberichte-240941.html
sorry, i didn't found it in english. Shortly it says: SLC is out HET-MLC
(only) is in.

Post by Michael Schuh
Intel SSD (actual series afaik MLC) compensate the different endurance
with more memory-chips and the controller software that round-robins
the writings over the entire disk except a reserved space for dying cells.
Same as it ever was. Wear-leveling.

Yup, whats the point now here?
Not everyone knows about this technique, so i mentioned it explicitly.
i guess some others, with may be less experience, are also reading our
messages.

Post by Michael Schuh
And yes there are manufacturers with much cleaner production and higer
quality of the memory-chips.
Did I not say, "Intel, Crucial/Micron, Samsung"?

if i remember correctly YES. did i said anything different? o_O
my statement was a confirmation and slightly explanation how it comes to
those quality differences. ;-)

it lets you guess also how good the quality of other products from the same
company is.

[Eugen Leitl]

Post by Jim Thompson

And yes there are manufacturers with much cleaner production and higer quality of the memory-chips.

Did I not say, "Intel, Crucial/Micron, Samsung"?

I still don't understand why you're trusting HDDs more than
SSDs. There are edsels and lemons both among vendors and
particular models.

The only protection is diversity, in-house QA with pre-aging,
and buying older, known good models and making sure you're not
getting slipped in a different one but one from the same lot.

Of course it might be different if you're only shipping 1 k
units yourself.

I'm sufficiently convinced by prosumer Intel and Samsung to
put them into DB systems (on RAID 10). They're immune to
vibration and typically tolerate higher temperatures than
HDDs. I've only lost one 1st gen Intel SSD from maybe 20-30
total in circulation.

[Eugen Leitl]

Post by Aaron C. de Bruyn
Just a note of personal experience. I've deployed ~20 pfSense firewalls
that had SSDs (both cheap and rated 'good' from Newegg) over the past 2

I try to stick to Intel and lately Samsung SSDs.

Post by Aaron C. de Bruyn
years. I am not convinced SSDs are more reliable. Nearly every one has
had an SSD die or become corrupt. We switched them all to USB sticks and
haven't had any more issues. Plus it's easier for us to ship a replacement
USB stick to the client and have them plug it in than to have them pop open
the case and replace the drive.
Maybe we've just had bad luck with SSDs, but I'm not convinced they are
ready.

I think my anecdote counters your anecdote.

[Dave Warren]

Post by Eugen Leitl

Post by Aaron C. de Bruyn
Just a note of personal experience. I've deployed ~20 pfSense firewalls
that had SSDs (both cheap and rated 'good' from Newegg) over the past 2

I try to stick to Intel and lately Samsung SSDs.

Post by Aaron C. de Bruyn
years. I am not convinced SSDs are more reliable. Nearly every one has
had an SSD die or become corrupt. We switched them all to USB sticks and
haven't had any more issues. Plus it's easier for us to ship a replacement
USB stick to the client and have them plug it in than to have them pop open
the case and replace the drive.
Maybe we've just had bad luck with SSDs, but I'm not convinced they are
ready.

I think my anecdote counters your anecdote.

My anecdote addition: I've had a couple cheapo SSDs fail in the worst
possible way: Silently. Data was being written incorrectly without
throwing errors, the errors were only noticed at the application level,
when data was read back and was found to be corrupt.

I wiped the drive with the manufacturer recommended tool and tried
again, filling the drive roughly 50% with disk images from our server,
then reading them back and confirming that we were getting bad data
without the hardware level throwing an error.

At that point I switched to Intel drives based on their reputation for
stability and reliability, and I haven't had a single issue with any
Intel SSD yet. I've got them in servers, workstations and laptops,
ranging from 80GB-240GB.

[Odhiambo Washington]

@Aaron,

Which brand of USB sticks are these you use? I've tried working with
Transcend and found the performance awful. I'll appreciate your
recommendation on USB sticks.

Post by Aaron C. de Bruyn
Just a note of personal experience. I've deployed ~20 pfSense firewalls
that had SSDs (both cheap and rated 'good' from Newegg) over the past 2
years. I am not convinced SSDs are more reliable. Nearly every one has
had an SSD die or become corrupt. We switched them all to USB sticks and
haven't had any more issues. Plus it's easier for us to ship a replacement
USB stick to the client and have them plug it in than to have them pop open
the case and replace the drive.
Maybe we've just had bad luck with SSDs, but I'm not convinced they are
ready.
-A

Post by Eugen Leitl

Post by Chris Bagnall
Which brings me to the question: the last time I performed a pfSense
'full' install (i.e. not embedded) was several years, and many
versions ago. What's the best practice when using an SSD? Use the
CD-based installer to do a 'full' install, or continue to use the
embedded NanoBSD image?

Modern SSDs are at least as reliable as HDs. I've used SSDs
with pfSense for years (including IDE DoMs) with full install
and never had a failure yet.

Post by Chris Bagnall
As an aside, there are several options on the "Advanced" tab
- Disable hardware checksum offload
- Disable hardware TCP segmentation offload
- Disable hardware large receive offload
Has anyone done any tests / is there a list maintained anywhere with
details of which NICs are "problematic" with these, and hence should
be disabled? The motherboard I'm using is a mix of Intel and Realtek
gigabit NICs (em and re respectively).

I've used Supermicro Atoms with 2 Intel NICs onboard and
with a dual-port Intel NIC added. I would be also interested in
suggested list of settings for Intel NICs.
_______________________________________________
List mailing list
http://lists.pfsense.org/mailman/listinfo/list

_______________________________________________
List mailing list
http://lists.pfsense.org/mailman/listinfo/list

[Aaron C. de Bruyn]

Verbatim "Tough-'n'-Tiny" flash drives. 2 GB and 4 GB.

http://www.newegg.com/Product/Product.aspx?Item=9SIA0SF0BP6305
http://www.newegg.com/Product/Product.aspx?Item=9SIA0SF0BP6306

Most of the ones we have in production are under 1 year old, but we had a
lot of SSDs fail before the 1-year mark.

I didn't really pay attention to the speed, but I write an image to the 2
GB drive in about 8 minutes. (Not a scientific number!)

-A

Post by Odhiambo Washington
@Aaron,
Which brand of USB sticks are these you use? I've tried working with
Transcend and found the performance awful. I'll appreciate your
recommendation on USB sticks.

Post by Aaron C. de Bruyn
Just a note of personal experience. I've deployed ~20 pfSense firewalls
that had SSDs (both cheap and rated 'good' from Newegg) over the past 2
years. I am not convinced SSDs are more reliable. Nearly every one has
had an SSD die or become corrupt. We switched them all to USB sticks and
haven't had any more issues. Plus it's easier for us to ship a replacement
USB stick to the client and have them plug it in than to have them pop open
the case and replace the drive.
Maybe we've just had bad luck with SSDs, but I'm not convinced they are
ready.
-A

Post by Eugen Leitl

Post by Chris Bagnall
Which brings me to the question: the last time I performed a pfSense
'full' install (i.e. not embedded) was several years, and many
versions ago. What's the best practice when using an SSD? Use the
CD-based installer to do a 'full' install, or continue to use the
embedded NanoBSD image?

Modern SSDs are at least as reliable as HDs. I've used SSDs
with pfSense for years (including IDE DoMs) with full install
and never had a failure yet.

Post by Chris Bagnall
As an aside, there are several options on the "Advanced" tab
- Disable hardware checksum offload
- Disable hardware TCP segmentation offload
- Disable hardware large receive offload
Has anyone done any tests / is there a list maintained anywhere with
details of which NICs are "problematic" with these, and hence should
be disabled? The motherboard I'm using is a mix of Intel and Realtek
gigabit NICs (em and re respectively).

I've used Supermicro Atoms with 2 Intel NICs onboard and
with a dual-port Intel NIC added. I would be also interested in
suggested list of settings for Intel NICs.
_______________________________________________
List mailing list
http://lists.pfsense.org/mailman/listinfo/list

_______________________________________________
List mailing list
http://lists.pfsense.org/mailman/listinfo/list

--
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
[phone]/[phone]
"I can't hear you -- I'm using the scrambler."
_______________________________________________
List mailing list
http://lists.pfsense.org/mailman/listinfo/list

[Chris Bagnall]

Post by Odhiambo Washington
Which brand of USB sticks are these you use? I've tried working with
Transcend and found the performance awful. I'll appreciate your
recommendation on USB sticks.

On this point, I've recently ordered some of these to try:
http://www.amazon.co.uk/gp/product/B007XFW2RO/ref=oh_details_o00_s02_i01?ie=UTF8&psc=1
and
http://www.amazon.co.uk/gp/product/B005FYNSZA/ref=oh_details_o00_s00_i00?ie=UTF8&psc=1

I'm that one or other of those will offer a good compromise between
cost, ease of installation/upgrade/replacement, performance, and they're
sufficiently small that they'd be virtually impossible to accidentally
"knock out" once installed.

Kind regards,

Chris

[Adam Thompson]

YMMV. I have Intel SSDs in several places, no problems after several years - including heavy usage in my laptop.
Imation SSDs all died prematurely, no exceptions.
Kingston varies depending on who the OEM of the week is.
Other brands are inconsistent and not predictable (the ones I've tried, that is).

Vendor QC makes a * huge * difference

[Diego Barrios]

Can you please provide us with a link to this Atom board you are using?


I`m at the same point with Alix hardware, very good and reliable hardware, but we need more NICs and firepower in small factor boards.


Seko


----- Original Message -----

From: "Chris Bagnall" <***@lists.minotaur.cc>
To: "pfSense support and discussion" <***@lists.pfsense.org>
Sent: Friday, June 7, 2013 8:40:34 PM
Subject: [pfSense] Best practice for SSD installs

Greetings list,

I've used pfSense embedded for many years on ALIX boards.

However, given the difficulty of getting those boards with 4 NICs, or
more than 256MB RAM, I've recently been exprimenting with an Atom-based
motherboard instead. Unfortunately, but unsurprisingly, the Atom board
in question doesn't have a Compact Flash slot, so my usual approach of
flashing the pfSense embedded image to a card isn't an option here.

The board supports CFast (which is hideously expensive in the UK) and/or
a standard 2.5" SATA device. Given a standard 2.5" 32GB SSD is
considerably less expensive than even a 4GB CFast card, I suspect I'll
be using SSDs for future installs.

Which brings me to the question: the last time I performed a pfSense
'full' install (i.e. not embedded) was several years, and many versions
ago. What's the best practice when using an SSD? Use the CD-based
installer to do a 'full' install, or continue to use the embedded
NanoBSD image?

One other thing I thought I might try is using an USB flash device. I
notice from the snapshots there's an image available for these devices,
but I can't seem to find much by the way of documentation online about
the benefits/pitfalls of this approach.

As an aside, there are several options on the "Advanced" tab relating to
NIC performance options:
- Disable hardware checksum offload
- Disable hardware TCP segmentation offload
- Disable hardware large receive offload
Has anyone done any tests / is there a list maintained anywhere with
details of which NICs are "problematic" with these, and hence should be
disabled? The motherboard I'm using is a mix of Intel and Realtek
gigabit NICs (em and re respectively).

Any suggestions gratefully received.

Kind regards,

Chris

[Vassilis V.]

Post by Diego Barrios
Can you please provide us with a link to this Atom board you are using?
I`m at the same point with Alix hardware, very good and reliable
hardware, but we need more NICs and firepower in small factor boards.
Seko

Atom boards I have had good experience with is the Supermicro X7SPE-H(F)
and the X7SPA-H(F) both have a single slot PCIe that can carry a quad
NIC so you would have a total of 6 NICs (Two are onboard). The (F)
models have IPMI onboard which is brilliant. I think they also have a
D525 series out by now.

The only issue I have had with firewalls on those systems is.. failing
SSDs :)


Vassilis

[Chris Bagnall]

Been away a few days, so only just catching up with this thread...

Post by Diego Barrios
Can you please provide us with a link to this Atom board you are using?

I am testing one of these in our office:
http://linitx.com/product/jetway-jbc362f362600-intel-atom-16ghz-barebone-system/13607
2 Realtek gigabit ports, both feeding into HP managed switches to VLANs
for the 3 WAN interfaces.

The other one I'm testing is this:
http://linitx.com/product/jetway-jnc9c550lf-15ghz-dual-core-atom-n550-mainboard-dual-gigabit-lan/12930
with the 3x Intel NIC daughterboard:
http://linitx.com/product/jetway-triple-intel-giga-lan-addon-board/12576

Kind regards,

Chris

[Jim Pingle]

Post by Chris Bagnall
Which brings me to the question: the last time I performed a pfSense
'full' install (i.e. not embedded) was several years, and many versions
ago. What's the best practice when using an SSD? Use the CD-based
installer to do a 'full' install, or continue to use the embedded
NanoBSD image?

The SSD hardware debates have been pretty well-covered by this thread,
I'll leave that aside since I don't have much to add (Except this: Avoid
Kingston at all costs)

As far as pfSense best practices, there are a few things you can do to
make sure the life of the SSD is preserved as long as possible. All of
this applies only to a full install, since NanoBSD already does as much
as possible to keep flash media from being written.

More than anything else, upgrade to 2.0.3 or 2.1. Before then, the
apinger (gateway) status was written to the disk once per second, which
adds up. Now it's done in a RAM disk all the time.

Limit your use of packages, especially packages that would cause a lot
of disk activity, such as squid. If you must use squid for access
control, disable caching and logging. Avoid packages that would
constantly write data, such as bandwidth graphing packages.

On 2.1 especially, you can also enable /var and /tmp to be RAM disks,
further lessening the disk writes. This keeps the log files, RRD graphs,
and other temporary files from being written to the disk constantly.
Since you'd be running a full install, be sure to allocate a decent
amount of space to then, running out can be quite bad. The graphs are
backed up on shutdown and restored at bootup when setup this way, as
they are on NanoBSD.

Some SSDs are rated for gzillions of writes, so the above are not as
much of a concern on quality SSDs (e.g. Intel), but I'd still play it
safe if it were mine.

One thing I haven't seen mentioned on the hardware side is to get a
drive that has a built-in capacitor that will flush the drive's cache to
disk on power loss. This can prevent drive corruption regardless of OS.
They tend to cost more, but they are a good investment especially if the
device is remote in a location that has dodgy power.

Jim

[Dave Warren]

Post by Jim Pingle
On 2.1 especially, you can also enable /var and /tmp to be RAM disks,
further lessening the disk writes. This keeps the log files, RRD graphs,
and other temporary files from being written to the disk constantly.
Since you'd be running a full install, be sure to allocate a decent
amount of space to then, running out can be quite bad. The graphs are
backed up on shutdown and restored at bootup when setup this way, as
they are on NanoBSD.

I'm really interested in this feature. I don't run pfSense on a SSD yet,
but it'll come sooner rather than later, we're replacing all of our
servers with SSD drives for boot and primary services (although not all
data)

What is "a decent amount of space"? Is 1GB enough or would more be safer?

Post by Jim Pingle
Some SSDs are rated for gzillions of writes, so the above are not as
much of a concern on quality SSDs (e.g. Intel), but I'd still play it
safe if it were mine.

The other factor, which works heavily in favour of a SSD on pfSense, is
that well written SSD controllers will distribute the writes across the
entire drive. pfSense will take less than 1GB of a 64GB drive, which
gives you 64GB worth of scratch space the SSD can play with.

Even for something that writes data constantly like squid, I'd be very
tempted to use a SSD due to the raw performance, although I'd probably
use a second SSD, one for boot and one for the squid cache, intending to
replace the squid-cache-SSD as needed.

SSDs are cheap enough that they're effectively disposable now (and I'm
not talking about the low-end cheapest-SSD-you-can-buy either), at least
for applications where there is a real and noticeable performance
difference.

[Jim Pingle]

Post by Dave Warren

Post by Jim Pingle
On 2.1 especially, you can also enable /var and /tmp to be RAM disks,
further lessening the disk writes. This keeps the log files, RRD graphs,
and other temporary files from being written to the disk constantly.
Since you'd be running a full install, be sure to allocate a decent
amount of space to then, running out can be quite bad. The graphs are
backed up on shutdown and restored at bootup when setup this way, as
they are on NanoBSD.

I'm really interested in this feature. I don't run pfSense on a SSD yet,
but it'll come sooner rather than later, we're replacing all of our
servers with SSD drives for boot and primary services (although not all
data)
What is "a decent amount of space"? Is 1GB enough or would more be safer?

A few hundred MB, minimum, if you have the RAM for it. NanoBSD by
default uses 40-60MB or so for those, I don't think I'd run with less
than 256MB or 512MB each. It may not always need that much, but the
first time you hit the limit when a package tries to install, you'll
regret not having used more space. :-)

[Jim Thompson]

The other factor, which works heavily in favour of a SSD on pfSense, is that well written SSD controllers will distribute the writes across the entire drive.

Well, yet unused sectors, until a TRIM command is issued, but yeah.

It's called "wear leveling".

[Dave Warren]

Post by Jim Thompson

The other factor, which works heavily in favour of a SSD on pfSense, is that well written SSD controllers will distribute the writes across the entire drive.

Well, yet unused sectors, until a TRIM command is issued, but yeah.
It's called "wear leveling".

Yes. But the point is that wear leveling works especially well in
environments where you leave a vast majority of the original drive empty
and unallocated.

Or, put another way, purely from a cell-dying-of-excessive-rewrites
point of view, if you have a 80GB partition, the 80GB drive will die
well before a 240GB drive that is otherwise identical, but has more
flash chips.

You can either use TRIM, or just use a small partition at the OS level
to achieve positive results; a small partition forces the OS to rewrite
existing data rather than writing to new, previously unallocated
sectors. When the SSD's firmware sees sectors being modified, it will
re-write the sector to an unused/unallocated sector on disk and garbage
collection can free up the previous sector on it's own pace.

[Eugen Leitl]

Post by Jim Pingle
One thing I haven't seen mentioned on the hardware side is to get a
drive that has a built-in capacitor that will flush the drive's cache to
disk on power loss. This can prevent drive corruption regardless of OS.
They tend to cost more, but they are a good investment especially if the
device is remote in a location that has dodgy power.

Crucial M500 is one of the first consumer drive with a buffer capacitor.
On the negative side it's write endurance is not particularly impressive.

[Seth Mos]

Post by Eugen Leitl

Post by Jim Pingle
One thing I haven't seen mentioned on the hardware side is to get a
drive that has a built-in capacitor that will flush the drive's cache to
disk on power loss. This can prevent drive corruption regardless of OS.
They tend to cost more, but they are a good investment especially if the
device is remote in a location that has dodgy power.

Crucial M500 is one of the first consumer drive with a buffer capacitor.
On the negative side it's write endurance is not particularly impressive.

Nope, the Intel 320 series had that one already, sadly, they skipped it
on the 330 and up.

We have about 300 Intel 320 series 80GB in the field, and only 1 made
itself a 8MB drive. So that wasn't so bad.

We have 12 320 Series 300GB in a raid 6 that's been humming well for
over a year now.

Our Dell optiplex and latitude ship with Samsung SSD drives that have
had no failures either (120), we refurbished another 70 with 120GB Intel
320 series, none of those failed either.

In the meantime we did lose a bunch of Seagate Barracuda ES drives, and
even managed to nail a Raid10 of just 10 drives. That's a impressive
feat to beat the odds.

No spinning rust is going to be purchased here anymore.

Cheers
Seth.

[Adam Thompson]

I've used the Sandisk one before. They're slow on reads and extremely slow on writes. On the other hand, none of them have died yet.
FWIW, I've had a Corsair equivalent plugged into my car stereo for about a year in temps from -40C to +50C and it still works fine.
-Adam

Post by Chris Bagnall

Post by Odhiambo Washington
Which brand of USB sticks are these you use? I've tried working with
Transcend and found the performance awful. I'll appreciate your
recommendation on USB sticks.

http://www.amazon.co.uk/gp/product/B007XFW2RO/ref=oh_details_o00_s02_i01?ie=UTF8&psc=1
and
http://www.amazon.co.uk/gp/product/B005FYNSZA/ref=oh_details_o00_s00_i00?ie=UTF8&psc=1
I'm that one or other of those will offer a good compromise between
cost, ease of installation/upgrade/replacement, performance, and they're
sufficiently small that they'd be virtually impossible to accidentally
"knock out" once installed.
Kind regards,
Chris
--
This email is made from 100% recycled electrons
_______________________________________________
List mailing list
http://lists.pfsense.org/mailman/listinfo/list

[Chris Bagnall]

Post by Adam Thompson
I've used the Sandisk one before. They're slow on reads and extremely slow on writes. On the other hand, none of them have died yet.

Noted - thanks for the feedback. I'll post a few speed tests to the list
once they've arrived and I've had chance to compare against the Integral
(and I think I have a similar, but pricier, Lexar unit here somewhere -
probably plugged into the back of the TV :-) )

I wonder if slow IO would actually matter much on a pfSense box, where
most things are (or at least could be) done from RAM. The only real
writes are logs, RRDs and config changes - and the logs could quite
easily go to ramdisk instead.

Kind regards,

Chris