[pfSense] 2.2.1 Site-to-Site IPsec VPN Connection Instability

[Bryan D.]

We've had a pfSense-to-pfSense "always on" IPsec VPN connecting 2 offices since 2008 (pfSense 1.2 IIRC) and it's:
- been ultra reliable (if VPN is down, suspect ISP issue or pfSense box failure)
- it's been quick to connect (about 1 second, almost unnoticeable)
- it's worked across numerous upgrades without issue (nice!)

Beginning with pfSense v2, we added multiple P2s at each end (still same reliability, etc.).

One of the offices has had its hardware updated and its pfSense updated to 2.2 then 2.2.1 (after testing to see whether we seemed to be affected by the "multiple P2 issue" noted in the upgrade page -- we're OK on that one). This connection has continued to work with the same characteristics as before. The 2.2.1 system is 64-bit and the other end is v2.1.5 32-bit

We recently added a second site-to-site IPsec VPN, essentially the same as the existing one except both sides are pfSense v2.2.1 (but other end is 32-bit) and stronger algorithms are being used and P1 is set to v2 (supposedly avoiding any "multiple P2" issues).

The new pfSense (v2.2.1 at both ends) "always on" (not!) IPsec VPN is:
- v-e-r-y s-l-o-w to connect: e.g. pinging when connection is down yields: once a connection after 12 seconds, once a connection after 22 seconds and dozens of connections after > 2.25 minutes
- completely unable to "stay connected": both sides have DPD enabled (5 sec./3 retries) and both sides can be initiators and both sides have 1 P2 set to ping the pfSense at the other end
- pressing the "connect button" on pfSense's IPsec status page yields an "instant connection" but there won't be any P2 traffic coming back for a while, which seems to consistently be the connection-delay issue

If one of the ends has regular (almost constant) traffic, the VPN stays connected. In testing, I've had one non-pfSense system pinging the pfSense at the other end and the VPN stays connected.

If VPN traffic pauses for a short time (ten minutes?), the connection is dropped. While that is not what's expected, given the config, it wouldn't be bad _if_ the connection was quickly re-established with traffic, but it's not.

I'm providing this information because of the discussions about issues with multiple P2s and the idea that they're solved by using v2 P1 at both ends. It's quite possible that:
- there are issues with multiple P2s even with v2 P1
- the DPD stuff isn't working
- the P2 "automatically ping host" stuff isn't working ... but pinging a host via a non-pfSense system does keep the connection alive

I'm willing to run some tests if someone wants to tell me what they want done. For about a week, the new v2.2.1 site-to-site VPN won't really be used, I can do almost anything that doesn't cause the other side to go dead (or I'd need to make a trip to the other site and that may not happen very quickly).

[mayak]

Post by Bryan D.
- been ultra reliable (if VPN is down, suspect ISP issue or pfSense box failure)
- it's been quick to connect (about 1 second, almost unnoticeable)
- it's worked across numerous upgrades without issue (nice!)
Beginning with pfSense v2, we added multiple P2s at each end (still same reliability, etc.).
One of the offices has had its hardware updated and its pfSense updated to 2.2 then 2.2.1 (after testing to see whether we seemed to be affected by the "multiple P2 issue" noted in the upgrade page -- we're OK on that one). This connection has continued to work with the same characteristics as before. The 2.2.1 system is 64-bit and the other end is v2.1.5 32-bit
We recently added a second site-to-site IPsec VPN, essentially the same as the existing one except both sides are pfSense v2.2.1 (but other end is 32-bit) and stronger algorithms are being used and P1 is set to v2 (supposedly avoiding any "multiple P2" issues).

<snip>

i have to say that i am also experiencing this. i'm in the process of installing smokeping to prove connectivity is good between the public ip endpoints between various vpns.

will report back with those results.

thanks

m

[mayak]

Post by mayak

Post by Bryan D.
- been ultra reliable (if VPN is down, suspect ISP issue or pfSense box failure)
- it's been quick to connect (about 1 second, almost unnoticeable)
- it's worked across numerous upgrades without issue (nice!)
Beginning with pfSense v2, we added multiple P2s at each end (still same reliability, etc.).
One of the offices has had its hardware updated and its pfSense updated to 2.2 then 2.2.1 (after testing to see whether we seemed to be affected by the "multiple P2 issue" noted in the upgrade page -- we're OK on that one). This connection has continued to work with the same characteristics as before. The 2.2.1 system is 64-bit and the other end is v2.1.5 32-bit
We recently added a second site-to-site IPsec VPN, essentially the same as the existing one except both sides are pfSense v2.2.1 (but other end is 32-bit) and stronger algorithms are being used and P1 is set to v2 (supposedly avoiding any "multiple P2" issues).

<snip>
i have to say that i am also experiencing this. i'm in the process of installing smokeping to prove connectivity is good between the public ip endpoints between various vpns.
will report back with those results.
thanks
m
__

it is happening -- three times since last post ... anyone else noticing vpn outages?

thanks

m

[Jeremy Bennett]

Yes. I have 5 sites (of various 2.0+ pfsense) connected via IPSec.
Historically, this setup was the definition of stable. Since adding a 2.2
box it has been flaky as all get out.

I'm in the process of upgrading each location to 2.2.1. Hoping that will be
the easy fix, if not, will have to start chasing it.

Post by mayak

Post by mayak

Post by Bryan D.
We've had a pfSense-to-pfSense "always on" IPsec VPN connecting 2
- been ultra reliable (if VPN is down, suspect ISP issue or pfSense box failure)
- it's been quick to connect (about 1 second, almost unnoticeable)
- it's worked across numerous upgrades without issue (nice!)
Beginning with pfSense v2, we added multiple P2s at each end (still same
reliability, etc.).
One of the offices has had its hardware updated and its pfSense updated
to 2.2 then 2.2.1 (after testing to see whether we seemed to be affected by
the "multiple P2 issue" noted in the upgrade page -- we're OK on that
one). This connection has continued to work with the same characteristics
as before. The 2.2.1 system is 64-bit and the other end is v2.1.5 32-bit
We recently added a second site-to-site IPsec VPN, essentially the same
as the existing one except both sides are pfSense v2.2.1 (but other end is
32-bit) and stronger algorithms are being used and P1 is set to v2
(supposedly avoiding any "multiple P2" issues).

<snip>
i have to say that i am also experiencing this. i'm in the process of
installing smokeping to prove connectivity is good between the public ip
endpoints between various vpns.
will report back with those results.
thanks
m
__

it is happening -- three times since last post ... anyone else noticing vpn outages?
thanks
m
_______________________________________________
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

[Justin Edmands]

I am having issues as well. 2.0.0 <--> 2.0.3 works fine. Upgraded to 2.2.1
and the connection always fails within 24 hours.

Please let me know how the 2.2.1 x 5 setup works.

Post by Jeremy Bennett
Yes. I have 5 sites (of various 2.0+ pfsense) connected via IPSec.
Historically, this setup was the definition of stable. Since adding a 2.2
box it has been flaky as all get out.
I'm in the process of upgrading each location to 2.2.1. Hoping that will
be the easy fix, if not, will have to start chasing it.

Post by mayak

Post by mayak

Post by Bryan D.
We've had a pfSense-to-pfSense "always on" IPsec VPN connecting 2
- been ultra reliable (if VPN is down, suspect ISP issue or pfSense box failure)
- it's been quick to connect (about 1 second, almost unnoticeable)
- it's worked across numerous upgrades without issue (nice!)
Beginning with pfSense v2, we added multiple P2s at each end (still
same reliability, etc.).
One of the offices has had its hardware updated and its pfSense updated
to 2.2 then 2.2.1 (after testing to see whether we seemed to be affected by
the "multiple P2 issue" noted in the upgrade page -- we're OK on that
one). This connection has continued to work with the same characteristics
as before. The 2.2.1 system is 64-bit and the other end is v2.1.5 32-bit
We recently added a second site-to-site IPsec VPN, essentially the same
as the existing one except both sides are pfSense v2.2.1 (but other end is
32-bit) and stronger algorithms are being used and P1 is set to v2
(supposedly avoiding any "multiple P2" issues).

<snip>
i have to say that i am also experiencing this. i'm in the process of
installing smokeping to prove connectivity is good between the public ip
endpoints between various vpns.
will report back with those results.
thanks
m
__

it is happening -- three times since last post ... anyone else noticing vpn outages?
thanks
m
_______________________________________________
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

_______________________________________________
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

[compdoc]

My ipsec connection goes down at least once a day now, maybe more. It’s a 2.2.1 to remote 2.1.5 connection.



After I open Status: IPsec and turn the connection off/on, all is well. Only have to do that with the 2.2.1 server - doesn't seem to affect the 2.1.5 side.







From: List [mailto:list-***@lists.pfsense.org] On Behalf Of Justin Edmands
Sent: Monday, March 23, 2015 3:00 PM
To: pfSense Support and Discussion Mailing List
Subject: Re: [pfSense] 2.2.1 Site-to-Site IPsec VPN Connection Instability



I am having issues as well. 2.0.0 <--> 2.0.3 works fine. Upgraded to 2.2.1 and the connection always fails within 24 hours.

Please let me know how the 2.2.1 x 5 setup works.

[Christopher CUSE]

Post by mayak

Post by Bryan D.
- been ultra reliable (if VPN is down, suspect ISP issue or pfSense box failure)
- it's been quick to connect (about 1 second, almost unnoticeable)
- it's worked across numerous upgrades without issue (nice!)
Beginning with pfSense v2, we added multiple P2s at each end (still same reliability, etc.).
One of the offices has had its hardware updated and its pfSense updated to 2.2 then 2.2.1 (after testing to see whether we seemed to be affected by the "multiple P2 issue" noted in the upgrade page -- we're OK on that one). This connection has continued to work with the same characteristics as before. The 2.2.1 system is 64-bit and the other end is v2.1.5 32-bit
We recently added a second site-to-site IPsec VPN, essentially the same as the existing one except both sides are pfSense v2.2.1 (but other end is 32-bit) and stronger algorithms are being used and P1 is set to v2 (supposedly avoiding any "multiple P2" issues).

<snip>
i have to say that i am also experiencing this. i'm in the process of installing smokeping to prove connectivity is good between the public ip endpoints between various vpns.
will report back with those results.
thanks
m

just got dropped again -- fourth time in last few hours -- something is definitely wrong.

upgraded all my pfsenses to 2.2.1 over the weekend.

m

[Vincent Hoffman-Kazlauskas]

Post by Christopher CUSE

Post by mayak

Post by Bryan D.
We've had a pfSense-to-pfSense "always on" IPsec VPN connecting 2
- been ultra reliable (if VPN is down, suspect ISP issue or pfSense box failure)
- it's been quick to connect (about 1 second, almost unnoticeable)
- it's worked across numerous upgrades without issue (nice!)
Beginning with pfSense v2, we added multiple P2s at each end (still
same reliability, etc.).
One of the offices has had its hardware updated and its pfSense
updated to 2.2 then 2.2.1 (after testing to see whether we seemed to
be affected by the "multiple P2 issue" noted in the upgrade page --
we're OK on that one). This connection has continued to work with
the same characteristics as before. The 2.2.1 system is 64-bit and
the other end is v2.1.5 32-bit
We recently added a second site-to-site IPsec VPN, essentially the
same as the existing one except both sides are pfSense v2.2.1 (but
other end is 32-bit) and stronger algorithms are being used and P1 is
set to v2 (supposedly avoiding any "multiple P2" issues).

<snip>
i have to say that i am also experiencing this. i'm in the process of
installing smokeping to prove connectivity is good between the public
ip endpoints between various vpns.
will report back with those results.
thanks
m

just got dropped again -- fourth time in last few hours -- something is definitely wrong.
upgraded all my pfsenses to 2.2.1 over the weekend.

We also seem to be having stability issues on our site to site vpn
(2.2.1 one end, 2.0.1 the other, plan was to upgrade soon) its only
stopping once every day or so afaik but it was prety rock solid before I
upgraded the end thats now 2.2.1, looking at adding some monitoring also
just to make sure of end to end connectivity before I investigate further.

Happy to supply more info if it'll help.

Vince

Post by Christopher CUSE
m
_______________________________________________
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

[Bryan D.]

Post by Christopher CUSE
just got dropped again -- fourth time in last few hours -- something is definitely wrong.
upgraded all my pfsenses to 2.2.1 over the weekend.

For me, the VPN drops in the absence of "end-to-end" traffic ... within minutes. The fact that both ends are config'd to ping and do DPD seems to be of no consequence. Our site-to-site VPNs have multiple P2s. As long as a connection exists, (in my limited testing) "activating" a new P2 seems to be "v2.1.5-reliable."

I set up a script (running on one of our severs) that pings and the connection has been up (with virtually no other traffic, because it's pre-production) for about 1.5 days. It dies within minutes without the pinging. The script did not work when run on the pfSense box, itself (though I really haven't thought it through and there could be a perfectly good reason why it wouldn't).


For anyone who's interested, here's the (simple) script:
---
#!/bin/sh
#set -x ## Uncomment to get a trace

# keep IPsec VPN tunnel(s) connected

#-------------------------------------------------------------------------------
# Run this script every minute via the following /etc/crontab entry
# (minus the first comment character):
#*/1 * * * * admin /usr/local/bin/keepAliveIPsec.sh & ## keep IPsec VPNs connected

# The space-separated list of hosts (IP or FQDN) that will be ping'd
HOSTS_TO_PING='172.24.24.1 172.24.28.1'

# Set the maximum number of seconds that a ping will wait for a response
PING_TIMEOUT='1'

# Set the interval, in seconds, between ping attempts to each group of hosts
PING_INTERVAL='3'

# NOTE that the total interval between pings for each host will be the
# PING_INTERVAL plus the sum of the response times for each host being ping'd --
# i.e., where the maximum response time is the PING_TIMEOUT and the minimum is
# the successful ping-response time (for each host being ping'd)
#-------------------------------------------------------------------------------

# Don't run if a keepAliveIPsec.sh process is already running
PROCS=`/bin/ps -ax -o pid,command`
OTHER_KEEPALIVE_PROCS=`\
echo "$PROCS" | /usr/bin/sed -e '/[ \t\/]keepAliveIPsec.sh/!d' \
-e '/^[ \t]*'"$$"'[ \t]/d'`
if test "$OTHER_KEEPALIVE_PROCS" != ""
then
#echo 'keepAliveIPsec.sh already running' # uncomment for testing
exit 1
fi

# Ping the required hosts, "forever"
while true
do
for HOST in $HOSTS_TO_PING
do
#/sbin/ping -c 1 -t "$PING_TIMEOUT" "$HOST" # uncomment for testing
/sbin/ping -c 1 -t "$PING_TIMEOUT" "$HOST" \
2>&1 >/dev/null # comment out for testing
done

#echo 'sleeping' # uncomment for testing
sleep "$PING_INTERVAL"
done

[Chris Buechler]

Post by Christopher CUSE

Post by mayak

Post by Bryan D.
We've had a pfSense-to-pfSense "always on" IPsec VPN connecting 2 offices
- been ultra reliable (if VPN is down, suspect ISP issue or pfSense box failure)
- it's been quick to connect (about 1 second, almost unnoticeable)
- it's worked across numerous upgrades without issue (nice!)
Beginning with pfSense v2, we added multiple P2s at each end (still same
reliability, etc.).
One of the offices has had its hardware updated and its pfSense updated
to 2.2 then 2.2.1 (after testing to see whether we seemed to be affected by
the "multiple P2 issue" noted in the upgrade page -- we're OK on that one).
This connection has continued to work with the same characteristics as
before. The 2.2.1 system is 64-bit and the other end is v2.1.5 32-bit
We recently added a second site-to-site IPsec VPN, essentially the same
as the existing one except both sides are pfSense v2.2.1 (but other end is
32-bit) and stronger algorithms are being used and P1 is set to v2
(supposedly avoiding any "multiple P2" issues).

<snip>
i have to say that i am also experiencing this. i'm in the process of
installing smokeping to prove connectivity is good between the public ip
endpoints between various vpns.
will report back with those results.
thanks
m

just got dropped again -- fourth time in last few hours -- something is definitely wrong.
upgraded all my pfsenses to 2.2.1 over the weekend.

Go to System>Advanced, System Tunables, and add a new tunable there.
Name net.key.preferred_oldsa, value 0, then save and apply changes.
That have any impact on things?

[Bryan D.]

Post by Chris Buechler
Go to System>Advanced, System Tunables, and add a new tunable there.
Name net.key.preferred_oldsa, value 0, then save and apply changes.
That have any impact on things?

Executive summary: no.

Here's what I did:
- created/applied tunable at both ends
- turned off our server's "keep VPN alive" pinger
- on my end, pressed the "disconnect" button for the 2.2.1-to-2.2.1 VPN (in Status -> IPsec)
- Status -> IPsec "immediately" indicated that the 2.2.1-to-2.2.1 VPN connection was established (this is always the case ... it sort o' fibs)
- started a once-per-second ping from my system (to the pfSense box at the other end)
- the first 88 pings were unanswered
- the VPN was "really connected" at about 89 seconds
- killed the pinging at about 103 seconds (and typed this stuff up, up to here)
- ran the pinger, again ... 7 successes ... interrupted the ping process
- waited about 3 minutes (remember, right now there's no other traffic across this VPN)
- ran the ping, again ... connection down, 34 pings unanswered
- VPN connected at about 35 seconds
- let the ping run for another minute ... stayed connected
- interrupted ping process
- waited about 3 minutes
- ran the ping, again ... connection down, 143 pings unanswered
- VPN connected at about 144 seconds

So far, I'd say that this change had zero effect. This appears to be the same behavior as though the tunable wasn't there.

Next, I did the following:
- pinger still running, VPN still connected
- pressed the "Restart IPsec service" button (Status -> IPsec) on the pfSense at the other end
- one ping slightly longer response, but no pings lost
- pressed the "Restart IPsec service" button (Status -> IPsec) on the pfSense at the my end
- one ping with no response but connection stayed up -- interestingly, Status -> IPsec reported an "established" time that indicated the original connection duraion (i.e., seemingly indicating that the restart didn't create a new connection)
- killed the pinger as I typed this stuff
- about 6 minutes later, ran a ping and the connection was still alive
- interrupted the ping after 7 pings
- waited about 3 minutes
- ran the ping, again ... connection still alive!
- I stopped the ping
- I thought that looked quite promising so, as a more rigorous set of tests, I was going to restart the pfSense box at the other end and completely stop, then restart the IPsec service at my end ... I typed that info in as my next steps
- however, when I went to access the pfSense box at the other end, about a minute later, the VPN was down
- I fired off another ping ... 126 pings unanswered
- VPN connected at about 127 seconds

Next, I did the following:
- rebooted the pfSense box at the other end
- stopped the IPsec service at my end
- waited long enough to ensure that the pfSense box at the other end had ample time to reboot (please!)
- started the IPsec service at my end
- started a once-per-second ping from my system and attempted to access both the pfSense box at the other end of the 2.2.1-to-2.2.1 VPN and the one at the other end of the 2.2.1-to-2.1.5 VPN
- got the pfSense box at the other end of the 2.2.1-to-2.1.5 VPN with about a 1 second delay, as "always" -- yes, experience with the previous IPsec does create high expectations! #;-))
- 54 pings unanswered
- the 2.2.1-to-2.2.1 VPN connected after about 55 seconds
- typed this stuff up and ran another ping (about 3-4 min. later)
- connection still up ... stopped ping
- waited about 3 minutes
- ran ping ... 30 pings unanswered
- VPN connected at about 31 seconds

In spite of a moment's promise, looks like this change no net effect. [hey, had to get in at least 1 pun]

FYI, since I implemented the pinger script on the server, I've accessed the pfSense box at the other end about 20 times and the VPN has been up every time. Without the pinger, I could count on it always being disconnected and having to wait .5 to 1.5 minutes for it to connect, after attempting to send some traffic.

[Chris Buechler]

Post by Bryan D.

Post by Chris Buechler
Go to System>Advanced, System Tunables, and add a new tunable there.
Name net.key.preferred_oldsa, value 0, then save and apply changes.
That have any impact on things?

Executive summary: no.

Clearly something not right there, but that exhausted my guesses
without having IPsec logs from both sides. If you can go through the
same steps (more or less, any set of steps that replicates a problem
is fine though), keep note of the times along the way so it can be
correlated to the logs, and send the logs and steps with times, that
should help.

[f***@gmail.com]

Enviado desde mi smartphone BlackBerry 10.
  Mensaje original  
De: Chris Buechler
Enviado: viernes, 27 de marzo de 2015 15:47
Para: pfSense Support and Discussion Mailing List
Responder a: pfSense Support and Discussion Mailing List
Asunto: Re: [pfSense] 2.2.1 Site-to-Site IPsec VPN Connection Instability

Post by Bryan D.

Post by Chris Buechler
Go to System>Advanced, System Tunables, and add a new tunable there.
Name net.key.preferred_oldsa, value 0, then save and apply changes.
That have any impact on things?

Executive summary: no.

Clearly something not right there, but that exhausted my guesses
without having IPsec logs from both sides. If you can go through the
same steps (more or less, any set of steps that replicates a problem
is fine though), keep note of the times along the way so it can be
correlated to the logs, and send the logs and steps with times, that
should help.
_______________________________________________
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

[Bryan D.]

While testing the previously discussed "stalling connections" with v2.2.1 IPsec -- which still exist with v2.2.2 (expected as the release notes give no indication of a fix) -- I noticed (what I suspect is) a new bug (https://redmine.pfsense.org/issues/4640).

After updating from 2.2.1 to 2.2.2, in VPN -> IPsec -> Advanced Settings, the check-box setting for "Disable Cisco Extensions" now toggles whatever the setting was for "Auto-exclude LAN address" and the checkbox for "Auto-exclude LAN address" ignores any attempts to set it on it's own.

Question

The "Auto-exclude LAN address" has an explanatory line of
---
Exclude traffic from LAN subnet to LAN IP address from IPsec.
---
which can be interpreted multiple ways (which LAN, local/remote? and which IPsec, server/client? ... or both?).

Since there's nothing on the associated WiKi help page, could someone provide a more detailed explanation of what this setting does and/or should be used for (and the default setting)?

[Bryan D.]

Re: https://www.netgate.com/blog/dns-over-tls-with-pfsense.html
---
Applying the suggested "Custom Options" to the Unbound/DNS Resolver configuration in pfSense 2.2.6 does not work, with logs indicating that "forward-ssl-upstream" is invalid.

I tried various incantations using "server:<newline>ssl-upstream: yes" with and without "ssl-port: 853" and, although the unbound service would then run, a DNS/host query always indicated that no hosts were found.

Does anyone know a configuration that will work with pfSense 2.2.6?

[Steve Yates]

Wild guess, but did you try it in 2.4.x?

--

Steve Yates
ITS, Inc.

-----Original Message-----
From: List <list-***@lists.pfsense.org> On Behalf Of Bryan D.
Sent: Wednesday, April 4, 2018 8:01 PM
To: pfSense Support and Discussion Mailing List <***@lists.pfsense.org>
Subject: [pfSense] DNS over TLS config for pfSense 2.2.6

Re: https://www.netgate.com/blog/dns-over-tls-with-pfsense.html
---
Applying the suggested "Custom Options" to the Unbound/DNS Resolver configuration in pfSense 2.2.6 does not work, with logs indicating that "forward-ssl-upstream" is invalid.

I tried various incantations using "server:<newline>ssl-upstream: yes" with and without "ssl-port: 853" and, although the unbound service would then run, a DNS/host query always indicated that no hosts were found.

Does anyone know a configuration that will work with pfSense 2.2.6?

_______________________________________________
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

[Dave Warren]

I'm running 2.4.3-RELEASE (amd64). I can't get it working here either
after a couple hours of poking at it on and off, it now looks like this
is actually a Cloudflare issue:

https://community.cloudflare.com/t/1-1-1-1-was-working-but-not-anymore/15136/4

"Thanks for the report! This is going to be fixed in the next upgrade
that’s being rolled out.
There was an interop issue in the last upgrade with Unbound as it sends
the frame size and the actual DNS message in two separate packets
instead of both at once."

So it looks like the immediate solution is to revert to port 53 and wait
for Cloudflare. I can also confirm that ***@853 does work here which
re-enforces that this is a Cloudflare specific issue.

Sorry, mine was indeed on 2.4.X. The daemon appeared to start up but any queries returned no records.

Post by Steve Yates
Wild guess, but did you try it in 2.4.x?
--
Steve Yates
ITS, Inc.
-----Original Message-----
Sent: Wednesday, April 4, 2018 8:01 PM
Subject: [pfSense] DNS over TLS config for pfSense 2.2.6
Re: https://www.netgate.com/blog/dns-over-tls-with-pfsense.html
---
Applying the suggested "Custom Options" to the Unbound/DNS Resolver
configuration in pfSense 2.2.6 does not work, with logs indicating that
"forward-ssl-upstream" is invalid.
I tried various incantations using "server:<newline>ssl-upstream: yes"
with and without "ssl-port: 853" and, although the unbound service would
then run, a DNS/host query always indicated that no hosts were found.
Does anyone know a configuration that will work with pfSense 2.2.6?
_______________________________________________
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold
_______________________________________________
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

_______________________________________________
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

[Bryan D.]

-----

So it looks like the following config works on pfSense 2.2.6's unbound/DNS Resolver (so should work with 1.1.1.1 when Cloudflare gets things fixed):
server:
ssl-upstream: yes
ssl-port: 853
forward-zone:
name: "."
forward-addr: ***@853

Thanks!

[Dave Warren]

Post by Bryan D.
-----
ssl-upstream: yes
ssl-port: 853
name: "."

Cloudflare has pushed an update, and things seem to be working from
here. For those having issues, try again now?

[Bryan D.]

Cloudflare has pushed an update, and things seem to be working from here. For those having issues, try again now?

Thanks for the "heads up." Works for me, also (i.e., on pfSense 2.2.6 configured as stated in previous posting).

[Dave Warren]

Post by Bryan D.

Cloudflare has pushed an update, and things seem to be working from here. For those having issues, try again now?

Thanks for the "heads up." Works for me, also (i.e., on pfSense 2.2.6 configured as stated in previous posting).

How's the speed? I'm seeing moderately slower results for queries that
go out to 1.1.1.1, whereas queries from the cache or stub zones (to
servers hosted out on the 'net) are very fast.

If I switch TLS off and go back to @53 it's faster, but ultimately not
as fast as just running recursion myself.

[Bryan D.]

FWIW, since my original report, I've noticed some other things:

- since it's not yet "deployed," the v2.2.1 (at both ends) site-to-site IPsec VPN has only 1 laptop and 1 wireless access point on the LAN and virtually nothing else happening on the WAN (it's tied to a cable modem)

- the condition, when I did the original report, was that the laptop was sleeping -- it's a Mac with network wake-up configured and, in that mode, they constantly bring the port up 'n down (hundreds of times per day, each, according to switches at this office)

- I needed to make some changes to that laptop so I had someone bring it over here ... and that significantly changed the VPN "up-ness" behavior:

+ now the VPN is _much_ more likely to be up when I attempt to use it (i.e., with no LAN-to-LAN non-pfSense traffic, assuming there is some generated by the VPN mechanisms, themselves), but ... wait for it ...

+ if I ping the pfSense at the other end and the VPN connection _is_ alive, it'll stay alive as long as I continue the once-a-second pinging (from a non-pfSense system on the LAN) ...

+ however, if I kill the ping, wait 2 or 3 minutes then ping it again, it'll be down ... i.e., the pinging activity seems to stimulate a connection failure once the pinging stops (this seems to be a consistent behavior) ...

+ or maybe what I'm seeing is "the norm" -- i.e., that, as soon as there's a lull in Lan-to-Lan traffic for a short time, the connection drops (even though the config includes DPD and ping'd host at each end)

e.g.:
[surprise, it's up]
______________________________________________________________________
/Users/admin (2015-03-23 @ 15:26:05)
root # ping 172.16.22.1
PING 172.16.22.1 (172.16.22.1): 56 data bytes
64 bytes from 172.16.22.1: icmp_seq=0 ttl=63 time=26.280 ms
64 bytes from 172.16.22.1: icmp_seq=1 ttl=63 time=17.740 ms
64 bytes from 172.16.22.1: icmp_seq=2 ttl=63 time=18.134 ms
^C
--- 172.16.22.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 17.740/20.718/26.280/3.936 ms


[now wait about 2.5 minutes ... and it's down]
______________________________________________________________________
/Users/admin (2015-03-23 @ 15:26:12)
root # ping 172.16.22.1
PING 172.16.22.1 (172.16.22.1): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
... <snip'd>
Request timeout for icmp_seq 95
Request timeout for icmp_seq 96
Request timeout for icmp_seq 97
64 bytes from 172.16.22.1: icmp_seq=98 ttl=63 time=15.365 ms
64 bytes from 172.16.22.1: icmp_seq=99 ttl=63 time=14.927 ms
64 bytes from 172.16.22.1: icmp_seq=100 ttl=63 time=13.905 ms
64 bytes from 172.16.22.1: icmp_seq=101 ttl=63 time=15.105 ms
64 bytes from 172.16.22.1: icmp_seq=102 ttl=63 time=17.298 ms
64 bytes from 172.16.22.1: icmp_seq=103 ttl=63 time=18.674 ms
64 bytes from 172.16.22.1: icmp_seq=104 ttl=63 time=16.015 ms
64 bytes from 172.16.22.1: icmp_seq=105 ttl=63 time=15.246 ms
64 bytes from 172.16.22.1: icmp_seq=106 ttl=63 time=15.009 ms
64 bytes from 172.16.22.1: icmp_seq=107 ttl=63 time=15.953 ms
64 bytes from 172.16.22.1: icmp_seq=108 ttl=63 time=17.085 ms
64 bytes from 172.16.22.1: icmp_seq=109 ttl=63 time=21.631 ms
64 bytes from 172.16.22.1: icmp_seq=110 ttl=63 time=16.873 ms
64 bytes from 172.16.22.1: icmp_seq=111 ttl=63 time=16.639 ms
64 bytes from 172.16.22.1: icmp_seq=112 ttl=63 time=15.385 ms
^C
--- 172.16.22.1 ping statistics ---
113 packets transmitted, 15 packets received, 86.7% packet loss
round-trip min/avg/max/stddev = 13.905/16.341/21.631/1.823 ms

______________________________________________________________________
/Users/admin (2015-03-23 @ 15:30:21)
root #

[Chris Buechler]

There's nothing to go on to offer any worthwhile suggestions. IPsec
logs best place to start.

Post by Bryan D.
- since it's not yet "deployed," the v2.2.1 (at both ends) site-to-site IPsec VPN has only 1 laptop and 1 wireless access point on the LAN and virtually nothing else happening on the WAN (it's tied to a cable modem)
- the condition, when I did the original report, was that the laptop was sleeping -- it's a Mac with network wake-up configured and, in that mode, they constantly bring the port up 'n down (hundreds of times per day, each, according to switches at this office)
+ now the VPN is _much_ more likely to be up when I attempt to use it (i.e., with no LAN-to-LAN non-pfSense traffic, assuming there is some generated by the VPN mechanisms, themselves), but ... wait for it ...
+ if I ping the pfSense at the other end and the VPN connection _is_ alive, it'll stay alive as long as I continue the once-a-second pinging (from a non-pfSense system on the LAN) ...
+ however, if I kill the ping, wait 2 or 3 minutes then ping it again, it'll be down ... i.e., the pinging activity seems to stimulate a connection failure once the pinging stops (this seems to be a consistent behavior) ...
+ or maybe what I'm seeing is "the norm" -- i.e., that, as soon as there's a lull in Lan-to-Lan traffic for a short time, the connection drops (even though the config includes DPD and ping'd host at each end)
[surprise, it's up]
______________________________________________________________________
root # ping 172.16.22.1
PING 172.16.22.1 (172.16.22.1): 56 data bytes
64 bytes from 172.16.22.1: icmp_seq=0 ttl=63 time=26.280 ms
64 bytes from 172.16.22.1: icmp_seq=1 ttl=63 time=17.740 ms
64 bytes from 172.16.22.1: icmp_seq=2 ttl=63 time=18.134 ms
^C
--- 172.16.22.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 17.740/20.718/26.280/3.936 ms
[now wait about 2.5 minutes ... and it's down]
______________________________________________________________________
root # ping 172.16.22.1
PING 172.16.22.1 (172.16.22.1): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
... <snip'd>
Request timeout for icmp_seq 95
Request timeout for icmp_seq 96
Request timeout for icmp_seq 97
64 bytes from 172.16.22.1: icmp_seq=98 ttl=63 time=15.365 ms
64 bytes from 172.16.22.1: icmp_seq=99 ttl=63 time=14.927 ms
64 bytes from 172.16.22.1: icmp_seq=100 ttl=63 time=13.905 ms
64 bytes from 172.16.22.1: icmp_seq=101 ttl=63 time=15.105 ms
64 bytes from 172.16.22.1: icmp_seq=102 ttl=63 time=17.298 ms
64 bytes from 172.16.22.1: icmp_seq=103 ttl=63 time=18.674 ms
64 bytes from 172.16.22.1: icmp_seq=104 ttl=63 time=16.015 ms
64 bytes from 172.16.22.1: icmp_seq=105 ttl=63 time=15.246 ms
64 bytes from 172.16.22.1: icmp_seq=106 ttl=63 time=15.009 ms
64 bytes from 172.16.22.1: icmp_seq=107 ttl=63 time=15.953 ms
64 bytes from 172.16.22.1: icmp_seq=108 ttl=63 time=17.085 ms
64 bytes from 172.16.22.1: icmp_seq=109 ttl=63 time=21.631 ms
64 bytes from 172.16.22.1: icmp_seq=110 ttl=63 time=16.873 ms
64 bytes from 172.16.22.1: icmp_seq=111 ttl=63 time=16.639 ms
64 bytes from 172.16.22.1: icmp_seq=112 ttl=63 time=15.385 ms
^C
--- 172.16.22.1 ping statistics ---
113 packets transmitted, 15 packets received, 86.7% packet loss
round-trip min/avg/max/stddev = 13.905/16.341/21.631/1.823 ms
______________________________________________________________________
root #
_______________________________________________
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

[Bryan D.]

Post by Chris Buechler
There's nothing to go on to offer any worthwhile suggestions. IPsec
logs best place to start.

If you can be more specific, I'll try to help. Sorry, but I don't have enough background with IPsec to ferret things out on my own. I did try setting both "Networking" and "IPsec Traffic" (in Advanced Settings) to Diag then to Raw, but saw no additional IPsec logging after pressing the "Restart IPsec service" button on that page.

[Bill Arlofski]

Post by Bryan D.
- been ultra reliable (if VPN is down, suspect ISP issue or pfSense box failure)
- it's been quick to connect (about 1 second, almost unnoticeable)
- it's worked across numerous upgrades without issue (nice!)
Beginning with pfSense v2, we added multiple P2s at each end (still same reliability, etc.).
One of the offices has had its hardware updated and its pfSense updated to 2.2 then 2.2.1 (after testing to see whether we seemed to be affected by the "multiple P2 issue" noted in the upgrade page -- we're OK on that one). This connection has continued to work with the same characteristics as before. The 2.2.1 system is 64-bit and the other end is v2.1.5 32-bit
We recently added a second site-to-site IPsec VPN, essentially the same as the existing one except both sides are pfSense v2.2.1 (but other end is 32-bit) and stronger algorithms are being used and P1 is set to v2 (supposedly avoiding any "multiple P2" issues).
- v-e-r-y s-l-o-w to connect: e.g. pinging when connection is down yields: once a connection after 12 seconds, once a connection after 22 seconds and dozens of connections after > 2.25 minutes
- completely unable to "stay connected": both sides have DPD enabled (5 sec./3 retries) and both sides can be initiators and both sides have 1 P2 set to ping the pfSense at the other end
- pressing the "connect button" on pfSense's IPsec status page yields an "instant connection" but there won't be any P2 traffic coming back for a while, which seems to consistently be the connection-delay issue
If one of the ends has regular (almost constant) traffic, the VPN stays connected. In testing, I've had one non-pfSense system pinging the pfSense at the other end and the VPN stays connected.
If VPN traffic pauses for a short time (ten minutes?), the connection is dropped. While that is not what's expected, given the config, it wouldn't be bad _if_ the connection was quickly re-established with traffic, but it's not.
- there are issues with multiple P2s even with v2 P1
- the DPD stuff isn't working
- the P2 "automatically ping host" stuff isn't working ... but pinging a host via a non-pfSense system does keep the connection alive
I'm willing to run some tests if someone wants to tell me what they want done. For about a week, the new v2.2.1 site-to-site VPN won't really be used, I can do almost anything that doesn't cause the other side to go dead (or I'd need to make a trip to the other site and that may not happen very quickly).

/me also raises hand

Since the 2.2 upgrade and now the 2.2.1 I am seeing the same symptoms with a
few of our endpoints. None of which are currently 2.2 nor 2.2.1, some of which
are m0n0walls. :)

Bill

[David Hatch]

We are having all the same symptoms above. All of our firewalls are
running 2.2.4. Everything that has 2 phase 2 entries is on IKE v2. We are
planning on changing the rest but we have 45 sites...it take a long
time...


Has anyone figured this out? It's driving me crazy and causing everything
to be so unreliable I've considered going back to 2.1.5 on all of my boxes
even if it takes me a month to do it. It's that bad. I'm experiencing all
of the OP's symptoms and nothing I can do will fix it short of pining from
a non-pfsense box inside the LAN to a remote location. Doing this
constantly will keep the connection solid, but that's about it. Any help
would be appreciated. Thanks.

[Bryan D.]

Post by David Hatch
We are having all the same symptoms above. All of our firewalls are
running 2.2.4. Everything that has 2 phase 2 entries is on IKE v2. ...
Has anyone figured this out? ... nothing I can do will fix it short of pining from
a non-pfsense box inside the LAN to a remote location. Doing this
constantly will keep the connection solid, but that's about it. Any help
would be appreciated. Thanks.

If you search the archives for the subject
2.2.1 Site-to-Site IPsec VPN Connection Instability
you'll find this is an issue that's been experienced by others.

Your "ping via an external system" workaround is what works (code in one of the posted emails).

On 2015-04-04 I provided Chris Buechler some logs he requested and on 2015-04-07 he responded with
---
thanks, that should help narrow things down. I don't see anything
obvious at a glance, will more closely correlate timestamps and across
logs when I have time at some point this week.
---
That was the last I heard.

I'm still on v2.2.2 and the issue still exists. From what you're saying, sounds like it's still in 2.2.4.

[m***@mchsi.com]

stop
----- Original Message -----
From: "Christopher CUSE" <***@ccuse.com>
To: ***@lists.pfsense.org
Sent: Monday, March 23, 2015 9:34:12 AM GMT -06:00 US/Canada Central
Subject: Re: [pfSense] 2.2.1 Site-to-Site IPsec VPN Connection Instability

Post by mayak

Post by Bryan D.
- been ultra reliable (if VPN is down, suspect ISP issue or pfSense box failure)
- it's been quick to connect (about 1 second, almost unnoticeable)
- it's worked across numerous upgrades without issue (nice!)
Beginning with pfSense v2, we added multiple P2s at each end (still same reliability, etc.).
One of the offices has had its hardware updated and its pfSense updated to 2.2 then 2.2.1 (after testing to see whether we seemed to be affected by the "multiple P2 issue" noted in the upgrade page -- we're OK on that one). This connection has continued to work with the same characteristics as before. The 2.2.1 system is 64-bit and the other end is v2.1.5 32-bit
We recently added a second site-to-site IPsec VPN, essentially the same as the existing one except both sides are pfSense v2.2.1 (but other end is 32-bit) and stronger algorithms are being used and P1 is set to v2 (supposedly avoiding any "multiple P2" issues).

<snip>
i have to say that i am also experiencing this. i'm in the process of installing smokeping to prove connectivity is good between the public ip endpoints between various vpns.
will report back with those results.
thanks
m

just got dropped again -- fourth time in last few hours -- something is definitely wrong.

upgraded all my pfsenses to 2.2.1 over the weekend.

m
_______________________________________________
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold