Cluster network configuration

[Alberto_De_Gaspari]

Good evening everyone. We are facing an issue regarding how to connect and reach the cluster from the web.
The cluster of 2 nodes is connected to a switch with the following configuration:

vlan100: Wan connection
vlan10: lan connection(where there are also 2 phisical servers which need to be reached by the vms)
vlan11: Pfsense Sync
untagged: Cluster ip's
untagged traffic used by the nodes to comunicate each other.

On the cluster we have deployed an HA pfsense running with carp on 2vms, one on each node.
Pfsense is configured like follows:
Wan connected on a vnet tagged on vlan100 by openvswitch
Lan connected on a vnet tagged on vlan10 by openvswitch
Sync connected on a vnet tagged on vlan11 by openvswitch
Manage connected to a vnet UNTAGGED

We have now a problem regarding the configuration of the cluster's ip: we were thinking about setting up the cluster with an ip assigned on the Manage network(intended as the untagged traffic running on the switch), which would became reachable from the outside once connected to a vpn exposed by pfSense.
Considering that the canary ip have to be on the manage network, in case of a sudden death of pfsense HA, or in case the cluster is starting without booting any of the 2 vm's hosting pfSense, the cluster will be missing it's canary and the ability to connect to the outside, without considering the ability to reach the cluster from outside.
Are we missing something or this is the only feasible solution applicable in our situation?
Does the canary ip need to be in the same subnet specified in the network configuration of the cluster or not?
Thank you in advance!
alberto

[penzoiders]

Hello @Alberto_De_Gaspari,

To reach the cluster from outside set a new interface on the pfSense and attach it to OpenVSwitch virtual network with NO VLAN ID (plain untagged OVS traffic will be able to reach Cluster Management IP).
Configure it with an IP in the same subnet of your Cluster Management IP and set the appropriate rules to reach it.
I would suggest using a VPN, but if you need to expose the NodeWeaver GUI directly you'll need to forward port 443 TCP (HTTPS) and 29876 TCP (websocket for VNCproxy) to the management IP.

Since build 1709131144 you can upload through the advanced setup menu and the upload folder your custom SSL certificates (just need to be valid and compatible with NGINX cerr+ca-bundle and plaintext key).

As for the Canary IP it's intended to be set to something OUTSIDE of the cluster to avoid master elections while switches are still booting since it must act as proof that the packet switching is correctly occurring.

You won't be able to properly boot a cluster after powerloss if the Canary is in a VM inside NodeWeaver. This for a simple reason: it won't be up and running after a blackout.
Moreover if you use a VM IP, especially a CARP floating IP, you will very likely have split brain issues in case of switch failure or node network isolation.

If you've set your canary to be the CARP IP of the pfSense VM go and change it straight away.

if you want to have something reliable as a Canary IP you may want to set an IP of some redundant hardware you have around your network, something as a dual-psu powered switch or a little always-on industrial device (note: do NOT use the management IP of the switch where nodes are attached, since this can come "online" before proper packet switching occours).

my2cents:
If you don't have a reliable candidate for Canary IP and really want it to be a multi device CARP/keepalived/whatever IP you can buy a bunch of Raspberry Pi / Pine64 / name-it-cheap SBC and ensure an High Available Canary IP with little money.
If you just need to setup a cheap CARP cluster for the Canary (a bit geeky, a bit to show off and maybe not needed, but "hell yeah nerdish!!!") you'll find this entry level Pine64 board for just 15$ (less than half the price of a raspberry, 40$ including case,psu,sdcard) https://www.pine64.org/?product=pine-a64-board

[penzoiders]

Does the canary ip need to be in the same subnet specified in the network configuration of the cluster or not?

yes, same subnet of Cluster Management IP.

[Alberto_De_Gaspari]

Thank you very much Penzoiders, every answer is super clear.

tl;dr; according to what said, and considering that it is impossibile to add the cluster directly to a vpn or at least protect it with nginx client certificate verification, the only feasibile solution is to add another piece of redundant hardware, right?

It was a good news overall, until i've read your last statement:
we are in a server farm under a floating ip dual network gateway. if it was possibile to use a different subnet i would be able to use the gw as canary, but with the notes you added the only possibility i see i adding another piece of hardware in the network, same subnet as my LAN network, to avoid exposing the cluster ip.
i think that all considered you should really add this to the requirements of the cluster:
exposing the cluster ip to the internet is the worst idea possible, as you said, so the only feasible(and best) solution implies the creation of a vpn. but if you want to create the vpn inside the cluster(as we would) you actually cant stop here, because the canary MUST be outside the cluster(for the reasons you stated, which are exactly what i was thinking about writing the first post, also considering the management ip of the switches) but in the same network of the cluster ip, so the lan, so it needs another piece of hw, which must be HA to respect dogmas of NW, which in my opinion adds complexity to the infrastructure.
I hoped the solution would be having the canary ip in a different subnetwork(pointing to the gateway, in my case), with also the addition that it would be very awesome if it could be possible to tag separately the canary ip(and network conf) and the cluster network configuration, having different network setup(which should include vlan tag) for the canary and the cluster network.
I also think that the https tunnel you are suggesting, if coming from a vm inside the cluster, would again make the cluster unreachable in case of the vm not booting for reasons(missing chunk or first boot of cluster), so also for this solution i need an external appliance.

I would really appreciate the possibility to avoid adding extra complexity to the network, any solution would be super-appreciated!

thank you in advance again!
alberto

[penzoiders]

Sorry for late re-reply

tl;dr; according to what said, and considering that it is impossibile to add the cluster directly to a vpn or at least protect it with nginx client certificate verification, the only feasibile solution is to add another piece of redundant hardware, right?

can't get this statement.
I was reccomending a VPN to access your cluster from outside.
Or a valid certificate to expose the webUI on the public IP (TCP 443 and 29876).

the Canary IP is a complete different subject.
If you have a physical router (even a clustered one) that can expose an IP (even a virtual one) on the same subnet of the NodeWeaver management IP to allow a simple ICMP ping from NodeWeaver to this IP: this is more than enough.
You don't need to NAT or to connect outside anything. just need something to ping across the switch to sense if packets are moving across ports (thus the switch is probably out of boot phase and up and running).

You don't need another piece of redundant whatever if you have already one.

[Alberto_De_Gaspari]

you are totally right, but this means exposing the ip of the cluster to the internet, which could be feasible if it is possible to access ssh only via public key and web interface via client side certificate validation, but this is not the case, so from what i see the only protection i have from the outside publishing the cluster in the network is only via password, which could actually be bruteforced or ddosed.
am i wrong if i say that the only possible solution with the stated network configuration is to directly expose the cluster over the internet or adding a piece of hardware to host the canary ip?
i'm setting up pfsense inside the cluster to provide a vpn that will allow me to reach the cluster ip in the lan, adding a piece of hw in the same network will allow me to set this as the canary ip outside the cluster, because to set the gateway of the farm as the canary ip would result in setting the cluster on the same subnet and network, which would result in having the cluster configured on a public ip(the ones provided by the farm's router), making it directly reachable with the problems stated at the beginning of this answer.
maybe i'm missing something, but i think that to avoid exposing the cluster to the network, which i don't consider a feasible solution considered the absence of security protocol apart from username and password, the only solution is to add an extra piece of hw to resolve the canary ip problem, right?
again, thank you in advance for your time, really appreciated!

[penzoiders]

but this means exposing the ip of the cluster to the internet

Nobody ever said that is necessary to expose the cluster to the internet.
If you set the Management IP in the LAN side of your physical reduntant router you can even make firewall rules to protect it from outer access.

I really can't get your point in this.

You don't need another piece of redundant hardware if you already have one in the same physical network. just expose a single IP that can be reached by the cluster from the redundant router LAN to the Cluster Management subnet. No need for that IP to be reachable by the outer network.