Failover Clustering Networking Basics and Fundamentals | Microsoft Community Hub

My name is John Marlin and I am with the High Availability and Storage Team.  With newer versions of Windows Server and Azure Stack HCI on the horizon, it’s time lớn head lớn the archives and dust off some old information as they are in need of updating.

In this blog, I want lớn talk about Failover Clustering and Networking. Networking is a fundamental key with Failover Clustering that sometimes is overlooked but can be the difference in success or failure. In this blog, I will be hitting on all facets from the basics, tweaks, multi-site/stretch, and Storage Spaces Direct.  By no means should this be taken as a “this is a networking requirement” blog.  Treat this as more of general guidance with some recommendations and things lớn consider.  Specific requirements for any of our operating systems (new or old) will be a part of the documentation (https://docs.microsoft.com) of the particular OS.

In Failover Clustering, all networking aspects are provided by our Network Fault Tolerant (NetFT) adapter. Our NetFT adapter is a virtual adapter that is created with the Cluster is created. There is no configuration necessary as it is self-configuring. When it is created, it will create its MAC Address based off of a hash of the MAC Address of the first physical network thẻ. It does have conflict detection and resolution built in. For the IP Address scheme, it will create itself an APIPA IPv4 (169.254.*) and IPv6 (fe80::*) address for communication.

Connection-specific DNS Suffix  . :

Description . . . . . . . . . . . : Microsoft Failover Cluster Virtual Adapter

Physical Address. . . . . . . . . : 02-B8-FA-7F-A5-F3

DHCP Enabled. . . . . . . . . . . : No

Autoconfiguration Enabled . . . . : Yes

Link-local IPv6 Address . . . . . : fe80::80ac:e638:2e8d:9c09%4(Preferred)

IPv4 Address. . . . . . . . . . . : 169.254.1.143(Preferred)

Subnet Mask . . . . . . . . . . . : 255.255.0.0

Default Gateway . . . . . . . . . :

DHCPv6 IAID . . . . . . . . . . . : 67287290

DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-26-6B-52-A5-00-15-5D-31-8E-86

NetBIOS over Tcpip. . . . . . . . : Enabled

The NetFT adapter provides the communications between all nodes in the cluster from the Cluster Service. To tự this, it discovers multiple communication paths between nodes and if the routes are on the same subnet or cross subnet. The way it does this is through “heartbeats” through all network adapters for Cluster use lớn all other nodes. Heartbeats basically serve multiple purposes.

1. Is this a viable route between the nodes?
2. Is this route currently up?
3. Is the node being connected lớn up?

There is more lớn heartbeats, but will defer lớn my other blog No Such Thing as a Heartbeat Network for more details on it.

For Cluster communication and heartbeats, there are several considerations that must be taken into tài khoản.

1. Traffic uses port 3343. Ensure any firewall rules have this port open for both TCP and UDP
2. Most Cluster traffic is lightweight.
3. Communication is sensitive lớn latency and packet loss. Latency delays could mean performance issues, including removal of nodes from membership.
4. Bandwidth is not as important as quality of service.

Cluster communication between nodes is crucial sánh that all nodes are currently in sync. Cluster communication is constantly going on as things progress. The NetFT adapter will dynamically switch intra-cluster traffic lớn another available Cluster network if it goes down or isn’t responding.

The communications from the Cluster Service lớn other nodes through the NetFT adapter looks lượt thích this.

• Cluster Service plumbs network routes over NIC1, NIC2 on NetFT

• Cluster Service establishes TCP connection over NetFT adapter using the private NetFT IP address (source port 3343)
• NetFT wraps the TCP connection inside of a UDP packet (source port 3343)
• NetFT sends this UDP packet over one of the cluster-enabled physical NIC adapters lớn the destination node targeted for destination node’s NetFT adapter
• Destination node’s NetFT adapter receives the UDP packet and then sends the TCP connection lớn the destination node’s Cluster Service

Heartbeats are always traversing all Cluster enabled adapters and networks. However, Cluster communication will only go through one network at a time. The network it will use is determined by the role of the network and the priority (metric).

There are three roles a Cluster has for networks.

Disabled for Cluster Communications – Role 0 - This is a network that Cluster will not use for anything.

Enabled for Cluster Communication only – Role 1 – Internal Cluster Communication and Cluster Shared Volume traffic (more later) are using this type network as a priority.

Enabled for client and cluster communication – Role 3 – This network is used for all client access and Cluster communications. Items lượt thích talking lớn a tên miền controller, DNS, DHCP (if enabled) when Network Names and IP Addresses come online. Cluster communication and Cluster Shared Volume traffic could use this network if all Role 1 networks are down.

Based on the roles, the NetFT adapter will create metrics for priority. The metric Failover Cluster uses is not the same as the network thẻ metrics that TCP/IP assigns. Networks are given a “cost” (Metric) lớn define priority. A lower metric value means a higher priority while a higher metric value means a lower priority.

These metrics are automatically configured based on Cluster network role setting.

Cluster Network Role of 1 = 40,000 starting value

Cluster Network Role of 3 = 80,000 starting value

Things such as Link tốc độ, RDMA, and RSS capabilities will reduce metric value. For example, let’s say I have two networks in my Cluster with one being selected and Cluster communications only and one for both Cluster/Client. I can run rẩy the following lớn see the metrics.

PS > Get-ClusterNetwork | ft Name, Metric

Name                 Metric

----                     ------

Cluster Network 1     70240

Cluster Network 2     30240

The NetFT adapter is also capable of taking advantage of SMB Multichannel and load balance across the networks. For NetFt lớn take advantage of it, the metrics need lớn be < 16 metric values apart. In the example above, SMB Multichannel would not be used. But if there were additional cards in the machines and it looked lượt thích this:

PS > Get-ClusterNetwork | ft Name, Metric

Name                 Metric

----                 ------

Cluster Network 1     70240

Cluster Network 2     30240

Cluster Network 3     30241

Cluster Network 4     30245

Cluster Network 5     30265

In a configuration such as this, SMB Multichannel would be used over Cluster Networks 2, 3 and 4. From a Cluster communication and heartbeat standpoint, multichannel really isn’t a big giảm giá. However, when a Cluster is using Cluster Shared Volumes or is a Storage Spaces Direct Cluster, storage traffic is going lớn need higher bandwidth. SMB Multichannel would fit nicely here sánh an additional network thẻ or higher tốc độ network cards are certainly a consideration.

In the beginning of the blog, I mentioned latency and packet loss. If heartbeats cannot get through in a timely fashion, node removals can happen. Heartbeats can be tuned in the case of higher latency networks. The following are mặc định settings for tuning the Cluster networks.

For more information on these settings, please refer lớn the Tuning Failover Cluster Network Thresholds blog.

Planning networks for Failover Clustering is dependent on how it will be used. Let’s take a look at some of the common network traffics a Cluster would have.

If this were a Hyper-V Cluster running virtual machines and Cluster Shared Volumes, Live Migration is going lớn occur.  Clients are also connecting lớn the virtual machines.

Cluster Communications and heart beating will always be on the wire.  If you are using Cluster Shared Volumes (CSV), there will be some redirection traffic.

If this were Cluster that used ISCSI for its storage, you would have that as a network.

If this was stretched (nodes in multiple sites), you may have the need for an additional network as the considerations for replication (such as Storage Replica) traffic.

If this is a Storage Spaces Direct Cluster, additional traffic for the Storage Bus Layer (SBL) traffic needs lớn be considered.

As you can see, there is a lot of various network traffic requirements depending on the type of Cluster and the roles running. Obviously, you cannot have a dedicated network or network thẻ for each as that just isn’t always possible.

We tự have a blog that will help with the Live Migration traffic lớn get some of the traffic isolated or limited in the bandwidth it uses. The blog Optimizing Hyper-V Live Migrations on an Hyperconverged Infrastructure goes over some tips lớn phối up.

The last thing I wanted lớn talk about is with stretch/multisite Failover Clusters. I have already mentioned the Cluster specific networking considerations, but now I want lớn talk about how the virtual machines react in this type environment.

Let’s say we have two datacenters and a four-node Failover Cluster with 2 nodes in each datacenter. As with most datacenters, they are in their own subnet and would be similar lớn this:

The first thing you want lớn consider is if you want security between the cluster nodes on the wire. As a mặc định, all Cluster communication is signed. That may be fine for some, but for others, they wish lớn have that extra level of security. We can phối the Cluster lớn encrypt all traffic between the nodes. It is simply a PowerShell command lớn change it. Once you change it, the Cluster as a whole needs lớn be restarted.

PS > (Get-Cluster).SecurityLevel = 2

0 = Clear Text

1 = Signed (default)

2 = Encrypt (slight performance decrease)

Here is a virtual machine (VM1) that has an IP Address on the 1.0.0.0/8 network and clients are connecting lớn it. If the virtual machine moves over lớn Site2 that is a different network (172.0.0.0/16), there will not be any connectivity as it stands.

To get around this, there are basically a couple options.

To prevent the virtual machine from moving from a Cluster-initiated move (i.e. drain, node shutdown, etc), consider using sites. When you create sites, Cluster now has site awareness. This means that any Cluster-initiated move will always keep resources in the same site. Setting a preferred site will also keep it in the same site. If the virtual machine was lớn ever move lớn the second site, it would be due lớn a user-initiated move (i.e. Move-ClusterGroup, etc) or a site failure.

But you still have the IP Address of the virtual machine issue lớn giảm giá with. During a migration of the virtual machine, one of the very last things is lớn register the name and IP Address with DNS. If you are using a static IP Address for the virtual machine, a script would need lớn be manually run rẩy lớn change the IP Address lớn the local site it is on. If you are using DHCP, with DHCP servers in each site, the virtual machine will obtain a new address for the local site and register it. You then have lớn giảm giá with DNS replication and TTL records a client may have. Instead of waiting for the timeout periods, a forced replication and TTL clearing on the client side would allow them lớn connect again.

If you tự not wish lớn go that route, a virtual LAN (VLAN) could be phối up across the routers/switches lớn be a single IP Address scheme. Doing this will not have the need lớn change the IP Address of the virtual machine as it will always remain the same. However, stretching a VLAN (not a recommendation by Microsoft) is not always easy lớn tự and the Networking Group within your company may not want lớn tự this for various reasons.  

Another consideration is implementing a network device on the network that has a third IP Address that clients connect lớn and it holds that actual IP Address of the virtual machine sánh it will route clients appropriately. For example:

 In the above example, we have a network device that has the IP Address of the virtual machine as 30.0.30.1. It will register this with all DNS and will keep the same IP Address no matter which site it is on. Your Networking Group would need lớn involved with this and need lớn control it. The chances of them not doing it is something lớn also consider if it can even done within your network.

We talked about virtual machines, but what about other resources, say, a tệp tin server?  Unlike virtual machine roles, roles such as a tệp tin server have a Network Name and IP Address resource in the Cluster. In Windows 2008 Failover Cluster, we added he concept of “or” dependencies. Meaning, we can depend on this "or" that.

In the case of the scenario above, your Network Name could be dependent on 1.0.0.50 “or” 172.0.0.50. As long as one of the IP Address resources is online, the name is online and what is published in DNS. To go a step further for the stretch scenario, we have two parameters that can be used.

RegisterAllProvidersIP: (default = 0 for FALSE)    

• Determines if all IP Addresses for a Network Name will be registered by DNS
• TRUE (1): IP Addresses can be online or offline and will still be registered
• Ensure application is phối lớn try all IP Addresses, sánh clients can connect quicker
• Not supported by all applications, kiểm tra with application vendor
• Supported by SQL Server starting with SQL Server 2012

HostRecordTTL: (default = 1200 seconds)

• Controls time the DNS record lives on client for a cluster network name
• Shorter TTL: DNS records for clients updated sooner
• Disclaimer: This does not tốc độ up DNS replication

By manipulating these parameters, you will have quicker connection times by a client. For example, I want lớn enable lớn register all the IP Addresses with DNS but I want the TTL lớn be 5 minutes. I would run rẩy the commands:

PS > Get-ClusterResource FSNetworkName | Set-ClusterParameter RegisterAllProvidersIP 1

PS > Get-ClusterResource FSNetworkName | Set-ClusterParameter HostRecordTTL 300

When setting the parameters, recycling (offline/online) of the resources is needed.

There is more I could go into here with this subject but need lớn signoff for now. I hope that this gives you some basics lớn consider when designing your Clusters while thinking of the networking aspects of it. Networking designs and considerations must be carefully thought out.

Happy Clustering !!

John Marlin

Senior Program Manager

High Availability and Storage

Follow u on Twitter: @johnmarlin_msft