On this page

VMware HCX networking

VMware HCX Network Extension (NE) stretches a Layer 2 network, either a VLAN-backed distributed port group or an NSX segment, from a source site to a destination site through a pair of HCX-NE appliances, so migrated VMs keep their IP addresses, MAC addresses and default gateway. In VCF Operations HCX 9.1 the NE appliances are rebuilt on the NSX Edge stack, one enhanced appliance can carry up to 128 extensions on a single vNIC, and VLAN networks can now be extended to native distributed port groups at the destination.

Network Extension is the part of HCX that application owners notice least when it works and most when it does not. This guide covers how the appliances move traffic, what Mobility Optimized Networking (MON) actually changes, how to size the MTU so packets do not fragment, how to scale and protect the appliances, and how to cut over an extended network so the gateway ends up at the destination and the extension can be removed.

Key points

  • NE bridges a source network to a matching network at the destination. The default gateway stays at the source until you cut it over.
  • Each NE appliance has an uplink (the encrypted tunnel to its peer, carried in UDP 4500) and sink port interfaces that sit on the extended networks.
  • MON stops east-west traffic in the cloud from hairpinning through the source gateway by adding /32 host routes on the destination Tier-1. It only works with NSX overlay destinations.
  • The HCX uplink MTU can be set between 1150 and 9000 bytes. A mismatch between peers, or an underlay smaller than the profile, is the most common cause of slow or broken extensions.
  • Unextending does not shut down your source gateway. HCX can connect the destination segment to the cloud gateway, but disabling the original gateway is your job.

What is HCX Network Extension?

Network Extension is a Layer 2 bridging service that you start from the source site. HCX creates a matching network at the destination and bridges the two, so a VM migrated to the destination, or created there on the extended network, behaves as if it sits on the same Layer 2 segment as the VMs it left behind. Broadcom's documentation is explicit that the default gateway for an extended network stays connected only at the source. Traffic from a migrated VM to any other subnet still leaves through the source router.

That single fact explains most NE design decisions. It is why extended networks let you migrate in waves without re-addressing anything, and it is also why latency, tromboning and the eventual gateway cutover all need planning.

Which networks can you extend? According to the HCX 9.1 user guide you can extend VLAN-tagged distributed port groups on a vSphere Distributed Switch, NSX overlay and NSX VLAN segments, and NSX VPC subnets. VPC subnet extension needs HCX and NSX 9.0 or later on the relevant sites. The destination can be an NSX overlay segment, an NSX VLAN segment, an NSX VPC subnet or, new in 9.1, a native distributed port group.

Source networkCan you extend it?Notes from Broadcom
VLAN-tagged distributed port groupYesUntagged port groups (VLAN None or 0) are greyed out
NSX overlay or NSX VLAN segmentYesNSX Manager 3.2.3 or later, registered with HCX
NSX VPC subnetYesHCX and NSX 9.0 or later on the source
vSphere Standard Switch port groupNoNot supported as a source
VMkernel, PVLAN, ephemeral bindingNoBlocked in the UI

Table 1. Source networks and Network Extension support in HCX 9.1. Source: Broadcom HCX 9.1 requirements and restrictions pages.

Important: Broadcom says never to extend the vSphere management network or any other VMkernel network (vMotion, vSAN, replication) with HCX. Network Extension also does not detect or prevent loops, IP conflicts or MAC conflicts, and you must not run NE and another bridge such as NSX L2 VPN on the same network at the same time. Broadcom warns that two bridging solutions together can cause a network outage.

How the Network Extension appliance works

NE runs as a pair of appliances, one at each site, deployed by the service mesh. Broadcom's troubleshooting KB 436416 describes two kinds of interface on each appliance:

  • The uplink, which sends and receives encapsulated traffic to the peer NE appliance. HCX wraps guest traffic in IPsec carried over UDP 4500 for the trip across the underlay. The uplink takes its IP address from the uplink network profile.
  • The sink port, which sends and receives the unencapsulated guest frames on the extended network itself. This is the interface that sits on your source VLAN or segment, and on the destination segment.

Older appliances use one sink port vNIC per extended network. HCX 9.1 enhanced appliances support multiple extensions per vNIC, up to 128 on a single vNIC, which many practitioners describe as a trunk interface. Broadcom's own term is "multiple extensions per vNIC". The appliance also has a management interface for talking to HCX Manager.

HCX Network Extension topology for one extended network At the source site, the default gateway stays in place and the source VLAN or segment connects to the source HCX-NE appliance through its sink port. The source NE uplink sends traffic through an IPsec tunnel carried in UDP 4500 across the underlay to the destination NE uplink. The destination NE sink port connects to an NSX segment, VLAN segment or distributed port group where the migrated VM keeps the same IP address and gateway. One extended network, end to end Source site Default gateway stays here (router or T1) Source VLAN or segment VMs not yet migrated HCX-NE (source) Sink port on the extended network Uplink on the uplink network profile Destination site Migrated VM same IP, same gateway NSX segment, VLAN segment or DVPG HCX-NE (destination) Sink port on the destination segment Uplink on the uplink network profile IPsec tunnel UDP 4500 over the underlay Layer 2 stretched IPs and MACs kept
Figure 1. An extended network: sink ports on each side carry guest frames, the uplinks carry the encrypted tunnel, and the gateway stays at the source. Source: Broadcom KB 436416 and the HCX 9.1 user guide.

Port group security settings on the source

On a distributed switch the sink port has to receive and send frames for MAC addresses that do not belong to the appliance. Broadcom KB 321612 says the distributed port group used for the extension needs MAC address changes and Forged transmits set to Accept. If either is set to Reject, ARP can look healthy while return traffic from the cloud side is dropped before it reaches the local VM. The upstream physical switch must also accept RARP packets, which HCX lists as a general requirement.

KB 344987 adds a less obvious one. On an NSX-enabled distributed switch with several non-LAG uplinks, mixing a "sink port" extension (from a plain DV port group) with a "mac-learning" extension (from an NSX VLAN segment) can make cloud VMs lose connectivity after a vMotion of the NE or the VM. Broadcom fixed it in NSX 3.2.4 and 4.2.0.

What changed for Network Extension in HCX 9.1?

VCF 9.1 shipped HCX as VCF Operations HCX with a new appliance design. Every new service mesh uses enhanced IX and NE appliances built on the NSX Edge stack, and traffic between enhanced IX and NE instances uses standard IPsec. Existing meshes keep the legacy PhotonOS appliances until you move them with the Migrating Network Extensions workflow, and Broadcom targets removal of the PhotonOS appliances for VCF 10.0. We cover that move step by step in our guide to migrating HCX service meshes to the enhanced appliances.

AreaLegacy NE (PhotonOS)Enhanced NE (9.1)
Default for new meshesNoYes
Extensions per applianceAbove 8 needs another applianceUp to 128 on one vNIC; 7 in single-extension mode
VLAN to native DVPGNoYes, from 9.1
In-Service upgrade IPs1 new management and 1 new uplink IP each1 new management and 1 intrasite control IP each; uplink IP reused
StatusDeprecated in 9.1Required before VCF 10

Table 2. Legacy and enhanced Network Extension appliances. Sources: VCF 9.1.0 release notes, HCX 9.1 requirements and In-Service upgrade pages, VMware Cloud Foundation blog.

The Migrating Network Extensions workflow moves extensions from one appliance pair to another with what Broadcom calls "minimal downtime". You can run it per appliance (Interconnect, View Appliances, the Migrate Network Extensions tab) or per network (Network Extension, Actions, Migrate Network Extensions). Both appliances must belong to the same network container and the target needs spare extension capacity. The same workflow is how you rebalance load across enhanced appliances.

128

extensions on one vNIC, enhanced NE

3

maximum destinations for one network

4

NE appliances in an HA group

1150-9000

uplink MTU range in bytes

How do you extend a network in HCX 9.1?

The workflow lives in the HCX Manager UI under Network Extension. The fields depend on the destination type, which catches people who are used to older versions where every extension asked for a gateway address.

  1. Open Extend Networks

    In HCX Manager go to Network Extension and click Extend Networks. Pick the service mesh; if only one exists it is selected for you.

  2. Select the source networks

    Use the Filters pane to hide networks that are ineligible, have no VMs or are already extended, then tick one or more networks and click Next.

  3. Fill in the destination details

    For an NSX overlay destination choose the Destination First Hop Router (a Tier-1 gateway), enter the gateway IP and prefix in CIDR form such as 192.168.10.1/24, pick the extension appliance and optionally enable Mobility Optimized Networking. For an NSX VLAN segment or vSphere DVPG destination, pick an appliance that supports VLAN and enter the destination VLAN ID instead.

  4. Leave the overlap options alone unless you mean it

    Settings - optional holds Allow Overlapping Source VLAN, Allow Overlapping IP Subnet, Allow Overlap with Network Profiles and Allow Overlapping Destination VLAN. Each one switches off a safety check.

  5. Validate, then submit

    Validate checks your inputs without extending anything. Submit starts the extension; follow it under Activity Logs. Requests queue if the same appliances are busy with a mesh edit, redeploy or another extension.

If the mesh has no appliance that supports VLAN destinations, you have to edit the service mesh to deploy one first. Broadcom's walkthrough on the VMware Cloud Foundation blog uses the same Gateway IP and Destination First Hop Router fields shown above.

Planning extended networks for a migration?

Get the NE design right before the first wave

Consult Circle designs HCX service meshes, Network Extension and MON for migrations into VCF 9.1, VMware Cloud on AWS and Azure VMware Solution, including the gateway cutover plan.

Talk to our HCX team

What is Mobility Optimized Networking and when do you need it?

Without MON, every packet a migrated VM sends to another subnet goes back across the tunnel to the source gateway. That is fine for traffic headed on-premises. It is wasteful when the other subnet is also in the cloud: two VMs that sit next to each other in the destination SDDC, on different segments, talk via the source router and cross the WAN twice. Broadcom calls this the network tromboning effect.

MON fixes it from the destination side. When you enable MON on an extension, HCX puts the segment's gateway IP on the destination Tier-1 with a /32 mask, and adds a /32 static route for each MON-enabled VM on that segment. The Tier-1 can then route between cloud segments locally. Broadcom notes that those VM routes are not advertised to the on-premises environment.

East-west traffic in the cloud with and without MON Left panel, without MON: VM A on an extended network sends traffic up the NE tunnel to the on-premises source gateway, which routes it back over the WAN to the cloud Tier-1 and on to VM B on a cloud segment. Right panel, with MON: HCX adds a /32 route for VM A on the cloud Tier-1, so VM A reaches VM B through the Tier-1 directly and the source gateway is not in the path. Without MON Source gateway on premises Cloud Tier-1 VM A extended net VM B cloud segment With MON Source gateway not in the path HCX adds a /32 route for VM A on the T1 Cloud Tier-1 VM A extended net VM B cloud segment 1. NE tunnel 2. back over WAN
Figure 2. Tromboning without MON, and the local path MON creates on the destination Tier-1. Source: Broadcom HCX 9.1 user guide, About Mobility Optimized Networking.

Which VMs get MON, and when

Enabling MON on a network does not move every VM's routing at once. The Target Router Location for each VM decides whether it uses the cloud Tier-1 or the on-premises router, and the default depends on how the VM arrived.

How the VM got thereTarget Router Location after MON is onWhat you do
HCX Bulk MigrationCloud Tier-1, set automaticallyNothing
HCX vMotion or RAVOn-premises routerSet the target manually in the UI or API
On the segment before MON was enabledOn-premises routerSet the target manually
Any VM, to turn MON offChoose On-premisesChange Target Router Location back

Table 3. MON router location by migration type. Source: Broadcom HCX 9.1, Configuring Mobility Optimized Networking.

Each MON-enabled VM needs VMware Tools, and Broadcom's MON troubleshooting KB explains why: HCX learns the VM's IP from the Tools report to vCenter, and if the IP is missing or outside the extended subnet the /32 route is not programmed and traffic goes back through the NE to on-premises. Secondary IP addresses and two vNICs on the same extended segment are not supported, and IPv6 traffic is not optimised.

MON requirements and limits

  • The destination needs an NSX Tier-1 connected to a Tier-0 before you enable MON, and the extension must terminate on that Tier-1.
  • MON is not supported when the destination is an NSX VPC subnet, an NSX VLAN segment or a vSphere DVPG. Plan MON only for overlay destinations.
  • The segment's default gateway IP and its DHCP server IP must be different addresses.
  • MON optimises traffic in the destination only. Traffic between a MON-enabled VM and a non-MON VM on extended networks is not optimised.
  • For a VM with several NICs on several MON-enabled networks, changing Target Router Location applies to all its interfaces.
  • Inbound traffic from the internet to MON-enabled VMs is not supported; outbound internet traffic through the Tier-0 needs NAT, per KB 321599.

Broadcom KB 88401 publishes MON scale figures per HCX Cloud Manager for HCX 4.5 and later: 400 MON-enabled VMs and 100 MON-enabled extensions with the default 4 vCPU and 12 GB manager, rising to 1,000 VMs when the manager is scaled up to 8 vCPU and 24 GB. That KB was written for the 4.x line. We have not found a 9.1-specific MON scale table, so check with Broadcom before you design near those numbers.

MON policy routes: the setting that causes asymmetric routing

Policy routes decide what happens to traffic from a MON-enabled VM whose destination is outside the Tier-1. They only apply once the VM's gateway has moved to the cloud. Broadcom KB 395509 spells out the logic, which is the reverse of what most people expect:

Match with Allow

  • Sent to the on-premises gateway through the NE tunnel
  • HCX creates Allow entries for 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16 after the first MON extension
  • So by default, all private destinations go back on-premises

Match with Deny, or no match

  • Sent to the destination Tier-0 and routed in the cloud
  • Internet-bound traffic falls here by default
  • The most specific prefix wins, so a /32 Deny beats a /8 Allow

Asymmetry comes in when outbound traffic leaves through the cloud Tier-0 but the rest of the network still routes the return traffic to the on-premises gateway. KB 439275 describes exactly this: VMs on VMware Cloud on AWS with MON lost internet access, and an endpoint security agent broke, because the core network sent replies to the on-premises router. Broadcom's three options were to send that traffic back through the tunnel, fix return routing in the core, or cut the gateway over. Stateful firewalls in the return path are what usually turns this from slow into broken.

Our recommendation: write down which destinations should leave locally before you enable MON, and add Deny entries for those only. Confirm with the network team that return routing for those flows lands in the cloud, then test with traceroute from a migrated VM. Broadcom also notes that after an NE HA failover, policy-routed MON traffic takes longer to recover than other traffic.

MTU, fragmentation and TCP MSS for extended networks

Every frame from an extended network gets new outer headers at the NE: an outer IP header, UDP 4500 and the IPsec overhead. A guest that sends a full 1500-byte packet therefore needs a larger packet on the underlay, or the NE has to fragment it, or the path drops it. Broadcom's performance tuning KB 389281 tells you to make the underlay MTU consistent and to allow for IPsec and encryption overhead.

How a guest packet is carried across the HCX tunnel The underlay path has an MTU set by its smallest hop. Inside it, the NE adds an outer IP header, UDP 4500 and IPsec headers in front of the guest frame from the extended network. TCP MSS clamping makes guest TCP segments smaller so the whole encapsulated packet fits the path MTU. If it does not fit, it is fragmented or dropped. Why a 1500-byte guest packet needs headroom Underlay path: the smallest MTU on any hop sets the limit Outer IP, UDP 4500 and IPsec headers added by the NE Guest frame from the extended network Ethernet header plus the guest IP packet TCP MSS clamping shrinks TCP segments so the whole packet fits the path Too big for the path: fragmented by the NE, or dropped if the path or its ICMP is broken
Figure 3. The encapsulation added by the NE, and where TCP MSS clamping helps. Broadcom does not publish a single overhead figure for the enhanced appliances, so size from the tests described below. Sources: Broadcom KBs 383237, 389281 and 436416.

Where the MTU is set

The uplink MTU comes from the HCX uplink network profile. Broadcom KB 383237 says it can be set between 1150 and 9000 bytes and that it governs the tunnels between IX-I and IX-R and between NE-I and NE-R. The local network profiles (management, vMotion, replication) should match the MTU of the matching VMkernel interfaces. Size the uplink to the largest MTU the path supports end to end: the KB's example is a local network at 9000 and a WAN at 1600, giving an uplink profile of up to 1600.

Why mismatches hurt so much

KB 379617 covers the common case where migrations and extended networks run slower than expected. The root cause is usually an MTU mismatch somewhere in the chain. Its example: one appliance has an uplink MTU of 1500 and its peer has 9000, so a packet of around 8960 bytes from the peer gets dropped, because path MTU discovery cannot see that one appliance is limited to 1500. A reverse-direction throughput of a quarter to a third of the forward rate in perftest results is, per Broadcom, abnormal and points to loss, often from fragmentation.

The KB's test sequence from the HCX Manager shell is short:

ccli
list
go <appliance-id>
pmtu
perftest all

pmtu reports the uplink path MTU in both directions and perftest all measures the IPsec tunnel and site throughput. Broadcom points out that PMTU testing will not catch a Layer 2 mismatch, such as an HCX uplink MTU larger than the distributed switch MTU, so check the switch and physical ports as well. The Transport Analytics view in the service mesh also monitors path MTU for every uplink.

TCP MSS clamping

TCP MSS Clamping (older guides call it TCP Flow Conditioning) sits under Traffic Engineering in the service mesh, next to Application Path Resiliency and Generic Receive Offload. The NE adjusts the maximum segment size during the TCP handshake so segments fit through the tunnel without fragmenting. The VMware Cloud Foundation blog's HCX 9.1 service mesh guide says it is enabled by default. It only helps TCP: UDP flows, and anything that sets the don't-fragment bit with large datagrams, still depend on the underlay MTU.

Important: enabling Application Path Resiliency needs UDP source ports 4500 to 4628 and destination port UDP 4500 open on firewalls at both sites, according to KB 389281. Open those before you tick the box, or the extra tunnels will not come up. Our HCX port requirements guide lists the other flows.

How to scale and protect Network Extension

Scaling out

Broadcom's restrictions page sets the ground rules. One NE appliance connects to one distributed switch or one NSX transport zone. For each site pair, a given network goes through exactly one appliance and is limited by that appliance's resources, so adding appliances does not speed up a single extended network. It spreads different networks across more appliances. Microsoft's Azure VMware Solution guidance gives a figure of 4 to 6 Gbps per NE appliance and recommends multiple NE appliances to distribute the load; Broadcom does not publish an equivalent figure for the 9.1 enhanced appliances, so treat it as a guide for legacy appliances in that platform.

KB 389281 lists the tuning options for HCX 4.10 and later. The starting point is to give a very busy VLAN its own NE appliance, then combine that with Generic Receive Offload, Application Path Resiliency (up to eight tunnels per IX and NE), TCP MSS Clamping and more appliances through Network Extension Appliance Scale Out. Microsoft adds that NE traffic is a single heavy flow, which can pin to one NSX Edge core on the destination side, and recommends reserving CPU for the NE and IX appliances and keeping them on different hosts from the NSX Edges.

New appliances do not pick up existing extensions by themselves. KB 392925 says the existing NE does not rebalance automatically. On legacy appliances you unextend and re-extend the network on the new appliance. On enhanced meshes, use the Migrating Network Extensions workflow instead. Our post on sizing IX and NE scale-out goes deeper on throughput planning.

Network Extension high availability

NE HA uses four appliances: two at the source and two at the destination, forming one HA group. Each source appliance pairs with one remote appliance, the pairs negotiate Active and Standby roles, and HCX uses DRS host anti-affinity to keep the Active and Standby appliances on separate hosts. A loss of heartbeats, or a fault detected on the Active appliance, makes the Standby pair take over and the group shows Degraded. There is no pre-emption and no automatic failback.

What HA protects

One appliance failure per group

Losing more than one appliance in the group at the same time disrupts the service. HA gives appliance resilience only; uplink resilience comes from Application Path Resiliency or multiple uplinks.

How to enable it

Spare appliances, no live extensions

Set the NE scale-out count to at least two. Appliances you activate HA on must have no networks extended over them, so build HA groups before you extend, then pick the HA group as the extension appliance.

Upgrades

Failover, not In-Service

In-Service mode is not available for HA groups. The Standby pair is upgraded, a switchover makes it Active, then the former Active pair is upgraded.

Limits

One Active, one Standby

KB 392925 says an HA group holds exactly one Active and one Standby appliance; multi-Active groups are not supported. HA also does not support Storage DRS anti-affinity.

For standalone appliances, In-Service mode is the way to upgrade or redeploy without a long outage. HCX builds the new appliances before removing the old ones, and Broadcom says the switchover takes a few seconds or less. Standard mode reuses the same IPs, so the old appliances are disconnected first, and extended networks can be disrupted for 30 seconds or more. In-Service needs the tunnels Up, management reachability, and no stretched DVPG with "Ephemeral - no binding".

How to cut over an extended network

Extensions are meant to last for the migration period. The cutover moves the default gateway to the destination and removes the extension. HCX does half of this for you.

When you unextend, the cloud segment is left disconnected from the cloud gateway by default, so the destination router does not start advertising a route that conflicts with the source network. If you tick Connect cloud network to cloud edge gateway after unextending, HCX connects the segment to the Tier-1 (or the VPC gateway for a VPC subnet), and with dynamic routing the subnet is advertised from the cloud side. What HCX does not do is touch your source gateway. KB 420204 states that the source Tier-1 connection stays active after unextend and that it is the user's responsibility to perform the gateway cutover and disable the gateway on premises, whether it is a physical router or an NSX Edge.

Gateway cutover sequence for an HCX extended network Six steps in order: 1, migrate every VM off the source network; 2, move DHCP, DNS and NTP dependencies to the destination; 3, check routing at the destination Tier-1 and Tier-0; 4, shut the source gateway interface and withdraw its route; 5, unextend with the connect cloud network to cloud edge gateway option; 6, test routing, ARP and DNS from the migrated VMs. Gateway cutover: our sequence 1. Migrate every VM off the source network, or accept they move L3 2. Move services DHCP, DNS and NTP to destination servers 3. Check routing Tier-1, Tier-0, BGP or OSPF at destination 4. Shut source gateway interface down and route withdrawn 5. Unextend with "connect cloud network" ticked 6. Test from VMs routing, ARP, DNS, north-south, east-west
Figure 4. The gateway cutover sequence we use. Steps 4 and 5 sit in the same short window; the outage for the subnet is the time between them. Based on Broadcom KBs 420204 and 439275 and the HCX 9.1 Removing a Network Extension page.

Broadcom's own gateway cutover outline in KB 439275 matches steps 1 to 3 and 5: migrate every VM that depends on the gateway, reconfigure anything using source-site DHCP, DNS or NTP, confirm BGP or OSPF on the destination, then in the source HCX Manager go to Services, Network Extension, expand the network, tick Connect cloud network to cloud edge gateway after unextending and click Unextend. Afterwards, check in the destination NSX Manager under Networking, Segments that Gateway Connectivity shows Enabled, and test north-south and east-west routing.

The ordering of steps 4 and 5 is our choice. Shutting the source gateway first means the subnet is never advertised from both sides at once. The cost is a short window where the extended VMs cannot leave the subnet. Keep that window to minutes by having both changes ready in the same change ticket.

ARP, DNS and the things that break after cutover

  • Gateway MAC change. With MON off, migrated VMs have been using the source router's MAC address as their gateway. After cutover the gateway IP answers from the destination router's MAC, so VMs need fresh ARP entries. Most guests re-ARP quickly, but appliances and some older operating systems hold entries longer. Our recommendation is to test a sample of each OS type in the window.
  • DHCP. The removal page warns that VMs relying on a source-site DHCP server, or on statically set DNS or NTP at the source, can lose those services. Plan DHCP relay or a destination server before the window.
  • Firewall and ACLs. Any rules on the source router interface for that subnet need an equivalent at the destination Tier-1 or distributed firewall.
  • Stragglers. Any VM still at the source on that VLAN is cut off from the migrated VMs once the bridge goes. Unextending stops cross-site communication on that network.

Rolling back

Broadcom does not publish a rollback procedure for a gateway cutover, so this is from our own runbooks. Before you unextend, write down the original extension settings (appliance, gateway CIDR, first hop router, MON state). If the cutover fails, disconnect the cloud segment from the Tier-1, bring the source gateway interface back, and extend the network again with the same settings. Note Broadcom's warning that connecting an extension to an existing destination segment with the same gateway IP and prefix disconnects the NSX router interface, which disrupts gateway traffic on the destination if that interface is in service. That is what you want during a rollback, but it is also why you should never re-extend a network whose cloud gateway is already carrying live traffic without a window.

If the normal unextend fails, the UI offers Force Unextend. Broadcom lists it for an appliance stuck in a state that blocks unextend, source components that no longer exist, or a source site that is offline; in that last case you unextend from the cloud side so the destination router stays connected for migrated VMs.

  • Every VM on the network migrated, or its owner has signed off a re-IP
  • DHCP, DNS and NTP for the subnet provided at the destination
  • Destination Tier-1 and Tier-0 routing and advertisement checked
  • Source gateway shutdown and route withdrawal scripted and approved
  • Original extension settings recorded for rollback
  • Test plan covering ARP, north-south and east-west traffic per OS type
  • MON policy routes reviewed, since they stop mattering once the gateway is local

Common Network Extension mistakes

These are the problems we see most often on HCX projects, each tied to a documented cause.

MTU

Uplink profile bigger than the path

A 9000-byte uplink profile over a 1500-byte WAN, or peers with different uplink MTUs. Run pmtu and perftest all before blaming the application.

vSphere

Port group security left at Reject

MAC address changes and Forged transmits must be Accept on the source port group, or return traffic is dropped while ARP looks fine.

MON

Assuming vMotioned VMs use MON

Only Bulk-migrated VMs switch to the cloud router automatically. vMotion and RAV VMs keep the on-premises gateway until you change them.

MON

Default policy routes left untouched

The RFC 1918 Allow entries send private traffic back on-premises. Adding Deny entries without fixing return routing creates asymmetric paths.

Cutover

Expecting HCX to shut the source gateway

It does not. Two live gateways for one subnet after unextend means split routing and hard-to-trace outages.

Upgrades

Upgrading the mesh in Standard mode

Extended networks can drop for 30 seconds or more. KB 407528 also describes cloud VMs losing their gateway after a mesh upgrade, fixed by unextending and re-extending.

Keeping MACs and IPs through cutover has its own traps with DHCP reservations and licence servers. Our post on Retain MAC, Network Extension and DHCP at cutover covers them, and the reserved MAC range error explains one validation failure that shows up on extended networks.

Extended networks still live months later?

Plan the gateway cutovers and switch off the bridge

We run cutover windows for HCX estates: dependency checks, MON clean-up, routing changes and rollback plans, so you can retire your Network Extension appliances on schedule.

Book a cutover review

For the wider migration picture, see what's new in VMware HCX 9.1, the 70-point HCX migration checklist and our guide to which HCX log to read when a migration fails.

Frequently Asked Questions about HCX Network Extension

Does HCX Network Extension move the default gateway to the destination?

No. The default gateway for an extended network stays at the source site, and migrated VMs keep routing through it until you enable MON for them or perform a gateway cutover and unextend the network.

How many networks can one HCX Network Extension appliance carry?

Up to 128 on a single vNIC with HCX 9.1 enhanced appliances in multiple-extensions-per-vNIC mode. Broadcom's requirements page says legacy appliances need an additional appliance beyond 8 networks, and enhanced appliances in single-extension mode beyond 7.

Does adding more NE appliances make one extended network faster?

No. Broadcom states that for each site pair a network goes through only one appliance and is limited by that appliance's resources. More appliances let you spread different networks across them.

Can I use MON when extending to a VLAN segment or a distributed port group?

No. HCX 9.1 does not support MON when the destination is an NSX VLAN segment, an NSX VPC subnet or a vSphere DVPG. MON needs an NSX overlay segment terminating on a Tier-1 that is connected to a Tier-0.

What MTU should the HCX uplink network profile use?

The largest MTU the underlay supports end to end, within the 1150 to 9000 byte range Broadcom allows. Both appliances in a pair should use the same value, and you should confirm it with the pmtu test from the HCX Manager CLI.

Does unextending a network shut down the on-premises gateway?

No. HCX can connect the destination segment to the cloud gateway if you tick the option, but Broadcom KB 420204 says disabling the source gateway, whether a physical router or an NSX Edge, is the user's responsibility.

How many appliances does Network Extension HA need?

Four: two at the source and two at the destination, forming one HA group with an Active pair and a Standby pair. HA tolerates one appliance failure in the group and does not fail back automatically.

Is TCP MSS clamping enabled by default for Network Extension?

Yes. VMware's HCX 9.1 service mesh guide states that TCP MSS Clamping is enabled by default under Traffic Engineering. It reduces fragmentation for TCP traffic only.

Sources

Written by the Consult Circle VMware HCX migration team. Last reviewed 3 November 2026.

Share this article: