Sizing an HCX Migration: IX and NE Scale-Out, Concurrent Migrations and Throughput

VMware HCX migration planning

To size a VMware HCX 9.1 migration, you size three things separately: the HCX Manager (which sets how many Bulk and Replication Assisted vMotion migrations can run at once), the Interconnect (IX) appliances (one per service mesh, each with its own concurrency and throughput ceiling) and the Network Extension (NE) appliances (one per extended VLAN or segment, plus standby appliances for HA). Wave duration then comes from the data each wave moves divided by the throughput you have measured with perftest, never the nominal link speed.

The limits below are for VCF Operations HCX 9.1.0, from the Broadcom Configuration Maximums site, the HCX 9.1 user guide and the Broadcom scale-out KBs. Figures that come only from an older KB or from Microsoft's Azure VMware Solution guidance are labelled, and so are our own opinions.

What decides the size of an HCX deployment?

Most sizing mistakes we see come from treating HCX as one box with one throughput number. The HCX Manager (Connector at the source, Cloud Manager at the destination) holds migration state, and its form factor decides how many Bulk and RAV migrations it can track at once. Each IX appliance moves migration data across the WAN with its own concurrency limit and throughput ceiling. NE appliances bridge Layer 2 networks and carry live traffic for VMs that have already moved. Under all of them sit the underlay network and the hosts and storage at both ends, which have to keep up with replication reads and writes. A wave plan has to respect the tightest of these at any moment, so we size each one and then check the plan against all of them together.

HCX Manager and appliance resource requirements

The HCX 9.1 user guide (Administering VCF Operations HCX, System Requirements) lists the virtual hardware for every appliance. Broadcom states that the same hardware applies to each additional appliance when you scale out, so multiply the IX and NE rows by the number you plan to deploy at each site.

Table 1. Appliance virtual hardware from the HCX 9.1 System Requirements page. The 9.0 page lists 2 GB of disk for IX and NE, so check the guide for the version you deploy.

Three notes on the same page affect capacity planning. Storage per appliance doubles during upgrade and redeploy, because HCX creates a second appliance for the operation. When you deploy the IX with WAN optimization, Broadcom's best practice is 100% CPU and memory reservation. And NE standby appliances in an HA group count towards the hardware total, so an HA-protected segment costs two NE appliances at each site.

Microsoft's Azure VMware Solution guidance notes that IX and NE appliances have no CPU reservation by default, recommends reserving CPU for them, and recommends keeping NE appliances and NSX Edge VMs on separate hosts. In our experience both points apply to on-premises VCF destinations too.

HCX Manager form factors and migration concurrency

KB 373010 ("Bulk Migration and Replication Assisted vMotion (RAV) scalability guide", HCX 4.10 or later) is the reference for scaling the manager. The limit applies per HCX Manager, regardless of how many site pairs, service meshes or IX appliances sit under it.

Table 2. HCX Manager form factors in KB 373010. The KB says 16 vCPU was tested for Large but recommends 32 where available.

The scale-up is a change to both the Connector and the Cloud Manager. It needs a maintenance window with no active migrations or configuration workflows, more vCPU, memory and disk on the manager VM, and a script run as root to set the form factor:

cd /usr/local/hcx/sbin
sh upscale_configs.sh medium    # or: sh upscale_configs.sh large
systemctl status app-engine
cat /common/scale-form-factor

The KB also warns that scale settings are not preserved across HCX upgrades, so add a check to your upgrade runbook. On Azure VMware Solution you do not run the script yourself: Microsoft provides the Set-HcxScaledCpuAndMemorySetting run command, which scales the Cloud Manager to Medium (8 vCPU, 24 GB, 120 GB) or Large (32 vCPU, 48 GB, 300 GB) and restarts it during the change.

Which HCX configuration maximums matter for wave planning?

The figures below come from configmax.broadcom.com for VCF Operations HCX 9.1.0 (Categories: HCX Sites and Managed Services, Migrations, Network Extension, Migration Centric Virtual Machine Limits). Recheck them for your exact release before you commit a plan, because Broadcom updates the site between versions.

Table 3. Selected VCF Operations HCX 9.1.0 configuration maximums from configmax.broadcom.com.

Two of these change how you build waves. First, Bulk and RAV share one pool. A wave of 150 RAV migrations and a wave of 150 Bulk migrations on the same manager already fill the default limit of 300, and configmax counts migrations "in any state", so a migration sitting in delta sync for a week waiting for its switchover window still uses a slot. Second, HCX vMotion is serial per service mesh. If a wave depends on HCX vMotion for zero-downtime moves, its duration is roughly the sum of every VM's vMotion time on that mesh, and the only way to run two at once is a second mesh.

RAV behaves differently at the end of the process. KB 373010 states that the concurrency limit covers the initial and delta sync of Bulk and RAV, while RAV switchovers are relocated one at a time. Several service meshes can run switchovers in parallel, but each mesh pair works through its own queue in sequence. If the business wants 40 VMs switched over inside a two-hour window, measure how long one switchover takes on a pilot and work out how many meshes you need to fit 40 of them.

For background on how each migration type moves data, see our guides to HCX Bulk Migration, Replication Assisted vMotion and HCX vMotion.

How do you scale out HCX Interconnect for migrations?

The 9.1 System Requirements page describes the default: one IX appliance serves every source and destination cluster in the service mesh, and you scale out by deploying additional service meshes and selecting them during migrations. Configmax adds the detail that an extra service mesh (with its own IX) can be created for each unique source and destination cluster pair, up to the 64-mesh maximum.

KB 392928 ("Scaling Interconnect (IX) Appliances for Load Balancing for HCX Migration") describes the procedure. You create additional service meshes in the HCX Manager UI, which deploys one IX pair per mesh, and then when you start a migration you pick the IX pair under Interconnect Options. The KB is explicit that HCX does not load balance migration traffic across additional IX appliances. The balancing is manual: you decide which wave or which VMs go through which mesh.

KB 442319 (service mesh selection logic, HCX 4.11) explains why that matters. If you leave the mesh unselected, HCX can pick any available mesh and may keep picking the same one for concurrent migrations. Broadcom's advice is to put the mesh choice into your automation or PowerCLI scripts so they alternate between meshes. It also recommends a dedicated service mesh for Network Extension and a separate one for IX migrations, so high-volume NE traffic does not compete with migration traffic on the same path.

The throughput ceiling of a single IX

Broadcom has published two throughput figures for one IX appliance, both written against the PhotonOS-era appliances. KB 373010 lists, for the default form factor, a maximum of 1 Gbps per migration workflow and 1.6 Gbps per IX appliance across any number of workflows. KB 416441 (HCX performance on underlay networks above 2 Gbps) describes an architectural boundary of about 2 Gbps for one IX at maximum concurrency, and states that adding CPU or memory to the IX does not raise transfer rates. Its recommendation for faster links is service mesh scale-out, with each IX on a different ESXi host, plus a check that storage at both ends can absorb the combined rate.

Neither KB gives figures for the enhanced IX appliances that HCX 9.1 builds on the NSX Edge stack. Broadcom's 9.1 release notes say the enhanced architecture performs better but publish no benchmark. Our recommendation: plan on the published 1.6 to 2 Gbps per IX until a perftest on your own enhanced mesh shows otherwise.

How do you scale Network Extension appliances?

NE scale-out happens inside the service mesh. The Service Mesh setting "Network Extension Appliance Scale Out" defaults to 1, and you raise it to deploy more NE appliances. The compute profile at each site must allow at least as many NE appliances as the mesh asks for. The 9.1 requirements page lets you deploy multiple NE appliances per vSphere Distributed Switch or NSX Transport Zone, and the 9.0 page states the per-segment rule directly: at most one NE per VLAN or NSX segment without HA, and at most two (one Active, one Standby) with HA. The NE count in the mesh must equal the total of standalone, HA Active and HA Standby appliances.

Configmax gives the sizing figures for 9.1.0: 4 to 6+ Gbps per NE appliance, 850 Mbps to 1.65 Gbps per network flow, 8 network extensions per NE when the destination uses NSX (10 interfaces minus uplink and management), 600 extensions and 128 NE appliances per HCX Manager, and 24 HA groups. Up to 10 NE appliances can be provisioned at once per service mesh. Microsoft's guidance for Azure VMware Solution quotes the same 4 to 6 Gbps per NE and recommends several NE appliances so that flows hash across different NSX Edges and DPDK cores, since a single NE to NE flow can pin to one Edge core.

Adding NE appliances does not move anything already extended. KB 392925 ("HCX Network Extension manual load balancing after Service Mesh scale-out") states that an NE hosting a stretched network does not rebalance when new NE appliances arrive. To use them, you unextend a segment and extend it again, choosing the target NE appliance in the wizard. That is a cutover for every VM on that segment at the destination, so plan it like one. HA groups are always one Active and one Standby, HCX picks the standby, and NE appliances selected for HA must have no networks extended over them. Our Network Extension guide covers MON, MTU and cutover in more detail.

Traffic Engineering options on the service mesh

The Service Mesh wizard has a Traffic Engineering section. In the HCX 4.10 user guide it lists Application Path Resiliency (APR), TCP MSS Clamping and Generic Receive Offload (GRO). APR creates up to eight transport tunnels between each IX and NE uplink IP pair, using a random source UDP port between 4500 and 4628 towards destination UDP 4500. It forwards traffic over one tunnel at a time and does not load balance across them, so APR adds resilience and path choice, not bandwidth. KB 396624 explains that a firewall blocking part of that port range leaves the tunnel status "degraded".

The same guide states that a service mesh can use up to three HCX uplinks and that they are not aggregated for throughput. KB 389281 (NE performance tuning, HCX 4.10 or later) lists MSS clamping and GRO as NE tuning options and recommends moving high-traffic VLANs onto a dedicated NE. We have not found a 9.1 page that documents these options on the enhanced appliances, so check the wizard on your own build before you design around them.

What bandwidth does HCX need?

The HCX 9.1 user guide publishes minimum underlay requirements per service. Broadcom states that operation below these values is not supported, that parallel migrations and extensions raise the requirement, and that the table applies to the service mesh appliances, not to HCX Manager traffic.

Table 4. Network underlay minimums from the HCX 9.1 user guide.

These are floors for a service to work at all. They say nothing about how long a wave will take. Set the MTU in the network profile before you deploy the IX appliances, because Broadcom notes that changing it later means redeploying the appliances and resyncing the mesh. Firewall rules for the uplinks are covered in our HCX port requirements guide.

WAN optimization in HCX 9.1

HCX 9.1 brings WAN optimization back, embedded in the Interconnect service mesh configuration. The VCF 9.1 release notes say it is available only on enhanced service meshes and not for OS Assisted Migration or HCX Assisted vMotion. The underlay table above lowers the vMotion and RAV minimum from 250 to 150 Mbps when WAN optimization is used, and Broadcom notes that it reduces bandwidth needs through deduplication and compression. It costs compute: the 9.1 requirements list the WAN Optimization appliance at 16 vCPU, 32 GB and 172 GB, with 100% CPU and memory reservation recommended. Legacy meshes need migrating to the enhanced architecture first, which our PhotonOS to enhanced appliance guide walks through.

In our experience the gain depends on how well the data deduplicates and compresses, so measure a pilot wave with and without it before you change the plan.

How to estimate HCX wave duration from data size and bandwidth

The base formula is simple. Convert the data to megabits and divide by the throughput you measured, in megabits per second:

hours = (TB x 8,000,000) / (measured Mbps x 3,600)

This uses decimal units (1 TB = 8,000,000 megabits). If your inventory reports TiB, multiply by 1.1 first. Use the space actually consumed on disk, not the provisioned size, for thin disks. KB 373010 warns that migration duration depends on many stages and is not a simple function of VM size and bandwidth, so treat the result as the base sync time and add the other stages on top.

A worked example (hypothetical numbers)

The figures in this example are made up to show the method. They are not a benchmark.

Suppose a wave contains 120 VMs with 18 TB of consumed storage, moving by Bulk Migration. The WAN link is 2 Gbps, and perftest on the planned service mesh reports a receiver rate of 1,400 Mbps from source to destination.

• One mesh: 18 x 8,000,000 = 144,000,000 megabits. Divided by 1,400 Mbps that is 102,857 seconds, about 28.6 hours of base sync.
• Concurrency check: 120 migrations fits within 200 per IX and 300 per manager, so one mesh can hold the whole wave.
• Two meshes: a second IX lets the wave use more of the 2 Gbps link. If perftest across both meshes together shows 1,800 Mbps, base sync drops to 144,000,000 / 1,800 = 80,000 seconds, about 22.2 hours.
• Add the rest: delta syncs while the wave waits for its switchover window, the switchover itself, and post-migration checks. Measure each on a pilot and add it to the base figure.

Two meshes do not double the rate, because the link and the source storage are shared, so run the second perftest with both meshes loaded together.

1. Inventory consumed storage per wave

   Use the space used on disk per VM, grouped by application and by switchover window.

2. Measure throughput per service mesh

   Run perftest or Transport Analytics on each mesh, at the time of day the wave will run, and record the receiver rate.

3. Calculate base sync time

   Apply the formula per mesh. If a wave needs more than one IX's worth of throughput or more than 200 concurrent migrations, split it across meshes.

4. Add switchover time

   Switchovers are serial per mesh for RAV and vMotion. Multiply the pilot switchover time by VMs per mesh.

5. Check against every limit

   Confirm the plan stays under the manager limit (300, or 1,000 extended), the per-IX limit of 200 and your host and storage headroom.

How to measure real HCX throughput with perftest

KB 320387 ("Steps to run 'Perftest' in HCX") documents the CLI test. You connect to the HCX Manager over SSH as root, enter the HCX CLI, list the appliances and select one by its ID:

ccli
list
go <appliance_id>
perftest site
perftest site -P <num_of_flows>
perftest ipsec
perftest uplink
perftest wanopt
perftest all

Each test runs in both directions. The KB tells you to use the receiver rate as the one-way throughput figure, since the sender rate is usually higher. Comparing the site, ipsec and uplink results helps locate the bottleneck: a good uplink result with a poor ipsec result points at the tunnel, while a poor uplink result points at the underlay. The KB does not name the ports perftest uses beyond UDP 4500 for the IPsec and FOU tests and says only that firewall rules must allow perftest traffic, so confirm the port list for your version on ports.broadcom.com before you open rules.

The HCX UI has a second tool. Transport Analytics (Infrastructure, Transport Analytics, Test Service Mesh Uplinks) tests each uplink against the thresholds of each enabled service and shows Available Bandwidth, Latency and Loss. For Bulk Migration it also suggests how many simultaneous migrations the underlay can support at the time of the test. Broadcom advises running it on its own, because parallel tests such as a CLI perftest skew the results.

Teams often skip the last question. KB 373010 notes that memory or I/O pressure on the source host can slow initial sync, and suggests a vCenter vMotion of the source VM to a less loaded host, which does not interrupt replication. KB 416441 makes the same point about storage: more meshes only help if the datastores at both ends can take the combined rate. If a wave stalls before any data moves, our guide to Bulk and RAV stuck at 0% base sync covers the usual causes.

Do and avoid when sizing HCX

HCX sizing checklist

• HCX version recorded and limits rechecked on configmax.broadcom.com for that release
• Manager form factor chosen for peak concurrent Bulk and RAV, on both Connector and Cloud Manager
• Number of migration service meshes set by the larger of: concurrency divided by 200, or target throughput divided by measured per-IX throughput
• NE scale-out count covers every extended VLAN, plus standby appliances for HA groups
• Host, CPU, memory and storage capacity booked at both sites for every appliance, doubled during upgrades
• Underlay minimums met for every service in use, and MTU set before IX deployment
• Perftest receiver rate recorded per mesh, at the wave's planned time of day
• Pilot wave timed end to end: base sync, delta sync and switchover

For the full programme view, including dependencies, licensing and rollback, use our 70-point HCX migration checklist and the step-by-step guide to migrating to VCF with HCX. If a wave has already failed, start with which HCX log to read.

Frequently Asked Questions about sizing HCX migrations

How many concurrent migrations can HCX run?

In VCF Operations HCX 9.1.0, configmax lists 300 concurrent Bulk and RAV migrations per HCX Manager, counted together and in any state, and 200 per IX appliance. With the extended scalability procedure in KB 373010 and a Large manager form factor, the manager limit rises to 1,000, while each IX stays at 200.

Does HCX load balance migrations across multiple IX appliances?

No. KB 392928 states that HCX does not load balance migration traffic on additional IX appliances, and KB 442319 says HCX can pick the same service mesh repeatedly if you leave the choice to it. Select the service mesh for each migration under Interconnect Options, or script the choice.

How many HCX vMotion migrations can run at the same time?

One per service mesh. Configmax for HCX 9.1.0 lists HCX vMotion as serial transfer and serial switchover per service mesh, with Cold migrations queued behind it. To run more at once, deploy more service meshes.

How much throughput can one HCX IX appliance deliver?

Broadcom's published figures are 1.6 Gbps per IX at the default form factor (KB 373010) and an architectural boundary of about 2 Gbps per IX at maximum concurrency (KB 416441). Both predate the 9.1 enhanced appliances, for which Broadcom has not published a figure, so measure your own mesh with perftest.

How much throughput does an HCX Network Extension appliance support?

Configmax for HCX 9.1.0 lists 4 to 6+ Gbps per NE appliance and 850 Mbps to 1.65 Gbps per network flow, with results varying by MTU, latency, traffic and CPU. Microsoft's Azure VMware Solution guidance quotes the same 4 to 6 Gbps per appliance.

Do new NE appliances take over existing extended networks automatically?

No. KB 392925 states that an NE hosting a stretched network does not rebalance after scale-out. You unextend the segment and extend it again, choosing the new NE appliance in the wizard.

What is the minimum bandwidth for HCX vMotion?

The HCX 9.1 user guide lists 250 Mbps for HCX vMotion and Replication Assisted vMotion, or 150 Mbps with WAN optimization, with at most 0.1% packet loss and 150 ms latency. Bulk Migration and OSAM need 50 Mbps with up to 1.0% loss.

How do I calculate how long an HCX migration wave will take?

Divide the consumed data, converted to megabits, by the perftest receiver rate in Mbps to get base sync time in seconds, then add delta sync, switchover and validation time measured on a pilot wave. KB 373010 notes that duration is not a simple function of size and bandwidth, so the pilot figures matter more than the formula.

Sources

• Broadcom Configuration Maximums: VCF Operations HCX 9.1.0
• Broadcom TechDocs: VCF Operations HCX 9.1 System Requirements
• Broadcom TechDocs: HCX 9.0 System Requirements
• Broadcom TechDocs: HCX 9.1 Network Underlay Minimum Requirements
• Broadcom: VCF 9.1.0.0 release notes, What's new in VCF Operations
• Broadcom TechDocs: Create a Service Mesh for vSphere-based Site Pairs (HCX 4.10)
• Broadcom TechDocs: Verifying Underlay Network Performance for Service Mesh Uplinks (HCX 4.9)
• Broadcom KB 373010: HCX Bulk Migration and RAV scalability guide (HCX 4.10 or higher)
• Broadcom KB 392928: Scaling Interconnect (IX) Appliances for Load Balancing for HCX Migration
• Broadcom KB 392925: HCX Network Extension manual load balancing after Service Mesh scale-out
• Broadcom KB 442319: HCX Service Mesh Selection Logic and Best Practices for VM Migrations
• Broadcom KB 416441: HCX performance outcomes for underlying network speed over 2 Gbps
• Broadcom KB 389281: HCX Network Extension Performance Tuning
• Broadcom KB 320387: Steps to run 'Perftest' in HCX
• Broadcom KB 396624: Transport Tunnels for HCX IX/NE appliances shows degraded
• Microsoft Learn: NSX Scale and Performance Recommendations for VMware HCX (Azure VMware Solution)
• Microsoft Learn: Use VMware HCX Run Commands (Azure VMware Solution)
• Microsoft Learn: Configure HCX network extension high availability (Azure VMware Solution)

Written by the Consult Circle VMware HCX migration team. Last reviewed 3 November 2026.