VMware Cloud Director Common Errors and Known Issues: Symptoms, Causes and Workarounds

CLOUD DIRECTOR TROUBLESHOOTING

A Consult Circle field reference to the Cloud Director failures that actually happen on provider platforms, organised by where they surface, with the checks that resolve each one.

Cloud Director fails in a fairly small number of ways, which is the useful thing about it. The messages are frequently generic, the logs are verbose, and the symptom usually appears several layers away from the cause, but the underlying set of problems is finite. Most of them come down to the transfer share, the database, task handling, placement, or certificates.

This reference is organised by where the problem surfaces rather than by component, because that is how you meet them: the cells will not start, the interface is unavailable, a task is hanging, a tenant cannot power something on. Each section gives the symptom, the causes worth checking in order of likelihood, and what to do.

Where a Broadcom knowledge base article covers a pattern in detail, it is referenced by number. Exact messages and file paths vary by version, so treat the shapes below as recognition patterns and confirm specifics against the documentation for the release you are running.

What this guide covers

• Cells that will not start, and the transfer share
• Service unavailable and interface problems
• Tasks that hang and never complete
• Upload and catalogue problems
• Placement, resource and relocate failures
• API and authentication errors
• Tenant-visible operation failures
• Stranded objects and deletion problems, plus prevention

Cells That Will Not Start

The first place to look is always the cell log, which records application initialisation and will usually name the subsystem that failed. Two causes dominate, and they have dominated for well over a decade.

Transfer spooling area not writable

The cell initialises and then stops, reporting that the transfer spooling area is not writable. This is the single most common cause of a cell failing to start and it is almost always the NFS mount rather than the application.

• Confirm the share is actually mounted on the cell rather than assuming it from the configuration file. A missing or inaccessible entry in the mount table is the usual finding.
• Check ownership and permissions. Cloud Director creates a dedicated user and group, and when the transfer directory is remounted over NFS both the user and group identifiers must be set correctly on the mount point. Setting only one of them will break transfer space validation, and this needs to apply recursively to sub-directories.
• Confirm the NFS server itself is healthy and reachable from every cell, not only from the one you are logged into.
• Where the datastore hosting the cells or the NFS server has had an outage, the cells may have started before the storage recovered. Restarting the Cloud Director service after the infrastructure is confirmed healthy usually resolves it.

Broadcom cover this pattern, including the ownership requirements, in KB 345742. It is worth reading once properly rather than repeatedly rediscovering the permissions detail.

Database connectivity

The other dominant cause. The cell cannot reach the database, or the database is up but cannot write. Two variants are worth distinguishing: the database service being down, which is obvious once you look, and table spaces having filled with no growth permitted, which is not obvious at all and presents as intermittent failure before it presents as an outage.

Involve whoever administers the database rather than guessing, then restart the Cloud Director service once the database is confirmed healthy. On appliance deployments, check the state of the embedded database cluster rather than assuming the primary is where you left it.

Appliance deployment failures

On appliance-based deployments, cell startup can fail because of the state of the shared transfer storage at deployment time. Before deploying a primary appliance the shared transfer storage must not already be populated, and before deploying a standby or additional cell it must contain the expected response file from the primary. Deploying in the wrong order, or reusing a share from a previous attempt, produces failures that look like application faults.

Also check the cluster failover mode after adding cells. New cells default to manual failover, and if the mode is inconsistent across the cluster it becomes indeterminate, which can lead to inconsistent state between nodes and cells following an old primary.

Service Unavailable and Interface Problems

A 503 from the interface with a Jetty error page generally means the application has not fully started rather than that the web layer is broken. Work backwards to the cell startup checks above, because the cause is nearly always the transfer share or the database.

Where the interface is available but slow, a common contributor on long-lived platforms is the size of the audit trail table. Provider estates accumulate audit records for years, and the effect on interface performance is gradual enough that it is rarely attributed correctly. Reducing it is a documented maintenance activity and worth scheduling rather than waiting for complaints.

When diagnosing anything interface-related on a load-balanced deployment, connect directly to a single cell to eliminate the load balancer from the picture. Behaviour that differs between the balanced address and a direct connection tells you where to look next.

Tasks That Hang and Never Complete

A task sitting in running that cannot be stopped from the interface is one of the more disruptive Cloud Director symptoms, because it blocks subsequent operations on the object and tenants can see it.

The most common underlying cause is the NFS transfer share becoming unmounted or unavailable on one or more cells, frequently after an NFS server problem. The task was accepted, the cell needs the share to proceed, and it waits. Broadcom document this pattern in KB 372537, and the resolution is to check the mount state on each cell and remount where necessary.

• Check the mount state on every cell rather than the one you happen to be on, since a single affected cell produces intermittent symptoms that look random.
• Confirm the NFS server is healthy before remounting, or the problem returns within hours.
• After remounting, expect a backlog of queued work to move. Give it time before concluding the fix did not work.
• Where tasks remain stuck after the share is healthy, escalate rather than manipulating them directly. Task state lives in the database and editing it by hand is a good way to turn a stuck task into a corrupted object.

How automation should recognise and report a stuck task, rather than waiting on it indefinitely, is covered in the Cloud Director API for day-to-day operations.

Upload and Catalogue Problems

Uploads stuck pending are among the more common provider support tickets, and the cause is usually environmental rather than a fault in the platform.

• Test with a small file first. If a small ISO completes and a large OVA does not, the problem is throughput or timeout rather than function.
• Measure latency between the uploading client and the platform with a capture rather than by impression, since upload behaviour is sensitive to it.
• Bypass the load balancer and connect to a single cell to establish whether the balancer is contributing.
• Deploy a test client on the same network as the platform. If uploads succeed from there and fail from the tenant, the answer is the path rather than the platform.
• Check transfer share capacity, since uploads land there first and a full share produces failures that look like network problems.

Broadcom cover this specific area in KB 386675, including the load balancer bypass method and the audit trail maintenance that helps interface responsiveness while you investigate.

Placement, Resource and Relocate Failures

No feasible placement solution

An operation fails reporting that no suitable resource was found, with candidate resource pools eliminated for not meeting a compute requirement. This surfaces most often on power on or deployment and it means the placement engine could not find anywhere that satisfies the constraints, not that the platform is out of capacity in general.

• Check the requested configuration against what the target clusters can actually provide. A request for more vCPUs than any host offers will eliminate every candidate.
• Check storage policy availability at the target, since a policy that is not present on candidate clusters removes them silently.
• Check any configured limits. Limits set through the cell management tool have, in some releases, produced placement behaviour that repeatedly relocated workloads across clusters before failing, and removing the configured limits resolved it.
• On multi-cluster estates with shared storage policies, the placement engine considers a wider candidate set than people expect, which makes the elimination list long and the actual constraint harder to spot.

Relocate task timeouts

Changing a VM placement or storage policy, or triggering a storage migration on the back end, produces a relocate workflow with a configurable timeout. On estates where relocations legitimately take longer than the default, the workflow times out even though the underlying operation is progressing.

The timeout is exposed as a configuration property adjustable through the cell management tool. Raising it is appropriate where your storage genuinely takes longer than the default allows. Raising it to mask a storage performance problem is not, and it is worth being honest about which situation you are in.

API and Authentication Errors

Table 1 - API and authentication failures, and what actually resolves them.

Certificate expiry deserves particular attention on provider platforms because it affects tenants directly and it is entirely predictable. Track expiry dates for the cells and any endpoint certificate as a scheduled activity rather than relying on somebody remembering.

Tenant-Visible Operation Failures

These are the ones that generate support tickets, and most have straightforward explanations that your first line can be trained to give.

• Operation rejected because the vApp is busy. Another task is in flight on the object. Cloud Director serialises operations per object, so the answer is to wait rather than to retry immediately, and tenants should be told that rather than left to guess.
• Cannot delete a vApp. It must be undeployed first. The error frequently reads as a permissions problem to a tenant, which sends them to your service desk rather than to the undeploy button.
• Disk resize rejected. Disks can grow but not shrink, and a change requiring a powered-off workload will fail where hot add was not enabled on the template. Both are worth validating in any self-service portal before submission.
• Guest shows old configuration after a change. Customisation is pending and will apply at the next power cycle. This is expected behaviour rather than a fault, and it accounts for a steady trickle of tickets.
• Graceful shutdown does not work. Tools are absent or not running in the guest. The tenant needs to know this is a guest condition rather than a platform one.
• Console access fails. Usually certificate trust or a network path to the console proxy rather than anything wrong with the workload.

Stranded Objects and Deletion Problems

Decommissioning a tenant is where Cloud Director estates reveal their history. Organisations and virtual data centres that will not delete cleanly are common, particularly on platforms that have been running for years and have been through several upgrades.

• Work through the deletion order deliberately: workloads undeployed and removed, then networks, then the virtual data centre, then the organisation. Attempting the outer object first produces errors that describe the symptom rather than the blocker.
• Objects invisible in the interface but present in the database are a recognised class of problem, typically left by an earlier failure. These need proper investigation rather than direct database manipulation.
• Where a deletion stalls in an intermediate state, resist the temptation to force it. A stranded object is inconvenient; a partially deleted one with dangling references is considerably worse.
• Record what you find. On a platform being migrated to VCF Automation, stranded objects are exactly the sort of thing an assessment should surface, and they are cheaper to resolve before a migration than during one.

That surfacing is one of the assessment items in the complete VCD to VCF Automation migration checklist.

A Diagnostic Sequence That Works

Applied in this order, most Cloud Director problems either resolve or become clearly escalatable within the hour.

1. Establish the timestamp and what changed. Most platform problems correlate with something outside Cloud Director: a storage event, a network change, a certificate expiry, a database maintenance window.
2. Check the transfer share mount state and writability on every cell.
3. Check database reachability and headroom, including table space growth.
4. Check certificate validity on the cells and any endpoint.
5. Check the cell log for initialisation state and the named subsystem where startup failed.
6. Establish whether the problem affects all cells or one, since a single affected cell produces symptoms that look intermittent and random.
7. Bypass the load balancer and test against a single cell directly.
8. For a task-level problem, follow the task rather than the object, and check whether other tasks on other objects are also affected.
9. Escalate with the timestamps, the affected cells and what you have already eliminated, rather than only the symptom.

Prevention

• Transfer share monitored for availability and free capacity, with alerting rather than periodic checking
• NFS mount entries verified as persistent across reboot on every cell
• Transfer share ownership and permissions verified recursively after any remount
• Database growth monitored, with table space expansion permitted rather than capped and forgotten
• Certificate expiry tracked as a scheduled activity for cells and endpoints
• Audit trail size managed on a schedule rather than when the interface becomes slow
• Storage policy availability verified consistently across all candidate clusters
• Relocate and workflow timeouts reviewed against what your storage actually delivers
• Cell cluster failover mode confirmed consistent after any cell is added
• Deletion runbook written for tenant decommissioning, with the correct object order
• A test client on the same network as the platform, available for isolating upload and latency problems
• Known issues reviewed against your specific version before every upgrade, since several notable Cloud Director defects have been version-specific

Frequently Asked Questions

Our cells will not start. Where do we look first?

The cell log, which records application initialisation and names the subsystem that failed. The two dominant causes are the transfer spooling area not being writable, usually an NFS mount or permissions problem, and database connectivity, either the service being down or table spaces having filled with no growth permitted.

Why do tasks hang in running and refuse to stop?

Most often because the NFS transfer share has become unmounted or unavailable on one or more cells, frequently after an NFS server problem. Check the mount state on every cell, not just the one you are logged into, and remount where necessary once the server is confirmed healthy. Broadcom document this in KB 372537.

Uploads are stuck pending. Is this a platform fault?

Usually not. Test with a small file to distinguish a throughput problem from a functional one, measure latency properly, bypass the load balancer to eliminate it, and check transfer share capacity. A test client on the same network as the platform quickly establishes whether the problem is the path or the platform.

What does a placement failure actually mean?

That the placement engine could not find a target satisfying the constraints, not that the platform is out of capacity generally. Check the requested configuration against what candidate clusters provide, check storage policy availability at the target, and check any configured limits, which have produced unexpected placement behaviour in some releases.

Should we edit stuck tasks in the database?

No. Task state lives in the database and manipulating it directly is a reliable way to convert a stuck task into a corrupted object with dangling references. Resolve the underlying cause, and escalate if the task remains stuck once the platform is healthy.

The interface has become slow over time. What causes that?

On long-lived provider platforms, audit trail table growth is a common contributor and the degradation is gradual enough that it is rarely attributed correctly. Managing it is a documented maintenance activity worth scheduling rather than waiting for tenant complaints.

How does this change once we migrate to VCF Automation?

Substantially, because the platform architecture is different. VCF Automation runs as containerised services on a Kubernetes cluster rather than as cells with a shared transfer share, so the failure modes above largely do not apply and a new set does. Diagnostic habits built around cells and NFS will need rebuilding.

Can Consult Circle help with a platform that keeps breaking?

Yes, and it is a common way we are brought in. The usual first step is a health review against the failure patterns above, which frequently identifies a recurring cause within a day. Where the underlying answer is that the platform has reached the end of what it can sensibly carry, that conversation tends to lead into the VCF Automation migration.

Where Consult Circle Fits

None of these problems are exotic. What makes them expensive is that recognising one takes ten minutes with prior exposure and the better part of two days without it, and provider platforms tend to produce them at inconvenient hours.

We run Cloud Director health reviews, support providers through recurring platform issues, and work on the migration to VCF Automation through our VMware Cloud Director services. Frequently the same engagement covers both, because a platform that keeps producing the same failures is usually also the platform whose migration is overdue.

Related guides in this series

• VMware Cloud Director to VCF Automation Migration: The Complete Guide for Service Providers
• VMware Cloud Director End of Life: What the VCF Automation Transition Means for Your Provider Business
• VCF Automation for VMware Cloud Director Administrators: What Carries Over and What Does Not
• Mapping VMware Cloud Director Constructs to VCF Automation
• Using the VCF Automation Migration Tool: Environment Assessment Through to Tenant Cutover
• VMware Cloud Director Migration Services: Scoping and Pricing a VCD to VCF Automation Programme
• Migrating Tenants Off Cloud Director: Communication, Sequencing and Cutover
• VMware Cloud Director 10.x vs VCF Automation 9.1: A Feature Parity Comparison for Service Providers
• The Complete VCD to VCF Automation Migration Checklist: 72 Checks for Service Providers
• The VMware Cloud Director API for Day-to-Day Operations: The Calls You Will Actually Use