Workspace creation always fails no matter what

Since you have already verified the underlying networking, AD connectivity, and the basic AMI prerequisites (UEFI, TPM 2.0), a "Pending to Failure" timeout strongly indicates that the AWS provisioning agent is being silently blocked inside the OS during the final bootstrap phase.

When an instance boots successfully but fails to signal "Ready," AWS Support typically recommends investigating the following common blockers:

• GPO Interference (Most Common): Because the instance joins your on-premises AD during provisioning, Group Policy Objects are applied immediately. If your GPOs restrict PowerShell execution policies (must be RemoteSigned or Bypass), disable WinRM, or modify Local Administrator / User Rights Assignments, the AWS provisioning scripts will crash silently.
  • Action: Create a dedicated testing OU for WorkSpaces, explicitly enable Block Inheritance to prevent any GPOs from applying, and try the launch again.
• Management Network Routing Overlap: WorkSpaces uses a secondary Elastic Network Interface (ENI) for management and health checks, operating in the 198.19.0.0/16 range.
  • Action: Ensure no custom routes, VPN clients, or GPOs are inadvertently routing this IP range back to your corporate network or blackholing it. If the agent cannot communicate over this ENI, the provisioning will time out.
• Windows 11 AppX Provisioned Packages: Win11 ISOs frequently update native AppX packages in the background. If a package update is running or pending during the Sysprep/Provisioning phase, the initialization will hang indefinitely.
  • Action: In your Image Builder pipeline, ensure all provisioned AppX packages are either fully updated or stripped out, and disable automatic Windows Store updates before sealing the AMI.
• BYOP Agent Initialization: Since you are using --ingestion-process BYOL_REGULAR_BYOP, the infrastructure expects a WorkSpaces Core deployment.
  • Action: Double-check that the Omnissa Horizon Agent is installed exactly according to the AWS WorkSpaces Core documentation and that no standard WorkSpaces streaming agents (like PCoIP/WSP) are installed, as they can conflict.

Tip to get logs: Since you cannot extract logs from a failed WorkSpace, launch your custom AMI as a regular EC2 instance in the exact same private subnet. Manually join it to your AD (in the same OU) and check the Windows Event Viewer. Often, you will immediately see GPO or service initialization failures that replicate the WorkSpaces bootstrap failure. Hope this helps !

In short The "deadlock" you are experiencing is simply because the backend is waiting for an agent to report back, but that agent hasn't been installed on your vanilla image yet!

In detail This actually explains exactly why your deployment is failing. The issue is that you are importing a completely "vanilla" image. You cannot import an AMI straight from a clean ISO into WorkSpaces without performing OS-level preparation first.

When I suggested logging into the machine, I didn't mean logging into the failed WorkSpace. I meant you need to launch your current AMI as a standard EC2 instance first to prepare it.

For your workflow to succeed, especially with --ingestion-process BYOL_REGULAR_BYOP (WorkSpaces Core), you are missing steps before the AMI import:

1. Install the Third-Party Agent: Because you are using WorkSpaces Core for Omnissa Horizon, the Horizon Agent must be installed manually on the image before you import it. The "vanilla" ISO does not have this, so the provisioning handshake will always fail.
2. Run the BYOL Checker Script: AWS provides a mandatory WorkSpaces BYOL Checker script that must be run on the Windows machine to validate that all services, firewalls, and AppX packages are in the correct state.
3. Sysprep: The machine must be sysprepped properly before sealing the final AMI. How to fix your workflow:
4. Stop your current Image Builder pipeline after it creates the initial AMI from your ISO.
5. Launch a regular EC2 instance from that new AMI into your public/private subnet.
6. Connect to this EC2 instance via RDP (you may need to retrieve the local Administrator password using the EC2 console and your key pair).
7. Install the Omnissa Horizon Agent.
8. Download and run the AWS WorkSpaces BYOL Checker script. Resolve any errors it flags.
9. Run Sysprep and shut down the EC2 instance.
10. Create a new AMI from this prepared EC2 instance.
11. Run your import-workspace-image command against this new, prepared AMI.

While you mentioned that your test instance has general internet access via a functional NAT GW and that your SGs are open, there are a few AWS-infrastructure layers missing from your description that are common culprits for this exact silent timeout.

The "deadlock" you are seeing almost always happens when the OS boots successfully, but the internal agent lacks either the direct API line-of-sight to the AWS Control Plane or the required IAM/KMS permissions to complete the provisioning handshake. Since AWS advised you to stick to your workflow, I recommend verifying the following points:

1. IAM Service Roles & KMS Policies You mentioned AD permissions, but what about AWS IAM? Did you verify that the AWSServiceRoleForWorkSpaces is correctly configured in your account? Furthermore, if your AMI or the destination WorkSpaces volume is encrypted, the WorkSpaces service needs explicit permissions (like kms:Decrypt) in the KMS Key Policy. If the service role lacks this, the instance boots, but the agent fails to mount or initialize the volume and times out silently.

2. Endpoint Reachability (Bypassing NAT) Even with a NAT Gateway, DNS resolution for internal AWS endpoints can sometimes fail depending on your VPC DHCP Options Set. Can you confirm that the provisioning agent has direct reachability to the AWS APIs? A bulletproof way to isolate this issue is to deploy Interface VPC Endpoints for com.amazonaws.[region].ec2, com.amazonaws.[region].ssm, and com.amazonaws.[region].kms in your private subnets. This forces the agent's API traffic over the internal AWS network instead of relying on the NAT Gateway routing.

3. Software & Agent Registration If you are importing a custom Win11 bundle, you need to ensure that the required AWS software (EC2Launch v2 and the SSM Agent) is not just installed, but capable of registering itself. If the SSM agent cannot reach the Systems Manager endpoints during the initial boot, the orchestration process will never receive the "Ready" signal. Since your underlying networking (AD, Subnets) is fine, focusing on the IAM and Endpoint reachability for the WorkSpaces service itself might finally clear this deadlock.

Hi Florian,

I went ahead and meticulously validated every single infrastructure and OS-layer component you mentioned, but unfortunately, the staging WorkSpace still times out and fails after sitting in a PENDING loop.

Here is the breakdown of the exact findings and steps I took based on your recommendations:

1. IAM Roles & KMS Policies: KMS Key Policies: Verified both our custom KMS key (WorkSpacesPoolDomainSecretKey) and the AWS managed key (aws/workspaces). The key policies are explicitly open. The custom key explicitly includes a statement granting workspaces.amazonaws.com the actions kms:Decrypt and kms:DescribeKey. The base AMI itself is verified as unencrypted, so there are no silent blockages during volume mounting.

Confirmed that the AWSServiceRoleForWorkSpaces exists and is fully active. Its trust relationship is properly mapped to workspaces-instances.amazonaws.com with sts:AssumeRole capabilities.

2. OS & Agent Validation via Test EC2 Instance: To rule out an issue with the automated agent injection during the EC2 Image Builder pipeline, I launched a temporary, standalone EC2 instance directly from my generated AMI. I RDP'd into the machine and verified that the AmazonSSMAgent service is not only present but natively Running. The component is fully structured and ready under C:\ProgramData\Amazon\EC2Launch. The image builder pipeline successfully baked these in. But I found that Amazon EC2Launch was corrupted, so I uninstalled it and installed the latest version, which resolved the issue. After that, I opened EC2Launch and ran "Sysprep with Shutdown", then created a new AMI from the powered-off instance and imported it into WorkSpaces > Images. Unfortunately, the import failed as well. The error message: Initialization failure: There was an unexpected error during the image import process. Please review our documentation at https://docs.aws.amazon.com/workspaces/latest/adminguide/byol-windows-images.html. Please ensure you have run a successful Sysprep and the BYOL Checker script (https://tools.amazonworkspaces.com/BYOLChecker.zip) has passed before retrying import. If the issue persist, please contact AWS Support (https://aws.amazon.com/contact-us).

3. Endpoint Reachability & Network Isolation: To bypass any potential NAT Gateway hairpining, routing quirks, or corporate DNS resolutions via our AD Connector's DHCP Options Set, I deployed Interface VPC Endpoints for the following services directly inside the targeted private subnets: com.amazonaws.eu-west-1.ec2 com.amazonaws.eu-west-1.ssm com.amazonaws.eu-west-1.kms

I attached our wide-open testing Security Group (0.0.0.0/0 Inbound/Outbound) to these Endpoints and ensured Private DNS Names were explicitly enabled to force the internal agent traffic completely over the private AWS backbone.

The Current Status: A 100% verified network layer (NAT GW, DNS, AD Domain joining working perfectly on manual test instances). Dedicated VPC Endpoints ensuring direct, internal API line-of-sight to the AWS Control Plane. Healthy, active SSM and EC2Launch v2 agents natively waiting inside the Windows 11 Enterprise OS. Correctly tuned IAM and KMS permissions.

The automated AWS provisioning engine still fails to register the handshake from the WorkSpace during the bootstrap phase, throwing the identical generic provisioning failure error. 😓🤦‍♂️

I have scheduled another call with the AWS Support team for tomorrow. I hope we'll finally be able to get this resolved.

Thanks again, Maor.

I understand you're experiencing persistent failures when creating WorkSpaces from your custom Windows 11 BYOL bundle. Based on your detailed troubleshooting, you've verified most infrastructure components, but the provisioning agent isn't completing initialization.

For creating WorkSpaces via CLI, the basic syntax supports specifying the DirectoryId, UserName, BundleId, and WorkspaceProperties (including RunningMode). However, the core issue you're facing appears to be with the image preparation rather than the creation command itself.

Since you're working with a BYOL workflow using BYOL_REGULAR_BYOP ingestion, there are critical image preparation requirements that may not be immediately obvious:

Key considerations for BYOL images:

1. Pre-installed components: The Windows image must have specific AWS components pre-installed and properly configured before import. A completely vanilla ISO may be missing required agents or configurations that the WorkSpaces provisioning process expects.

2. EC2Launch v2 configuration: The image needs EC2Launch v2 properly configured to communicate with the WorkSpaces service during bootstrap. A clean ISO won't have this configured correctly for WorkSpaces.

3. Sysprep state: The image must be in the correct Sysprep state. If you're creating an AMI from a vanilla ISO through Image Builder, the generalization and specialization passes may not align with what WorkSpaces expects.

4. BYOL-specific requirements: BYOL images have additional requirements beyond standard EC2 AMIs, particularly around licensing validation and agent communication.

Recommended path forward:

Since you cannot access logs from the failed WorkSpace and AWS Support has confirmed server-side log collection isn't available for this workflow, I recommend:

1. Start with a working base: Instead of a vanilla ISO, consider starting with an existing WorkSpaces image or a properly prepared Windows 11 image that already has the necessary AWS components.

2. Review the ImportWorkspaceImage requirements: Ensure your AMI preparation process includes all prerequisites specific to the BYOL_REGULAR_BYOP ingestion process.

3. Contact AWS Support with specific details: Since you're already working with support, provide them with the specific AMI ID, bundle ID, and request they investigate whether there are missing components in your source AMI that prevent the provisioning agent from initializing. Request guidance on the exact image preparation steps required for Windows 11 BYOL with Omnissa Horizon integration.

4. Test with a minimal customization: If possible, try importing an image that's been minimally modified from a known-working state rather than building from a completely clean ISO.

The fact that the EC2 instance launches successfully but the WorkSpaces agent never signals ready strongly suggests the image is missing required components or configurations specific to the WorkSpaces BYOL workflow, rather than an infrastructure or networking issue.

For your ongoing support case, continue adding correspondence through the AWS Support Center at https://console.aws.amazon.com/support/ with any additional findings or to request specific guidance on image preparation requirements for your use case.
Sources
Use CreateWorkspaces with a CLI - AWS SDK Code Examples
WorkSpaces examples using AWS CLI - AWS SDK Code Examples

Hi Florian,

Thank you for your quick response.

I have a dedicated OU for the account without any special policies applied, as this is our internal test domain and there are almost no GPOs configured at all. However, I will check it anyway but pretty sure that's not the case.

I'm not sure how I can verify the ENI-related items, as I do not have the option to RDP to the machine at any stage of the process. The same applies to the AppX and Agent Initialization steps.

As I described in my workflow, there is no point in the process where I can remotely connect to the machine. Once the ISO is uploaded to the S3 bucket, I create an AMI from it. Without performing any additional actions after the AMI creation, I immediately import it into WorkSpaces Images using the following command:

aws workspaces import-workspace-image --ec2-image-id ami-<our_ami_id> --ingestion-process BYOL_REGULAR_BYOP --image-name "Windows11-Enterprise-25H2-Enterprise" --image-description "Win11 25H2 Ent" --region eu-west-1

After that, I create a bundle through the GUI and then attempt to create a WorkSpace using the following command:

aws workspaces create-workspaces --workspaces "DirectoryId=<id-our_directory_id>,UserName='[UNDEFINED]',BundleId=wsb-<bundle_id>,WorkspaceName=HorizonGoldenImage-Public-Bundle,WorkspaceProperties={RunningMode=MANUAL}"

Because of this workflow, I currently don't see any opportunity to log in to the machine and validate the items you mentioned.

I'm still struggling with this issue for more than two weeks now, and I have already tried the Instant method exactly as you described, multiple times.

I removed all Appx packages, installed the Horizon Agent using the Horizon 8 method for WorkSpaces Core, and ran the AWS Image Builder tool. All steps passed successfully. After that, I ran Sysprep using the image checker tool, and the machine was shut down as expected.

I then created an AMI from it and proceeded to create the WorkSpaces image. However, at that stage it fails and returns an error after 2-3 hours of a Pending status.

By the way, all official and unofficial documentation and blog posts suggest first creating a clean WorkSpace, then connecting to it via RDP and performing all required configuration steps there so I'm really confused at this point.

I also have an open support case, and they advised me not to use the Instant method, confirming that my current workflow is correct. They are still investigating the issue.

Thanks.