"pcluster create" results in error during startup - How to address ?

Question

**I am attempting to create a small Parallel cluster using OSX and anaconda. I installed and configured parallelcluster with no errors. The resulting simple config file looks like:**

```
[aws]
aws_region_name = us-east-1

[global]
cluster_template = default
update_check = true
sanity_check = true

[aliases]
ssh = ssh {CFN_USER}@{MASTER_IP} {ARGS}

[cluster default]
key_name = newmac
base_os = ubuntu1604
scheduler = sge
initial_queue_size = 2
max_queue_size = 2
maintain_initial_size = true
vpc_settings = default

[vpc default]
vpc_id = vpc-058cb4b123bf54848
master_subnet_id = subnet-04998fb2f4bc80ccc
compute_subnet_id = subnet-03c7e26fb2ea81430
use_public_ips = false
```

**When I went to create the cluster here is what I see:**  
  
(base) esteban$ pcluster create -c /Users/esteban/.parallelcluster/config first-cluster  
Beginning cluster creation for cluster: first-cluster  
WARNING: The configuration parameter 'scheduler' generated the following warnings:  
The job scheduler you are using (sge) is scheduled to be deprecated in future releases of ParallelCluster. More information is available here: https://github.com/aws/aws-parallelcluster/wiki/Deprecation-of-SGE-and-Torque-in-ParallelCluster  
Creating stack named: parallelcluster-first-cluster  
Status: parallelcluster-first-cluster - ROLLBACK_IN_PROGRESS                      
Cluster creation failed.  Failed events:  
  - AWS::AutoScaling::AutoScalingGroup ComputeFleet Resource creation cancelled  
  - AWS::CloudFormation::WaitCondition MasterServerWaitCondition Received FAILURE signal with UniqueId i-0438bb35816b58aa5

**Okay so I can still login although there are no SGE binaries installed.**  
   
$ pcluster ssh first-cluster -i ~/.ssh/newmac.pem

**So I just deleted the cluster which appears to have (I hope) cleaned up all the associated services.**   
  
pcluster delete -c /Users/esteban/.parallelcluster/config first-cluster  
  
Any ideas ? Thanks  
  
Edited by: Stevie on Jul 23, 2020 11:42 AM  
  
Edited by: Stevie on Jul 23, 2020 11:46 AM  
  
Edited by: Stevie on Jul 23, 2020 11:47 AM

enrico-aws · Answer

Hi Stevie,  
  
How did you created the VPC, NAT gateway and the network components?  
You can use your own default resources, create them by hand or create them through the _pcluster configure_ command.  
  
In any case the _pcluster create_ command expects these resources are already created so the _delete_ command doesn't delete them.  
  
It is by design because the same VPC could be used for multiple clusters, and there is a limit on the number of VPC, Elastic IPs, etc the users can create, so we decided to decouple VPC and network resources from the clusters ones, it is not related to the issue with your previous failed creation.  
  
Let us know if it helps.

Stevie · Answer

Thank you. I had to move on to another project for a bit. I will take into consideration the suggested remedies. One thing that I noticed after trying to delete the failed cluster was that although it appeared to have deleted the cluster - I had to go in and manually remove components (VPC, NAT gateway, etc). In my case, it seemed that the failed create impaired the ability of pcluster delete to back out all supporting components.

AWS-User-4920370 · Answer

Hi, Stevie.  
  
The error message you posted, MasterServerWaitCondition Received FAILURE signal with UniqueId INSTANCE_ID, means there was an error when one of the nodes in the cluster was running its configuration tasks after booting. Usually this is due to a failure on the head node of the cluster.  
  
You can start debugging the issue by first passing the '-nr/--norollback` flag to `pcluster create` and then, if cluster creation fails again, you can log into the head node and look for the cause of the failure in /var/log/cfn-init.log.  
  
This wiki page contains tips for debugging common cluster creation failures: https://github.com/aws/aws-parallelcluster/wiki/Stack-Creation-Failures  
  
Let us know what you find in the logs, and we'll try and provide more specific guidance based on that.  
  
~Tim

"pcluster create" results in error during startup - How to address ?

Relevant content