To manage bucket encryption using condition in CloudFormation template

Hi AWS, I am trying to impose a condition on S3 BucketEncryption property whether it should be customer managed (SSE-KMS) or AWS managed key (SSE-S3). The code for the template is:

# version: 1.0
AWSTemplateFormatVersion: "2010-09-09"
Description: Create standardized S3 bucket using CloudFormation Template

Parameters:
  BucketName:
    Type: String
    Description: "Name of the S3 bucket"
  KMSKeyArn:
    Type: String
    Description: "KMS Key Arn to encrypt S3 bucket"
    Default: ""
  SSEAlgorithm:
    Type: String
    Description: "Encryption algorithm for KMS"
    AllowedValues:
      - aws:kms
      - AES256

Conditions:
  KMSKeysProvided: !Not [!Equals [!Ref KMSKeyArn, ""]]

Resources:
  S3Bucket:
    Type: 'AWS::S3::Bucket'
    DeletionPolicy: Retain
    UpdateReplacePolicy: Retain
    Properties:
      BucketName: !Ref BucketName
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      BucketEncryption: 
        ServerSideEncryptionConfiguration: 
        - !If
          - KMSKeysProvided
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: !Ref SSEAlgorithm
              KMSMasterKeyID: !Ref KMSKeyArn
            BucketKeyEnabled: true
          - !Ref "AWS::NoValue"

When I am selecting the SSEAlgorithm as AES256 I am receiving this error Property ServerSideEncryptionConfiguration cannot be empty. I know KMSMasterKeyID should not be present when the SSEAlgorithm is of AES256 type but I am confused how to get rid of this error.

Please help.