ALB/Lambda CloudFormation circular dependency

I'm trying to create a CFn stack including a Lambda function being called by ALB. Part of this requires giving permission for the ALB to invoke the Lambda function.

However, when I do this in CloudFormation I get an error:

API: elasticloadbalancingv2:RegisterTargets elasticloadbalancing principal does not have permission to invoke <Lambda ARN> from target group <ALB Target Group ARN>

This is happening because CFn is creating the ALB target group before the permission is being created. Normally, I'd put a DependsOn in the target group definition so that but when I do that CFN tells me that there is a circular dependency - and there is because the permission references both the Lambda function and the target group.

I could remove the reference to the target group from the permission (and indeed this does fix the problem) but doing so would allow anything in the account to invoke the function (please correct me if I'm wrong here).

Is there a way around this? Briefest CFn template below:

AWSTemplateFormatVersion: "2010-09-09"

Resources:
    ALB:
        Type: "AWS::ElasticLoadBalancingV2::LoadBalancer"
        Properties:
            Name: "test-alb"
            Scheme: "internal"
            Type: "application"
            Subnets: 
              - "subnet-01c47f76"
              - "subnet-957eecf0"

    ALBListener:
        Type: "AWS::ElasticLoadBalancingV2::Listener"
        Properties:
            LoadBalancerArn: !Ref ALB
            Port: 80
            Protocol: "HTTP"
            DefaultActions: 
              - 
                Type: "forward"
                ForwardConfig: 
                    TargetGroups:
                      - TargetGroupArn: !Ref ALBTargetGroup

    ALBLambdaPermission:
        DependsOn:
          - LambdaFunction
        Type: AWS::Lambda::Permission
        Properties:
            FunctionName: !GetAtt LambdaFunction.Arn
            Action: lambda:InvokeFunction
            Principal: elasticloadbalancing.amazonaws.com
            SourceArn: !Ref ALBTargetGroup

    ALBTargetGroup:
        DependsOn: ALBLambdaPermission
        Type: "AWS::ElasticLoadBalancingV2::TargetGroup"
        Properties:
            HealthCheckPath: "/"
            TargetType: "lambda"
            Targets: 
              - 
                Id: !GetAtt LambdaFunction.Arn

    LambdaRole:
      Type: AWS::IAM::Role
      Properties:
        AssumeRolePolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Effect: Allow
              Principal:
                Service:
                  - lambda.amazonaws.com
              Action:
                - sts:AssumeRole
        ManagedPolicyArns:
          - !Sub "arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"


    LambdaFunction:
        Type: "AWS::Lambda::Function"
        Properties:
            FunctionName: "alb-test-lambda"
            Handler: "index.lambda_handler"
            Code: 
              ZipFile: |
                def lambda_handler(event, context):
                    return  {
                        "statusCode": 200,
                        "body": "Hello from Lambda!",
                        "headers": {
                            "Content-Type": "text/html"
                        }
                    }
            Role: !GetAtt LambdaRole.Arn
            Runtime: "python3.6"