Deploy aws managed flink in VPC - VPC configurations are not supported for SQL applications

0

I try to deploy aws managed flink application into aws VPC (public subnet) and notice the following in AWS documentation

VPC configurations are not supported for SQL applications. (https://docs.aws.amazon.com/managed-flink/latest/java/vpc.html)

and my flink application use flink.table.api. Is that categorized as SQL application ? and when I tried, I get the following error

So my question is:

  • why SQL application is not supported by vpc ?
  • Is there any workaround in my case

Note: this is an error when cluster trying to start a job. rather than an error when executing the job itself

 {
    "applicationARN": "arn:aws:kinesisanalytics:us-east-1:accountid:application/example",
    "applicationVersionId": "2",
    "locationInformation": "org.apache.flink.runtime.webmonitor.handlers.JarRunOverrideHandler.lambda$handleRequest$4(JarRunOverrideHandler.java:245)",
    "logger": "org.apache.flink.runtime.webmonitor.handlers.JarRunOverrideHandler",
    "message": "Error occurred when trying to start the job",
    "messageSchemaVersion": "1",
    "messageType": "ERROR",
    "threadName": "Flink-DispatcherRestEndpoint-thread-2",
    "throwableInformation": "java.util.concurrent.CompletionException: org.apache.flink.util.FlinkRuntimeException: Could not execute application.\n\tat java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)\n\tat java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\nCaused by: org.apache.flink.util.FlinkRuntimeException: Could not execute application.\n\tat org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:88)\n\tat org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70)\n\tat org.apache.flink.runtime.webmonitor.handlers.JarRunOverrideHandler.lambda$handleRequest$3(JarRunOverrideHandler.java:239)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)\n\t... 6 more\nCaused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Failed to execute sql\n\tat org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)\n\tat org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)\n\tat org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105)\n\tat org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84)\n\t... 9 more\nCaused by: org.apache.flink.table.api.TableException: Failed to execute sql\n\tat org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1048)\n\tat org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:864)\n\tat org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1099)\n\tat org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:735)\n\tat com.data.FlinkMain.run(FlinkMain.java:43)\n\tat com.datas.FlinkMain.main(FlinkMain.java:24)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566)\n\tat org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)\n\t... 12 more\nCaused by: org.apache.flink.util.FlinkException: Failed to execute job 'insert-into_default_catalog.default_database._spendingbyday$1,default_catalog.default_database.customertransaction$1,default_catalog.default_database.spendingbycategory$1'.\n\tat org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2253)\n\tat org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:189)\n\tat org.apache.flink.table.planner.delegation.DefaultExecutor.executeAsync(DefaultExecutor.java:110)\n\tat org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1020)\n\t... 22 more\nCaused by: java.lang.RuntimeException: Error while waiting for job to be initialized\n\tat org.apache.flink.client.ClientUtils.waitUntilJobInitializationFinished(ClientUtils.java:151)\n\tat org.apache.flink.client.deployment.application.executors.EmbeddedExecutor.lambda$submitAndGetJobClientFuture$2(EmbeddedExecutor.java:141)\n\tat org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:73)\n\tat java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642)\n\tat java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)\n\t... 1 more\nCaused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Invocation of [LocalRpcInvocation(RestfulGateway.requestJobStatus(JobID, Time))] at recipient [pekko.ssl.tcp://flink@flink-jobmanager:6123/user/rpc/dispatcher_1] timed out. This is usually caused by: 1) Pekko failed sending the message silently, due to problems like oversized payload or serialization failures. In that case, you should find detailed error information in the logs. 2) The recipient needs more time for responding, due to problems like slow machines or network jitters. In that case, you can try to increase pekko.ask.timeout.\n\tat java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)\n\tat java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)\n\tat org.apache.flink.client.deployment.application.executors.EmbeddedExecutor.lambda$submitAndGetJobClientFuture$0(EmbeddedExecutor.java:146)\n\tat org.apache.flink.client.ClientUtils.waitUntilJobInitializationFinished(ClientUtils.java:135)\n\t... 5 more\nCaused by: java.util.concurrent.TimeoutException: Invocation of [LocalRpcInvocation(RestfulGateway.requestJobStatus(JobID, Time))] at recipient [pekko.ssl.tcp://flink@flink-jobmanager:6123/user/rpc/dispatcher_1] timed out. This is usually caused by: 1) Pekko failed sending the message silently, due to problems like oversized payload or serialization failures. In that case, you should find detailed error information in the logs. 2) The recipient needs more time for responding, due to problems like slow machines or network jitters. In that case, you can try to increase pekko.ask.timeout.\n\tat com.sun.proxy.$Proxy43.requestJobStatus(Unknown Source)\n\tat org.apache.flink.client.deployment.application.executors.EmbeddedExecutor.lambda$submitAndGetJobClientFuture$0(EmbeddedExecutor.java:144)\n\t... 6 more\nCaused by: org.apache.pekko.pattern.AskTimeoutException: Ask timed out on [Actor[pekko://flink/user/rpc/dispatcher_1#554204934]] after [60000 ms]. Message of type [org.apache.flink.runtime.rpc.messages.LocalFencedMessage]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.\n"
}
1 Answer
1

Are you trying to communicate with some other resource within the VPC or something in the internet?

If it's in the internet you need to place your workload in a private subnet, create a NAT gateway in a public subnet, and add a route on the route table of the private subnet to go to 0.0.0.0/0 via the NAT gateway.

profile pictureAWS
EXPERT
answered 2 months ago
  • thanks for the reply. I am unable to start the application (flink application) within VPC at first place. And I try to connect to a database with public ip from vpc public subnet. I guess that should invalidate the private subnet issues ?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions