By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Quicksight Not able to maintain connection to Presto for queries running more than 5 minutes

0

I have a 3TB Data kept in S3 location , I have created a Presto cluster in EMR and it queries the S3 data with every kind of filter condition ,,, but it takes 5-10 min (approx) to give results. Now I have a Quicksight dashboard need to get these queries data to be represented in my Visuals with the inputs provided in the form of controls from the panel . Dataset's Data source is presto on EMR , but it is taking 4-5 min to give the output . In the meantime the Quicksight gives an error "Quicksight is taking too long to get visual , please connect with quicksight team".

when I checked in my presto server the query fired by quicksight is still running , but the Quisight dashboard has already failed.

i have updated some query.execution.max.time for presto to 30 minutes and timeout also more than 30minutes in presto config file in server , but still it is not helping the cause.

Please help to suggest How can i get data in quicksight dashboard for queries running more than 5 min in presto server ?

Note: I am aware of hard limit of 2 minutes from Quicksight , but with this way Quicksight will never be able to grow as a dashboarding interface as long running queries of 5-10 minutes are expected in inudstries.

Topics
Analytics
Tags
Amazon EMRAmazon QuickSight
Language
English
rePost-User-9261459
asked a year ago249 views
1 Answer
0

Hello,

Regarding your query, if you are using a Direct Query Data set in Quicksight, you could be facing a timeout while loading your analysis and dashboards. As you are aware, there is a timeout of 2 minutes in Quicksight while generating visuals.

In this scenario, Quicksight would execute a query when you refresh the visual and if the query execution could not be completed within 2 minutes, the Quicksight would issue a timeout and the analysis will not be able to load.

Also, the timeout of 2 minutes is a hard limit and currently Quicksight does not support configuring a manual timeout for heavy queries.

[+] - https://docs.aws.amazon.com/quicksight/latest/user/data-source-limits.html

Additionally, I would recommended the below workaround to avoid this:

The following are steps that you can take to limit the amount of data that you're importing into Quicksight:

  1. Unselect columns that you don't need.

  2. Add filters to your data set in Quicksight.

  3. Use predicates in your custom SQL query such as WHERE and HAVING.

    [+] - https://repost.aws/knowledge-center/quicksight-resolve-query-timeout-issues

That being said, to further look into this issue more closely, if you would like to do a resource based troubleshooting, please raise a support case with AWS for further information. If a support case has already been created please be assured that we will get back to you and assist you in the best way possible.

AWS
Sakshi_J
answered a year ago
AWS
EXPERT
Fabrizio@AWS
reviewed a year ago
  • rePost-User-9261459
    a year ago

    But we have 10 TB of data in S3 , and for internal business data visual query we want to use presto as a cluster to scan the data from s3 as time is actually not the concern for the users to get the output.

    Quicksight is a reporting platform from where users are expected to add new filter conditions and presto will fire the query on those filters. currently the scan of 10 TB Data usually takes more than 5 min for each of these queries , Quicksight doesnt seem to be the right option then ? As to again the store the 10TB of data in SPICE will actually add on to the costs which we are trying to save .

    Please suggest a workaround where data storage is expected to be S3 and queries can take more than 5 min to be displayed in quicksight visuals.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content