AWS re:Postを使用することにより、以下に同意したことになります AWS re:Post 利用規約
AWS re:Postre:Post

How can I throttle the write operation from a glue import job to RDS mysql

0

I want to import 20 million records from a csv to RDS using glue. The glue job will run once a quarter only. I want to be able to throttle the write operations from glue to RDS to limit the IOPS to reduce costs. How can this be done? Is there a way to limit the number of concurrent connections from glue to RDS or any configuration that can help reduce the IOPS. One way to accomplish this is to split the csv into smaller chunks and then process smaller files - but it seems like an overkill just to reduce the IOPS. Update - RDS is cross account and a shared resource so I cannot change any RDS configs

トピック
移行と近代化分析データベース
タグ
Amazon Relational Database Service (RDS)AWS GlueMySQL
言語
English
rePost-User-1191271
質問済み 7ヶ月前249ビュー
1回答
0

Normally you don't limit IOPS since the DB will do that "throttling" by taking longer, the main impact in the number of connections in parallel, which you can control by controlling the number of cores in the cluster or the number of partitions (using repartition or coalesce) in the data (whichever is lower).

profile pictureAWS
エキスパート
Gonzalo Herreros
回答済み 7ヶ月前
  • rePost-User-1191271
    7ヶ月前

    Sorry I should have mentioned, RDS is cross account and a shared database so I cannot change RDS config

  • Gonzalo Herreros エキスパート
    7ヶ月前

    That's not what I suggested (that would throttle all users equally), all I talked about is done in the Glue code

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ