Best AWS service to pull data from external api

Question

I am wondering what is the best way to pull data front external APIs using AWS services to store it in S3, dynamodb or RDS in a periodic way. 
Is it lambda? Glue? Eventbridge? Appflow with custom connector or another service. It is very weird not to have a simple way (service) to do that. Now even with lambda you need layering to import python "Requests" library.
Any idea and experience doing that?

Riku_Kobayashi · Accepted Answer

Hello.

I think it depends on the API being executed, but I would use Lambda.  
It is true that you need to create a layer in Python to run the API, but using AWS SAM you can easily create a layer with the modules listed in "requirements.txt".  
https://docs.aws.amazon.com/lambda/latest/dg/layers-sam.html

For the template below, place "requirements.txt" in the "my_layer/" directory.  
```
Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.8
      Layers:
        - !Ref MyLayer

MyLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      Description: Layer description
      ContentUri: 'my_layer/'
      CompatibleRuntimes:
        - python3.8
    Metadata:
      BuildMethod: python3.8
```

However, in the case of Lambda, the maximum execution time is 15 minutes, so if the processing takes longer than that, I think it is better to use ECS Fargate etc.

Antonio_Lagrotteria · Answer

Depends on your use case, but done options can be api destinations with event bridge: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-api-destinations.html or step functions call to third party : https://docs.aws.amazon.com/step-functions/latest/dg/connect-third-party-apis.html.

Lambda would also work as described, but you need to either install dependencies or as in node, use low level native https package.

For python you can also refer to this: https://repost.aws/questions/QUrw_ULTk6RpSQ9iMkkbElbg/external-rest-api-call-in-python-with-lambda-function

Best AWS service to pull data from external api

Relevant content