By using AWS re:Post, you agree to the Terms of Use

Questions tagged with Amazon CloudWatch Logs

Sort by most recent

Browse through the questions and answers listed below or filter and sort to narrow down your results.

create-export-task | Filter CloudWatch logs using JMESpath

My objective is to create a mechanism for exporting CloudWatch logs to S3 on a case-by-case basis. Given my logs appear in the following format: ``` { "level": "error", "message": "Oops", "errorCode": "MY_ERROR_CODE_1" } { "level": "info", "message": "All good" } { "level": "info", "message": "Something else" } ``` I'd like the export to **only** include the error logs. Using [create-export-task](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/logs/create-export-task.html), is it possible to use the `query` param to filter the response data given the above log structure? I'm not sure whether the log structure is incorrect for this use or if I have misunderstood the purpose of the query param. My JMESPath attempts so far have been unsuccessful. Some attempts include: ``` aws logs create-export-task \ --log-group-name myGroup \ --log-stream-name-prefix myGroup-test \ --from 1664537580000 \ --to 1664537640000 \ --destination myGroup-archive-ab1 \ --destination-prefix test \ --query '{Message: message, Error: errorCode}' ``` and same command, but with the following query `--query '{Message: .message, Error: .errorCode}'` which produces the following error: *Bad value for --query {Message: .message, Error: .errorCode}: invalid token: Parse error at column 10, token "." (DOT), for expression: "{Message: .message, Error: .errorCode}"*
0
answers
0
votes
7
views
asked 8 hours ago

SageMaker MultiDataModel deployment error during inference. ValueError: Exactly one .pth or .pt file is required for PyTorch models: []

Hello, I've been trying to deploy multiple PyTorch models on one endpoint on SageMaker from a SageMaker Notebook. First I tested deployment of single models on single endpoints, to check if everything works smoothly and it did. I would create a PyTorchModel first: ``` import sagemaker from sagemaker.pytorch import PyTorchModel from sagemaker import get_execution_role from sagemaker.multidatamodel import MultiDataModel from sagemaker.serializers import JSONSerializer from sagemaker.deserializers import JSONDeserializer import boto3 role = get_execution_role() sagemaker_session = sagemaker.Session() pytorch_model = PyTorchModel( entry_point='inference.py', source_dir='code', role=role, model_data='s3://***/model/model.tar.gz', framework_version='1.11.0', py_version='py38', name='***-model', sagemaker_session=sagemaker_session ) ``` MultiDataModel inherits properties from Model classes, so I used the same PyTorch model that I used for single model deployment. Then I would define the MultiDataModel the following way: ``` models = MultiDataModel(name='***-multi-model', model_data_prefix='s3://***-sagemaker/model/', model=pytorch_model, sagemaker_session=sagemaker_session ) ``` All it should need is the prefix to the S3 bucket of the model artifacts saved as tar.gz files (the same files used for single model deployment), the previously defined PyTorch model, a name and a sagemaker_session. To deploy it: ``` models.deploy(initial_instance_count =1, instance_type='ml.m4.xlarge', serializer=JSONSerializer(), deserializer=JSONDeserializer(), endpoint_name='***-multi-model-deployment', ) ``` The deployment goes well, as there are no failures and the endpoint is InService by the end of this step. However the error occurs when I try to run inference on one of the models: ``` import json body = {"url":"https://***image.jpg"} #url to an image online payload = json.dumps(body) client = boto3.client('sagemaker-runtime') response = client.invoke_endpoint( EndpointName = "***-multi-model-deployment", ContentType = "application/json", TargetModel = "/model.tar.gz", Body = payload) ``` This prompts an error message: ``` ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (500) from model with message "{ "code": 500, "type": "InternalServerException", "message": "Failed to start workers for model ec1cd509c40ca81ffc3fb09deb4599e2 version: 1.0" } ". See https://***.console.aws.amazon.com/cloudwatch/home?region=***#logEventViewer:group=/aws/sagemaker/Endpoints/***-multi-model-deployment in account ***** for more information. ``` The Cloudwatch logs show this error in particular: ``` 22-09-26T15:51:40,494 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.8/site-packages/ts/model_service_worker.py", line 210, in <module> 2022-09-26T15:51:40,494 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - worker.run_server() 2022-09-26T15:51:40,494 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.8/site-packages/ts/model_service_worker.py", line 181, in run_server 2022-09-26T15:51:40,495 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2022-09-26T15:51:40,495 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.8/site-packages/ts/model_service_worker.py", line 139, in handle_connection 2022-09-26T15:51:40,495 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2022-09-26T15:51:40,495 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.8/site-packages/ts/model_service_worker.py", line 104, in load_model 2022-09-26T15:51:40,495 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - service = model_loader.load( 2022-09-26T15:51:40,495 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.8/site-packages/ts/model_loader.py", line 151, in load 2022-09-26T15:51:40,495 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - initialize_fn(service.context) 2022-09-26T15:51:40,495 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.8/site-packages/sagemaker_pytorch_serving_container/handler_service.py", line 51, in initialize 2022-09-26T15:51:40,495 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - super().initialize(context) 2022-09-26T15:51:40,495 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.8/site-packages/sagemaker_inference/default_handler_service.py", line 66, in initialize 2022-09-26T15:51:40,495 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - self._service.validate_and_initialize(model_dir=model_dir) 2022-09-26T15:51:40,495 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.8/site-packages/sagemaker_inference/transformer.py", line 162, in validate_and_initialize 2022-09-26T15:51:40,495 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - self._model = self._model_fn(model_dir) 2022-09-26T15:51:40,495 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.8/site-packages/sagemaker_pytorch_serving_container/default_pytorch_inference_handler.py", line 73, in default_model_fn 2022-09-26T15:51:40,495 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - raise ValueError( 2022-09-26T15:51:40,496 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - ValueError: Exactly one .pth or .pt file is required for PyTorch models: [] ``` It seems like it's having problems loading the model, saying only one .pth file is required, however in the invocation function i point to the exact model artifact present at that S3 bucket prefix. I'm having a hard time trying to fix this issue, so it would be very helpful if anyone had some suggestions! Instead of giving the MultiDataModel a model, I also tried providing it an ECR docker image with the same inference code, but I would get the same error during invocation of the endpoint.
1
answers
0
votes
32
views
asked 7 days ago

Cloudwatch Log Insights - Line Chart Issues

Hello :wave: We are trying to use Cloudwatch Logs together with Cloudwatch Log Insights to Query through application's logs and visualise the API usage by different consumers. A single log consists of fields such as: - consumer (api user) - controller (base endpoint on which the consumer made a request) - action (exact action on which someone sent a request) Sample entry could look like this (obviously it truncated for the sake of explaining the problem) ``` { consumer: 'test-app-1', controller: 'api/test-controller', action: 'test-action } ``` Now, I would like to find a way in which I can group this in a line chart, for example requests made to different controllers by the consumers over time or exact action requests by consumers over time. I am not sure how to write the query, if I am to be honest, as ``` filter @logStream = 'someLogStream' | fields @timestamp, @message | stats count() by consumer, controller, action ``` does not generate a line chart over time, and adding ``` filter @logStream = 'someLogStream' | fields @timestamp, @message | stats count() by consumer, controller, action, bin(15m) ``` does not change the error message on visualisation screen, which states that: >The data is not suitable for a line chart. Try a bar chart, or group your result by bin function. Can anyone advise on what should I do? For starters, requests over time per consumer would be a nice metric and I believe that once we get to that point, we will be able to modify it per action / per controller. I think that I lack proficiency in the Insights syntax to achieve what I want. Thanks!
1
answers
0
votes
27
views
asked 13 days ago