Kinesis Analytics for SQL Application Issue

0

Hello, I am having trouble to properly handle query with tumbling window. My application sends 15 sensor data messages per second to Kinesis Data Stream, which is used as an input stream for Kinesis Analytics application. I am trying to run an aggregation query using a GROUP BY clause to process rows in a tumbling window by 60 second interval. The output stream then sends data to a lambda function. I expect that the messages should arrive at lambda every 60 seconds but instead, they arrive much faster, almost every second, and the aggregations don't work as expected. Here is the CloudFormation template that I am using:

ApplicationCode: CREATE OR REPLACE STREAM "SENSORCALC_STREAM" (
"name" VARCHAR(16),
"facilityId" INTEGER, "processId" BIGINT, "sensorId" INTEGER NOT NULL, "min_value" REAL, "max_value" REAL, "stddev_value" REAL);

    CREATE OR REPLACE PUMP "SENSORCALC_STREAM_PUMP" AS 
    INSERT INTO "SENSORCALC_STREAM" 
    SELECT STREAM      
        "name", 
        "facilityId",             
        "processId",             
        "sensorId",        
        MIN("sensorData") AS "min_value",
        MAX("sensorData") AS "max_value",
        STDDEV_SAMP("sensorData") AS "stddev_value"
        
    FROM "SOURCE_SQL_STREAM_001"
    GROUP BY "facilityId","processId", "sensorId", "name",
         STEP("SOURCE_SQL_STREAM_001".ROWTIME BY INTERVAL '60' SECOND);

KinesisAnalyticsSensorApplicationOutput: Type: "AWS::KinesisAnalytics::ApplicationOutput" DependsOn: KinesisAnalyticsSensorApplication Properties: ApplicationName: !Ref KinesisAnalyticsSensorApplication Output: Name: "SENSORCALC_STREAM" LambdaOutput: ResourceARN: !GetAtt SensorStatsFunction.Arn RoleARN: !GetAtt KinesisAnalyticsSensorRole.Arn DestinationSchema: RecordFormatType: "JSON"

I would really appreciate your help in pointing what I am missing. Thank you, Serge

Serge
preguntada hace 2 años104 visualizaciones
No hay respuestas

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas