Error using EvaluateDataQuality when trying to compare array

Question

When i create a rule to compare arrays directly  like here https://docs.aws.amazon.com/glue/latest/dg/data-quality-getting-started.html it works perfectly.

When i try use 
```python
    table_ruleset = """ Rules=[my_rules]""""
    EvaluateDataQuality().process_rows(
frame=empty,
        ruleset=table_ruleset,
        publishing_options={
            "dataQualityEvaluationContext": "data_evaluations",
            "enableDataQualityCloudWatchMetrics": True,
            "enableDataQualityResultsPublishing": True,
        },
        additional_options={"performanceTuning.caching": "CACHE_NOTHING"},
    )
```
its failing i think parsing the text
my_rules is equal to:

```python
my_rules  = """ [  CustomSql "select array_agg(code) from table where IN ( 'CANCELLED', 'REPROGRAMATED', 'GRACE' );" =[], CustomSql "select array_agg(code) from table" = [] ] """
```

but im reciving

```
IllegalArgumentException: Parsing Error: No rules or analyzers provided., line 3:338 no viable alternative at input 'Rules=[CustomSql "select array_agg(code) from table where IN ( 'CANCELLED', 'REPROGRAMATED', 'GRACE' );" =['
```
Im doing something wrong? Or is this a problem with the library

Accepted Answer

I don't think that's allowed (I don't see what you mean by "compare arrays directly works"), arrays/lists are for the IN operand not = , the order would likely break it and empty arrays are not allowed either.  
You should be able to redefine your query so it returns a number of a boolean, for instance if you want to check that code array return empty: 
```
CustomSql "select size(array_agg(code)) from table where IN ( 'CANCELLED', 'REPROGRAMATED', 'GRACE' );" = 0
```
or just:
```
CustomSql "select count(code) from table where IN ( 'CANCELLED', 'REPROGRAMATED', 'GRACE' );" = 0
```

Error using EvaluateDataQuality when trying to compare array

相關內容