Inconsistent documentation for DQDL rule type IsComplete

Question

I get strange errors in the data quality tab of the glue studio, which seem to contradict the documentation in https://docs.aws.amazon.com/glue/latest/dg/dqdl.html#dqdl-rule-types-IsComplete:

```
Rule_13 IsComplete "abc"
Rule failed	Unsupported nested column type of column abc: ArrayType(DoubleType,true)!
```
Contradicts: "Supported column types: Any column type"

```
Rule_14 IsComplete "xyz"
Rule failed	Value: 0.6140889354693816 does not meet the constraint requirement!
```
Contradicts: "Checks whether all of the values in a column are complete (non-null)."

Is there any better way to understand what is going wrong in my case?

Answer

Hi, 
thank you for your feedback on the AWS Glue Data Quality preview. As for your specific questions:

1. in the [first page of the documentation on the DataQuality evaluation](https://docs.aws.amazon.com/glue/latest/dg/glue-data-quality.html) we mention that :"Data quality rules can't evaluate nested or list-type data sources.".  So later we refer to all supported column types, still we will keep into consideration your feedback.
2.  The error message you get is specifying that **the rule evaluation failed** because 61.4% of the rows are actually null and that violates the constraint for the rule.

hope this helps.

Inconsistent documentation for DQDL rule type IsComplete

相关内容