[SPARK-38669][ML] Validate input dataset of ml.clustering and ml.recommendation

[zhengruifeng]

What changes were proposed in this pull request?

Validate input dataset of ml.clustering

Why are the changes needed?

when input dataset contains invalid values, fail fast and output relevant error message

Does this PR introduce any user-facing change?

No

How was this patch tested?

added testsuites

[zhengruifeng]

cc @srowen @huaxingao

[srowen]

Basically the same change as last time? that's fine if this is the remainder.

[zhengruifeng]

@srowen ALS is also added here, now all classification/regression/clustering/recommendation support dataset validation. I think there will be another PR to cleanup unused internal functions.

[srowen]

The approach seems OK; looks like some related text failures though

[zhengruifeng]

retest this please

[srowen]

I'm still seeing ...

annotations failed mypy checks:
python/pyspark/pandas/ml.py:62: error: Incompatible types in assignment (expression has type "Index", variable has type "MultiIndex")  [assignment]
python/pyspark/pandas/frame.py:6133: error: unused "type: ignore" comment
python/pyspark/pandas/frame.py:6292: error: Incompatible types in assignment (expression has type "Index", variable has type "MultiIndex")  [assignment]
Found 3 errors in 2 files (checked 3[25](https://github.com/zhengruifeng/spark/runs/5751276900?check_suite_focus=true#step:15:25) source files)

But I don't think that's related? I wonder if we just need to try tomorrow

[zhengruifeng]

I also think it should be irrelevant, let me rebase the PR.

[srowen]

Merged to master

[zhengruifeng]

thanks for reviewing