How does a custom type dataset call the groupBy method?

I create a custom type dataset by spark.

public class KeyValuePair {      String source;      String target;      int value;      getter...      setter...  }  ...  List<KeyValuePair> list = generateList();  Dataset<KeyValuePair> dataset = spark.createDataset(list, Encoders.kryo(KeyValuePair.class));  Dataset<Row> agg = dataset.groupBy("source", "target").agg(avg("value"));  

When the dataset calls the method groupBy, it will throw an exception. Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot resolve column name "source" among (value)

I wonder whether a custom type dataset can call the method groupBy. Is Row type dataset the only one can call this method without throw an exception?

How can I aggregate a custom type dataset?

https://stackoverflow.com/questions/66575894/how-does-a-custom-type-dataset-call-the-groupby-method March 11, 2021 at 11:08AM