Understanding WeightedKappaLoss using Keras

I'm using Keras to try to predict a vector of scores (0-1) using a sequence of events.

For example, X is a sequence of 3 vectors comprised of 6 features each, while y is a vector of 3 scores:

X  [    [1,2,3,4,5,6], <--- dummy data    [1,2,3,4,5,6],    [1,2,3,4,5,6]  ]    y  [0.34 ,0.12 ,0.46] <--- dummy data  

I want to adress the problem as ordinal classification, so if the actual values are [0.5,0.5,0.5] the prediction [0.49,0.49,0.49] is better then [0.3,0.3,0.3]. My Original solution, was to use sigmoid activation on my last layer and mse as the loss function, so the output is ranged between 0-1 for each of the output neurons:

def get_model(num_samples, num_features, output_size):      opt = Adam()      model = Sequential()            model.add(LSTM(config['lstm_neurons'], activation=config['lstm_activation'], input_shape=(num_samples, num_features)))      model.add(Dropout(config['dropout_rate']))        for layer in config['dense_layers']:        model.add(Dense(layer['neurons'], activation=layer['activation']))        model.add(Dense(output_size, activation='sigmoid'))      model.compile(loss='mse', optimizer=opt, metrics=['mae', 'mse'])        return model  

My Goal is to understand the usage of WeightedKappaLoss and to implement it on my actual data. I've created this Colab to fiddle around with the idea. In the Colab, my data is a sequence shaped (5000,3,3) and my targets shape is (5000, 4) representing 1 of 4 possible classes.

I want the model to understand that it needs to trim the floating point of the X in order to predict the right y class:

[[3.49877793, 3.65873511, 3.20218196],   [3.20258153, 3.7578669 , 3.83365481],   [3.9579924 , 3.41765455, 3.89652426]], ----> y is 3 [0,0,1,0]    [[1.74290875, 1.41573056, 1.31195701],   [1.89952004, 1.95459796, 1.93148095],   [1.18668981, 1.98982041, 1.89025326]], ----> y is 1 [1,0,0,0]  

New model code:

def get_model(num_samples, num_features, output_size):      opt = Adam(learning_rate=config['learning_rate'])      model = Sequential()            model.add(LSTM(config['lstm_neurons'], activation=config['lstm_activation'], input_shape=(num_samples, num_features)))      model.add(Dropout(config['dropout_rate']))        for layer in config['dense_layers']:        model.add(Dense(layer['neurons'], activation=layer['activation']))        model.add(Dense(output_size, activation='softmax'))      model.compile(loss=tfa.losses.WeightedKappaLoss(num_classes=4), optimizer=opt, metrics=[tfa.metrics.CohenKappa(num_classes=4)])        return model  

When fitting the model I can see the following metrics on TensorBoard:

I'm not sure about the following points and would appreciate clarification:

• Am I using it right?
• In my original problem, I'm predicting 3 scores, as opposed of the Colab example, where I'm predicting only 1. If I'm using WeightedKappaLoss, does it mean I'll need to convert each of the scores to a vector of 100 one-hot encoding?
• Is there a way to use the WeightedKappaLoss on the original floating point scores without converting to a classification problem?

https://stackoverflow.com/questions/65305864/understanding-weightedkappaloss-using-keras December 15, 2020 at 08:28PM