Unsupervised Clustering with Keras

[gaoxx643]

Is your feature request related to a problem? Please describe.
Pattern recognition in user behavior data is frequently performed by DiDi's data scientists to help the decision makers better understand the dynamics of demand and supply.

Describe the solution you'd like
Autoencoder is desired as it is a data compression algorithm that can help extract the patterns in data.

Code Snippet

nfeat=144
encoding_dim = 2
#layers number
input_dim = tf.keras.Input(shape=(144,))
#encoded1 = tf.keras.layers.Dense(100, activation=tf.compat.v2.keras.activations.relu, name='encoded1')(input_dim)
#encoded1 = tf.keras.layers.Dense(100, activation='relu', name='encoded1')(input_dim)
encoded2 = tf.keras.layers.Dense(7, activation='relu', name='encoded2')(input_dim)
#decoded1 = tf.keras.layers.Dense(100, activation='relu', name='decoded1')(encoded2)
decoded2 = tf.keras.layers.Dense(144, activation='relu', name='decoded2')(encoded2)

model = tf.keras.Model(inputs=input_dim, outputs=decoded2)
model.summary()
model.compile("adam",
              loss='mean_squared_error',
              metrics=['accuracy'])
history = model.fit(x,x,epochs=EPOCHES,
          batch_size=BATCH_SIZE,
          verbose=1)

encoded_layer_model = tf.keras.Model(inputs=model.input,
                           outputs=model.get_layer('encoded2').output)
x_encoded = encoded_layer_model.predict(x)

[Yancey0623]

Maybe we can refine the above code to a custom Keras Mode and add it to https://github.com/sql-machine-learning/models/tree/develop/sqlflow_models so that we can call it in SQLFlow like:

-- TRAIN
SELECT * FROM table TRAIN sqlflow_models.SVDModel WITH nfeet=144, encoding_dim=2 INTO db.my_model;
--- PREDICT
SELECT * FROM table PREDICT db.my_pred_result WITH is_predict = true USING db.my_model;

The code snippet:

class SVDModel(tf.keras.Model):
    def __init__(self, is_predict=False):
        super(SVDModel, self).__init__()
        self.encode1 = tf.keras.layers.Dense(7, activation='relu', name='encode1')
        self.decode1 = tf.keras.layers.Dense(144, activation='relu', name='decode1')
        self.is_predict = is_predict
    def call(self, inputs, training=False):
        x = self.encode1(inputs)
        if self.is_predict:
            return x
        x = self.decode1(x)
        return x
    def default_optimizer(self):
        return tf.keras.optimizers.Adam()

[Echo9573]

We want SQLFlow to recognize the patterns of each sample. Therefore, after dimension reduction, cluster analysis of the data is required.

Code snippet

encoded_layer_model = tf.keras.Model(inputs=model.input,  outputs=model.get_layer('encoded2').output)
x_encoded = encoded_layer_model.predict(x)

kmeans = KMeans(n_clusters=5,random_state=100).fit(x_encoded.values)
kmeans_group = kmeans.labels_

df_clusters = pd.concat([df, pd.Series(kmeans_group)], axis=1)

PS: df.values==x， type(df)==pd.DataFrame , type(x)==np.array
finally, we hope SQLFlow can output df_clusters

[Yancey0623]

Hi @Echo9573 ,can you show more information about the code snippet, such as the definition of df, encoded_df ... so that we can reproduce it on the host.

[Yancey0623]

This feature is completed, c.f. #768, I will close this issue.