Bdtjtk

I have csv file with two columns:

category, description

1030 categories in the file and only about 12,600 lines

I need to get a model for text classification, trained on this data. I use keras with LSTM model.

I found an article describing how to make a binary classification, and slightly modified it to use several categories.

My code:

import pandas as pd

import numpy as np

from keras.models import Sequential

from keras.layers import Dense, Embedding, LSTM

from numpy import array

from keras.preprocessing.text import one_hot

from sklearn.preprocessing import LabelEncoder

from keras.preprocessing import sequence 

import keras



df = pd.read_csv('/tmp/input_data.csv')



#one hot encode your documents



# integer encode the documents

vocab_size = 2000

encoded_docs = [one_hot(d, vocab_size) for d in df['description']]



def load_data_from_arrays(strings, labels, train_test_split=0.9):

    data_size = len(strings)

    test_size = int(data_size - round(data_size * train_test_split))

    print("Test size: {}".format(test_size))



    print("nTraining set:")

    x_train = strings[test_size:]

    print("t - x_train: {}".format(len(x_train)))

    y_train = labels[test_size:]

    print("t - y_train: {}".format(len(y_train)))



    print("nTesting set:")

    x_test = strings[:test_size]

    print("t - x_test: {}".format(len(x_test)))

    y_test = labels[:test_size]

    print("t - y_test: {}".format(len(y_test)))



    return x_train, y_train, x_test, y_test





encoder = LabelEncoder()

categories = encoder.fit_transform(df['category'])

num_classes = np.max(categories) + 1

print('Categories count: {}'.format(num_classes))

#Categories count: 1030



X_train, y_train, x_test, y_test = load_data_from_arrays(encoded_docs, categories, train_test_split=0.8)



# Truncate and pad the review sequences 



max_review_length = 500 

X_train = sequence.pad_sequences(X_train, maxlen=max_review_length) 

x_test = sequence.pad_sequences(x_test, maxlen=max_review_length) 



y_train = keras.utils.to_categorical(y_train, num_classes)

y_test = keras.utils.to_categorical(y_test, num_classes)

print('y_train shape:', y_train.shape)

print('y_test shape:', y_test.shape)



# Build the model 

embedding_vector_length = 32 

top_words = 10000



model = Sequential() 

model.add(Embedding(top_words, embedding_vector_length, input_length=max_review_length)) 

model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2)) 

model.add(Dense(num_classes, activation='softmax')) 

model.compile(loss='categorical_crossentropy',optimizer='adam', metrics=['accuracy']) 

print(model.summary())



_________________________________________________________________

Layer (type)                 Output Shape              Param #   

=================================================================

embedding_8 (Embedding)      (None, 500, 32)           320000    

_________________________________________________________________

lstm_8 (LSTM)                (None, 100)               53200     

_________________________________________________________________

dense_8 (Dense)              (None, 1030)              104030    

=================================================================

Total params: 477,230

Trainable params: 477,230

Non-trainable params: 0

_________________________________________________________________

None



#Train the model

model.fit(X_train, y_train, validation_data=(x_test, y_test), epochs=5, batch_size=64) 



Train on 10118 samples, validate on 2530 samples

Epoch 1/5

10118/10118 [==============================] - 60s 6ms/step - loss: 6.5086 - acc: 0.0019 - val_loss: 10.0911 - val_acc: 0.0000e+00

Epoch 2/5

10118/10118 [==============================] - 63s 6ms/step - loss: 6.3281 - acc: 0.0028 - val_loss: 10.8270 - val_acc: 0.0000e+00

Epoch 3/5

10118/10118 [==============================] - 63s 6ms/step - loss: 6.3120 - acc: 0.0024 - val_loss: 11.0078 - val_acc: 0.0000e+00

Epoch 4/5

10118/10118 [==============================] - 64s 6ms/step - loss: 6.2891 - acc: 0.0030 - val_loss: 11.8264 - val_acc: 0.0000e+00

Epoch 5/5

10118/10118 [==============================] - 69s 7ms/step - loss: 6.2559 - acc: 0.0032 - val_loss: 12.1625 - val_acc: 0.0000e+00



#Evaluate the model

scores = model.evaluate(x_test, y_test, verbose=0) 

print("Accuracy: %.2f%%" % (scores[1]*100))



Accuracy: 0.00%

What mistake did I make when preparing the data?
why accuracy is always 0?

I guess that your vocab_size is way too low. If you are dealing with usual text, try 10.000 - 100.000 as a starting point.

What one_hot does is to use the hashing trick. That means all of your words are hashed and projected into an 2000 vector space. It does not only mean that your dict is 2000 words long, it does mean every word will be projected to into this space, which effectively causes a lot of collisions, where words have the same index and are considered as equal in the LSTM.

Furthermore you should take a look at the transformed text, just too get an understanding of what happens here. To do so, build an reverse lookup and transform all the indices back.

As a further improvement it is feasible to preprocess the text with common techniques like stemming, normalizing etc. and the usage of a vocabulary or discard bag of words and use word embeddings.