@Author：Runsen

本次博客将介绍如何在Keras中，对文本进行处理Text preprocessing

Text preprocessing

• Keras API
• 文档: https://keras.io/preprocessing/text/

from tensorflow.keras.preprocessing.text import Tokenizer, text_to_word_sequence, one_hot
from tensorflow.keras.preprocessing.sequence import pad_sequences

Tokenization of a sentence

• Tokenizatio：将字符序列转换成符号序列的过程 (https://en.wikipedia.org/wiki/Lexical_analysis#Token)

sentences = ['Curiosity killed the cat.', 'But satisfaction brought it back']
tk = Tokenizer()    # create Tokenizer instance
tk.fit_on_texts(sentences)    # tokenizer should be fit with text data in advance

• 文本建模的一种简单方法是为每个句子创建整数序列

• 通过这样做，可以保留有关单词顺序的信息

seq = tk.texts_to_sequences(sentences)
print(seq)

[[1, 2, 3, 4], [5, 6, 7, 8, 9]]

One-hot encoding

有时，只需要检查某个词是否出现在句子中，这种文本处理的方式被称为“一One-hot encoding ”。

• 如果单词出现在句子中，它被编码为“1”
• 如果不是，则编码为“0”

mat = tk.sequences_to_matrix(seq)
print(mat)

[[0. 1. 1. 1. 1. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 1. 1. 1. 1. 1.]]

Padding sequences

• 通常，为了保持句子的维数，会执行zero padding
• Padding 就像填充图像格式，但适用于序列

# if set padding to 'pre', zeros are appended to start of sentences
pad_seq = pad_sequences(seq, padding='pre')     
print(pad_seq)

[[0 1 2 3 4]
[5 6 7 8 9]]

、pad_seq = pad_sequences(seq, padding='post')
print(pad_seq)

[[1 2 3 4 0]
[5 6 7 8 9]]

Word Embeddings

• 另一种单词嵌入的方法是通过在word embedding中预先训练单词向量（例如word2vec、GloVe、fasttext等）

Word vectors

单词嵌入是将每个单词转换成一个固定维度（单词）向量的过程。嵌入空间（即向量空间）的维数是一个超参数；可以将维数设置为任何正整数

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import *
from tensorflow.keras.datasets import reuters
from tensorflow.keras.preprocessing import sequence
from tensorflow.keras.utils import to_categorical

Embedding layer

因此，Embedding layer具有三维张量

• Output 输出形状= (batch_size, input_length, output_dim)
• input_dim 输入尺寸：输入空间的维数（感兴趣的唯一标记的数量）
• output_dim 输出尺寸：嵌入空间的维数
• input_length 输入长度：输入序列的长度（如果没有，可以改变）

#输入长度不变时
model = Sequential()
model.add(Embedding(input_dim = 10, output_dim = 5, input_length = 3))
model.output_shape

(None, 3, 5)

# when input length varies
model = Sequential()
model.add(Embedding(input_dim = 10, output_dim = 5, input_length = None))    
model.output_shape

(None, None, 5)

在model中，通常采用 embedding layer作为第一层对文本格式的数据进行建模

# parameters to import dataset
num_words = 3000
maxlen = 50(X_train, y_train), (X_test, y_test) = reuters.load_data(num_words = num_words, maxlen = maxlen)
X_train = sequence.pad_sequences(X_train, maxlen = maxlen, padding = 'post')
X_test = sequence.pad_sequences(X_test, maxlen = maxlen, padding = 'post')
y_train = to_categorical(y_train, num_classes = 46)
y_test = to_categorical(y_test, num_classes = 46)
print(X_train.shape)
print(X_test.shape)
print(y_train.shape)
print(y_test.shape)

(1595, 50)
(399, 50)
(1595, 46)
(399, 46)

input_dim = num_words
output_dim = 100     # we set dimensionality of embedding space as 100
input_length = maxlen
def reuters_model():model = Sequential()model.add(Embedding(input_dim = input_dim, output_dim = output_dim, input_length = input_length))model.add(GRU(50, return_sequences = False))model.add(Dense(100))model.add(Activation('relu'))model.add(Dense(46, activation = 'softmax'))model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])return model
model = reuters_model()model.summary()

model.fit(X_train, y_train, epochs = 10, batch_size = 256, verbose = 1)
result = model.evaluate(X_test, y_test)
print('Test Accuracy: ', result[1])

Test Accuracy: 0.7468671798706055