Question :
I’ve recently reviewed an interesting implementation for convolutional text classification. However all TensorFlow code I’ve reviewed uses a random (not pretrained) embedding vectors like the following:
with tf.device('/cpu:0'), tf.name_scope("embedding"):
W = tf.Variable(
tf.random_uniform([vocab_size, embedding_size], 1.0, 1.0),
name="W")
self.embedded_chars = tf.nn.embedding_lookup(W, self.input_x)
self.embedded_chars_expanded = tf.expand_dims(self.embedded_chars, 1)
Does anybody know how to use the results of Word2vec or a GloVe pretrained word embedding instead of a random one?
Answer #1:
There are a few ways that you can use a pretrained embedding in TensorFlow. Let’s say that you have the embedding in a NumPy array called embedding
, with vocab_size
rows and embedding_dim
columns and you want to create a tensor W
that can be used in a call to tf.nn.embedding_lookup()
.

Simply create
W
as atf.constant()
that takesembedding
as its value:W = tf.constant(embedding, name="W")
This is the easiest approach, but it is not memory efficient because the value of a
tf.constant()
is stored multiple times in memory. Sinceembedding
can be very large, you should only use this approach for toy examples. 
Create
W
as atf.Variable
and initialize it from the NumPy array via atf.placeholder()
:W = tf.Variable(tf.constant(0.0, shape=[vocab_size, embedding_dim]), trainable=False, name="W") embedding_placeholder = tf.placeholder(tf.float32, [vocab_size, embedding_dim]) embedding_init = W.assign(embedding_placeholder) # ... sess = tf.Session() sess.run(embedding_init, feed_dict={embedding_placeholder: embedding})
This avoid storing a copy of
embedding
in the graph, but it does require enough memory to keep two copies of the matrix in memory at once (one for the NumPy array, and one for thetf.Variable
). Note that I’ve assumed that you want to hold the embedding matrix constant during training, soW
is created withtrainable=False
. 
If the embedding was trained as part of another TensorFlow model, you can use a
tf.train.Saver
to load the value from the other model’s checkpoint file. This means that the embedding matrix can bypass Python altogether. CreateW
as in option 2, then do the following:W = tf.Variable(...) embedding_saver = tf.train.Saver({"name_of_variable_in_other_model": W}) # ... sess = tf.Session() embedding_saver.restore(sess, "checkpoint_filename.ckpt")
Answer #2:
I use this method to load and share embedding.
W = tf.get_variable(name="W", shape=embedding.shape, initializer=tf.constant_initializer(embedding), trainable=False)
Answer #3:
The answer of @mrry is not right because it provoques the overwriting of the embeddings weights each the network is run, so if you are following a minibatch approach to train your network, you are overwriting the weights of the embeddings. So, on my point of view the right way to pretrained embeddings is:
embeddings = tf.get_variable("embeddings", shape=[dim1, dim2], initializer=tf.constant_initializer(np.array(embeddings_matrix))
Answer #4:
2.0 Compatible Answer: There are many PreTrained Embeddings, which are developed by Google and which have been Open Sourced.
Some of them are Universal Sentence Encoder (USE), ELMO, BERT
, etc.. and it is very easy to reuse them in your code.
Code to reuse the PreTrained Embedding
, Universal Sentence Encoder
is shown below:
!pip install "tensorflow_hub>=0.6.0"
!pip install "tensorflow>=2.0.0"
import tensorflow as tf
import tensorflow_hub as hub
module_url = "https://tfhub.dev/google/universalsentenceencoder/4"
embed = hub.KerasLayer(module_url)
embeddings = embed(["A long sentence.", "singleword",
"http://example.com"])
print(embeddings.shape) #(3,128)
For more information the PreTrained Embeddings developed and opensourced by Google, refer TF Hub Link.
Answer #5:
With tensorflow version 2 its quite easy if you use the Embedding layer
X=tf.keras.layers.Embedding(input_dim=vocab_size,
output_dim=300,
input_length=Length_of_input_sequences,
embeddings_initializer=matrix_of_pretrained_weights
)(ur_inp)
Answer #6:
I was also facing embedding issue, So i wrote detailed tutorial with dataset.
Here I would like to add what I tried You can also try this method,
import tensorflow as tf
tf.reset_default_graph()
input_x=tf.placeholder(tf.int32,shape=[None,None])
#you have to edit shape according to your embedding size
Word_embedding = tf.get_variable(name="W", shape=[400000,100], initializer=tf.constant_initializer(np.array(word_embedding)), trainable=False)
embedding_loopup= tf.nn.embedding_lookup(Word_embedding,input_x)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for ii in final_:
print(sess.run(embedding_loopup,feed_dict={input_x:[ii]}))
Here is working detailed Tutorial Ipython example if you want to understand from scratch , take a look .