Bdtjtk

I've written an RNN that looks at paragraphs in character level, and would like to save it to use later. Some code is as follows:

cell = tf.nn.rnn_cell.LSTMCell(state_size, state_is_tuple=True)

batch_size = tf.placeholder(tf.int32, , name='batch_size')

multi_cell = tf.nn.rnn_cell.MultiRNNCell([cell] * num_layers, state_is_tuple=True)

init_state = multi_cell.zero_state(batch_size, dtype=tf.float32)

rnn_outputs, final_state = tf.nn.dynamic_rnn(multi_cell, rnn_inputs, initial_state=init_state)



with tf.variable_scope('softmax'):

    W = tf.get_variable('W', [state_size, num_classes])

    b = tf.get_variable('b', [num_classes], initializer=tf.constant_initializer(0.0))



rnn_outputs = tf.reshape(rnn_outputs, [-1, state_size])

y_reshaped = tf.reshape(y, [-1])



logits = tf.matmul(rnn_outputs, W) + b

predictions = tf.nn.softmax(logits, name="predictions")



total_loss = tf.reduce_mean(

    tf.nn.sparse_softmax_cross_entropy_with_logits(

        logits=logits, 

        labels=y_reshaped

    )

)

train_step = tf.train.AdamOptimizer(learning_rate).minimize(total_loss)

And then I use tf.train.Saver() and saver.save(sess, "path/to/save") to save my model.

Then I try to load my model in another script and generate text using the code below:

tf.reset_default_graph()

imported_meta = tf.train.import_meta_graph("path/to/save/save_file.meta")



with tf.Session() as sess:

    imported_meta.restore(sess, tf.train.latest_checkpoint("path/to/save"))

    x = sess.graph.get_tensor_by_name("input_placeholder:0")

    batch_size_tensor = sess.graph.get_tensor_by_name("batch_size:0")

    predictions = sess.graph.get_tensor_by_name("predictions:0")



    state = None

    current_char = vocab_to_index[start_char]



    for i in range(num_chars):

        if state is not None:

            feed_dict={batch_size_tensor: batch_size, x: [[current_char]], init_state: state}

        else:

            feed_dict={batch_size_tensor: batch_size, x: [[current_char]]}



        rnn_outputs, state = sess.run(

            [predictions, final_state], 

            feed_dict

        )

Basically what I want to do here is to input a character, then generate a character based on the previous one, and again. After the initial character, the final_state out of dynamic_rnn should be sess.run() and feed into the next generating process as init_state. However, I could not find a way to save init_state and final_state defined in the training code to load into the test code, there is no "name" argument like for tf.nn.softmax for those operations.

What I want to have is some code like final_state = sess.graph.get_operation_by_name('final_state') so that I could sess.run(final_state) and feed that back as init_state.

I've tried using tf.add_to_collection("some_name", final_state) in the training code and tf.get_collection("some_name"), but the error says collection "some_name" cannot be found in the test graph.

Have anyone who has written a text generation model hit this problem during generation stage? Or how do people generate text / save and load output from dynamic_rnn?

Thanks a lot in advance!