I have been accepted to Google Summer of Code program to work on TensorFlow for three months. I will be working on TensorFlow’s reinforcement learning library TF-Agents. In this post, I briefly summmarize the steps I took to setup the TF-Agents environment for future reference.
Create Anaconda Environment
I use conda to manage Python environments, so I first created a new conda environment for GSoC:
conda create -n gsoc
Output
Collecting package metadata: done
Solving environment: done
## Package Plan ##
environment location: /home/ryanlee/anaconda3/envs/gsoc
Proceed ([y]/n)? y
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate gsoc
#
# To deactivate an active environment, use
#
# $ conda deactivate
I then activated the environment.
source activate gsoc
Since I will use pip to install packages, I needed to install pip to separate my Python libraries.
conda install pip
Install TensorFlow
To install TF-Agents, I need to install TensorFlow first. I installed the tf-nightly package since I installed it on a laptop without GPU.
pip install tf-nightly # To use only CPU pip install tf-nightly-gpu # To use both CPU and GPU
Output
Collecting tf-nightly
Using cached https://files.pythonhosted.org/packages/0d/b6/ac3fce5f57a0cd1262ff0a8bf75a3ccc35bacf454e2dab898e6d7f14dc6e/tf_nightly-1.14.1.dev20190508-cp37-cp37m-manylinux1_x86_64.whl
Collecting protobuf>=3.6.1 (from tf-nightly)
Using cached https://files.pythonhosted.org/packages/19/a5/ac51df34cdf4739574492ed4903c11dadd72a7bec4a31bb0496f4f50fc19/protobuf-3.7.1-cp37-cp37m-manylinux1_x86_64.whl
Requirement already satisfied: wheel>=0.26 in /home/ryanlee/anaconda3/envs/gsoc/lib/python3.7/site-packages (from tf-nightly) (0.33.1)
Collecting termcolor>=1.1.0 (from tf-nightly)
Collecting keras-applications>=1.0.6 (from tf-nightly)
Using cached https://files.pythonhosted.org/packages/90/85/64c82949765cfb246bbdaf5aca2d55f400f792655927a017710a78445def/Keras_Applications-1.0.7-py2.py3-none-any.whl
Collecting google-pasta>=0.1.2 (from tf-nightly)
Using cached https://files.pythonhosted.org/packages/f9/68/a14620bfb042691f532dcde8576ff82ee82e4c003cdc0a3dbee5f289cee6/google_pasta-0.1.6-py3-none-any.whl
Collecting astor>=0.6.0 (from tf-nightly)
Using cached https://files.pythonhosted.org/packages/35/6b/11530768cac581a12952a2aad00e1526b89d242d0b9f59534ef6e6a1752f/astor-0.7.1-py2.py3-none-any.whl
Collecting numpy<2.0,>=1.14.5 (from tf-nightly)
Using cached https://files.pythonhosted.org/packages/bb/76/24e9f32c78e6f6fb26cf2596b428f393bf015b63459468119f282f70a7fd/numpy-1.16.3-cp37-cp37m-manylinux1_x86_64.whl
Collecting wrapt>=1.11.1 (from tf-nightly)
Collecting keras-preprocessing>=1.0.5 (from tf-nightly)
Using cached https://files.pythonhosted.org/packages/c0/bf/0315ef6a9fd3fc2346e85b0ff1f5f83ca17073f2c31ac719ab2e4da0d4a3/Keras_Preprocessing-1.0.9-py2.py3-none-any.whl
Collecting grpcio>=1.8.6 (from tf-nightly)
Using cached https://files.pythonhosted.org/packages/44/3c/0f680a3e2e7720dc1b37bf3163b1f62f0f847dc081a17f2a2f4389e86a38/grpcio-1.20.1-cp37-cp37m-manylinux1_x86_64.whl
Collecting tb-nightly<1.15.0a0,>=1.14.0a0 (from tf-nightly)
Using cached https://files.pythonhosted.org/packages/28/ea/961dab12737fc150d457e72eb1dd39cf6964b9be752955ea25a3daaf4a91/tb_nightly-1.14.0a20190508-py3-none-any.whl
Collecting absl-py>=0.7.0 (from tf-nightly)
Collecting tf-estimator-nightly (from tf-nightly)
Using cached https://files.pythonhosted.org/packages/9b/c3/2aecf2510a292a905af1a93ad510ff9dcfa23a2a10d0a0124c8392e6e4da/tf_estimator_nightly-1.14.0.dev2019050801-py2.py3-none-any.whl
Collecting six>=1.10.0 (from tf-nightly)
Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting gast>=0.2.0 (from tf-nightly)
Requirement already satisfied: setuptools in /home/ryanlee/anaconda3/envs/gsoc/lib/python3.7/site-packages (from protobuf>=3.6.1->tf-nightly) (41.0.1)
Collecting h5py (from keras-applications>=1.0.6->tf-nightly)
Using cached https://files.pythonhosted.org/packages/8e/fd/2ca5c4f4ed33ac4178f9c4d551e3946ab480866e3cd67a65a67a4bb35367/h5py-2.9.0-cp37-cp37m-manylinux1_x86_64.whl
Collecting markdown>=2.6.8 (from tb-nightly<1.15.0a0,>=1.14.0a0->tf-nightly)
Using cached https://files.pythonhosted.org/packages/f5/e4/d8c18f2555add57ff21bf25af36d827145896a07607486cc79a2aea641af/Markdown-3.1-py2.py3-none-any.whl
Collecting werkzeug>=0.11.15 (from tb-nightly<1.15.0a0,>=1.14.0a0->tf-nightly)
Using cached https://files.pythonhosted.org/packages/18/79/84f02539cc181cdbf5ff5a41b9f52cae870b6f632767e43ba6ac70132e92/Werkzeug-0.15.2-py2.py3-none-any.whl
Installing collected packages: six, protobuf, termcolor, numpy, h5py, keras-applications, google-pasta, astor, wrapt, keras-preprocessing, grpcio, absl-py, markdown, werkzeug, tb-nightly, tf-estimator-nightly, gast, tf-nightly
Successfully installed absl-py-0.7.1 astor-0.7.1 gast-0.2.2 google-pasta-0.1.6 grpcio-1.20.1 h5py-2.9.0 keras-applications-1.0.7 keras-preprocessing-1.0.9 markdown-3.1 numpy-1.16.3 protobuf-3.7.1 six-1.12.0 tb-nightly-1.14.0a20190508 termcolor-1.1.0 tf-estimator-nightly-1.14.0.dev2019050801 tf-nightly-1.14.1.dev20190508 werkzeug-0.15.2 wrapt-1.11.1
Install TF-Agents
I followed the TF-Agents installation guide. First, I cloned the repository:
git clone https://github.com/tensorflow/agents.git
Then, I installed tf_agents from the cloned repository.
cd agents/ pip install -e .
Output
Obtaining file:///home/ryanlee/git/tf-agents
Requirement already satisfied: absl-py>=0.6.1 in /home/ryanlee/anaconda3/envs/gsoc/lib/python3.7/site-packages (from tf-agents-nightly==0.2.0.dev20190509) (0.7.1)
Collecting gin-config==0.1.3 (from tf-agents-nightly==0.2.0.dev20190509)
Using cached https://files.pythonhosted.org/packages/8c/be/c984b1c8a7ba1c385b32bf39c7a225cd9f713d49705898309d01b60fd0e7/gin_config-0.1.3-py3-none-any.whl
Requirement already satisfied: numpy>=1.13.3 in /home/ryanlee/anaconda3/envs/gsoc/lib/python3.7/site-packages (from tf-agents-nightly==0.2.0.dev20190509) (1.16.3)
Requirement already satisfied: six>=1.10.0 in /home/ryanlee/anaconda3/envs/gsoc/lib/python3.7/site-packages (from tf-agents-nightly==0.2.0.dev20190509) (1.12.0)
Collecting tfp-nightly (from tf-agents-nightly==0.2.0.dev20190509)
Using cached https://files.pythonhosted.org/packages/a8/5d/4435668ff4b7b1544bc546978c1b743ec9bfcb0925632252adde4c78484f/tfp_nightly-0.7.0.dev20190508-py2.py3-none-any.whl
Collecting decorator (from tfp-nightly->tf-agents-nightly==0.2.0.dev20190509)
Using cached https://files.pythonhosted.org/packages/5f/88/0075e461560a1e750a0dcbf77f1d9de775028c37a19a346a6c565a257399/decorator-4.4.0-py2.py3-none-any.whl
Collecting cloudpickle>=0.6.1 (from tfp-nightly->tf-agents-nightly==0.2.0.dev20190509)
Using cached https://files.pythonhosted.org/packages/c6/d9/d45cdb70f3d86480f02f220bc2ec6da69a45de4a5bb61a49fd4a5106ada8/cloudpickle-1.0.0-py2.py3-none-any.whl
Installing collected packages: gin-config, decorator, cloudpickle, tfp-nightly, tf-agents-nightly
Running setup.py develop for tf-agents-nightly
Successfully installed cloudpickle-1.0.0 decorator-4.4.0 gin-config-0.1.3 tf-agents-nightly tfp-nightly-0.7.0.dev20190508
Verify Installation
To make sure that I installed TF-Agents correctly, I trained a DQN agent on CartPole environment. Since CartPole is from OpenAI Gym, I first installed the gym package:
pip install gym
Then, I ran the example script provided in TF-Agents repository:
python tf_agents/agents/dqn/examples/v1/train_eval_gym.py \ --root_dir=$HOME/tmp/dqn/gym/cart-pole/ \ --alsologtostderr
Output
W0509 11:08:39.915435 140633859897088 deprecation_wrapper.py:119] From tf_agents/agents/dqn/examples/v1/train_eval_gym.py:293: The name tf.enable_resource_variables is deprecated. Please use tf.compat.v1.enable_resource_variables instead.
W0509 11:08:39.969043 140633859897088 deprecation.py:506] From /home/ryanlee/git/tf-agents/tf_agents/utils/common.py:137: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
2019-05-09 11:08:40.033286: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2893490000 Hz
2019-05-09 11:08:40.033533: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x559d27472b60 executing computations on platform Host. Devices:
2019-05-09 11:08:40.033554: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined>
I0509 11:08:42.020883 140633859897088 common.py:845] No checkpoint available at /home/ryanlee/tmp/dqn/gym/cart-pole/train
I0509 11:08:42.025129 140633859897088 common.py:845] No checkpoint available at /home/ryanlee/tmp/dqn/gym/cart-pole/train/policy
I0509 11:08:42.029206 140633859897088 common.py:845] No checkpoint available at /home/ryanlee/tmp/dqn/gym/cart-pole/train/replay_buffer
2019-05-09 11:08:42.283903: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1336] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0509 11:08:42.686741 140633859897088 common.py:812] uninitialized_vars:
I0509 11:08:42.686947 140633859897088 common.py:814] <tf.Variable 'save_counter_1:0' shape=() dtype=int64>
W0509 11:08:42.988529 140631959648000 backprop.py:820] The dtype of the watched tensor must be floating (e.g. tf.float32), got tf.int64
W0509 11:08:42.989435 140631959648000 backprop.py:820] The dtype of the watched tensor must be floating (e.g. tf.float32), got tf.int32
W0509 11:08:42.989624 140631959648000 backprop.py:820] The dtype of the watched tensor must be floating (e.g. tf.float32), got tf.int64
W0509 11:08:42.989779 140631959648000 backprop.py:820] The dtype of the watched tensor must be floating (e.g. tf.float32), got tf.int32
W0509 11:08:42.990668 140631959648000 backprop.py:820] The dtype of the watched tensor must be floating (e.g. tf.float32), got tf.int32
I0509 11:08:44.620225 140633859897088 metric_utils.py:47] Step = 0
AverageReturn = 9.399999618530273
AverageEpisodeLength = 9.399999618530273
I0509 11:08:47.787508 140633859897088 train_eval_gym.py:258] step = 1000, loss = 2.022921
I0509 11:08:47.867046 140633859897088 train_eval_gym.py:264] 324.246 steps/sec
I0509 11:08:47.867327 140633859897088 train_eval_gym.py:266] collect_time = 2.0146474838256836, train_time = 1.069429636001587
I0509 11:08:50.807419 140633859897088 train_eval_gym.py:258] step = 2000, loss = 6.163950
I0509 11:08:50.807928 140633859897088 train_eval_gym.py:264] 414.814 steps/sec
I0509 11:08:50.808043 140633859897088 train_eval_gym.py:266] collect_time = 1.6414339542388916, train_time = 0.7692830562591553
Check TensorBoard
The example script writes log files that can be visualized in TensorBoard.
tensorboard --logdir $HOME/tmp/dqn/gym/cart-pole/eval
Output
TensorBoard 1.14.0a20190508 at http://ryanlee-ThinkPad-T430s:6006/ (Press CTRL+C to quit)
To check TensorBoard visualizations, navigate to localhost:6006.
Encountered Errors
1. No attribute __version__
AttributeError: module 'tensorflow' has no attribute '__version__'
I got this error two times:
- When I tried to install TF-Agents, and
- When I tried running the TF-Agents example script
1-1. Check TensorFlow Version
First, check that you DON’T have tensorflow package and DO have tf-nightly or tf-nightly-gpu package.
pip list | grep tensorflow # Should output nothing pip list | grep tf-nightly # Should output 1 line (for computers without GPU) pip list | grep tf-nightly-gpu # Should output 1 line (for computers with GPU)
If you have tensorflow installed, remove it.
pip uninstall tensorflow
If you don’t have tf-nightly or tf-nightly-gpu installed, install it.
pip install tf-nightly # To use only CPU pip install tf-nightly-gpu # To use both CPU and GPU
1-2. Deactivate and Activate conda Environment
This error sometimes also be fixed by simply reactivating the Anaconda environment.
source deactivate source activate gsoc