Warm welcome to 2017 – TensorFlow!

A warm welcome to all The Information Age readers, followers or just curious about. We are back and within a new year. 2016 was the year of major developments in Deep Learning and Artificial Intelligence. There were countless stories, papers and decisions within those fields, and this Blog just scratch the very outer surface of what was going on; moreover I just started Blogging on a daily basis in the last three months of 2016, but the attentive reader would have noticed that I posted almost every week about deep learning and AI, maybe more than twice a week.

For 2017 I intend to keep The Information Age attentive about AI. But the spirit and purpose of the Blog will continue to be about broader issues about Information Technology, Information Theory and theories, Computer Science and Data Science, and related fields of study that might be concurrent and intertwined with those subjects; for instance I have published some interesting posts about Quantum Computing and Quantum Computation, which are fields  I see promising to be crucially important for all that this blog is about.

I decided today to share a post from the O’Reilly FinTech newsletter. I subscribe and receive on a regular basis almost all of O’Reilly Media newsletter dealing with technology issues, and I recommend it keenly. This newsletter is about FinTech, the applications of cutting-edge technologies to Financial Services businesses; but within the list of links proposed in the newsletter we could find today post on the important Deep Leaning library TensorFlow. This should not come as a surprise as deep learning and machine learning frameworks already are important technologies for FinTech applications, and the future possible applications are yet to be discovered, with knowledgeable people predicting more to come. The post I will share and pass some of the highlights below is a sort of another easy introduction to TensorFlow, the potential and significance of the library, and this isn’t about hype but about insisting on about coming back to basics. There is a need to do this because of the steep learning curve to even starting to understand the basics, let alone mastering the entire library. So reminders and refocusing arent hype, indeed.

The post is interesting also from a software engineering perspective, as the Python scripts are thoroughly highlighted:

Hello, TensorFlow!

Building and training your first TensorFlow graph from the ground up.

The TensorFlow project is bigger than you might realize. The fact that it’s a library for deep learning, and its connection to Google, has helped TensorFlow attract a lot of attention. But beyond the hype, there are unique elements to the project that are worthy of closer inspection:

  • The core library is suited to a broad family of machine learning techniques, not “just” deep learning.
  • Linear algebra and other internals are prominently exposed.
  • In addition to the core machine learning functionality, TensorFlow also includes its own logging system, its own interactive log visualizer, and even its own heavily engineered serving architecture.
  • The execution model for TensorFlow differs from Python’s scikit-learn, or most tools in R.

Cool stuff, but—especially for someone hoping to explore machine learning for the first time—TensorFlow can be a lot to take in.

How does TensorFlow work? Let’s break it down so we can see and understand every moving part. We’ll explore the data flow graph that defines the computations your data will undergo, how to train models with gradient descent using TensorFlow, and how TensorBoard can visualize your TensorFlow work. The examples here won’t solve industrial machine learning problems, but they’ll help you understand the components underlying everything built with TensorFlow, including whatever you build next!


Names and execution in Python and TensorFlow

The way TensorFlow manages computation is not totally different from the way Python usually does. With both, it’s important to remember, to paraphrase Hadley Wickham, that an object has no name (see Figure 1). In order to see the similarities (and differences) between how Python and TensorFlow work, let’s look at how they refer to objects and handle evaluation.


Figure 1. Names “have” objects, rather than the reverse. Image courtesy of Hadley Wickham, used with permission.


Internally, Python manages all your objects and keeps track of your variable names and which objects they refer to. The TensorFlow graph represents another layer of this kind of management; as we’ll see, Python names will refer to objects that connect to more granular and managed TensorFlow graph operations.


When you enter a Python expression, for example at an interactive interpreter or Read Evaluate Print Loop (REPL), whatever is read is almost always evaluated right away. Python is eager to do what you tell it. So, if I tell Python to foo.append(bar), it appends right away, even if I never use foo again.


TensorFlow separates the definition of computations from their execution even further by having them happen in separate places: a graph defines the operations, but the operations only happen within a session. Graphs and sessions are created independently. A graph is like a blueprint, and a session is like a construction site.


Back to our plain Python example, recall that foo and bar refer to the same list. By appending bar into foo, we’ve put a list inside itself. You could think of this structure as a graph with one node, pointing to itself. Nesting lists is one way to represent a graph structure like a TensorFlow computation graph.

>>> foo.append(bar)
>>> foo
## [[...]]


The nodes of the TensorFlow graph are called “operations,” or “ops.” We can see what operations are in the graph with graph.get_operations().


>>> graph.get_operations()


Currently, there isn’t anything in the graph. We’ll need to put everything we want TensorFlow to compute into that graph. Let’s start with a simple constant input value of one.


>>> input_value = tf.constant(1.0)


That constant now lives as a node, an operation, in the graph. The Python variable name input_value refers indirectly to that operation, but we can also find the operation in the default graph.

>>> operations = graph.get_operations()
>>> operations
## []
>>> operations[0].node_def
## name: "Const"
## op: "Const"
## attr {
##   key: "dtype"
##   value {
##     type: DT_FLOAT
##   }
## }
## attr {
##   key: "value"
##   value {
##     tensor {
##       dtype: DT_FLOAT
##       tensor_shape {
##       }
##       float_val: 1.0
##     }
##   }
## }


TensorFlow uses protocol buffers internally. (Protocol buffers are sort of like a Google-strength JSON.) Printing the node_def for the constant operation above shows what’s in TensorFlow’s protocol buffer representation for the number one.


People new to TensorFlow sometimes wonder why there’s all this fuss about making “TensorFlow versions” of things. Why can’t we just use a normal Python variable without also defining a TensorFlow object? One of the TensorFlow tutorials has an explanation:


To do efficient numerical computing in Python, we typically use libraries like NumPy that do expensive operations such as matrix multiplication outside Python, using highly efficient code implemented in another language. Unfortunately, there can still be a lot of overhead from switching back to Python every operation. This overhead is especially bad if you want to run computations on GPUs or in a distributed manner, where there can be a high cost to transferring data.


TensorFlow also does its heavy lifting outside Python, but it takes things a step further to avoid this overhead. Instead of running a single expensive operation independently from Python, TensorFlow lets us describe a graph of interacting operations that run entirely outside Python. This approach is similar to that used in Theano or Torch.



TensorFlow can do a lot of great things, but it can only work with what’s been explicitly given to it. This is true even for a single constant.


If we inspect our input_value, we see it is a constant 32-bit float tensor of no dimension: just one number:


>>> input_value
## <tf.Tensor 'Const:0' shape=() dtype=float32>


Note that this doesn’t tell us what that number is. To evaluate input_value and get a numerical value out, we need to create a “session” where graph operations can be evaluated and then explicitly ask to evaluate or “run” input_value. (The session picks up the default graph by default.)


>>> sess = tf.Session()
>>> sess.run(input_value)## 1.0


It may feel a little strange to “run” a constant. But it isn’t so different from evaluating an expression as usual in Python; it’s just that TensorFlow is managing its own space of things—the computational graph—and it has its own method of evaluation.

Log Out

Ok, I will log out for now. I still want to come back to the post to further learn about TensorFlow! I hope that everyone have read this brief flavored introduction about this library and have felt the same impetus to learn more, specially if your keen on data engineering and software development topics. Let us not forget though that to proper learn all there is to learn about Deep Learning and Artificial Intelligence, from a basic background perspective, TensorFlow might not be the proper and best option to go along with. Hello 2017!

Here’s a final version of the code. It’s fairly minimal, with every part showing useful (and understandable) TensorFlow functionality.

import tensorflow as tf

x = tf.constant(1.0, name='input')
w = tf.Variable(0.8, name='weight')
y = tf.mul(w, x, name='output')

y_ = tf.constant(0.0, name='correct_value')
loss = tf.pow(y - y_, 2, name='loss')
train_step = tf.train.GradientDescentOptimizer(0.025).minimize(loss)

for value in [x, w, y, y_, loss]:
    tf.scalar_summary(value.op.name, value)

summaries = tf.merge_all_summaries()

sess = tf.Session()
summary_writer = tf.train.SummaryWriter('log_simple_stats', sess.graph)

for i in range(100):
    summary_writer.add_summary(sess.run(summaries), i)

featured image: Training Course: Deep Learning and TensorFlow (with Hands-On Exercises)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s