Deep Learning is modular: Diogo Moitinho de Almeida #reworkDL

I share today a video from the #reworkDL ( Deep Learning Summit London 2016) once again. The speaker raised my attention for two reasons: his is a very Portuguese name, and he is a Math Olympiad champion, achieving a higher distinction on the International Math Olympiad.

From the YouTube description of this video:

Diogo Moitinho de Almeida is a data scientist, software engineer, and hacker. He has previously been a medalist at the International Math Olympiad ending a 13-year losing streak for the Philippines, received the top prize in the Interdisciplinary Contest in Modeling achieving the highest distinction of any team from the Western Hemisphere, and won a Kaggle competition setting a new state-of-the-art for black box identification of causality and getting the opportunity to speak at the Conference on Neural Information Processing Systems. As a lifelong learner and big fan of online education, he has taken more classes online than he has getting his undergraduate degree at the Rensselaer Polytechnic Institute. He loves all things software and enjoys contributing to open source, giving talks on things that he’s built, and improving his Emacs setup.

Diogo is a Data Scientist at Enlitic, considered one of the most innovative companies within the new ecosystem of companies emerging around Deep Learning. Enlitic is active in Healthcare. Indeed this is one area of applications of Artificial Intelligence (AI) and Deep Learning with the most promise. The field of Medicine is going through a quiet revolution and technology is one of the driving forces of this change. Medicine has been a part of our human Civilization that has changed little for the past, say, 5000 years. The next 50 years may completely change Medicine. The doctor diagnosing a disease and prescribing a treatment model has its days counted. Instead in the future we will witness (those on the lucky side of being around 50 plus years from now…) a shift towards a deeper connection between patient and his physician, with the relation being almost one of a partnership, or an equal ground of knowledge of what needs to be done. AI and technology will certainly be part of that. This will be both good for the patient and the doctor, freeing the physician to pursue further and further research and development, for instance in molecular medicine that also implements deep learning and AI frameworks. A typical win-win scenario. The devils might emerge in details and imperfections…

The talk precisely points to some limitations in current deep learning frameworks. This isn’t to be viewed as a negative or pessimistic position though. Pointing out limitations or shortcomings are actually essential for anything to evolve further, we must remember. One source of difficulty is the modular nature of current deep learning frameworks. Modularity in networks creates scalability challenges, but this is the cost of having the advantages of connected nodes of operational units. Diogo then lists other challenges related with flexibility, optimization and the like:

Issues with:

  • Optimization


  • Efficiency


  • Creativity


  • Flexibility


The surprising pessimistic views about the limitations of the deep learning frameworks that have emerged like TensorFlow, Caffe, Theano, Torch, etc. are worth a mention and an eye opener. Diogo thinks the inflexibility these frameworks create is largely unnecessary. Batch normalization was refered to as one of the biggest achievements in deep learning and the mentioned frameworks still have some efficiency issues with this process. The rest of the talk lists the other problems like hyperparameter scheduling, etc… Important to watch for the interested. Can we do better? I believe that we can!   😉

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s