Chapter 1: Introduction to Deep Reinforcement LearningChapter Goal: Introduce the reader to field of reinforcement learning and setting the context of what they will learn in rest of the bookSub -Topics1. Deep reinforcement learning2. Examples and case studies3. Types of algorithms with mind-map4. Libraries and environment setup5. Summary
Chapter 2: Markov Decision ProcessesChapter Goal: Help the reader understand models, foundations on which all algorithms are built. Sub - Topics 1. Agent and environment2. Rewards3. Markov reward and decision processes4. Policies and value functions5. Bellman equations
Chapter 3: Model Based Algorithms Chapter Goal: Introduce reader to dynamic programming and related algorithms Sub - Topics:
1. Introduction to OpenAI Gym environment2. Policy evaluation/prediction3. Policy iteration and improvement4. Generalised policy iteration5. Value iteration
Chapter 4: Model Free ApproachesChapter Goal: Introduce Reader to model free methods which form the basis for majority of current solutionsSub - Topics: 1. Prediction and control with Monte Carlo methods2. Exploration vs exploitation3. TD learning methods4. TD control5. On policy learning using SARSA6. Off policy learning using q-learning
Chapter 5: Function Approximation Chapter Goal: Help readers understand value function approximation and Deep Learning use in Reinforcement Learning. 1. Limitations to tabular methods studied so far2. Value function approximation3. Linear methods and features used4. Non linear function approximation using deep Learning
Chapter 6: Deep Q-Learning
Chapter Goal: Help readers understand core use of deep learning in reinforcement learning. Deep q learning and many of its variants are introduced here with in depth code exercises. 1. Deep q-networks (DQN)2. Issues in Naive DQN 3. Introduce experience replay and target networks4. Double q-learning (DDQN)5. Duelling DQN6. Categorical 51-atom DQN (C51)7. Quantile regression DQN (QR-DQN)8. Hindsight experience replay (HER)
Chapter 7: Policy Gradient Algorithms Chapter Goal: Introduce reader to concept of policy gradients and related theory. Gain in depth knowledge of common policy gradient methods through hands-on exercises1. Policy gradient approach and its advantages2. The policy gradient theorem3. REINFORCE algorithm4. REINFORCE with baseline5. Actor-critic methods6. Advantage actor critic (A2C/A3C)7. Proximal policy optimization (PPO)8. Trust region policy optimization (TRPO)
Chapter 8: Combining Policy Gradients and Q-Learning Chapter Goal: Introduce reader to the trade offs between two approaches ways to connect together the two seemingly dissimilar approaches. Gain in depth knowledge of some land mark approaches.1. Tradeoff between policy gradients and q-learning2. The connection3. Deep deterministic policy gradient (DDPG)4. Twin delayed DDPG (TD3)5. Soft actor critic (SAC)
Chapter 9: Integrated Learning and Planning Chapter Goal: Introduce reader to the scalable approaches which are sample efficient for scalable problems.1. Model based reinforcement learning
About the Author:

Nimish is a passionate technical leader who brings to table extreme focus on use of technology for solving customer problems. He has over 25 years of work experience in the Software and Consulting. Nimish has held leadership roles with P&L responsibilities at PwC, IBM and Oracle. In 2006 he set out on his entrepreneurial journey in Software consulting at SOAIS with offices in Boston, Chicago and Bangalore. Today the firm provides Automation and Digital Transformation services to Fortune 100 companies helping them make the transition from on-premise applications to the cloud.

He is also an angel investor in the space of AI and Automation driven startups. He has co-founded Paybooks, a SaaS HR and Payroll platform for Indian market. He has also cofounded a Boston based startup which offers ZipperAgent and ZipperHQ, a suite of AI driven workflow and video marketing automation platforms. He currently hold the position as CTO and Chief Data Scientist for both these platforms.

Nimish has an MBA from Indian Institute of Management in Ahmedabad, India and a BS in Electrical Engineering from Indian Institute of Technology in Kanpur, India. He also holds multiple certifications in AI and Deep Learning.