Class meeting for 10-605 Parameter Servers

This is one of the class meetings on the schedule for the course Machine Learning with Large Datasets 10-605 in Fall_2016.

Slides

• Lecture: Powerpoint, PDF.

Quiz

• Today's quiz

Optional Readings

• Parameter Server for Distributed Machine Learning

• Strategies and Principles of Distributed Machine Learning on Big Data

• Petuum: A new platform for distributed machine learning on big data

Things to remember

• Architecture of a generic parameter server (PS), with get/put access to parameters
• Pros/cons of asynchronous vs bounded asynchronous vs fully synchronous PS
• Pros/cons of PS model versus Hadoop plus IPM
• Stale synchronous parallel (SSP) computation model
• Data-parallel versus model-parallel algorithms
  • Data-parallel example: SGD on sharded data
  • Model-parallel example: Lasso accounting for parameter dependencies and parameter importance