Class meeting for 10-405 Workflows For Hadoop

From Cohen Courses
Revision as of 10:10, 5 March 2018 by Wcohen (Talk | contribs)

Jump to: navigation, search

This is one of the class meetings on the schedule for the course Machine Learning with Large Datasets 10-405 in Spring 2018.





Also discussed

Things to Remember

  • Combiners and how/when they improve efficiency
  • What dataflow languages are, what sort of abstract operations they use, and what the complexity of these operations is.
  • How joins are implemented in dataflow (and the difference between map-side and reduce-side joins)
  • What the PageRank algorithm is
  • Common ways of representing graphs in map-reduce system
    • A list of edges
    • A list of nodes with outlinks
  • Why iteration is often expensive in pure dataflow algorithms.
  • How Spark differs from and/or is similar to other dataflow algorithms
    • Actions/transformations
    • RDDs
    • Caching
  • Definition of a similarity join/soft join.
  • How to implement k-means in a map-reduce setting with dataflow
    • Not discussed in class, but in the slide deck!