Class meeting for 10-405 Workflows For Hadoop

From Cohen Courses
Jump to: navigation, search

This is one of the class meetings on the schedule for the course Machine Learning with Large Datasets 10-405 in Spring 2018.





Also discussed

Things to Remember

  • Combiners and how/when they improve efficiency
  • What dataflow languages are, what sort of abstract operations they use, and what the complexity of these operations is.
  • How joins are implemented in dataflow
    • The difference between map-side and reduce-side joins and how they are implemented
    • When to use map-side vs reduce-side joins
  • Definition of a similarity join/soft join.
  • Complexity of operations like similarity join, TFIDF computation, etc.
  • What the PageRank algorithm is
  • Common ways of representing graphs in map-reduce system
    • A list of edges
    • A list of nodes with outlinks
  • Why iteration is often expensive in pure dataflow algorithms.
  • How Spark differs from and/or is similar to other dataflow algorithms
    • Actions/transformations
    • RDDs
    • Caching
  • How to implement k-means in a map-reduce setting with dataflow
    • Not discussed in class, but in the slide deck!