Difference between revisions of "Class meeting for 10-405 Workflows For Hadoop"

From Cohen Courses
Jump to: navigation, search
(Slides)
(Things to Remember)
Line 26: Line 26:
 
=== Things to Remember ===
 
=== Things to Remember ===
  
* The TFIDF representation for documents.
+
* Combiners and how/when they improve efficiency
 
* What dataflow languages are, what sort of abstract operations they use, and what the complexity of these operations is.
 
* What dataflow languages are, what sort of abstract operations they use, and what the complexity of these operations is.
 
* How joins are implemented in dataflow (and the difference between map-side and reduce-side joins)
 
* How joins are implemented in dataflow (and the difference between map-side and reduce-side joins)

Revision as of 11:10, 5 March 2018

This is one of the class meetings on the schedule for the course Machine Learning with Large Datasets 10-405 in Spring 2018.

Contents

Slides

Quizzes

Readings

Also discussed

Things to Remember

  • Combiners and how/when they improve efficiency
  • What dataflow languages are, what sort of abstract operations they use, and what the complexity of these operations is.
  • How joins are implemented in dataflow (and the difference between map-side and reduce-side joins)
  • What the PageRank algorithm is
  • Common ways of representing graphs in map-reduce system
    • A list of edges
    • A list of nodes with outlinks
  • Why iteration is often expensive in pure dataflow algorithms.
  • How Spark differs from and/or is similar to other dataflow algorithms
    • Actions/transformations
    • RDDs
    • Caching
  • Definition of a similarity join/soft join.
  • How to implement k-means in a map-reduce setting with dataflow
    • Not discussed in class, but in the slide deck!