Straggler Mitigation through Unequal Error Protection for Distributed Matrix Multiplication

Busra Tegin, Eduin E. Hernandez, Stefano Rini, Tolga M. Duman

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

2 Scopus citations

Abstract

Large-scale machine learning and data mining methods routinely distribute computations across multiple agents to parallelize processing. The time required for computation at the agents is affected by the availability of local resources giving rise to the straggler problem in which the computation results are held back by unresponsive agents. For this problem, linear coding of the matrix sub-blocks can be used to introduce resilience toward straggling. The Parameter Server (PS) utilizes a channel code and distributes the matrices to the workers for multiplication. It then produces an approximation to the desired matrix multiplication using the results of the computations received at a given deadline. In this paper, we propose to employ Unequal Error Protection (UEP) codes to alleviate the straggler problem. The resiliency level of each sub-block is chosen according to its norm as blocks with larger norms have higher effects on the result of the matrix multiplication. We validate the effectiveness of our scheme both theoretically and through numerical evaluations. We derive a theoretical characterization of the performance of UEP using random linear codes, and compare it the case of equal error protection. We also apply the proposed coding strategy to the computation of the back-propagation step in the training of a Deep Neural Network (DNN), for which we investigate the fundamental trade-off between precision and the time required for the computations.

Original languageEnglish
Title of host publicationICC 2021 - IEEE International Conference on Communications, Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781728171227
DOIs
StatePublished - Jun 2021
Event2021 IEEE International Conference on Communications, ICC 2021 - Virtual, Online, Canada
Duration: 14 Jun 202123 Jun 2021

Publication series

NameIEEE International Conference on Communications
ISSN (Print)1550-3607

Conference

Conference2021 IEEE International Conference on Communications, ICC 2021
Country/TerritoryCanada
CityVirtual, Online
Period14/06/2123/06/21

Keywords

  • Approximate matrix multiplication
  • Distributed computation
  • Straggling servers
  • Unequal error protection

Fingerprint

Dive into the research topics of 'Straggler Mitigation through Unequal Error Protection for Distributed Matrix Multiplication'. Together they form a unique fingerprint.

Cite this