Parallel Computing

Parallel computing is about getting a problem to run faster (or at all) by splitting it across many processors, which forces you to think carefully about how work decomposes, how data moves, and where the real bottleneck actually lives — usually communication, not computation. You will spend most of the semester on four projects implementing the same kinds of kernels (matrix multiplies, reductions, broadcasts) across MPI, OpenMP, and CUDA, then analyzing speedup, efficiency, and scalability to explain why your numbers look the way they do. It is a graduate-level follow-on to algorithms and architecture, and the mental model you build here — decomposition, mapping, overhead, isoefficiency — is what underlies almost every modern HPC, distributed-systems, and large-scale ML workload.