mpi deadlock with matrix-vector multiplication

Question

I'm trying to implement the matrix-vector MPI program in the 'Using MPI Portable Parallel Programming with the Message-Passing Interface Second Edition William Gropp Ewing Lusk Anthony Skjellum' book, page 35 here.

It is a self scheduling program in which the 0 rank process broadcast the vector to all other processes and sends a row of the matrix to each of them. The processes then compute the dot product between the vector and the row and send the result back to rank 0 process. The example in the book is in fortran and i ported it in c++ but it deadlocks after the first send

#include "mpi.h"
#include <iostream>
#include <vector>

int main(){

    int W_rank, W_size;
    int rows = 9;
    int cols = 9;

    std::vector<double> b(cols);
    std::vector< std::vector<double> > a(rows);
    std::vector<double> c(rows);
    std::vector<double> buffer(cols);

    for(auto& row : a) row.resize(cols);

    MPI_Init(NULL, NULL);
    MPI_Comm_size(MPI_COMM_WORLD, &W_size);
    MPI_Comm_rank(MPI_COMM_WORLD, &W_rank);

    if(W_rank == 0){
        
        std::fill(b.begin(), b.end(), 1);
        
        for(auto& row : a){
            std::fill(row.begin(), row.end(), 1);
        }

        int numsent = 0;
        
        MPI_Bcast(b.data(), b.size(), MPI_DOUBLE, 0, MPI_COMM_WORLD);
        
        for(int i=0; i<W_size; ++i){
            for(int j=0; j<cols; ++j){
                buffer[j] = a[i][j];
            }
            MPI_Send(buffer.data(), buffer.size(), MPI_DOUBLE, i, i, MPI_COMM_WORLD);///////deadlocks here///////////
            ++numsent;
        }

        double ans;
        MPI_Status status;
        for(int i=0; i<rows; ++i){
            MPI_Recv(&ans, 1, MPI_DOUBLE, MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, &status);
            int sender = status.MPI_SOURCE;
            int tag = status.MPI_TAG;
            c[tag] = ans;
            if(numsent<rows){
                for(int j=0; j<cols; ++j){
                    buffer[j] = a[numsent][j];
                }
                MPI_Send(buffer.data(), buffer.size(), MPI_DOUBLE, sender, numsent, MPI_COMM_WORLD);
                ++numsent;
            }
            else{
                MPI_Send(MPI_BOTTOM, 0, MPI_DOUBLE, sender, -1, MPI_COMM_WORLD);
            }
        }
    }
    else{

        MPI_Bcast(b.data(), b.size(), MPI_DOUBLE, 0, MPI_COMM_WORLD);
        
        MPI_Status status;
        
        while(1){
            MPI_Recv(buffer.data(), buffer.size(), MPI_DOUBLE, 0, MPI_ANY_TAG, MPI_COMM_WORLD, &status);//////deadlocks here///////////////////
            if(status.MPI_TAG == -1) break;
            int row = status.MPI_TAG;
            double ans = 0;
            for(int i=0; i<cols; ++i){
                ans += buffer[i]*b[i];
            }
            MPI_Send(&ans, 1, MPI_DOUBLE, 0, row, MPI_COMM_WORLD);
        }
       
    }
    
    MPI_Finalize();

    return 0;
}

Answer 1

Your handling of MPI tags is dangerous or wrong. First of all, tags have to be nonnegative, so your -1 tag is illegal. If internally it's interpreted as unsigned it becomes a very large tag. And secondly: there is an upper bound to tag values, so you using numsent as the tag is clever, but potentially a problem. I would use tag=0 for regular messages and tag=1 for termination, or something like that.