I am using MatrixTable to create inputs and outputs for my AI model. I want to randomly shuffle all rows because of that in the MatrixTable. Is there a way to do that? If not, do you know if there is a way to convert it to a spark object and do it there instead somehow? I tried the following:
tot_num = mt_3.count() sample_size = int(0.3 * tot_num) mt_3 = mt_3.sample_rows(p=sample_size / tot_num, seed=42)
But when I check the output:
mt_3.show() it does not randomly arrange them, I see only chromosome 1 top rows.