These two errors are symptoms of the same problem – a computational graph that is too large/deep. This isn’t your fault, and we should handle these kind of massive queries gracefully by using data structures on the heap instead of using the function stack in Python and the scala backend.
How many families do you have? This part is likely triggering the depth error:
denseMatrix = denseByFamily[ 0 ]
for ii in range(1 , len( denseByFamily ) ):
denseMatrix = full_outer_join_mt( denseMatrix, denseByFamily[ ii ] )
denseMatrix.write( outputDenseMatrix, overwrite = True )
It’s possible that redesigning this to do a tree merge could help, something like this:
but with full_outer_join_mt instead of union_cols.