I have a quality-of-life question. When running Hail on DNAnexus, my notebook often gets disconnected while code is running (e.g. I need to close my laptop). I understand that I can use a JupyterLab magic command, e.g.
# a bunch of hail commands in a cell
and subsequently use out.show() to view the output. This works when the cell doesn’t produce an error. When the cell produces an error, though, out is essentially empty. It would be very useful to capture the error so that I have an idea of how to troubleshoot. Any suggestions?
I’ll post this question to the DNAnexus forums as well.
I think someone from DNAnexus commented that they themself never work in DNAnexus notebooks since they are unreliable. Better to work in a normal notebook and just upload it after you are done. Don’t ask me why this is the case.
FWIW, this is what we recommend. Jupyter Notebooks are just too unreliable for exactly the reasons you’re describing. I don’t know why the capture magic doesn’t pick up the exceptions. Hail’s exceptions are just normal Python exceptions :/.
Does DNANexus have a “submit script” method? This is how we generally work at Broad. We submit scripts which write to well-known locations. When the script finishes, we can kick up a notebook locally to inspect the (small) analysis results.