Hi! I wanted to get some insights on the optimal cloud configurations for exporting hail table or hail matrix table? Attached is a screenshot of the cloud analysis environment configuration I can work with. Are there certain things I need to consider while trying to make sure it runs smoothly? Thanks!
The most cost effective way to use Hail is with an autoscaling policy. I’m not sure if that is available within Terra.
I wrote some verbiage about how to use Hail on the cloud here. It’s a bit out of date (e.g. “preemptible” cores are now “spot” cores) but should get you started. There is no single “optimal” configuration. The best configuration depends on your data, the code you’re executing, and where in the execution process you are. Autoscaling policies are really helpful because they allow the cluster to automatically adapt to your data and the code you’re executing rather than you having to manually modify the cluster for every operation.