# Merge multiple sparse MT to one sparse MT **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [September 14, 2020, 8:37pm UTC](https://discuss.hail.is/t/merge-multiple-sparse-mt-to-one-sparse-mt/1658 "2020-09-14T20:37:15Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![jinasong](https://avatars.discourse-cdn.com/v4/letter/j/9fc348/32.png) [@jinasong](https://discuss.hail.is/u/jinasong)\ **Post date:** [September 14, 2020, 8:37pm UTC](https://discuss.hail.is/t/merge-multiple-sparse-mt-to-one-sparse-mt/1658/1 "2020-09-14T20:37:15Z")
Hello, Just in case you did not see my question in the reply in my other post, I ask it again here. I updated Hail, as of version 0.2.56 and tested run\_combiner() function with 100, 1k, and 10k gvcf files each (average size : 6G). Run\_combiner() runs for 100 gvcfs and 1k gvcfs completed successfully through multiple attempts of failed subtasks showing similar messages as before. Run time in a new version was faster than in the previous Hail version. Thanks much for your and your team’s work. The job for 10k gvcfs has been failed. The first round of 100 batches, merging 100 gvcfs to a sparse MT, completed successfully, but it failed when starting the second round with the error message as below. If you let me know **how to resolve this issue** , I will really appreciate it. – Caused by: java.io.IOException: All datanodes [DatanodeInfoWithStorage[\*\*\*\*\*\*\*,DISK]] are bad And, I found out 100 sparse MTs generated by the first round in a run\_combiner() run in my temp storage. **Is it possible to combine 100 MTs to 1 MT with any other Hail function**? Thank you. -Jina ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [September 15, 2020, 3:43pm UTC](https://discuss.hail.is/t/merge-multiple-sparse-mt-to-one-sparse-mt/1658/2 "2020-09-15T15:43:47Z")
Thanks very much for your patience on this. Could you share your call the the combiner? What temporary directory are you using? The “data nodes are bad” error might mean that nodes are full, I think that’s the exception HDFS throws when it runs out of storage space. ---
**Author:** ![jinasong](https://avatars.discourse-cdn.com/v4/letter/j/9fc348/32.png) [@jinasong](https://discuss.hail.is/u/jinasong)\ **Post date:** [September 15, 2020, 6:49pm UTC](https://discuss.hail.is/t/merge-multiple-sparse-mt-to-one-sparse-mt/1658/3 "2020-09-15T18:49:48Z")
Hi Tim @tpoterba, Thanks for your reply. I have attached a Python script to call the run\_combiner function and my commands to start a cluster and to submit a job. I hope it would helpful for solving my issue. By the way, “out of storage space” you mentioned, where does it mean and how can I increase it? -Jina \< Cluster on GCP\> - my-auto-policy [myauto] : max primary : 10 , max secondary : 1000 > hailctl dataproc start [mycluster] --vep GRCh38 --labels=mt=hm8-10k --autoscaling-policy=[myauto] --master-machine-type=n1-highmem-8 --worker-machine-type=n1-highmem-8 --properties=dataproc:dataproc.logging.stackdriver.enable=true,dataproc:dataproc.monitoring.stackdriver.enable=true \< Job \> > hailctl dataproc submit [mycluster] run\_gvcf\_combiner.py \< Function call in run\_gvcf\_combiner.py \> output\_file = ‘gs://[my\_bucket]/[MT\_folder]/10k\_20200817.mt’ # output destination temp\_bucket = ‘gs://[my\_bucket]/[temp\_folder]/’ # bucket for storing intermediate files hl.experimental.run\_combiner(inputs, out\_file=output\_file, tmp\_path=temp\_bucket, branch\_factor=100, batch\_size=100, reference\_genome=‘GRCh38’, use\_genome\_default\_intervals=True) ---
**Author:** ![chrisvittal](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/chrisvittal/32/327_2.png) [@chrisvittal](https://discuss.hail.is/u/chrisvittal)\ **Post date:** [September 15, 2020, 7:50pm UTC](https://discuss.hail.is/t/merge-multiple-sparse-mt-to-one-sparse-mt/1658/4 "2020-09-15T19:50:36Z")
Is that the full autoscaling policy? What’s the `workerConfig` `minInstances`? ---
**Author:** ![jinasong](https://avatars.discourse-cdn.com/v4/letter/j/9fc348/32.png) [@jinasong](https://discuss.hail.is/u/jinasong)\ **Post date:** [September 15, 2020, 8:20pm UTC](https://discuss.hail.is/t/merge-multiple-sparse-mt-to-one-sparse-mt/1658/5 "2020-09-15T20:20:33Z")
You can see my autoscaling policy here. - workerConfig: maxInstances: 10 minInstances: 2 weight: 1 - secondaryWorkerConfig: maxInstances: 1000 weight: 1 - basicAlgorithm: cooldownPeriod: 2m yarnConfig: – scaleUpFactor: 1.0 – scaleDownFactor: 1.0 – gracefulDecommissionTimeout: 120s ---
**Author:** ![jinasong](https://avatars.discourse-cdn.com/v4/letter/j/9fc348/32.png) [@jinasong](https://discuss.hail.is/u/jinasong)\ **Post date:** [September 21, 2020, 4:40pm UTC](https://discuss.hail.is/t/merge-multiple-sparse-mt-to-one-sparse-mt/1658/6 "2020-09-21T16:40:21Z")
Hi @tpoterba and @chrisvittal, If you let me know how to solve the “out of storage space” error or(and) how to merge multiple sparse matrix tables to one sparse matrkx table, I will really appreciate it. I am looking forward to your reply. -Jina