# Speed up of mt.entries().to\_pandas()

**URL:** <https://discuss.hail.is/t/speed-up-of-mt-entries-to-pandas/1646>\
**Category:** Hail Query & hailctl\
**Created:** [September 11, 2020, 10:57am UTC](https://discuss.hail.is/t/speed-up-of-mt-entries-to-pandas/1646 "2020-09-11T10:57:14Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![nonchev](https://avatars.discourse-cdn.com/v4/letter/n/54ee81/32.png) [@nonchev](https://discuss.hail.is/u/nonchev)\
**Post date:** [September 11, 2020, 10:57am UTC](https://discuss.hail.is/t/speed-up-of-mt-entries-to-pandas/1646/1 "2020-09-11T10:57:14Z")

</div>

Hello,

i have wrote this code:

> hl\_samples = hl.array(samples) - individuals  
> for mt\_path in mt\_paths:  
> mt = hl.methods.read\_matrix\_table(mt\_path)
> 
> ```
> # filter samples
> mt = mt.filter_cols(hl_samples.contains(mt.s))
> df = mt.entries().to_pandas()
> 
> # format df
> df[['ref','alt']] = pd.DataFrame(df.alleles.tolist(), index=df.index)
> df['GT'] = df['GT.alleles'].apply(_sum)
> df = df.drop(['alleles', 'GT.alleles', 'GT.phased'], axis=1)
> df.columns = ['chrom', 'pos', 'sample', 'DP', 'GQ', 'ref', 'alt', 'GT']
> df = df[['sample', 'chrom', 'pos', 'ref', 'alt', 'GT', 'DP', 'GQ']]
> 
> # append to final df with all variants per sample
> res_df = res_df.append(df, ignore_index = True)
> 
> ```

However `df = mt.entries().to_pandas()` takes 95% of the execution time and this is very slow.  
Generally I am converting to panda.dataframe just because I didnt fount a way to do this df operations in HAIL.  
Can you suggest me how can I speed it up?

Cheers,

---

<div class="post-metadata">

**Author:** ![nonchev](https://avatars.discourse-cdn.com/v4/letter/n/54ee81/32.png) [@nonchev](https://discuss.hail.is/u/nonchev)\
**Post date:** [September 11, 2020, 11:06am UTC](https://discuss.hail.is/t/speed-up-of-mt-entries-to-pandas/1646/2 "2020-09-11T11:06:31Z")

</div>

res\_df would look like this in the end:

 ![Screenshot 2020-09-11 at 14.05.58](https://canada1.discourse-cdn.com/flex036/uploads/hail/original/1X/cbabdf11cfd50066146c9e9456bc57a775c8d72b.png)

---

<div class="post-metadata">

**Author:** ![johnc1231](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/johnc1231/32/286_2.png) [@johnc1231](https://discuss.hail.is/u/johnc1231)\
**Post date:** [September 11, 2020, 1:35pm UTC](https://discuss.hail.is/t/speed-up-of-mt-entries-to-pandas/1646/3 "2020-09-11T13:35:52Z")

</div>

First of all, hail is lazily executed. So when you run `to_pandas`, it actually executes your whole pipeline at once (reading, filtering, writing to pandas). No matter what you do, that’s going to show up as the majority of the execution time.

Secondly, `entries` is always going to be relatively slow. You’re restructuring all your data, and converting it to an inefficient representation. The point of having `MatrixTable`s is to avoid having to make a giant `entries` table.

Thirdly, you can do everything you’re doing here in hail. Consider referencing the cheat sheets here: [https://hail.is/docs/0.2/cheatsheets.html](https://hail.is/docs/0.2/cheatsheets.html) to see functions available on `MatrixTable`s.

You’re essentially just using Hail as a VCF importer in the above code, and then doing everything in Pandas. Don’t do that, it’s going to be a bad experience.