Split multi: index out of bound on chrX - LPL column

Hello I am working with VariantDataset and I want to use vds = hl.vds.split_multi(vds)
It works ok, but I have problem with chrX with LPL entries.

When I split multi on chrX the following error shows up:
Error summary: HailException: array index out of bounds: index=0, length=0

… line 189, in \n .map(lambda idx: old_entry.LPL[idx]

If I select entries like this:
vds.variant_data = vds.variant_data.select_entries(vds.variant_data.LA, vds.variant_data.LGT, vds.variant_data.LAD, vds.variant_data.RGQ, vds.variant_data.gvcf_info,
vds.variant_data.AF, vds.variant_data.DP, vds.variant_data.F1R2, vds.variant_data.F2R1, vds.variant_data.GP, vds.variant_data.GQ,
vds.variant_data.ICNT, vds.variant_data.MB, vds.variant_data.MIN_DP, vds.variant_data.PRI, vds.variant_data.PS,
vds.variant_data.SB, vds.variant_data.SPL, vds.variant_data.SQ
)
It solve the problem and the split_multi go though even on chrX. But I have notice that removing LPL change my GQ values.

How to make that GQ do not change with removing LPL or how to repair LPL so it could be kept for split_multi.

Preview of LPL in my dataset:

Preview of LA in my dataset:

Whole error message:
{
“name”: “HailUserError”,
“message”: “Error summary: HailException: array index out of bounds: index=0, length=0\n------------\nHail stack trace:\n File "/tmp/ipykernel_24639/3739325340.py", line 3, in \n vds_chrY.variant_data = hl.experimental.sparse_split_multi(vds_chrY.variant_data)\n\n File "/usr/local/lib/python3.8/dist-packages/hail/experimental/vcf_combiner/sparse_split_multi.py", line 206, in sparse_split_multi\n entries: ds[entries].map(transform_entries)\n\n File "/usr/local/lib/python3.8/dist-packages/hail/experimental/vcf_combiner/sparse_split_multi.py", line 199, in transform_entries\n return hl.bind(with_local_a_index, lai)\n\n File "/usr/local/lib/python3.8/dist-packages/hail/expr/functions.py", line 515, in bind\n lambda_result = to_expr(f(args))\n\n File "/usr/local/lib/python3.8/dist-packages/hail/experimental/vcf_combiner/sparse_split_multi.py", line 186, in with_local_a_index\n hl.range(0, 3).map(lambda i: hl.min(\n\n File "/usr/local/lib/python3.8/dist-packages/hail/experimental/vcf_combiner/sparse_split_multi.py", line 187, in \n hl.range(0, hl.triangle(hl.len(old_entry.LA)))\n\n File "/usr/local/lib/python3.8/dist-packages/hail/experimental/vcf_combiner/sparse_split_multi.py", line 189, in \n .map(lambda idx: old_entry.LPL[idx])))))\n\n File "/usr/local/lib/python3.8/dist-packages/hail/expr/expressions/typed_expressions.py", line 481, in getitem\n return self.method("indexArray", self.dtype.element_type, item)\n\n File "/usr/local/lib/python3.8/dist-packages/hail/expr/expressions/base_expression.py", line 695, in method\n x = ir.Apply(name, ret_type, self.ir, *(a.ir for a in args))\n\n File "/usr/local/lib/python3.8/dist-packages/hail/ir/ir.py", line 2628, in init\n self.save_error_info()\n",
“stack”: "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mHailUserError\u001b[0m Traceback (most recent call last)\nFile \u001b[0;32m~/.local/lib/python3.8/site-packages/IPython/core/formatters.py:707\u001b[0m, in \u001b[0;36mPlainTextFormatter.call\u001b[0;34m(self, obj)\u001b[0m\n\u001b[1;32m 700\u001b[0m stream \u001b[39m=\u001b[39m StringIO()\n\u001b[1;32m 701\u001b[0m printer \u001b[39m=\u001b[39m pretty\u001b[39m.\u001b[39mRepresentationPrinter(stream, \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mverbose,\n\u001b[1;32m 702\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mmax_width, \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mnewline,\n\u001b[1;32m 703\u001b[0m max_seq_length\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39mmax_seq_length,\n\u001b[1;32m 704\u001b[0m singleton_pprinters\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39msingleton_printers,\n\u001b[1;32m 705\u001b[0m type_pprinters\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39mtype_printers,\n\u001b[1;32m 706\u001b[0m deferred_pprinters\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39mdeferred_printers)\n\u001b[0;32m–> 707\u001b[0m printer\u001b[39m.\u001b[39;49mpretty(obj)\n\u001b[1;32m 708\u001b[0m printer\u001b[39m.\u001b[39mflush()\n\u001b[1;32m 709\u001b[0m \u001b[39mreturn\u001b[39;00m stream\u001b[39m.\u001b[39mgetvalue()\n\nFile \u001b[0;32m~/.local/lib/python3.8/site-packages/IPython/lib/pretty.py:410\u001b[0m, in \u001b[0;36mRepresentationPrinter.pretty\u001b[0;34m(self, obj)\u001b[0m\n\u001b[1;32m 407\u001b[0m \u001b[39mreturn\u001b[39;00m meth(obj, \u001b[39mself\u001b[39m, cycle)\n\u001b[1;32m 408\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mcls\u001b[39m \u001b[39mis\u001b[39;00m \u001b[39mnot\u001b[39;00m \u001b[39mobject\u001b[39m \\n\u001b[1;32m 409\u001b[0m \u001b[39mand\u001b[39;00m callable(\u001b[39mcls\u001b[39m\u001b[39m.\u001b[39m\u001b[39m__dict
\u001b[39m\u001b[39m.\u001b[39mget(\u001b[39m’\u001b[39m\u001b[39m__repr
\u001b[39m\u001b[39m’\u001b[39m)):\n\u001b[0;32m–> 410\u001b[0m \u001b[39mreturn\u001b[39;00m repr_pprint(obj, \u001b[39mself\u001b[39;49m, cycle)\n\u001b[1;32m 412\u001b[0m \u001b[39mreturn\u001b[39;00m default_pprint(obj, \u001b[39mself\u001b[39m, cycle)\n\u001b[1;32m 413\u001b[0m \u001b[39mfinally\u001b[39;00m:\n\nFile \u001b[0;32m~/.local/lib/python3.8/site-packages/IPython/lib/pretty.py:778\u001b[0m, in \u001b[0;36m_repr_pprint\u001b[0;34m(obj, p, cycle)\u001b[0m\n\u001b[1;32m 776\u001b[0m \u001b[39m"""A pprint that just redirects to the normal repr function."""\u001b[39;00m\n\u001b[1;32m 777\u001b[0m \u001b[39m# Find newlines and replace them with p.break()\u001b[39;00m\n\u001b[0;32m–> 778\u001b[0m output \u001b[39m=\u001b[39m \u001b[39mrepr\u001b[39;49m(obj)\n\u001b[1;32m 779\u001b[0m lines \u001b[39m=\u001b[39m output\u001b[39m.\u001b[39msplitlines()\n\u001b[1;32m 780\u001b[0m \u001b[39mwith\u001b[39;00m p\u001b[39m.\u001b[39mgroup():\n\nFile \u001b[0;32m/usr/local/lib/python3.8/dist-packages/hail/matrixtable.py:2573\u001b[0m, in \u001b[0;36mMatrixTable.Show.repr\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 2572\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39m__repr\u001b[39m(\u001b[39mself\u001b[39m):\n\u001b[0;32m-> 2573\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m\u001b[39m__str__\u001b[39;49m()\n\nFile \u001b[0;32m/usr/local/lib/python3.8/dist-packages/hail/matrixtable.py:2567\u001b[0m, in \u001b[0;36mMatrixTable.Show.str\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 2566\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39m__str_\u001b[39m(\u001b[39mself\u001b[39m):\n\u001b[0;32m-> 2567\u001b[0m s \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mtable_show\u001b[39m.\u001b[39;49m\u001b[39m__str__\u001b[39;49m()\n\u001b[1;32m 2568\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mdisplayed_n_cols \u001b[39m!=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mactual_n_cols:\n\u001b[1;32m 2569\u001b[0m s \u001b[39m+\u001b[39m\u001b[39m=\u001b[39m \u001b[39mf\u001b[39m\u001b[39m"\u001b[39m\u001b[39mshowing the first \u001b[39m\u001b[39m{\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mdisplayed_n_cols \u001b[39m}\u001b[39;00m\u001b[39m of \u001b[39m\u001b[39m{\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mactual_n_cols \u001b[39m}\u001b[39;00m\u001b[39m columns\u001b[39m\u001b[39m"\u001b[39m\n\nFile \u001b[0;32m/usr/local/lib/python3.8/dist-packages/hail/table.py:1358\u001b[0m, in \u001b[0;36mTable.Show.str\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 1357\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39m__str_\u001b[39m(\u001b[39mself\u001b[39m):\n\u001b[0;32m-> 1358\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_ascii_str()\n\nFile \u001b[0;32m/usr/local/lib/python3.8/dist-packages/hail/table.py:1384\u001b[0m, in \u001b[0;36mTable._Show._ascii_str\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 1381\u001b[0m \u001b[39mreturn\u001b[39;00m s[:truncate \u001b[39m-\u001b[39m \u001b[39m3\u001b[39m] \u001b[39m+\u001b[39m \u001b[39m"\u001b[39m\u001b[39m…\u001b[39m\u001b[39m"\u001b[39m\n\u001b[1;32m 1382\u001b[0m \u001b[39mreturn\u001b[39;00m s\n\u001b[0;32m-> 1384\u001b[0m rows, has_more, dtype \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mdata()\n\u001b[1;32m 1385\u001b[0m fields \u001b[39m=\u001b[39m \u001b[39mlist\u001b[39m(dtype)\n\u001b[1;32m 1386\u001b[0m trunc_fields \u001b[39m=\u001b[39m [trunc(f) \u001b[39mfor\u001b[39;00m f \u001b[39min\u001b[39;00m fields]\n\nFile \u001b[0;32m/usr/local/lib/python3.8/dist-packages/hail/table.py:1368\u001b[0m, in \u001b[0;36mTable._Show.data\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 1366\u001b[0m row_dtype \u001b[39m=\u001b[39m t\u001b[39m.\u001b[39mrow\u001b[39m.\u001b[39mdtype\n\u001b[1;32m 1367\u001b[0m t \u001b[39m=\u001b[39m t\u001b[39m.\u001b[39mselect(\u001b[39m
\u001b[39m\u001b[39m*\u001b[39m{k: hl\u001b[39m.\u001b[39m_showstr(v) \u001b[39mfor\u001b[39;00m (k, v) \u001b[39min\u001b[39;00m t\u001b[39m.\u001b[39mrow\u001b[39m.\u001b[39mitems()})\n\u001b[0;32m-> 1368\u001b[0m rows, has_more \u001b[39m=\u001b[39m t\u001b[39m.\u001b[39;49m_take_n(\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mn)\n\u001b[1;32m 1369\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_data \u001b[39m=\u001b[39m (rows, has_more, row_dtype)\n\u001b[1;32m 1370\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_data\n\nFile \u001b[0;32m/usr/local/lib/python3.8/dist-packages/hail/table.py:1515\u001b[0m, in \u001b[0;36mTable._take_n\u001b[0;34m(self, n)\u001b[0m\n\u001b[1;32m 1513\u001b[0m has_more \u001b[39m=\u001b[39m \u001b[39mFalse\u001b[39;00m\n\u001b[1;32m 1514\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[0;32m-> 1515\u001b[0m rows \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mtake(n \u001b[39m+\u001b[39;49m \u001b[39m1\u001b[39;49m)\n\u001b[1;32m 1516\u001b[0m has_more \u001b[39m=\u001b[39m \u001b[39mlen\u001b[39m(rows) \u001b[39m>\u001b[39m n\n\u001b[1;32m 1517\u001b[0m rows \u001b[39m=\u001b[39m rows[:n]\n\nFile \u001b[0;32m:2\u001b[0m, in \u001b[0;36mtake\u001b[0;34m(self, n, _localize)\u001b[0m\n\nFile \u001b[0;32m/usr/local/lib/python3.8/dist-packages/hail/typecheck/check.py:577\u001b[0m, in \u001b[0;36m_make_dec..wrapper\u001b[0;34m(original_func, args, kwargs)\u001b[0m\n\u001b[1;32m 574\u001b[0m \u001b[39m@decorator\u001b[39m\n\u001b[1;32m 575\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mwrapper\u001b[39m(__original_func, \u001b[39m\u001b[39margs, \u001b[39m\u001b[39m\u001b[39m\u001b[39mkwargs):\n\u001b[1;32m 576\u001b[0m args, kwargs \u001b[39m=\u001b[39m check_all(__original_func, args, kwargs, checkers, is_method\u001b[39m=\u001b[39mis_method)\n\u001b[0;32m–> 577\u001b[0m \u001b[39mreturn\u001b[39;00m original_func(\u001b[39m*\u001b[39;49margs, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs)\n\nFile \u001b[0;32m/usr/local/lib/python3.8/dist-packages/hail/table.py:2188\u001b[0m, in \u001b[0;36mTable.take\u001b[0;34m(self, n, _localize)\u001b[0m\n\u001b[1;32m 2154\u001b[0m \u001b[39m@typecheck_method\u001b[39m(n\u001b[39m=\u001b[39m\u001b[39mint\u001b[39m, _localize\u001b[39m=\u001b[39m\u001b[39mbool\u001b[39m)\n\u001b[1;32m 2155\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mtake\u001b[39m(\u001b[39mself\u001b[39m, n, _localize\u001b[39m=\u001b[39m\u001b[39mTrue\u001b[39;00m):\n\u001b[1;32m 2156\u001b[0m \u001b[39m"""Collect the first n rows of the table into a local list.\u001b[39;00m\n\u001b[1;32m 2157\u001b[0m \n\u001b[1;32m 2158\u001b[0m \u001b[39m Examples\u001b[39;00m\n\u001b[0;32m (…)\u001b[0m\n\u001b[1;32m 2185\u001b[0m \u001b[39m List of row structs.\u001b[39;00m\n\u001b[1;32m 2186\u001b[0m \u001b[39m """\u001b[39;00m\n\u001b[0;32m-> 2188\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mhead(n)\u001b[39m.\u001b[39;49mcollect(_localize)\n\nFile \u001b[0;32m:2\u001b[0m, in \u001b[0;36mcollect\u001b[0;34m(self, _localize, _timed)\u001b[0m\n\nFile \u001b[0;32m/usr/local/lib/python3.8/dist-packages/hail/typecheck/check.py:577\u001b[0m, in \u001b[0;36m_make_dec..wrapper\u001b[0;34m(original_func, args, kwargs)\u001b[0m\n\u001b[1;32m 574\u001b[0m \u001b[39m@decorator\u001b[39m\n\u001b[1;32m 575\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mwrapper\u001b[39m(__original_func, \u001b[39m\u001b[39margs, \u001b[39m\u001b[39m\u001b[39m\u001b[39mkwargs):\n\u001b[1;32m 576\u001b[0m args, kwargs \u001b[39m=\u001b[39m check_all(__original_func, args, kwargs, checkers, is_method\u001b[39m=\u001b[39mis_method)\n\u001b[0;32m–> 577\u001b[0m \u001b[39mreturn\u001b[39;00m original_func(\u001b[39m*\u001b[39;49margs, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs)\n\nFile \u001b[0;32m/usr/local/lib/python3.8/dist-packages/hail/table.py:1987\u001b[0m, in \u001b[0;36mTable.collect\u001b[0;34m(self, _localize, _timed)\u001b[0m\n\u001b[1;32m 1985\u001b[0m e \u001b[39m=\u001b[39m construct_expr(rows_ir, hl\u001b[39m.\u001b[39mtarray(t\u001b[39m.\u001b[39mrow\u001b[39m.\u001b[39mdtype))\n\u001b[1;32m 1986\u001b[0m \u001b[39mif\u001b[39;00m _localize:\n\u001b[0;32m-> 1987\u001b[0m \u001b[39mreturn\u001b[39;00m Env\u001b[39m.\u001b[39;49mbackend()\u001b[39m.\u001b[39;49mexecute(e\u001b[39m.\u001b[39;49m_ir, timed\u001b[39m=\u001b[39;49m_timed)\n\u001b[1;32m 1988\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[1;32m 1989\u001b[0m \u001b[39mreturn\u001b[39;00m e\n\nFile \u001b[0;32m/usr/local/lib/python3.8/dist-packages/hail/backend/py4j_backend.py:104\u001b[0m, in \u001b[0;36mPy4JBackend.execute\u001b[0;34m(self, ir, timed)\u001b[0m\n\u001b[1;32m 102\u001b[0m \u001b[39mreturn\u001b[39;00m (value, timings) \u001b[39mif\u001b[39;00m timed \u001b[39melse\u001b[39;00m value\n\u001b[1;32m 103\u001b[0m \u001b[39mexcept\u001b[39;00m FatalError \u001b[39mas\u001b[39;00m e:\n\u001b[0;32m–> 104\u001b[0m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_handle_fatal_error_from_backend(e, ir)\n\nFile \u001b[0;32m/usr/local/lib/python3.8/dist-packages/hail/backend/backend.py:189\u001b[0m, in \u001b[0;36mBackend._handle_fatal_error_from_backend\u001b[0;34m(self, err, ir)\u001b[0m\n\u001b[1;32m 184\u001b[0m error_message \u001b[39m=\u001b[39m \u001b[39mstr\u001b[39m(err)\n\u001b[1;32m 185\u001b[0m message_and_trace \u001b[39m=\u001b[39m (\u001b[39mf\u001b[39m\u001b[39m’\u001b[39m\u001b[39m{\u001b[39;00merror_message\u001b[39m}\u001b[39;00m\u001b[39m\n\u001b[39;00m\u001b[39m’\u001b[39m\n\u001b[1;32m 186\u001b[0m \u001b[39m’\u001b[39m\u001b[39m------------\u001b[39m\u001b[39m\n\u001b[39;00m\u001b[39m’\u001b[39m\n\u001b[1;32m 187\u001b[0m \u001b[39m’\u001b[39m\u001b[39mHail stack trace:\u001b[39m\u001b[39m\n\u001b[39;00m\u001b[39m’\u001b[39m\n\u001b[1;32m 188\u001b[0m \u001b[39mf\u001b[39m\u001b[39m’\u001b[39m\u001b[39m{\u001b[39;00mbetter_stack_trace\u001b[39m}\u001b[39;00m\u001b[39m’\u001b[39m)\n\u001b[0;32m–> 189\u001b[0m \u001b[39mraise\u001b[39;00m HailUserError(message_and_trace) \u001b[39mfrom\u001b[39;00m \u001b[39mNone\u001b[39m\n\n\u001b[0;31mHailUserError\u001b[0m: Error summary: HailException: array index out of bounds: index=0, length=0\n------------\nHail stack trace:\n File "/tmp/ipykernel_24639/3739325340.py", line 3, in \n vds_chrY.variant_data = hl.experimental.sparse_split_multi(vds_chrY.variant_data)\n\n File "/usr/local/lib/python3.8/dist-packages/hail/experimental/vcf_combiner/sparse_split_multi.py", line 206, in sparse_split_multi\n entries: ds[entries].map(transform_entries)\n\n File "/usr/local/lib/python3.8/dist-packages/hail/experimental/vcf_combiner/sparse_split_multi.py", line 199, in transform_entries\n return hl.bind(with_local_a_index, lai)\n\n File "/usr/local/lib/python3.8/dist-packages/hail/expr/functions.py", line 515, in bind\n lambda_result = to_expr(f(*args))\n\n File "/usr/local/lib/python3.8/dist-packages/hail/experimental/vcf_combiner/sparse_split_multi.py", line 186, in with_local_a_index\n hl.range(0, 3).map(lambda i: hl.min(\n\n File "/usr/local/lib/python3.8/dist-packages/hail/experimental/vcf_combiner/sparse_split_multi.py", line 187, in \n hl.range(0, hl.triangle(hl.len(old_entry.LA)))\n\n File "/usr/local/lib/python3.8/dist-packages/hail/experimental/vcf_combiner/sparse_split_multi.py", line 189, in \n .map(lambda idx: old_entry.LPL[idx])))))\n\n File "/usr/local/lib/python3.8/dist-packages/hail/expr/expressions/typed_expressions.py", line 481, in getitem\n return self._method("indexArray", self.dtype.element_type, item)\n\n File "/usr/local/lib/python3.8/dist-packages/hail/expr/expressions/base_expression.py", line 695, in _method\n x = ir.Apply(name, ret_type, self._ir, *(a._ir for a in args))\n\n File "/usr/local/lib/python3.8/dist-packages/hail/ir/ir.py", line 2628, in init\n self.save_error_info()\n”
}

Hello again,

is there any update on my problem?

Have a nice day,
Veronika