Issues with tiebreaker expression


#1

I get the following error after importing a table and trying to use the tiebreaker expression in idb_prune:


#2

Hi Mathias,
Can you copy the full text of the command and the error trace, rather than a screen shot? You can use triple backticks to format code blocks. Thanks!


#3

We’d also strongly recommend switching to 0.2 – we will deprecate (stop supporting) 0.1 soon. This will unfortunately mean rewriting pipelines and recreating data files.


#4

I cant really copy the code as text, so I hope that giving this screenshot will help? Otherwise I have to type it all by hand.


#5

Ah, the problem is with your filterSamplesExpr - the inbreeding aggregator returns a structure with several values.


#6

Probably you want to compare the f statistic to something?


#7

That helped, however, I keep having problems with a NullPointerException when using idb_prune.


#8

How many samples do you have?

Also, do you have plans to move to 0.2? We’ll be fully deprecating 0.1 soon.


#9

I also hope to move to 0.2, it is not entirely up to me. 18.000 samples.


#10

I’m guessing that the java process died during the local step of IBD prune (or possibly LD prune).

How much memory does the master machine have?


#11

I am working on 3 nodes with 350 gb RAM each at the moment. It is on a small data subset with only few variants included.


#12

For some reason a second run does not end in the same error. But the ibd_prune has been running for hours now on the small subset, is there an issue or is it just very inefficient?


#13

I believe we made the maximal_independent_set algorithm immensely faster from 0.1 to 0.2.

It’s possible that the 0.1 algorithm is unusable on sample sizes greater than a few thousand.

The ibd_prune step is the same computation whether you have 1000 variants or 100M - pairwise comparisons of 18K samples.


#14

@Mathias_Hansen - what is it going to take to get you to 0.2?

We are imminently going to announce end-of-life for 0.1 – meaning no support for external users running 0.1.