Trouble trying to subset a vcf

Hail Query & hailctl
1

I am trying to subset a vcf using the below code.

    args = parse_command_line()

    hl.init(default_reference='GRCh38')

    # read in the vcf. Replace nul with .
    mt = hl.import_vcf(args.input_vcf, force=True, find_replace=('nul', '.'))

    # read in the sample list
    with open(args.subset_list) as f:
        samples = set(f.read().splitlines())

    # filter to just the samples in our list
    mt = mt.filter_cols(hl.literal(samples).contains(mt.s), keep=True)

    # Optional(?): remove all sites that are non-variant in the subset
    # *** Do we want this? ***
    mt = hl.variant_qc(mt)
    mt = mt.filter_rows(mt.variant_qc.n_non_ref > 0, keep=True)

    # write out a new VCF
    # Using the original vcf metadata in the new subsetted vcf
    # we can make sure our vcfs are consistent with what we usually deliver.
    # The sample names included in the call set are NOT in the vcf metadata, so we
    # do not have to worry about samples that were in original vcf but not in subsetted vcf
    # being included.
    hl.export_vcf(mt, args.output_vcf, metadata=hl.get_vcf_metadata(args.input_vcf))

I have run this many times without issue, but I am running into the below when trying to run now.

FATAL: caught exception java.lang.UnsatisfiedLinkError: /tmp/libboot4088385528771424649.so: /tmp/libboot4088385528771424649.so: failed to map segment from shared object: Operation not permitted
java.lang.UnsatisfiedLinkError: /tmp/libboot4088385528771424649.so: /tmp/libboot4088385528771424649.so: failed to map segment from shared object: Operation not permitted
	at java.lang.ClassLoader$NativeLibrary.load(Native Method)
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
	at java.lang.Runtime.load0(Runtime.java:809)
	at java.lang.System.load(System.java:1086)
	at is.hail.nativecode.NativeCode.<clinit>(NativeCode.java:24)
	at is.hail.nativecode.NativeBase.<init>(NativeBase.scala:20)
	at is.hail.annotations.Region.<init>(Region.scala:180)
	at is.hail.annotations.Region$.apply(Region.scala:16)
	at is.hail.annotations.Region$.scoped(Region.scala:18)
	at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:7)
	at is.hail.backend.Backend.execute(Backend.scala:86)
	at is.hail.backend.Backend.executeJSON(Backend.scala:92)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:748)
ERROR:root:Exception while sending command.
Traceback (most recent call last):
  File "/home/unix/samn/miniconda3/envs/hail/lib/python3.7/site-packages/py4j/java_gateway.py", line 1159, in send_command
    raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty

Could someone help me out with this?

2

Hey @samn ,

Sorry to hear you’re having trouble. This likely means the system you’re running on has some restrictions on loading shard libraries from /tmp. Did you update the OS recently? Is this a shared system and maybe the administrator changed permissions? Are you inside a docker container?

I think this is often the result of someone applying DOD-required security hardening to a machine.

If you have sudo privileges to this machine, you can remove this security feature like this:

sudo mount /tmp -o remount,exec

Modern versions of Hail do not rely on native code in the same way and should not have this problem. I think you’re using a version of Hail older than 0.2.43. Versions of Hail since and including 0.2.43 should not have this problem. I’d also say Hail has included a number of improvements since May 2020 that might improve performance of your pipeline as well!

3

Updating hail looks like it did the trick. Thank you!