# PCA job aborted from SparkException **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [June 2, 2020, 11:00pm UTC](https://discuss.hail.is/t/pca-job-aborted-from-sparkexception/1433 "2020-06-02T23:00:55Z")\ **Posts on this page:** 7\ **Page:** 3
**Author:** ![acererak](https://avatars.discourse-cdn.com/v4/letter/a/3bc359/32.png) [@acererak](https://discuss.hail.is/u/acererak)\ **Post date:** [July 15, 2020, 7:10pm UTC](https://discuss.hail.is/t/pca-job-aborted-from-sparkexception/1433/41 "2020-07-15T19:10:40Z")
Yep, thank you! I am running some other analyses right now but I’ll post a log tomorrow if I still have issues upon rerunning. ---
**Author:** ![acererak](https://avatars.discourse-cdn.com/v4/letter/a/3bc359/32.png) [@acererak](https://discuss.hail.is/u/acererak)\ **Post date:** [July 16, 2020, 1:12pm UTC](https://discuss.hail.is/t/pca-job-aborted-from-sparkexception/1433/42 "2020-07-16T13:12:11Z")
@tpoterba it looks like upload\_log was removed from the codebase ```auto --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) in ----> 1 hl.upload_log(bucket+"/pca_run.log") AttributeError: module 'hail' has no attribute 'upload_log' ``` I found this issue that was closed in April of this year: [https://github.com/hail-is/hail/issues/7392](https://github.com/hail-is/hail/issues/7392) Notably, since using 0.2.49 I was successfully able to run hl.ld\_prune() (runtime was approx 4hrs), which brought my variant row count down from 580k to 340k with r2=0.2. Previously this function produced a SparkException so we are getting somewhere! Even with another overnight run, the hl.hwe\_normalized\_pca() looks like it is a stuck job. Happy to generate a log file for this if there is a way. ---
**Author:** ![acererak](https://avatars.discourse-cdn.com/v4/letter/a/3bc359/32.png) [@acererak](https://discuss.hail.is/u/acererak)\ **Post date:** [July 16, 2020, 1:14pm UTC](https://discuss.hail.is/t/pca-job-aborted-from-sparkexception/1433/43 "2020-07-16T13:14:54Z")
Is the logging from the hl.init() statement the same log? Looks like I can access that from the terminal of the Terra virtual machine and it looks quite large now. ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [July 16, 2020, 1:25pm UTC](https://discuss.hail.is/t/pca-job-aborted-from-sparkexception/1433/44 "2020-07-16T13:25:36Z")
Oops, sorry, got confused – hl.copy\_log is what I meant to point you to. That’s a convenience around using another utility to copy the log from the local disk of the driver machine to a Google bucket / remote FS. Yeah, the log on that machine is what we want – I do expect it to be somewhat large. ---
**Author:** ![acererak](https://avatars.discourse-cdn.com/v4/letter/a/3bc359/32.png) [@acererak](https://discuss.hail.is/u/acererak)\ **Post date:** [July 16, 2020, 1:26pm UTC](https://discuss.hail.is/t/pca-job-aborted-from-sparkexception/1433/45 "2020-07-16T13:26:11Z")
> [@tpoterba](#): > > Yeah, the log on that machine is what we want – I do expect it to be somewhat large. Should I post in here? ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [July 16, 2020, 1:28pm UTC](https://discuss.hail.is/t/pca-job-aborted-from-sparkexception/1433/46 "2020-07-16T13:28:30Z")
Probably won’t fit. Can you email as attachment/drive link to [hail-team@broadinstitute.org](mailto:hail-team@broadinstitute.org)? ---
**Author:** ![acererak](https://avatars.discourse-cdn.com/v4/letter/a/3bc359/32.png) [@acererak](https://discuss.hail.is/u/acererak)\ **Post date:** [July 28, 2020, 7:29pm UTC](https://discuss.hail.is/t/pca-job-aborted-from-sparkexception/1433/47 "2020-07-28T19:29:47Z")
For anyone else following, wanted to post an update on this. I finally was able to run a PCA job after some help from Tim. We found that my MatrixTable, assembled via import\_plink() was not partitioned very efficiently ( you can check this via mt.n\_partitions(). We were able to both repartition the data and save/checkpoint as a MatrixTable for added efficiency with ```auto gen = gen.repartition(250).checkpoint('gs://some/[path.mt](http://path.mt)', overwrite=True) ``` I did have some trouble with the shuffling step of the repartition due to using preemptible nodes, so Tim provided me with the following code for a “no shuffle” repartition: ````auto def no_shuffle_repartition(mt, path1, path2, n_parts): mt = mt.checkpoint(path1) return hl.read_matrix_table(path1, _intervals=mt._calculate_new_partitions(n_parts)).checkpoint(path2) ``` ```` [Previous page](https://discuss.hail.is/t/pca-job-aborted-from-sparkexception/1433.md?page=2)