nope! I’m using that in a separate pipeline. also, as a quick update, I was having the same Method too large issue when trying to checkpoint a separate table. I reverted back to 0.2.74 and was able to write out the other table (sorry didn’t keep the logs) and am trying to rerun this script on 0.2.74 as well
I was able to checkpoint the table when running the script with 0.2.74 – will send the new log now
This is probably related to whole-stage codegen, which is now live in 75.
sorry, what does that mean? is there something I can change to avoid triggering this error?
You can turn this off with a feature flag:
hl._set_flags(no_whole_stage_codegen='1')
edit: set_flags to _set_flags
1 Like
I wouldn’t do this in general, though, and we’ll fix this method size problem – whole stage code generation is a significant performance improvement in many pipelines!
1 Like
cool, thank you!!