How to use variables to represent data fields

Help [0.1]
1

I am trying to store Column Annotation names as variables so that I can loop through the table and filter out NaN values and then store that specific annotation for a short period.

# creates a list with just the genes and no other column / clinical annotations
x = []
for i in vds2_subset.col:
    x = np.append(x,i)

genes = list(filter(lambda k: 'ENSG' in k, x))

all_p_values = []
for i in genes:
    ## filter nan values
    print (i)
    filtered_protein = vds2_subset.filter_cols(vds2_subset.i > -30, keep = True) 
    x = filtered_protein.sex_bool.collect() #sex data
    y = filtered_protein.i.collect() #protein expression data

Error I get:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-229-cbe3b4bf9e4f> in <module>()
     10     ## filter nan values
     11     print (i)
---> 12     filtered_protein = vds2_subset.filter_cols(vds2_subset.i > -30, keep = True)
     13     x = filtered_protein.sex_bool.collect() #sex data
     14     y = filtered_protein.i.collect() #protein expression data

/home/hail/hail.zip/hail/table.py in __getattr__(self, item)
     84             return self.__dict__[item]
     85         else:
---> 86             raise AttributeError(get_nice_attr_error(self, item))
     87 
     88     def _copy_fields_from(self, other: 'ExprContainer'):

AttributeError: MatrixTable instance has no field, method, or property 'i'
    Hint: use 'describe()' to show the names of all data fields.

Can you make suggestions are what I can do instead or what I am doing wrong

2

Given this and your last post, I think what you really want to do is load the sample-by-gene expression file as a MatrixTable using hl.import_matrix_table. Then, you can apply all these transformations together. Finally, you can go back to a table with mt.make_table().

3

I think also our infrastructure is going to have trouble applying thousands of filter operations.