This is a tricky one. It really depends on the true purpose of the loan. (In my opinion)
I’ve seen/dealt with loans before where a “farmer/rancher” purchased land and a dwelling was on that land. His intended purpose was to purchase this for the ag usage, not the dwelling and the acreage was over 25 so we exempted it.
I’ve had loans where the intended purpose was to purchase a home that came with (an example) 30 acres. Yes, those 30 acres were for ag/farm usage but the purpose for the customer was to purchase a home. Therefore, not exempt.
I think on the basis of refinance and improvements it’s really going to depend on the purpose of the loan. I wouldn’t look at these loans, realize they have farm land and stop there. It’s going to depend a lot on what your financial institution wants to do too.