I agree with rcooper’s answer, but relief may be on the way. The CFPB recently proposed changes to the serving rules. The proposed revision clarifies the exemption for institutions with small servicing operations. The proposal clarifies that loans serviced on a charitable basis will not be included. To be eligible you cannot impose fees for servicing the loans.
For more information see my blog article on this topic at https://jholzknecht.wordpress.com/2013/04/22/cfpb-proposes-more-changes-to-the-qm-and-servicing-changes/