An automated valuation model that performs well in purchase-loan testing may deliver different results on home equity and refinance transactions, where there is no new sale price to help establish the property’s value.
That distinction is central to new MISMO testing guidance and to lenders’ decisions about which automated valuation models, or AVMs, to use. For brokers and LOs, those decisions can affect the value used to calculate loan-to-value ratios, the equity a borrower can access, and whether a file requires additional valuation work.
Veros Real Estate Solutions on Oct. 8 endorsed MISMO’s Automated Valuation Model Testing Guidance and Best Practices, which provides a standardized framework for evaluating valuation accuracy and whether a model’s Common Confidence Scores reflect its performance. MISMO released the guidance Sept. 29.
Reena Agrawal, senior research economist at Veros, told NMP why the benchmarks used to evaluate a model matter.
“It is easier for an AVM to test well against sales benchmarks and more difficult to test well against appraisal benchmarks,” Agrawal said. “Therefore, an AVM which tests consistently well against both is likely to have better underlying models.”
When Sales Data Influences The Test
In purchase transactions, AVM vendors sometimes have access to the sale prices being used as testing benchmarks, Agrawal said. That can happen because of a delay between the transaction and the test or because a vendor receives sales data early.
Access to that information may improve measured accuracy without showing how the model would perform on a transaction lacking a recent sale price.
HELOC and refinance files present a different challenge. Neither the AVM nor the appraiser establishing the benchmark has a new sale price for the subject property to draw on, Agrawal said.
The difference makes benchmark selection particularly important for lenders evaluating models for home equity and refinance lending. Strong purchase-testing results alone may offer an incomplete picture of how a model will perform on those files.
According to Agrawal, MISMO recommends selecting benchmark samples that reflect the model’s actual or intended use, including appraisal benchmarks for HELOC and refinance transactions. Relevant property types, geography, price ranges, and other characteristics also should factor into the sample.
For originators, the practical issue is whether a lender’s valuation process has been tested against properties and transactions resembling the borrower’s file.
Set Standards Before Testing
Agrawal said lenders should establish performance thresholds and testing procedures before evaluating a model, with standards tied to their risk tolerance and the way they intend to use the AVM.
Testing samples should reflect actual business use. When practical, lenders also should use all applicable purchase and appraisal benchmarks, since larger samples produce more reliable results, she said.
“Testing should also address potential benchmark leakage and be refreshed regularly; MISMO recommends testing accuracy at least quarterly,” Agrawal said.
Benchmark leakage occurs when information used to judge a model’s accuracy also becomes available to the model being tested. Addressing that risk helps lenders distinguish performance that reflects the model’s valuation capabilities from results influenced by access to the benchmark.
MISMO’s framework also addresses the timing of valuations and the metrics used to measure typical errors, the spread of results and outliers. The guidance is intended to support both lenders conducting testing internally and those commissioning work from third-party reviewers.
Testing The Confidence Score
The guidance builds on MISMO’s Common Confidence Score, which reached recommendation status in April following implementation by providers including Clear Capital, Cotality, ICE, Quantarium, and Veros. The score establishes a common scale for communicating AVM reliability across vendors.
The testing framework addresses two related questions: how closely an AVM’s estimates track benchmark values, and whether its confidence scores are calibrated to the accuracy they indicate.
That second check matters when lenders use confidence scores to determine whether an automated value is acceptable or whether a property needs further review.
Agrawal also serves as vice chair of MISMO’s AVM Development Workgroup, which developed the guidance with participation from lenders, valuation providers, testing firms, and investors.
What It Means For A Loan File
Better testing could help lenders identify the most accurate AVM for a particular application and determine where additional valuation support is needed, Agrawal said.
“For borrowers, more reliable valuation testing could reduce the risk that an inaccurate AVM affects a credit decision,” she said. “Whether an appraisal is required ultimately depends on the lender's policies and the transaction itself.”
For brokers and LOs, that makes the lender’s valuation requirements part of the early conversation with a home equity or refinance borrower. An automated estimate may support the file, but the lender’s acceptance criteria and requirements for further valuation work will help determine how the transaction proceeds.
The guidance gives lenders a more consistent way to evaluate those decisions, and a reason to look beyond a vendor’s overall accuracy results to how the model performs on the loans they actually originate.
