There is no doubt that the growing complexity of financial transactions has dramatically increased firms reliance on models. In turn, the risks that arise from improper model use have also escalated. Below we’ll explore five critical best practices that will help you get a better handle on model risk. Whether your models are developed and maintained in-house, or managed externally by third-party vendors, it is essential that these practices are followed in order to effectively manage model risk.
- Understand Model Alternatives: It is fundamentally important to evaluate the conceptual soundness and applicability of models, which will require specifying alternative models for comparison. Examples include choosing between parametric or non-parametric families of distributions, or all joint distributions that have the same marginals for aggregation of risks in a portfolio. If the portfolio contains complex derivatives, there are many different pricing models to choose from. Developing the tools to calculate quantitative measures of model risk is a highly technical process and requires significant expertise in both mathematical finance and software development. Overall the evaluation should consider model effectiveness, limitations, reliability, computational speed, and available data.
- Realize that Model Risk Affects both Pricing and Risk Models: In the aftermath of the recent financial crisis, getting a handle on model uncertainty when assessing regulatory capital requirements is crucial. In the current landscape of risk-sensitive capital charges, there are now multiple types of model risk to assess. Capital charges based on real-world risk-measures such as VaR require the ability to model portfolio return distributions - in a way that is compatible with risk-neutral XVA value adjustments on MTM book pricing, which are in turn based on modeling counterparty credit spreads, expected future funding costs, and the results of pricing models for the underlying trades. The potential compounding effects of this complex modeling environment require good communication between the owners of the business requirements, quantitative experts, and development team members, irrespective of their organizational or geographical position.
- Pay Attention to the Testing Process: The models should be developed with a well-defined process that sets clear roles and responsibilities. Model developers should use test-driven development, originating from a number of sources, including: external benchmarks (e.g., the market, published results), prototypes or other implementations, edge cases (i.e., limiting cases), expected behavior (e.g., sensitivity analysis) and stress testing.
Model validation is an iterative process. Any issues that are identified will lead to further tests. The process should also be repeated after any changes to the pricing system.
- Use an Independent Model Validation Role: The development process should include a model validation role that is independent from the model development role. This role is responsible for reviewing the tests created by the quant developers, for creating additional tests to their own satisfaction, and for making the final assertion that test coverage and depth is sufficient. The individuals filling this quant validation role should be no less quantitatively capable than the quant developers, and should have separate incentives.
- Engage in Ongoing Monitoring and Learning: To minimize model risk, it is important to monitor the model’s performance over time. Incorporating feedback from live usage of pricing models is of high value in detecting problems early on, and should be accompanied by procedures for triggering investigations and resolutions of issues, and incorporating new tests as they become apparent. Risk models should also be continually back-tested using historical data where available, including procedures to back-fill and maintain historical data when incorporating new instruments into the portfolio, in order to ensure a high-level of confidence in the model risk assessment.
The identification of roles and responsibilities for the model validation team is a key step in the overall process. In large projects you may assign one or more individuals to each role. In smaller projects each person may cover more than one role.
The most important aspect that permeates these practices is having a well-defined process – an on-going process that continues beyond the initial development and deployment phase. Within this, there are certain roles, some of which must be independent. And finally, the process must be supported by efficient and cross-referencing systems. Paying attention to these considerations is essential to exercising greater control over model risk.
Interested in learning five more proven best practices that will help you keep model risk in check? Download our mini whitepaper on Managing Model Risk.
