Model Risk Management for Stress Testing

The Prudential Regulation Authority (PRA) is consulting on proposals to update its guidance on Model Risk Management (MRM)  practices for stress testing.

The PRA says the proposals reflect the aims of a current review by the European Banking Agency (EBA) of the 2010 Committee of European Banking Supervisors (CEBS) Guidelines on Stress Testing (GL32).

Four principles have been published:

Principle 1: Banks have an established definition of a model and maintain a model inventory

P1.1 Definition of a model:

Banks should establish their own definition of a model. When identifying models banks are expected to take into consideration:

 (a) Calculation methods or systems that are based on statistical, financial or economic assumptions (eg impairment models, income models)
(b) Calculation mechanisms used to transform a set of parameters or values into a quantitative measure (eg scenario expansion models, probability of default models)
(c) Frameworks or systems where qualitative judgement is applied to generate quantitative results (eg where adjustments are made to address known model limitations)
(d) Calculation mechanisms where outputs of other models are used to calculate financial/risk measures (eg expected loss which uses the output of probability of default, loss given default and exposure at default models)
In cases where calculation mechanisms are not classified as models, banks should ensure the risks associated with the implementation and use of such calculations are adequately understood, controlled, and documented as part of an established management control process.
P1.2 Model inventory:

Banks should maintain a comprehensive set of information on models ‘implemented for use’, ‘under development’, or ‘recently retired’. The information should clearly identify model owners and users, and should also include all model uses and dependencies, ie models that depend or use the output of other models. A designated internal party should be responsible for maintaining the bank-wide inventory of all models. Any variation of a model which requires separate validation and approval should be classified as a separate model.

Principle 2: Banks have implemented an effective governance framework, policies, procedures and controls to manage their model risk

P2.1 Board of directors and senior management responsibility:

The board of directors should establish a framework for the management of model risk and this should be adequately documented. Senior management is responsible for the execution and maintenance of the framework and should designate the roles and responsibilities for the framework to model owners, model users, and control and compliance functions. The board of directors and senior management are expected to provide challenge to model outputs and understand model capabilities, the model limitations, and the potential impact of model uncertainty.

P2.2 Model risk management policies:

These should cover all aspects of model risk management, including model definitions; model development standards; model change; implementation; use; validation; review; and management sign-off. The policies should set out appropriate governance and challenge frameworks, and the roles and responsibilities of model owners, model users, and control and compliance functions. The prioritisation, scope and frequency of validation, review, and monitoring activities should also be set out in the policies.

P2.3 Model owners and control functions:

Model owners should have accountability for model use and performance. Model owners should be responsible for ensuring that models are appropriately developed, implemented, used as intended, have undergone appropriate validation and approval, and are recorded and maintained in the model inventory. Control staff should have the authority to restrict the use of models and monitor any limits on model use.

 P2.4 Role of Internal Audit (IA):
 IA should assess the overall effectiveness of the model risk management framework. IA should evaluate and independently verify whether model risk management practices are comprehensive, rigorous, and effective.
P2.5 Use of external resources:

If external resources are used for any model development, validation, or review activities, banks should be able to verify that these are conducted in accordance with their model risk management standards. Designated internal staff should be responsible for the work delivered by the external party, and should be able to address any issues identified either with model development or as a result of model validation.

Principle 3: Banks have implemented a robust model development and implementation process and ensure appropriate use of models

 P3.1 Model purpose and design:

The purpose, design, choice of parameters, mathematical theory, and underlying assumptions of a model should be appropriately documented and conceptually sound (appropriate for the intended business purpose), and supported by published research and generally accepted industry practice where appropriate. Particular emphasis should be placed on model limitations and, where possible, model results should be supported by a comparison with alternative theories/approaches, or by assessing the sensitivities of changes in model inputs.

 P3.2 Use of data:

The data used to develop a model should be assessed for quality and relevance. Where adjustments are made, proxies are used, or where the data are not representative of the bank’s portfolio or asset mix, the impact should be justified and documented so that users are aware of the potential model limitations.

 P3.3 Testing:
Appropriate testing of models should be conducted to take into account potential limitations, assess their robustness and stability over time, and across a variety of economic and market conditions, in particular those relating to periods of stress. Testing activities should be appropriately documented.
P3.4 Documentation:

Banks should have sufficiently detailed model documentation so that an independent third party with relevant expertise is able to understand how the model operates, identify its key assumptions and limitations, and replicate any parameter estimation and model results. Where a bank uses vendor models, it should have appropriate documentation on the approach to be able to validate the model.

 P3.5 Use of judgement:

Any judgements or model overlays that are used to modify the parameters, inputs and/or outputs of a model due to known model limitations should form a part of the development process, should be appropriately understood and documented, and should be subject to review and challenge by independent parties.

 P3.6 Supporting systems:
Model calculations should be implemented in information systems or environments which have been thoroughly tested for this purpose. The findings of any system/implementation tests should be documented.
P3.7 Business involvement:

Frontline business should play an integral part in the design and testing of models and should challenge the methods, the underlying assumptions, and the output of the models – both at inception and on an ongoing basis.

 P3.8 Model uncertainty:

Banks should demonstrate that model uncertainties are adequately understood, managed, monitored, reported, and accounted for in the results. Where conservatism is used to mitigate model uncertainty, banks should justify and document any such adjustments and demonstrate that the adjustments are intuitive from a business and economic perspective.

 P3.9 Monitoring:

Banks should perform periodic monitoring of model performance with a frequency commensurate with the nature and materiality of the models and risks, with due consideration given to model complexity.

Principle 4: Banks undertake appropriate model validation and independent review activities to ensure sound model performance and greater understanding of model uncertainties

 P4.1 Scope of validation and review:
 All model components (inputs, calculations and reporting outputs) should be subject to independent validation for both in-house developed models and vendor models. Any validation work undertaken by model developers and users as well as any material changes to already validated models or overlays should be subject to review by an independent party. The extent of validation and independent review should be appropriate with the overall use, complexity, and materiality of the models or changes to a model.
P4.2 Independence:

The staff performing model reviews should be independent of the model development process to be able to provide a robust and objective view. The effectiveness of the independent challenge should be judged by the quality of the issues identified and the actions taken by model owners and management to address them.

 P4.3 Staff competence and influence:

Banks should consider whether validation staff have: the necessary knowledge, skills, and expertise to perform model validations; an adequate degree of familiarity with the business, product, risk, and intended use of the model; and sufficient influence and stature within the bank to ensure that issues and deficiencies are escalated and addressed in a timely manner.

 P4.4 Treatment of model issues/deficiencies:

When significant model deficiencies and/or errors are identified during the validation process, banks should consider whether the use of models should either be prohibited or only be permitted under strict controls and mitigants. The process of managing identified model issues should include the tracking of the outstanding issues and should be adequately documented.

 P4.5 Frequency of model validation:

Banks should undertake regular revalidation of models to track known limitations and to identify potential new issues. Periodic reviews should be carried out with a frequency and level of rigour commensurate with the overall use, complexity, and materiality of the models.