CP6/22 – Model Risk Management principles for Banks

The proposed principles are comprehensive and fit for purpose, if we assume the application of proportionality. Similar frameworks have been developed and used by model vendors (such as Kadre) and sophisticated users of predictive Credit Risk models, both Capital and Decision models, for many years.
This is clearly a step up in expectations and undoubtedly a review of Model Risk Management practices in Australia would find deficiencies similar to those identified by the PRA in the UK. Even banks who have adopted the IRB regime are unlikely to apply all of the sub-Principles to all of their decision models e.g., do they apply them to the management of their Fraud Detection models or potentially their Marketing prospecting models?
The following Appendix is a precis of the principles detailed in CP6/22 and is intended to provide a high-level insight into the requirements and what constitutes Model Risk Management best practice.

Whilst CP6/22 is a Consultation Paper from the UK, and so will not directly apply to most of us, it does suggest that there will be increasing scrutiny on the effective development and management of all critical Decision Models, which are increasingly important to all lenders, not just those regulated by Prudential Regulators. The principles and subsequent policies and processes recommended by the PRA should be considered by all Decision Model users, and of course Kadre are here to help!

 

The Principles (This summary paraphrases the PRA CP and is not complete.)

Principle 1 – Model identification and model risk classification

Principle 1.1 Model definition – firms should adopt the prescribed definition of a model and consider applying similar controls for other decision rules or algorithms affecting significant business decision even if they don’t qualify as models under this definition.

Principle 1.2 Model inventory – firms should maintain a comprehensive inventory of all models in use, in development or decommissioned. It should include the purpose and use of the model, note any simplifications and limitations, document the findings from the model validation and note governance details such as the individuals responsible for validation, dates of validation and frequency of future validation.

Principle 1.3 Model tiering – Risk based tiering should be adopted to help identify the models that pose the most risk to the business based on both the materiality of the model’s application and the complexity associated with the model. The materiality could address quantitative measure such as the size of the portfolio or number of customers or transaction it is applied to as well as qualitative factors relating to the purpose of the model and its relative potential impact on the business decisions.

Complexity should consider the inherent risk associated with the model such as the nature of the data used in development or implementation e.g. unstructured data or alternative data being high complexity, or the challenges relating to the interpretability, explainability, transparency or potential for bias in the model.

The approach to tiering should be periodically validated and individual tiering assignments should be independently reassessed periodically.

 

Principle 2 – Governance

Principle 2.1 Board of Directors’ responsibilities – the Model Risk Management framework should be subject to leadership from the board of directors. It should be designed to promote an understanding of model risk in aggregate and on an individual model basis. The board should set a model risk appetite, highlighting measures that define the effectiveness of the design and operation of the MRM framework, approve use for decision making, identify the limits on model use, exceptions and overall compliance, thresholds for acceptable model performance and tolerance for errors as well as use of mitigants and oversight of expert judgement. The Board is expected to receive regular reports on the firms model risk profile compared to the stated risk appetite and they are expected to provide challenge to the model outputs, understand the capabilities and limitations and operating boundaries as well as consequences of poor model performance.

Principle 2.2 Senior Management Function (SMF) accountability for model risk management framework – the firm should identify the most relevant SMF to assume overall responsibility for the management of model risk (similar to BEAR requirements for single point accountability). These responsibilities could include establishing policies and procedures to operationalise the MRM Framework, assigning roles and responsibilities, ensuring effective challenge, independent valuation, reviewing model results and internal audit reports, ensuring remedial action is undertaken, and that resourcing is adequate for the effective implementation of the framework.

Principle 2.3 Policies and procedures – are required to formalise the MRM Framework and support effective implementation and review of the firm’s model risk appetite and profile. These should be approved by the board and reviewed on a regular basis. Policies should be cross-references and aligned with other relevant parts of the broader risk management regime.

Policies and procedures should address all aspects of the model lifecycle including definition of model, tiering approach, model development standards, data quality procedures, standards for model validation and performance monitoring, model risk mitigants and the approval process for new or amended models.

The SMF should ensure adequacy of the board-level policies.

Principle 2.4 Roles and responsibilities – roles and responsibilities for each stage of the model life cycle should be documented including identification of requisite skills, experience and expertise. Model performance monitoring can be undertaken by the model owner, users or developers but the adequacy of model monitoring should be assessed by the model validators.

Model owners should be identified and documented for all models. Their accountabilities include ensuring the models as monitored against the risk appetite, that the model is assigned the correct tier, it is recorded in the inventory and information about the model is up to date and accurate.

Model users should also be identified and documented. Model users are accountable for ensuring that the model us is consistent with the intended purpose and any limitations are taken into account when outputs used.

Model developers should be identified and documented. They are accountable for ensuring that research, development, evaluation and testing are all conducted in adherence to the firms standards.

Model validation should be undertaken by staff with requisite skill and expertise and sufficient familiarity with the business application of the model. They should have the necessary organisational standing and incentives to ensure limitations or inappropriate use are reported in a prompt and timely manner.

Principle 2.5 Internal Audit – should periodically assess the effectiveness of the MRM framework. This should be documented and reported to the board and relevant committees. It should verify that the internal policies and procedures are comprehensive and able to identify and manage model risks, that controls and validation activities are adequate, that validation staff have the necessary experience, expertise, standing and incentives and that the model owners and risk control functions comply with internal policies and procedures.

Principle 2.6 Use of externally developed models, third party and vendor products – notes that boards and executive management are responsible for the management of model risk even when they have entered into an outsourcing or third party arrangement. This would include satisfying themselves that any vendor models have been validated to the same standards they apply and verification of the relevance of any vendor supplied data.

Subsidiaries relying on parent group developed models should be able to demonstrate that the parent has implemented the MRM framework to the standards of the firm.

 

Principle – 3 Model development, implementation and use

Principle 3.1 Model purpose and design – all models should have a clear purpose and design objectives relevant to the intended use. The choice of modelling technique should be conceptually sound and supported by published research or generally accepted industry practice. The output should be compared to other approaches where possible. Emphasis should be placed on communicating to the model users the limitations of the model under different circumstances.

Principle 3.2 The use of the data – data used in the development of the models should be consistent with that available during model use and consistent with the chosen methodology. There should be no inappropriate bias, and it will need to be compliant with data privacy regulations.

If the development data is not representative of the use the potential impact should be assessed and relevant limitations on use and mitigations should be applied. Adjustments to data used in the model development should be documented and validated. Any higher risk data sources e.g. unstructured data should be identified.

Principle 3.3 Model development testing – Model quality should be demonstrated during the development process. A monitoring pack should be defined by the developers, including the operating boundaries within which a model is expected to perform.

Models should be tested during development against the models design objectives based on a range of tests including backward looking out of time samples, forward looking plausible stress scenarios, sensitivity analysis to determine the boundaries of acceptable performance and compare the model with challenger models (n.b. they do not refer explicitly refer to comparison with existing models or Champion Challenger running of models, both of which would be consider by many to be better practice)

Models with dynamic calibration should recalculate performance tests each time the calibration is undertaken! This has potential to impact techniques such as machine learning where ever they are used to refine or recalibrate models at high frequency or have been implemented in continuous learning mode. Development testing should apply to both new builds and material changes to existing models.

Principle 3.4 Model adjustments and expert judgement – Any adjustments to any aspect of the model should be considered, justified, documented and, if material, validated. (n.b. the principles are silent on Reject Inference processes, this principle could apply)

Principle 3.5 Model Development documentation – should be comprehensive and up to date, including description of data, choice of methodology, performance testing and model limitations.

Principle 3.6 Supporting systems – should be thoroughly tested and periodically reassessed for suitability.

 

Principle 4 – Independent model validation – should be ongoing, independent and effectively challenge model development and use.

Principle 4.1 The independent validation function – should be objective and unbiased, sharing responsibility with model owners for model performance monitoring and process verification. They should be independent from the model development process and model owners. IRB firms need to demonstrate different reporting lines to the owners and developers. All firms should have an independent party such as Internal Audit periodically review the overall effectiveness of the model validation process and its outcomes.

Principle 4.2 Independent review – All models should be subject to independent review of all aspects of model development, implementation and use as well as the determination of model tier.

Principle 4.3 Process verification – All model processes and systems implementations should be subject to verification including model inputs, calculations and reporting outputs.

Principle 4.4 Model performance monitoring – should be undertaken regularly with reference to the criteria designed during the development of the model. A range of tests should be undertaken including benchmarking to alternative models, sensitivity tests, over-ride analysis and evaluation of new data.

The frequency of model testing should be determined with consideration to the model tier and monitoring reports should be independently reviewed.

Principle 4.5 Periodic revalidation – should be undertake regularly with a frequency determined by the model tier.

Principle 5 – Model risk mitigants – have established policies and procedures for applying model risk mitigants when models are under-performing. Conduct independent review of post-model adjustments (PMAs).

Principle 5.1 Model risk mitigants – can be applied if a model is underperforming. The firm should have established policies and procedures that define the circumstances for when and how mitigations can be applied. A clear rationale should be documented and approved by the relevant authority. They should be systematic and applied transparently and subject to independent review.

Principle 5.2 Restrictions on model use – should apply when a model is shown to have significant deficiencies or errors. The process for managing model deficiencies should be documented and any deficiencies should be reported to key stakeholders.

Principle 5.3 Exceptions and escalations – A policy should exist that documents when models that are not approved, validated, used outside of intended purpose or where they consistently breach performance expectations can be allowed for use. Exceptions should be temporary, reported and supported by stakeholders and maximum tolerances should be defined.