ebook img

Internal Model Validation PDF

31 Pages·2017·0.35 MB·English
by  PatelAlpesh
Save to my drive
Quick download
Download
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.

Preview Internal Model Validation

Internal Model Validation Model Validation Guidance July 2017 THIS PAGE IS INTENTIONALLY BLANK 2 Contents 1 Introduction 5 1.1 Purpose 5 1.2 Scope 5 2 Executive summary 6 3 Validation report format 7 4 Components of validation 8 4.1 Introduction 8 4.2 Independence 8 4.3 Risk coverage and risk ranking 9 4.3.1 Risk coverage 9 4.3.2 Risk ranking 9 4.4 Overview of the validation cycle 10 4.4.1 Analysis of validation test results 10 4.4.2 Escalation 11 4.5 Pass/fail criteria 11 4.6 Expert judgement 12 4.6.1 Expert judgements used in the internal model 12 4.6.2 Expert judgement used in validation 13 4.7 Limitations 14 4.8 Targeted validation 14 4.8.1 Reliance on previous validation 15 5 Key technical validation tests 16 5.1 Sensitivity tests 16 5.1.1 Deterministically varying a set of assumptions (“ST-1”) 16 5.1.2 Plausible alternative set of assumptions (“ST-2”) 16 5.2 Stability tests 17 5.3 Stress and scenario tests 18 5.4 Reverse stress tests (RSTs) 19 5.5 P&L attribution 20 5.6 Testing against experience 21 5.7 Benchmarking 21 5.8 Other tests 22 6 Thematic findings 23 6.1 One year validation 23 6.2 Dependencies and aggregation 24 6.3 Outwards reinsurance 25 6.4 Catastrophe risk 26 6.4.1 Model version 26 6.4.2 Model completeness 26 6.5 Operational risk 27 6.6 Credit risk 27 6.7 Events not in data (ENIDs) 28 6.8 Special Purpose Arrangements (SPAs) 28 Appendix A: Reverse stress tests: frequently asked questions 30 3 THIS PAGE IS INTENTIONALLY BLANK 4 1 Introduction 1.1 Purpose This document provides guidance to managing agents in respect of internal model validation requirements under Solvency II. It builds on previous Lloyd’s guidance and findings from Lloyd’s review of the market’s validation reports. This model validation guidance should be read in conjunction with the Lloyd’s Minimum Standards, previous Lloyd’s guidance and workshops on model validation for a complete understanding of the regulatory requirements in the Lloyd’s market. These can be found on the Lloyd’s Model Validation section on Lloyds.com. 1.2 Scope This document offers guidance on validation of the methods, assumptions and expert judgement used in the internal model. The following areas are not covered in detail in this document; additional guidance has been provided by Lloyd’s in the sources listed below: - Targeted validation - Data - Catastrophe risk - Major model change - Lloyd’s Minimum Standard 5: model scope, change and use - Lloyd’s Minimum Standard 6: modelling, design and implementation - Lloyd’s Minimum Standard 7: validation 5 2 Executive summary The overall purpose of the validation exercise is to ensure that the internal model provides a realistic and robust assessment of all the material risks faced by the syndicate. Validation is an iterative process of identification of model limitations and the implementation of improvements. The most appropriate validation tools for achieving this task will vary by syndicate, depending on the size and complexity of its risks. This guidance is intended to assist managing agents with the selection and application of the validation tools most suited to their syndicates’ risk profiles. Regulatory expectations have also been evolving; the Supervisory Statement SS17/16 from the Prudential Regulatory Authority (PRA) discusses model validation in section 7 and the internal model change policy in section 9. Managing agents have made good progress in embedding the validation process within their business since Solvency II was implemented. Some of the improvements noted by Lloyd’s in the latest round of validation report reviews were: adequate management level information was included in validation reports; - limitations of the model and validation process and failures in validation tests were - appropriately highlighted in the validation report; the purpose and use of validation tests was clearer; - more clarity in the governance structure and escalation process was noted; - generally, the pass/fail criteria for quantitative validation tests were more explicit. - Based on the latest validation report reviews, Lloyd’s is advising agents to give special consideration to the two areas listed below. These are both discussed in detail later in this guidance. 1. Reverse stress testing: several inconsistencies were identified in execution of this particular validation test. Agents should aim to define the scope and execute the test appropriately in line with Lloyd’s guidance; 2. Structure of the report: in some cases the validation report did not include sufficient technical detail on certain topics (e.g. expert judgement, sensitivity tests, one year SCR, P&L attribution) to assessment of regulatory requirements. In addition to the main validation report (provided to the Board), this may be submitted either as part of a separate (more technical) report or as appendices to the main validation report. The current state of model validation in the market is deemed to be well-progressed and embedded within agents’ operations. This latest guidance is intended to continue the advancement of model validation and improve efficiency and effectiveness of the process. Lloyd’s intends to move to a more targeted approach to validation of material risks faced by the syndicate. Further details can be found in section 4.8. 6 3 Validation report format A good validation report should (at the very least): - be concise; - aim for a top-down view of validation; - not exclude relevant quantitative details of capital and validation; - be treated as a Board report; - focus on material risks; - emphasise failures; - outline the limitations to the validation process. This is a list of suggestions and should not be treated as an exhaustive list of requirements. The technical details of validation tests and process are also required to be documented. Lloyd’s has recommended different approaches to documenting and reporting details of a validation process in previous workshops (validation workshop 2016 and validation workshop 2014). In the latest round of reviews of agents’ validation reports, Lloyd’s observed a marked improvement in the size and content of the main validation report. Most reports successfully summarised the key validation findings in a concise manner. However, in some cases, the level of quantitative detail was not adequate. It is important to include sufficient quantitative detail, as the Board must have sight of key details to allow them take strategic and financial decisions. Lloyd’s is required to ensure, for regulatory compliance, that validation tests are carried out consistent with the guidance issued. Lloyd’s identified that technical details are sometimes not included in the regulatory submissions. This has led to challenges and required additional work by those agents. To address this, detailed information on the validation tests, audit trail and other technical details may be included in a separate (more technical) report or as appendices to the main report. This would ensure a concise validation report, summarising key details of the process, is maintained. The main validation report should still retain relevant quantitative information that helps identify possible limitations in the process or model. The additional report (or appendices) would provide evidence that would assist in assessment of regulatory compliance. Throughout this guidance, the terminology ‘validation report’ is intended to specify the Board report, while ‘validation pack’ is the regulatory submission to Lloyd’s (including both the validation report and the additional supporting technical documentation). As part of the validation pack, agents are requested to document how feedback from Lloyd’s (and other stakeholders) during the previous validation cycle have been considered as part of the development of the latest validation cycle. 7 4 Components of validation 4.1 Introduction It is essential for the managing agent, for Lloyd’s, and for compliance with Solvency II that syndicates’ internal models are thoroughly validated. The requirements for model validation under Solvency II are extensive, and meeting them has brought many challenges for the industry. This is especially true for Lloyd's due to the unique and complex nature of the risks written in the market. Managing agents have invested significant time and effort over the last few years in order to meet the requirements of Solvency II, including those for validation. The validation exercise consists of many components which are covered in more detail in the following sections. The key high level validation requirements include: independence in order to ensure objective challenge; - a process to ensure that all material risks are covered in the model; - a risk ranking exercise and a demonstration that the validation effort has been - proportional to the materiality of the risk; validation tools and tests to be well defined and appropriate for the risks being - validated; outcomes of tests to be clearly explained and justified, and the path from a “fail” - outcome to escalation and model change (if appropriate) should be clearly mapped; and documentation of the validation process and the reasons for the steps taken. - 4.2 Independence Validation requires objective challenge. Agents should be able to demonstrate that the individuals responsible for validation have sufficient independence from the design, build, parameterisation and implementation of the model component being validated. They may make use of tests carried out by those responsible for the modelling, but not rely entirely on them. There is no requirement for external validation to form part of the validation process. A number of agents do however use third parties in the validation process in order to supplement internal resources. Where reports prepared by third parties (both internal and external) are used in the validation process, the agent should make clear: the scope of the work performed; - the reliance placed on the work; and - why the validator feels able to rely on the work performed by the third party (this may - take the form of the experience and qualifications of the external parties being set out). 8 4.3 Risk coverage and risk ranking 4.3.1 Risk coverage The objective of the validation process is to test that all material risks to the syndicate are adequately assessed in the internal model. The initial steps in the validation process should therefore be a gap analysis to test whether all material risks are indeed covered. A typical and acceptable way to do this would involve: an identification of risks to the business; - an identification of which of these risks are not covered by the internal model; and - an assessment of whether the risks not covered are material. - The risk identification process should identify all sources of loss to which the syndicate could have non-trivial exposure. This process should not be restricted to insurance risks; it should, for example, include considerations such as the terms and conditions of the cover issued, data and operational systems, the current legal environment, recent market experience, and so on. It should also take into account the possibility of new sources of loss not experienced by the syndicate or market in the past, for example emerging risks. The P&L attribution exercise has historically been useful in identifying risk coverage gaps – see section 5.5 for further details. If material risks have been excluded as a result of this gap analysis, the model needs to be amended to include them. Many agents have maintained a risk register as a way of identifying risks faced by the syndicate. Lloyd’s considers this to be an appropriate approach under Solvency II. The format of the register should be the one most suitable to agents. The risk register may also be used to identify which risks are currently “in” or “out” of the internal model. The risks not covered by the internal model should be assessed for materiality to the syndicate. This requires explicit criteria for assessing materiality and/or risk indicators for determining materiality. It is agents’ responsibility to determine and describe the materiality threshold for their syndicates. The materiality threshold should be closely linked to those for major model changes. Historically, some of the largest losses to the market have come from sources of loss that were only partially modelled or non-modelled. Examples include asbestos, multi-year reserve deteriorations, and the World Trade Center terrorist attacks. For this reason, Lloyd’s considers a comprehensive risk identification process to be the basis for sound validation. Lloyd’s recommends that agents develop a comprehensive process for identifying potential risks to the business and assess their materiality on an ongoing basis (at least annually). This should cover changes in the risk profile (for example, an increase in cyber risk), and provide re-affirmation that risks previously assessed as non-material remain so. 4.3.2 Risk ranking The risk coverage exercise described above should be proportional to the size and complexity of the business and confirm whether all material risks are captured. The purpose of risk ranking, in the context of validation, is to identify those areas requiring the most extensive validation. Risk ranking should therefore be one of the first steps in the validation of the internal model. In addition to this, agents can also perform risk ranking at the end of a validation exercise in order to check that the ranking is as expected at the conclusion of the process. 9 Agents should be able to produce rankings consistent with the groupings used to manage the business (for example, by class of business within reserve risk or by currency/asset class within market risk). It is not necessary to rank every risk component of the internal model, although the methodology should have the capability to do so. The ranking methodology may be approximate, but should be broadly consistent across risk types. There is overlap between risk ranking and the assessment of risk coverage described above. Both can be done using approximate metrics, such as the materiality of the risk being considered. In this context, agents should view risk ranking as a tool for making the validation process more efficient. It will enable them to allocate resources to areas which are quantitatively or qualitatively material to the internal model. Lloyd’s recommends that the validation policy includes a description of how the outputs of the risk ranking exercise are used within the validation process. 4.4 Overview of the validation cycle The validation cycle is at the centre of the validation process. It is comprised of four steps: (1) the application of the validation test or tool; (2) the analysis of test results; (3) the escalation of test results to appropriate individuals in the business; and (4) the implementation of any changes necessitated by the validation test outcome. Definition and implementation of the validation cycle has been embedded in the validation process through multiple iterations. Agents have provided relatively clear descriptions of steps (1) and (2). Many agents provided sufficient information on step (3), but a practical challenge often is to understand the right level of detail of the governance process that should be included in the report. Details of governance are often included in the validation policy. Adding a summary in the validation report and clearly signposting where a fuller description might be found may overcome challenges associated with step (3). It can be a challenge to demonstrate the escalation process as many issues identified will have already been addressed during the validation cycle and are therefore resolved by the time the final validation report is completed. Examples of validation test failures and the process followed for such cases is one way to evidence the governance structure in place. Finally, Lloyd’s recommends that each validation test shows evidence of the four stages of the validation cycle. In particular, a test outcome and the reasons for that outcome should be clear. 4.4.1 Analysis of validation test results The results of validation tests should be justified with clear rationale in conjunction with the pass/fail criteria discussed in section 4.5; this ensures the effectiveness of the test. A good way to provide adequate rationale is to support qualitative conclusions with quantitative analysis or vice versa; two examples are provided below to illustrate this. 10

Description:
adequate management level information was included in validation reports;. - limitations of the model few years in order to meet the requirements of Solvency II, including those for validation. The validation .. assess different areas of the model (e.g. reinsurance) under extreme conditions. Stres
See more

The list of books you might like

Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.