AuditOne Compliance Advisory: Model Validations and Related Exercises

AuditOne Advisory

From Bud Genovese, Chairman

Model validations to meet model risk management and governance requirements are clarified in this reprint of WIB’s April 2017 Compliance Digest article written by Jeremy Taylor, Co-CEO, AuditOne LLC. Please feel free to forward it to the appropriate people in your financial institution. Thank you. -Bud

Model Validations and Related Exercises

Jeremy Taylor, Co-CEO, AuditOne LLC

The last 2 – 3 years have seen a burst of emphasis on ensuring a disciplined process for financial institutions’ (FIs’) management of model risk, in accordance with OCC Bulletin 2011-12 (which superseded 2000-16) and FRB SR 11-07.  Neither the FDIC nor the NCUA have issued similar formal guidance, but they have signaled their concurrence in other documents (e.g., the Winter 2005 issue of the FDIC’s Supervisory Insights).

This emphasis has come, not surprisingly, in response to FIs’ growing reliance on models, whether purchased or developed in-house, for a wide range of risk management and other purposes.  While community banks are typically less model-reliant, there are still expectations that they adopt appropriate governance and controls for their models, examples of which are cited below.

While the OCC and FRB documents address all the required elements of model governance, what has become evident is some confusion among model users over terminology and over the particular requirements for verifying satisfactory model performance.  Specifically, we see frequent failure to distinguish clearly between what each of the following is intended to do and when/where it may be needed:

  • Validation
  • Certification
  • Audit
  • Independent review
  • Service Organization Controls report (SOC 1/SSAE 16 and/or SOC 2)

This blurring in terminology can in turn translate into either gaps in required documenting or, perhaps, overkill.  The resulting confusion can easily carry over into an FI’s dialogue with its regulators and auditors as well.

The OCC and FRB guidance clearly states that: “All model components, including input, processing, and reporting, should be subject to validation.”  But who is to do this and how?  For purchased models, it is critical to recognize that the structure, features and core functionality are common to all users; it therefore makes sense for the model vendor to arrange for a validation report to be prepared independently (e.g., by a consulting firm with specialized expertise in that area), to be made available to all users.  This makes more sense than every user going out and independently commissioning such a report.  Its purpose is essentially to answer the question, “Does the model do what it says it does?”  Such a report will attest to those validation concerns relevant to all users, which takes in a model’s logical structure; the underlying math, finance, statistics, etc.; their translation into algorithms and coding; the reliability of those calculations; the reporting options available; alternative/competing models and approaches; etc.

So why, then, the need for audit?  Because it is important to recognize that this list of validation concerns is only a partial list.  There is another, quite different, set of concerns that an off-the-shelf validation report will not touch: the user-specific concerns.  Is the model appropriate for the user and has it been set up properly?  This is where an auditor will focus, taking the validation report as a starting point (including verification that any issues identified in the report have been following up on by the user).  The audit will then address such critical issues as:

  • Is the model appropriate for the FI’s needs?
  • Have the data feeds been set up correctly?
  • Are there controls to ensure continuing data input integrity?
  • Are the assumptions reasonable and properly supported?
  • Is the model producing results that make sense, verified by back-test?

Unfortunately, the audit exercise described above is often referred to as a “validation.”  It also sometimes gets called (e.g., in the 1996 Joint Policy Statement on Interest Rate Risk) an “independent review.”  Let’s call just call it an audit, and the first exercise discussed above a certification.  Again, this distinction only becomes necessary for purchased models; in-house models won’t have the same bifurcation.

For most of the vendor models that our firm encounters – such as for IRR simulations, AML suspicious activity monitoring, ALLL general reserve requirements – independent model certifications are indeed typically available.  To throw in another wrinkle, there are situations where an outside vendor may also need to consider obtaining and making available a Service Organization Controls (SOC) 1 and/or SOC 2 report.  This will come into play where the vendor is not just offering usage of a model but also an outsourced service to take its client’s (e.g., a bank’s) data, run it through the model, and produce reporting to send to the user.  In those cases, the vendor’s internal controls relevant to defined principles – security, availability, processing integrity, confidentiality and privacy – become a separate concern that a model certification report may address only in part.

A SOC 1 is performed in accordance with Statement on Standards for Attestation Engagements (SSAE) 16 and is needed when the output relates directly to the financial statements of a client; the ALLL is a good example.  A SOC 2 is broader in its assessment of controls and does not have the financial statement focus.  It was created in response to technology entities such as data centers and cloud-based systems to create an audit that would assess the effectiveness of their controls according to the defined principles listed above, applied across all functional areas and not just those related to financial reporting.  For this reason, some service organizations have both a SOC 1/SSAE 16 and a SOC 2 performed to satisfy different audiences.  The clients of the service provider may need the SOC 1/SSAE 16 for their financial accountants while providing the SOC 2 to the information security officer of the client firm.

Whatever label we apply to these different types of exercise with their different goals and audiences, what is clear is regulators’ growing focus on ensuring a systematic and comprehensive approach to addressing the relevant concerns.

Published in Western Independent Bankers Association’s Compliance Digest, Issue 23 – April 2017.


AuditOne LLC’S Analysis of IRR model assumptions 2017

AuditOne Advisory

From Bud Genovese, Chairman

Kruskal Hewitt, a Senor Associate based in our New York City office, has written an article below that  summarizes key components used for interest rate risk (IRR) model assumptions and limit-setting. Mr. Hewitt tracked this data from the numerous IRR audits we perform each year which we’ve compiled (anonymously) into a very useful database. Please feel free to forward this informative column to any appropriate people in your financial institution.  Thank you.  –Bud


AuditOne LLC is a leading provider of outsourced internal audit services for community banks, credit unions and other financial institutions.  Please refer to our website  for further information.  Among our practice areas is interest rate risk (IRR).  US financial institutions are expected to have an annual internal audit of their modeling, monitoring and control of IRR.  Key to IRR modelling are various forward-looking assumptions required for the simulations of net interest income and economic value of equity under interest rate shock scenarios.      

AuditOne has compiled (anonymously) data from 94 of our IRR clients on IRR limits and assumptions.  These are institutions where we have used data from the most recent AuditOne IRR audit, no further back than 2014.  AuditOne believes this database is relevant to AuditOne clients because it covers a relatively narrow range of asset size, geography and business lines.


NII:  Net interest income (NII) is a current period (generally, one-year and two-year) estimate of interest-sensitive revenues and expenses under alternate interest rate scenarios.  (The tables beginning on page two all apply to one-year horizons.)

EVE:  Economic value of equity (EVE) is a theoretic valuation of the institution where cash flows from all assets and liabilities are discounted to their net present value, then summed together.

INSTANTANEOUS vs. RAMPED CHANGES:  The figures showing in the tables below are mostly for instantaneous (or immediate) rate shocks (85 clients).  These assume rates change instantly by the full shock amount, as opposed to a gradual rate rise (ramp) over time, typically a 12-month ramp.

BETA:  This represents the assumed percent of a market rate change that is reflected in administered rates – most importantly, deposit rates.  If the driver rate is Fed Funds and the beta for saving accounts is 45, then for every 100 basis point rise in Fed Funds, savings account rates are assumed to rise 45 basis points.  There may also be assumptions about lags in administered rate changes, but we have not captured these in our database.

AVERAGE LIFE:  Non-maturity deposits (NMDs) have no contractual maturity and therefore form a stable, longer-term funding source.  In order to get a meaningful estimate of EVE, NMDs are assigned an assumed average life by account type.

The standard rate shock set-up assumes the yield curve shifts in parallel fashion over the entire maturity spectrum.  However, many institutions also run simulations based on flatteners, steepeners and other non-parallel shocks. These can be helpful for assessing specific, balance sheet vulnerabilities. But we advise against basing IRR limits on non-parallel shocks because shock details are too hard to define in policy.

STATIC vs. DYNAMIC BALANCE SHEET:  For NII simulations, the balance sheet can either be static (constant), with like replacement of run-off assets or liabilities, or it can incorporate growth (e.g., budgeted balances).  The 2010 Interagency Guidance specified that a static balance sheet must be used, though simulations can also be run off a dynamic balance sheet.


The following results have been presented here across the whole database.  However, we would be happy to recalculate any of the results for subsets of institutions based on asset size, primary regulator and/or model vendor.  Please contact either Jeremy Taylor at 949-981-0420 or Kevin Watson at 562-802-3581.  See the database mix summary section below for the key identifiers.

We have presented only average (mean) figures in the tables below.  We also computed medians, but these were very close to the corresponding average for all but one data set.  The one exception was NMD average life where the average exceeded the median by a meaningful amount for each of the four deposit categories.  This means that the top half of institutions (in terms of their average life assumptions) kewed more, or had more extreme values than, the bottom half.

NII-at-risk (one-year) simulation policy limits

EVE-at-risk simulation policy limits

Beta assumptions 

Average life assumptions (in months)

Interest rate shock application (for limits)

Parallel versus non-parallel shock assumptions

Balance sheet growth assumptions


The following tables describe the 94 institutions in the database.  All dollar figures are in millions.

Database mix by asset size

Database mix by primary regulator

Database mix by model vendor

Kruskal has been a Senior Associate with AuditOne since 2014, specializing in ALM (asset/liability management) audit and consulting work. He has considerable experience in the treasury and trading areas, including derivatives, investments and foreign exchange, in addition to interest rate and liquidity risk. Prior to AuditOne, he was with a Japanese utility, managing market and credit risk. Before that his background included market risk management with a large US regional bank and with multinational banks in the US, Asia and Europe. Kruskal holds a BA in Mathematics and an MBA from Northeastern University. His certifications include PRM (Professional Risk Manager), FRM (Financial Risk Manager), and CALMS (Certified ALM Specialist).