Adversarial Evaluation
Engine
A deterministic framework for engineering complex professional scenarios that systematically expose reasoning failures in large language models.
RLHF & AI Evaluation Engineering
Modern AI systems learn through a pipeline: pre-training on internet-scale data, supervised fine-tuning on curated examples, and reinforcement learning from human feedback (RLHF). The quality of that feedback — the evaluation data — determines whether models develop genuine reasoning or sophisticated pattern matching.
Evaluation engineering sits at the most critical point in this pipeline. It's the process of designing tasks complex enough to distinguish real understanding from surface-level pattern completion — and building the verification infrastructure to objectively measure the difference.
Why Current Benchmarks Fail
Most AI evaluation tasks test surface-level capabilities. Models pass these tests through pattern recognition, not reasoning — creating a dangerous illusion of competence that breaks down in professional settings where precision matters.
- Single-document Q&A
- Multiple choice formats
- Surface-level extraction
- No cross-reference needed
- No calculation chains
- No policy conflict resolution
- Multi-document synthesis (5-7 files)
- Cross-file conflict resolution
- 12-step calculation chains
- Policy authority navigation
- Constraint satisfaction under ambiguity
- Portfolio-level risk analysis
Adversarial Design Philosophy
Each trap category targets a specific reasoning weakness in large language models. The system doesn't just “make hard tasks” — it systematically engineers failure modes that expose the gap between pattern matching and genuine professional reasoning.
Policy Authority Conflict
Two documents disagree on a threshold. One governs as the more recent issuance. Models must identify which source is authoritative.
Non-Recurring Income Trap
A one-time insurance settlement is embedded in operating income. Must be excluded from underwriting NOI — the footnote is easy to miss.
Competing Appraisals
Two property appraisals using different methodologies produce values 12%+ apart. Policy mandates using the lower when spread exceeds 10%.
Synonym Variation
Three terms for the same concept across different files — 'total fixed charges,' 'annual loan obligation,' and 'recurring contractual obligations' — each with slightly different scope.
Calculation Cascade
One wrong input propagates through 12 downstream calculations. Wrong appraisal → wrong LTV → wrong tier → wrong rate → wrong DSCR → wrong decision.
Portfolio Concentration
A borrower qualifies on every individual metric but triggers a sector concentration limit buried in a table footnote of a different document entirely.
Deterministic Verification Pipeline
Every number in the system is traceable. Raw data flows through a deterministic engine, is independently verified, generates professional documents, and produces a self-scoring golden deliverable — all before a single rubric criterion is written.
Commercial Loan Underwriting Analysis
A credit analyst evaluates three loan applications against lending policy, competing appraisals, and internal risk guidelines. One definitional error cascades through 12 downstream calculations, flipping the lending decision entirely.
Meridian_Commercial_Lending_Policy_v4.2
Meridian Commercial Bank Commercial Lending Policy Manual Revision 4.2 Effective Date: January 15, 2025 Approved by: Credit Policy Committee Classification: Internal — Confidential This document sets forth the policies and procedures governing all commercial lending activities of Meridian Commercial Bank. All lending officers, credit analysts, and risk management personnel are required to comply with the standards herein. © 2025 Meridian Commercial Bank. All rights reserved. Table of Contents
1.1 This Commercial Lending Policy Manual (the "Policy") establishes the minimum standards, procedures,
and guidelines applicable to all commercial real estate lending activities conducted by Meridian Commercial Bank (the "Bank"). This Policy supersedes all prior versions and amendments, except as specifically noted in subsequent Risk Bulletins issued by the Credit Policy Committee.
1.2 The Policy applies to all new loan originations, renewals, modifications, and extensions of credit
involving commercial real estate collateral with an aggregate exposure exceeding Two Hundred Fifty Thousand Dollars ($250,000).
1.3 Approval authority is vested as follows:
(a) Loans up to $500,000: Branch Lending Officer with VP Credit concurrence (b) Loans $500,001 to $2,000,000: Senior Lending Officer with Credit Committee review (c) Loans $2,000,001 to $5,000,000: Credit Committee approval required (d) Loans exceeding $5,000,000: Board Credit Sub-Committee approval required
1.4 All exceptions to this Policy must follow the Exception Approval Process set forth in Section 9 and
require documentation of the business justification, risk mitigants, and compensating factors.
2.1 All commercial real estate loan applications shall be accompanied by a complete financial package
including, at minimum: (i) three years of audited or compiled financial statements, (ii) current rent roll or tenant schedule, (iii) property operating statements for the most recent twenty-four month period, (iv) personal financial statements of all guarantors, and (v) an independent appraisal report prepared by a state-licensed or certified appraiser.
2.2 The credit analyst shall verify all financial data against original source documents. Self-reported income
figures shall be reconciled against tax returns, bank statements, or independently audited financial statements. Any discrepancies exceeding five percent (5%) must be resolved and documented prior to credit committee presentation.
2.3 For properties with mixed-use designations, the analyst shall determine the proportional allocation of
rentable square footage by use classification (residential, retail, office, industrial) and apply the most restrictive policy parameter applicable to the dominant use classification, unless otherwise specified by current Risk Bulletins.
Section 3: Risk Classification Framework
The Bank employs a four-tier risk classification system for commercial real estate loans. Each tier establishes minimum pricing floors, enhanced monitoring requirements, and escalation procedures. Assignment to a risk tier is based on the calculated loan-to-value ratio at origination.
3.1 Risk Tier Definitions
Risk tier assignment is determined by the loan-to-value ratio calculated in accordance with Section 4.2. The following table summarizes tier parameters: Risk Tier Classification LTV Range Rate Floor Tier 1 Low Risk LTV £ 60.0% 5.90% Tier 2 Moderate Risk 60.1% – 70.0% 6.75% Tier 3 Elevated Risk 70.1% – 80.0% 7.50% Tier 4 High Risk > 80.0% 8.25% The rate floor represents the minimum acceptable interest rate for new originations within each tier. Relationship pricing adjustments may be applied above, but not below, the applicable floor rate.
4.1 General Collateral Standards
All commercial real estate collateral shall be appraised by an independent, state-certified appraiser who holds the MAI designation from the Appraisal Institute or equivalent professional credential. The appraisal must comply with the Uniform Standards of Professional Appraisal Practice (USPAP) and be dated within one hundred eighty (180) days of the anticipated closing date. When two or more independent appraisals are obtained and the spread between appraised values exceeds ten percent (10%), the lower of the two values shall be used for LTV calculation purposes. The spread shall be computed as the absolute difference between the highest and lowest appraised values, divided by the lower value. This requirement ensures conservative collateral valuation in cases where significant professional disagreement exists regarding market value. In cases where the spread is ten percent (10%) or less, the Bank may, at the discretion of the Credit Committee, use the arithmetic mean of the two appraised values or the lower value, whichever the Committee deems most appropriate given market conditions.
4.2 Maximum Loan-to-Value by Property Classification
The following maximum loan-to-value ratios apply to new originations by property classification. The LTV ratio is calculated as the proposed loan amount divided by the appraised value determined in accordance with Section 4.1: Property Classification Maximum LTV Commercial Office 75% Retail 70% Industrial / Warehouse 75% Mixed-Use 70% Multifamily (5+ units) 80% Loans that exceed the maximum LTV for their property classification may be considered only through the Exception Approval Process described in Section 9, and must include compensating factors such as additional collateral, personal guarantees, or cash reserves equal to at least six months of debt service.
5.1 DSCR Thresholds and Loan Decisioning
The debt service coverage ratio ("DSCR") is the primary measure of a borrower's ability to service proposed indebtedness from property operating income. The DSCR is calculated as Net Operating Income divided by Total Fixed Charges (as defined in Section 5.3). Loan applications shall be evaluated against the following DSCR thresholds: DSCR Range Decision Classification Action Required ‡ 1.25x Clean Approval Standard approval process; no additional conditions 1.00x – 1.24x Conditional Approval Requires enhanced monitoring, additional reserves, or credit enhancement < 1.00x Deny Application does not meet minimum debt service requirements
5.2 Income Verification Standards
Net Operating Income for DSCR calculation purposes shall be derived from verifiable, recurring property income sources. The analyst shall exclude all non-recurring items, including but not limited to: insurance claim proceeds, legal settlement recoveries, one-time tenant improvement reimbursements, and gain on sale of assets. Income adjustments must be documented in the credit memorandum with reference to supporting evidence. For owner-occupied properties where facility revenue is imputed from practice or business operations, the analyst shall use market-rate comparables to establish a reasonable imputed rental value, giving appropriate consideration to actual operating expense burden and historical occupancy patterns.
5.3 Definition of Total Fixed Charges
For purposes of calculating the debt service coverage ratio, "Total Fixed Charges" shall include: (a) the borrower's annual loan payment obligation under the proposed facility, (b) annual capital lease obligations, and (c) any other recurring contractual payment obligations with a remaining term exceeding twelve months. The inclusion of capital lease obligations and long-term contractual commitments ensures that the DSCR reflects the borrower's complete recurring payment burden. Short-term obligations with a remaining term of twelve months or less may be excluded at the discretion of the credit analyst, provided such exclusion is documented and justified in the credit memorandum.
6.1 Standard amortization for commercial real estate loans shall be twenty-five (25) years, with a balloon
payment due at the end of the stated loan term. Loan terms shall generally range from five (5) to ten (10) years.
6.2 Interest rates shall be determined by the applicable risk tier rate floor (Section 3.1) plus any relationship
pricing adjustments approved by the Senior Lending Officer. Floating rate structures require a minimum floor rate equal to the applicable tier floor.
6.3 Prepayment provisions shall include a minimum lockout period of twelve (12) months and declining
prepayment premiums as follows: Year 1 — 5%; Year 2 — 4%; Year 3 — 3%; Year 4 — 2%; Year 5 and beyond — 1% or par, whichever is greater.
7.1 The loan file for each commercial real estate transaction shall contain, at minimum, the following
documents: loan application, financial statements, rent roll, operating statements, appraisal report(s), title commitment, survey, environmental assessment (Phase I), insurance certificates, and legal entity documentation.
7.2 All financial statements used for underwriting purposes must be prepared by a licensed Certified Public
Accountant (CPA). Compilation-level financial statements are acceptable for loan requests under $2,000,000; review or audit engagements are required for larger exposures.
7.3 The credit memorandum shall document all underwriting calculations, identify all sources of data, note
any discrepancies between documents, and provide a clear recommendation with supporting rationale. The memorandum must explicitly address: LTV compliance, DSCR adequacy, risk tier assignment, portfolio concentration impact, and any applicable Risk Bulletin modifications.
8.1 Geographic Diversification
The Bank shall maintain geographic diversification in its commercial real estate portfolio. No single metropolitan statistical area (MSA) shall account for more than thirty-five percent (35%) of total commercial real estate outstandings. Concentration approaching this threshold shall be reported quarterly to the Credit Policy Committee.
8.2 Sector Concentration Limits
No single sector classification shall represent more than twenty-five percent (25%) of total portfolio outstanding. Sector classifications include: Commercial Office, Retail, Industrial/Warehouse, Multifamily, Medical Office, Hospitality, and Mixed-Use. Concentration levels shall be evaluated at the time of origination and monitored quarterly. When a proposed origination would cause a sector classification to exceed the twenty-five percent (25%) limit, the loan must be processed through the Exception Approval Process (Section 9), regardless of individual loan merit. The exception request must include a portfolio-level risk assessment and proposed mitigants.
8.3 Single Borrower Exposure
Aggregate exposure to any single borrower and related entities shall not exceed ten percent (10%) of the Bank's total risk-based capital. For purposes of this section, related entities include any entity in which the borrower holds a controlling interest or shares common management.
9.1 Any loan that does not conform to the standards set forth in this Policy may be considered for exception
approval under the following conditions: (a) The exception request includes a written justification from the originating officer; (b) Compensating factors are identified and documented; (c) The exception is approved at one level above normal approval authority; (d) The exception is reported to the Credit Policy Committee within thirty (30) days.
9.2 Exception approvals are not precedent-setting and shall not be cited as justification for future exceptions.
Each exception request shall be evaluated independently based on its specific facts and circumstances.
10.1 All commercial real estate lending activities shall comply with applicable federal and state regulations,
including but not limited to: the Interagency Guidelines for Real Estate Lending Policies, FDIC regulations, OCC guidance on CRE concentrations, and state-specific usury and lending statutes.
10.2 The Chief Risk Officer shall ensure that this Policy is reviewed and updated at least annually, or more
frequently as warranted by changes in regulatory guidance, market conditions, or portfolio performance.
APPENDIX A: Standard Forms and Templates
The following standard forms are maintained by the Commercial Lending Operations department and are available on the Bank's intranet: (cid:127) Form CRE-100: Commercial Loan Application (cid:127) Form CRE-200: Credit Memorandum Template (cid:127) Form CRE-300: Appraisal Review Checklist (cid:127) Form CRE-400: Environmental Due Diligence Checklist (cid:127) Form CRE-500: Portfolio Concentration Monitoring Report (cid:127) Form CRE-600: Exception Approval Request (cid:127) Form CRE-700: Annual Loan Review Template All forms are subject to periodic revision. Lending staff should verify they are using the most current version prior to submission.
APPENDIX B: Risk Bulletins
Risk Bulletin 2025-03 Date Issued: March 01, 2025 Subject: Revised Maximum LTV for Mixed-Use Properties with Significant Commercial Component Issuing Authority: Credit Policy Committee To: All Commercial Lending Officers, Credit Analysts, and Risk Management Personnel Based on the Credit Policy Committee's review of current market conditions, increased vacancy rates in commercial-dominant mixed-use properties, and recent regulatory guidance regarding commercial real estate concentration risk, the following modification to the Commercial Lending Policy is effective immediately: Effective immediately, the maximum loan-to-value ratio for mixed-use properties where commercial use exceeds thirty percent (30%) of total rentable square footage is reduced from seventy percent (70%) to sixty-five percent (65%). This modification applies to all new originations, renewals, and material modifications of existing facilities involving mixed-use properties where commercial tenancy (including office, retail, and other non-residential uses) represents more than thirty percent (30%) of total rentable square footage as determined by the current rent roll or lease schedule. Mixed-use properties where commercial use is thirty percent (30%) or less of total rentable square footage remain subject to the standard seventy percent (70%) maximum LTV as set forth in Policy Section 4.2. This bulletin supplements Policy Section 4.2 and shall remain in effect until superseded by a subsequent Risk Bulletin or formal Policy revision. Questions regarding the application of this bulletin should be directed to the Credit Policy Committee or the Chief Risk Officer. Authorized by: James R. Morrison, SVP, Chief Risk Officer Distribution Date: March 01, 2025
What This Proves
Large language models can produce impressively fluent responses to complex professional scenarios — and still get the answer fundamentally wrong. The gap between fluency and accuracy is where adversarial evaluation operates.
This system demonstrates that with careful engineering — deterministic answer keys, independent verification, professionally realistic documents, and self-scoring golden deliverables — it is possible to construct evaluation tasks where model failure is measurable, reproducible, and traceable to specific reasoning breakdowns.
The human remains in the loop not as a limitation, but as a feature. The judgment required to design these scenarios, verify their correctness, and evaluate model responses is precisely the kind of reasoning that current AI systems have not yet mastered. That is both the problem this work addresses and the proof that it matters.