Evaluating AI Tools for Business Use — A Framework for BAs — AI for Business Analysts

What you'll learn

Apply the six-dimension AI tool assessment framework — capability, reliability, explainability, integration, vendor risk, and ethical considerations — to a real tool evaluation
Design evaluation test cases from requirements rather than from vendor-suggested scenarios, including in-scope, edge-case, out-of-scope, and verifiable-answer categories
Write AI system requirements that go beyond functional requirements to include accuracy thresholds, explainability requirements, data governance, and override protocols
Identify the questions that standard procurement processes miss when evaluating AI vendors and explain why they matter for long-term performance

Organizations are acquiring AI tools at a rate that outpaces their ability to evaluate them properly. Procurement processes designed for standard software are poorly suited to AI systems, which differ from conventional software in ways that matter for reliability, explainability, and risk. Business analysts — with their structured analytical thinking, their requirements expertise, and their ability to interrogate claims systematically — are among the most valuable participants in AI tool evaluations. The BA who can evaluate AI solutions rigorously is not just contributing to a procurement decision; they are protecting the organization from a class of technology risk that most procurement frameworks were not built to assess.

The BA's Role in AI Tool Evaluations

AI tool evaluations in most organizations are led by technology teams and informed by vendor demonstrations. Both of these inputs are valuable and both have structural limitations. Technology teams assess integration, security, and architecture. Vendor demonstrations show the tool performing well on prepared examples. Neither reliably answers the question that matters most for the business: will this tool produce reliable, useful, and governable outputs in our specific operational context?

Business analysts contribute what technology teams and vendor demonstrations cannot: requirements-led assessment. A BA approaching an AI tool evaluation asks: what does this tool need to do, for whom, under what conditions, and how will we know when it is not doing it well enough? This is the framing that converts a vendor demonstration into an informed procurement decision.

The BA's role in an AI evaluation typically spans three activities: translating business requirements into evaluation criteria, designing the test cases that probe the tool against those criteria, and documenting the gaps between vendor claims and observed performance in a format that informs the selection decision.

A Structured Assessment Framework for AI Tools

A rigorous AI tool assessment evaluates six dimensions. No tool excels across all six — the framework helps the evaluation team understand the trade-offs clearly rather than being influenced primarily by the tool that performs best on the dimensions the vendor chose to demonstrate.

Capability assesses whether the tool can perform the functions the business requires at the required level of accuracy and reliability. Test the tool on your actual data — your documents, your formats, your queries — not the vendor's demonstration data. The difference in performance between the two is diagnostic of how well-generalised the tool's capability is.

Reliability assesses how the tool behaves when inputs are ambiguous, incomplete, or outside its optimal range. Every AI tool performs well on clean, well-structured inputs within its training distribution. The reliability question is what it does when inputs are messy, edge-case, or unexpected — which is exactly what operational reality will produce over time.

Explainability assesses whether the tool can communicate the basis for its outputs in a way that supports human review and accountability. For regulated industries, AI outputs that cannot be explained and audited are frequently non-compliant. For any organization, AI outputs that cannot be reviewed create accountability gaps when those outputs are wrong.

Integration assesses the technical and operational feasibility of connecting the tool to the systems and workflows it needs to be part of. A tool that requires significant custom integration work, that does not connect to your existing data sources, or that requires workflow restructuring to use effectively has a higher total implementation cost than the license fee suggests.

Vendor risk assesses the commercial and operational risk associated with the vendor relationship. A small AI vendor with an impressive tool creates a dependency risk. What happens to your implementation if the vendor is acquired, pivots, or fails? What are the data portability provisions in the contract? What is the exit path if the tool does not perform as expected in production?

Ethical considerations assesses whether the tool's design and outputs introduce bias, fairness, privacy, or transparency risks relevant to your use case. An AI tool used in any people-facing context — HR, customer service, credit assessment — requires explicit assessment of the bias and fairness risks embedded in its training data and model design.

Tip

Design your AI tool evaluation test cases from your requirements, not from the vendor's suggested evaluation scenarios. Vendors design demonstration and evaluation scenarios that show their tool well. Your test cases should specifically probe the requirements that are most important and the failure modes that are most consequential. Include at minimum: a set of representative in-scope cases, a set of edge cases at the boundary of the stated scope, a set of out-of-scope inputs to understand how the tool handles queries it should decline, and a set of cases where the correct answer is known and verifiable. This structure surfaces how the tool performs across its full operating range rather than only its optimum range.

Knowledge check

An organization is evaluating an AI document classification tool for use in their legal team. The vendor's demonstration uses well-formatted contracts from major commercial deals. The BA designs an additional test set using the organization's own historical documents, which include older scanned PDFs and internally drafted letters. The tool's accuracy drops significantly on the internal test set. What does this finding most directly inform?

Select one answer.

Writing Requirements for AI Systems

AI systems require requirements that go beyond what conventional software requirements capture. In addition to the functional requirements (what the tool must do), AI requirements must address accuracy and confidence thresholds (how accurate must the output be, and how should the tool communicate its confidence level?), explainability requirements (must the tool be able to justify its outputs?), data governance requirements (how is input data handled, stored, and retained?), and override and escalation requirements (under what conditions must a human be able to review and override the AI output?).

These requirements are not typically included in standard requirements frameworks and must be explicitly added when the system being specified is an AI tool.

Questions Procurement Processes Miss

Standard procurement processes assess price, security, support, and integration. The questions that procurement processes most frequently fail to ask of AI vendors are the ones that matter most for long-term performance and risk management.

What is the model trained on, and how recent is the training data? A tool trained on data that does not include your industry, your document types, or your regulatory environment will underperform in ways that become apparent only after deployment.

How does the model handle inputs outside its training distribution? Ask for failure examples, not just success examples. The vendor's willingness to provide honest failure examples is itself a signal of their reliability as a partner.

What is the retraining frequency and the process for reporting performance degradation? AI models drift over time as the world changes relative to their training data. What is the vendor's commitment to keeping the model current?

Who is liable if the AI output causes a business harm? This question is rarely asked and the answer is revealing about how seriously the vendor has thought about the risk transfer implications of their product.

Preventing a misinformed AI procurement decision in a legal team

Business Analyst, professional services firm

Context

A professional services firm's legal operations team was evaluating two AI contract review tools. The technology team had run vendor demonstrations for both and preferred the tool that showed higher accuracy on the example contracts provided. The BA was asked to support the evaluation by developing the requirements-led assessment. The firm's contract library included documents from the 1990s onwards, a mix of negotiated and standard-form agreements, and a significant proportion of non-standard indemnity clauses specific to their practice areas.

Action

The BA designed a test set based on requirements rather than vendor scenarios: representative in-scope cases from the firm's own contract library, edge cases involving the non-standard indemnity clauses, out-of-scope inputs to test how each tool handled documents it should decline, and cases with known correct answers. Both tools were tested on the firm's actual document types. The preferred tool's accuracy dropped materially on the firm's older scanned documents and non-standard clauses. The second tool performed more consistently across the full test set. The BA also asked both vendors the procurement-missed questions: training data currency, failure mode examples, retraining frequency, and liability provisions.

Outcome

The selection decision reversed from the vendor-demonstration preference to the second tool, based on consistent real-world performance and clearer liability provisions in the contract. The legal team lead noted that without the BA-led requirements test, the firm would have purchased a tool that would have underperformed on a substantial portion of their actual work.

Warning

AI tool demonstrations are optimized to show the tool at its best on examples the vendor controls. Do not make a selection decision based primarily on a vendor demonstration without supplementing it with independent testing on your own data, reference conversations with existing customers in comparable use cases, and a structured assessment against your requirements framework. The gap between demonstrated performance and production performance is often significant — and is most significant for organizations whose use case, data, or operational environment differs from the vendor's primary customer base.

Quick check

During an AI tool evaluation, the vendor demonstrates impressive performance on the sample documents they have provided. The BA designs a parallel test using the organization's own historical documents and observes significantly lower accuracy. What does this finding most likely indicate?

Select one answer.

Exercise

~30 min

Your Task

Choose an AI tool your organization uses or is evaluating. Design a set of test cases based on your own requirements: at least two representative in-scope cases, two edge cases at the boundary of the stated scope, two out-of-scope inputs to see how the tool handles queries it should decline, and two cases where you already know the correct answer and can verify the output. Run all eight test cases and score the tool's performance against the six-dimension framework from this lesson. The result is a structured evaluation finding that reflects your actual operational context rather than the vendor's demonstration conditions.

Success looks like

Your eight test cases are drawn from your own operational context — not from the vendor's suggested evaluation scenarios or documentation examples
Your scoring across the six dimensions identifies at least one trade-off or gap that would not have been visible from the vendor demonstration alone
Your evaluation finding is specific enough to inform a procurement recommendation — it identifies what the tool does well, where it underperforms, and what that means for the intended use case

Watch out for

Using the vendor's sample data or suggested test cases rather than your own — this defeats the purpose of the exercise and will not surface the performance gap between demonstration and production conditions
Scoring the tool on capability alone and treating high capability as sufficient — the framework requires assessment across all six dimensions, including the vendor risk and ethical considerations that are easy to overlook when a tool performs well technically

Hint

Start with the failure modes that would be most consequential for your use case — the out-of-scope and edge-case tests are often more diagnostic than the representative in-scope tests, because they reveal how the tool behaves when reality departs from its optimum conditions.

Key takeaways

BAs contribute to AI evaluations what technology teams and vendor demonstrations cannot: requirements-led assessment that tests whether the tool produces reliable, useful, and governable outputs in the organization's specific operational context.
Evaluate AI tools across six dimensions — capability, reliability, explainability, integration, vendor risk, and ethical considerations — because trade-offs across these dimensions are how informed selection decisions are made.
AI system requirements must go beyond functional requirements to include accuracy thresholds, explainability requirements, data governance requirements, and override and escalation protocols — elements that standard requirements frameworks do not capture.
Test AI tools on your own data in your own operational context — the performance gap between vendor demonstration examples and real-world inputs is diagnostic of capability generalisation and frequently significant.
The questions procurement processes most often miss — training data currency, failure mode behavior, retraining frequency, and liability for AI-caused harm — are among the most important for long-term performance and risk management.