Start with one workflow
Choose a specific task and user group before scoring an entire organisation. Record how the work happens today, where time is spent and which errors cause harm. Readiness is use-case-specific: a team may be ready for internally reviewed drafting while being unready to automate a customer-facing decision.
Use the questions below in a working session with the business owner, data owner, technology team and someone who performs the task. For each question, record the evidence, the owner and the next action.
1. Is the intended outcome clear?
Identify the user, the task and an observable result. “We need an AI assistant” is not a problem definition. Reducing the time needed to find an approved procedure, while preserving answer accuracy, can be tested. Record the baseline and measurement method before the pilot.
2. Is the data suitable and permitted?
Identify the required information, its source, who can authorise access and how it is updated. Inspect a representative sample for gaps, conflicts and language differences. Do not put sensitive information into an external service before agreeing appropriate access and handling.
3. How will you evaluate quality and failure?
Create a representative set of tasks, including difficult cases and Arabic and English examples where relevant. Define an acceptable answer and how uncertainty should be handled. Identify situations requiring human review, refusal or escalation. Keep an evaluation set separate from the examples used to develop the solution.
4. Who will operate it?
Name the service owner, support arrangements, integration dependencies, access controls and fallback process. Monitoring should examine quality, cost and latency together. A system that works in a demonstration may create an operating burden the team cannot sustain.
5. Can users work with the result?
Explain where the output appears and how an employee reviews and corrects it. Test the time required for review, not just generation. Gather feedback from actual users and define training and decision responsibilities before expansion.
6. Does the value justify the full cost?
Compare the expected improvement with integration, model usage, human review, support and change-management costs. Record the assumed usage volume and a cost range. Include process simplification or conventional automation as alternatives if they can achieve the same outcome.
Make and record a decision
- Pilot: the owner, data, evaluation criteria and fallback are clear enough for a controlled test.
- Resolve a blocker first: a specific gap, such as missing data permission or a baseline, prevents a useful test. Assign an owner and evidence required to close it.
- Defer: the benefit is weak or the consequences of error cannot be adequately contained. Record what would change the decision.
A hypothetical worked example
Consider a team searching internal procedures in Arabic and English. A pilot could be limited to approved documents, require a source for each answer and leave final approval with the employee. Compare answer accuracy and task time with the current search process, including outdated documents and questions the source material cannot answer. This is an illustration, not a client result.
How to use this guide
This is a qualitative framework prepared by Barry Advisory. It is not a statistically validated maturity benchmark or a certification. It complements the organisation’s technical, legal and security assessments.
To turn the assessment into an engagement scope, explore AI strategy advisory and founder Fred Barry’s experience.