The short answer
A proof of concept should answer one question: is this worth building, and what will it cost to run? To get a real answer, choose one contained use case with a named owner, write down success thresholds before you start, test on your own data handled safely, measure against an evaluation set of real examples, and keep it to a few weeks.
It should end with results against each threshold, a go or no-go recommendation and a production cost estimate. If it ends with "everyone liked the demo", it has not done its job.
Choose a use case that can lead somewhere
Good first candidates are frequent (small savings add up), measurable today (so you have a baseline), easy for a person to check, backed by data you already have, owned by someone who will decide and run it, and connected to known systems, so there is a path to production.
Answering staff questions from approved policies, extracting fields from supplier invoices, drafting email replies for staff to approve and summarizing case history are typical examples. Leave decisions about individuals, payments and anything that must be right without review for later. Our AI readiness checklist helps you shortlist.
Define success before you start
| Measure | What to set | How to measure it |
|---|---|---|
| Accuracy | The share of test cases that must be right, and which errors are automatic failures | Score outputs against the evaluation set |
| Time saved | Minutes per item compared with today | Time the current process before the proof of concept starts |
| Cost per transaction | The most a question, document or task may cost to process | Actual model and service charges during testing |
| User acceptance | Whether pilot users would choose to use it | Structured feedback and the share of drafts accepted with light edits |
| Safety | No answers from content a user cannot open; resistance to prompt injection | Permission and security tests |
Agree what "go", "no-go" and "go with changes" mean, and have the business owner sign off. Thresholds set after the results are in are not thresholds.
A hypothetical HR policy assistant. The evaluation set holds 120 real staff questions, 20 of which have no answer in the policies. The team agrees to go ahead only if at least 90% of answerable questions are answered correctly with a citation, at least 18 of the 20 unanswerable ones get "I could not find that", no answer draws on a document the user cannot open, and the measured cost per question fits the budget finance approved. Your own thresholds depend on the task and the cost of an error.
Use your own data, safely
Demo data hides the hard cases, so test on your own documents and records, handled carefully:
- Minimize. Use anonymized, synthetic or de-identified data where personal information is not needed for the purpose, including in prompts. Keeping only the personal information you need is a basic principle of the FTC's data security guidance for businesses (FTC).
- Use enterprise services. Choose services whose terms exclude your content from model training, and confirm where processing happens. Microsoft, for example, documents that "Global" deployments may process prompts in any geography where the model is deployed (Microsoft Learn), so deployment type and region are deliberate choices.
- Limit access. Only the project team can reach the data, in an approved environment, with a plan to delete it at the end.
- Or keep it fully private. If the data cannot go to an outside AI provider even for a test, run the proof of concept on an open-weight model on a temporary GPU server in your own cloud account. See private AI and self-hosted models.
- Review privacy early. State privacy laws and sector laws apply to personal information used in AI systems, and protected health information needs a business associate agreement with any AI vendor that handles it (see AI and HIPAA). This is general information, not legal advice.
Build an evaluation set
The evaluation set is the most valuable thing a proof of concept produces, because you reuse it every time the model, prompts or sources change.
- Collect real examples: questions staff actually ask, documents you actually receive.
- Have subject-matter experts write the expected answer or correct fields for each.
- Include awkward cases: questions with no answer, conflicting documents, poor scans, off-topic requests, attempts to reach restricted content and instructions hidden inside a document.
- Hold some examples back for the final test, so you are not just tuning to questions you have already seen.
- Score consistently, with people, automated evaluators or both. Microsoft's guidance describes evaluating with your own datasets before deployment, then monitoring quality in production (Microsoft Learn).
For assistants that answer from documents, our guide to retrieval-augmented generation explains what is being tested.
Keep it small and time-boxed
A few weeks, one use case, a small group of users and weekly results. Do not build the full interface or connect every source. Do sketch the production integration and measure running costs, because the next decision depends on them. Microsoft's cost guidance recommends the same habit: run a small, representative test workload and compare actual charges with your estimate (Microsoft Learn). If results are clearly short of the thresholds halfway through, stop and decide whether to change approach or drop it.
What you should receive at the end
- Results against each threshold, with examples of failures.
- A go, no-go or go-with-changes recommendation, with reasons.
- A production cost estimate in US dollars (USD), with one-time build and ongoing usage, hosting, licenses and support shown separately (see what an AI project costs).
- Risks, plus the controls and human review points production would need.
- An integration outline: systems, access and owner.
- A list of what was not tested.
Rather talk it through? If you have an AI idea and want evidence before committing a budget, we can scope a time-boxed proof of concept on your own data, with thresholds agreed up front. Talk to a Promatics specialist
Why many pilots never reach production
| Cause | How to avoid it |
|---|---|
| No owner | Name a business owner who decides and will run it |
| Demo data | Test on your own data, including edge cases |
| No integration plan | Identify systems, APIs and permissions during the proof of concept |
| No budget for running costs | Measure cost per transaction and present it with the build estimate |
| Success never defined | Write thresholds down before starting |
How Promatics runs a proof of concept
- Agree it in writing: use case, owner, thresholds, data handling, timebox and an estimate in US dollars (USD).
- Prepare safely: an approved data sample, de-identified where possible, in an enterprise environment, and an evaluation set built with your experts.
- Build and test: a working prototype, weekly results and tuning, then a final run on held-back examples plus permission and prompt injection tests.
- Report: results against thresholds, our recommendation, a production design and cost estimate, and the risks.
- Your decision: if the answer is no, we say so. If it is yes, we build it through generative AI development or AI agents, and support it after launch.
When to bring in help
You can run a simple trial yourself when testing Copilot or a built-in AI feature on non-sensitive content with a few users. Bring in help when the proof of concept needs personal or confidential data, when production would connect to your ERP or CRM, when leadership needs evidence it can rely on, or when nobody in-house has built an evaluation set before.
Plan your proof of concept
- One use case and a named business owner.
- Today's baseline measured: volume, time per item, error rate.
- Thresholds for accuracy, time saved, cost per transaction and user acceptance, signed off.
- Automatic-failure conditions listed.
- Data sample approved and de-identified where possible.
- Enterprise service, region and retention confirmed.
- Evaluation set with edge cases and a held-back portion.
- Timebox and weekly check-ins agreed.
- Production systems and integration path identified.
- Decision meeting booked for the end.
Sources and further reading
Product capabilities and guidance change. These are the primary sources this article relies on, checked on the review date above.
- Data security guidance for business, Federal Trade Commission
- Data, privacy, and security for Foundry Models sold by Azure in Microsoft Foundry, Microsoft Learn
- Observability in generative AI, Microsoft Learn
- Plan and manage costs for Microsoft Foundry, Microsoft Learn
This article is general information, not legal, accounting or security advice for your specific situation. Examples are hypothetical unless stated otherwise.