Scorecard dimensions
- Data protection and residency
- Training use of customer content
- Security certifications and practices
- Transparency of limitations
- Human oversight features
- Accessibility and language support
- Commercial fairness and lock-in risk
- Alignment with your prohibited-use list
Process
RFI → shortlist → security/privacy review → pilot → scorecard → decision memo. Keep the memo. Future staff need to know why a vendor was refused. Pair with the buying path in AI procurement for dioceses.
Red flags
Vague subprocessors, pressure to skip legal review, “we train on your data” without clear opt-out, no export path, or dismissiveness about special-category data in church contexts. A logo of a dove does not substitute for a DPA.
Questions to bring to every call
Where is inference executed? Where are prompts stored? Who can read them? Is customer content used for training? What is deleted on contract end? Can we force EU-only processing? How are API keys rotated? What happens during a subprocessor change? Write answers into the procurement file the same day—memory is not a control.
Scoring without self-deception
Weight privacy and exit rights at least as heavily as features. A tool that dazzles in demo but traps your data is not a bargain. Require written answers; verbal assurances evaporate at renewal. If two vendors score close, pick the one whose support model and documentation you can actually understand under stress.
Share the scorecard with leadership before the pilot ends—not after the contract is emotionally already signed.