AI’s Own Builders Are Sounding the Alarm. Clubs Should Listen.

Some of the people building the world’s most powerful AI systems are warning that the industry is moving dangerously fast. For clubs being offered the next intelligent assistant or automated management feature, those warnings deserve attention.

In September, The Times reported that researcher Jacob Coxon had resigned from Anthropic after previously working at OpenAI. He accused the companies of “gambling with our lives” in their pursuit of increasingly powerful AI. His warning concerned the direction of advanced AI development and the prospect of systems exceeding human control. The Times .

Anthropic co-founder Dario Amodei has also called for a slower pace. In a September essay, he wrote: “We must slow the pace at which we improve the capabilities of AI models.” He argues that safeguards need time to catch up with capabilities and calls for independent evaluators inside frontier AI companies. We Must Pace the Frontier .

For club boards, the debate raises an immediate question: if the people developing this technology are calling for greater scrutiny, what scrutiny should a software company complete before connecting it to your members’ information?

A club can face serious consequences long before any prediction about superintelligence is settled. A private record disclosed to the wrong person, an incorrect account action, or a misleading message sent to hundreds of households is enough.

Consider what your club may already hold. Alongside names and email addresses, there may be children’s birth dates, household relationships, emergency contacts, accommodation requests, incident reports, staff records, and payment histories. Check-in records can reveal attendance patterns. Notes added to help staff handle a difficult situation may describe circumstances a family would never want circulated.

To someone reviewing a software demonstration, these are fields in a database. To a member, they may be the details of a divorce, a financial difficulty, or a child’s needs. A feature that makes those records easier to search also needs boundaries around who can retrieve them.

The security organization OWASP identifies sensitive information disclosure as a risk in applications using large language models. A poorly protected application can expose private information through its responses. Its guidance calls for limiting access to necessary data and enforcing access controls. OWASP: Sensitive Information Disclosure .

Imagine a seasonal employee asking an assistant about a household and receiving confidential notes the employee cannot open through the normal member screen. If that happens, the feature has crossed a boundary the ordinary software was meant to enforce. Permission limits must follow the information into the AI feature.

Clubs also need to understand whether information leaves their management platform. A familiar logo on the screen does not identify every company involved in processing a request. If a feature sends member information to an external AI provider, the club needs to know which information, for what purpose, and under what terms.

A promise that information will not be used to train a model answers one question. It does not, by itself, explain retention, provider access, logs, deletion, or the other services involved. The UK’s National Cyber Security Centre advises organizations considering sensitive AI uses to examine how providers manage that information and who may access it. NCSC guidance .

Imagine a manager trying to save an hour on renewal notices. They upload a full household export to an unapproved chatbot, unaware that it includes private notes and children’s details. The immediate concern is the unnecessary disclosure itself. Whether that service uses the information for training is a separate question. The same scrutiny belongs inside software products that send information on the manager’s behalf.

The stakes rise again when an assistant can take action. Drafting a renewal reminder and sending it to the entire membership require different authority. Summarizing an account and changing its balance require different authority. A tool introduced to answer questions should not quietly receive the power to modify records, issue credits, or change access privileges.

OWASP describes excessive agency as a risk arising when AI applications receive too much functionality, permission, or autonomy. It recommends narrow permissions and approval controls for consequential actions. Applied to clubs, that means a vendor should be able to demonstrate exactly what an assistant can do and which actions require an authorized person to approve them. OWASP: Excessive Agency .

There is also the question of whose instructions the assistant follows. Researchers call one class of attack prompt injection: content in a document, webpage, or message can contain instructions that steer a vulnerable AI application away from its intended task. The consequences depend on the information and tools it can reach. OWASP: Prompt Injection .

For a club, that could mean a malicious instruction arriving inside an ordinary-looking document. Before connecting an assistant to member records, a board should ask whether the vendor has tested that possibility. If the assistant follows the instruction, what prevents it from reaching or sending information it should never disclose?

An attacker is not required for an AI feature to cause trouble. NIST identifies confidently incorrect output and excessive human reliance on AI as distinct risks. A plausible response can persuade someone to act before checking the underlying record. NIST Generative AI Profile .

At a busy front desk, imagine an assistant incorrectly stating that a waiver is complete. Or a generated account summary that leaves out a payment arrangement. Or a message that confidently promises an exception the club never approved. The consequences would be personal: a family embarrassed, a member misinformed, or a staff member acting on a false account of the facts.

This is where clubs should be wary of a provider’s rush to add AI. The warning sign is a feature arriving before the provider can explain its data access, demonstrate its permission limits, show how errors are handled, or identify the person responsible for the release. A launch announcement cannot answer those questions. Neither can the reputation of the underlying AI model.

A board should be willing to pause an adoption when the answers are vague. Ask the vendor to demonstrate the feature using a restricted staff account. Ask what information reaches outside providers. Ask who approves changes affecting members. Ask what happens when the model returns a wrong answer or becomes unavailable. Ask whether the club can disable the AI feature and continue its ordinary work.

Members may never see the sales presentation or know which model sits behind their club’s software. They will still experience the consequences of the decisions made on their behalf. If private information is exposed, explaining that the feature was new or the provider was moving quickly will offer little comfort.

Before the next AI feature is switched on, a club should be able to look its members in the eye and explain what has been connected, what has been shared, and why the risk was acceptable.

If it cannot, the feature is not ready for their trust.