Your privacy obligations when using AI with customer data

Every time someone on your team pastes a customer email into an AI tool, your business may be making a disclosure of personal information under Australian privacy law. If your business is an "APP entity" under the Privacy Act 1988 (Cth) (the Act), that single action can trigger a chain of obligations, and if something goes wrong it can expose you to penalties of up to $50 million, a regulator's investigation, and a damages claim from the individual involved.

The Australian Privacy Principles (APPs) in Schedule 1 of the Act govern how APP entities collect, use, disclose and secure personal information. Using AI, whether a chatbot answering customer queries, a document summariser, a marketing tool, or a model trained on your own data, intersects with the APPs in four ways that matter most: use and disclosure (APP 6), cross-border disclosure (APP 8), security (APP 11) and transparency (APP 1). This article sets out who the obligations apply to, what each duty requires, what happens if you breach them, and the practical steps to take before you use AI with customer data.

Are you an APP entity?

An APP entity is an agency or organisation covered by the Privacy Act. For a business, the key question is whether you are a "small business operator", because small business operators are generally exempt from the APPs.

The test is annual turnover. Under s 6D of the Privacy Act 1988 (Cth), a business is a small business if its annual turnover for the previous financial year was $3 million or less. If your turnover was more than $3 million, you are not a small business operator and the APPs apply to you in full. If your business is new, the test looks at the current year's turnover instead.

Your business is also an APP entity even with turnover of $3 million or less if any of the following applies:

Health services: you provide a health service and hold health information other than in an employee record
Trading in personal information: you buy or sell personal information
Commonwealth contracts: you are contracted to provide services under a Commonwealth contract

A few other categories exist as well, such as credit reporting bodies and operators connected to certain regulated activities. Two practical notes. First, turnover is measured on the previous financial year, so a business that tips over $3 million this year is caught from the following year, not the day the money lands. Second, the small business exemption is the subject of ongoing reform proposals and may not survive in its current form, so the safer assumption for any business using AI with customer data is that the APPs apply to you.

What counts as personal information

Personal information is information or an opinion about an identified individual, or an individual who is reasonably identifiable. The definition is broad. It plainly covers the obvious items, including names, email addresses, phone numbers, home addresses, dates of birth and photographs, but it also covers anything from which a person can reasonably be worked out: order histories, support tickets, chat transcripts, voice recordings, location data and device identifiers.

For AI purposes, the breadth matters because of what flows into your tools. A customer query pasted into a chatbot contains personal information. A document uploaded for summarisation often contains it. An email thread used to draft a reply contains it. The information does not stop being personal information because it is sitting in an AI prompt rather than a customer database.

Two categories deserve special attention. Sensitive information, including health information and biometric information used for identification, attracts stricter rules under the APPs: it can generally only be collected with consent, and any secondary use must be directly related to the primary purpose. And while properly de-identified information falls outside the Act, genuine de-identification is harder than it sounds: it needs to be done properly, and documented, or the information will not actually be outside the Act's reach.

Only use personal information for the purpose it was collected (APP 6)

APP 6 is the principle most often breached by AI use. It says that if you hold personal information collected for a particular purpose (the primary purpose), you must not use or disclose it for another purpose (a secondary purpose) unless the individual consented, or an exception applies.

The main exception is where the individual would reasonably expect the use or disclosure, and the secondary purpose is related to the primary purpose or, for sensitive information, directly related to it. Other exceptions cover uses required or authorised by law.

Applied to AI:

Using a customer's details to answer their support request through an AI-assisted tool is consistent with the primary purpose of providing support.
Using accumulated customer data to train a model, build a profile, or run market analysis is a secondary purpose. Unless the customer would reasonably expect it (unlikely for model training) or has consented, it is a breach of APP 6.
Each disclosure to an AI vendor is itself a "disclosure" under the Act, and needs its own basis.

The practical takeaway: decide, before you use AI, whether each use is primary-purpose or needs consent. If it is a secondary purpose and you cannot obtain consent, you cannot do it.

Know where your data goes before it leaves Australia (APP 8)

Most commercial AI platforms are hosted overseas, and that is where APP 8 bites. Under APP 8, before you disclose personal information to an overseas recipient, you must take such steps as are reasonable in the circumstances to ensure the recipient does not breach the APPs in relation to the information.

The catch is in s 16C of the Act: if the overseas recipient mishandles the information, the recipient's act is treated as your act, and a breach of the APPs by the recipient is a breach by you. "The AI vendor handles it" is not a defence. You remain accountable for what the vendor does with the information.

You do not need to comply with APP 8 if you reasonably believe the recipient is subject to a law or binding scheme that protects the information in a way that is at least substantially similar to the APPs, if the individual consents to the disclosure with notice of the risks, or if the disclosure is required or authorised by law. In practice, for most AI tools:

You will need to assess the vendor's jurisdiction, its data protection framework and any contractual protections before you disclose anything.
If the vendor stores or processes data in several countries, assess each location.
A consent-based exception is hard to rely on for data you have already collected, because consent must be voluntary and informed, which is difficult to retrofit after collection.

Secure the personal information you hold (APP 11)

APP 11 requires you to take such steps as are reasonable in the circumstances to protect personal information from misuse, interference, loss, and unauthorised access, modification or disclosure. It also requires you to destroy or de-identify information you no longer need, and it makes clear these steps include technical and organisational measures.

Applied to AI, the security duty raises questions you should be able to answer:

Does the AI tool retain the data you input? Many tools use prompts and uploaded documents for training or keep them for extended periods. If customer data sits in a vendor's training set, you have not secured it.
Can you control deletion? A tool with no deletion or export function is hard to reconcile with the duty to destroy or de-identify information you no longer need.
What happens to data when a staff member or customer uses the tool? A single careless prompt can put personal information where you cannot retrieve it.
Is access to the tool controlled, and are staff trained on what can and cannot be input?

The safest default for customer data is: de-identify before you input, or do not input at all.

Say what you are doing (APP 1)

APP 1 requires you to manage personal information openly and transparently. That means having a clearly expressed and up-to-date privacy policy, and practices and procedures that ensure you comply with the APPs and can deal with complaints.

If you use AI with personal information, your policy and your collection notices should say so: what the AI is used for, whether data goes overseas, whether it is used for training, and how individuals can complain or correct information. A policy that says nothing about AI while your team pastes customer data into tools is not up to date in any practical sense, and if a complaint or an investigation follows, the gap between your written policy and your actual practice is exactly what gets scrutinised.

What happens if you get it wrong

The regulator is the Office of the Australian Information Commissioner (OAIC). It can investigate complaints and conduct its own investigations, make determinations, accept enforceable undertakings, and take civil penalty proceedings in court.

The headline figure is the penalty for a serious interference with privacy under s 13G of the Act. For a body corporate, the maximum civil penalty is the greatest of $50 million, three times the benefit obtained from the conduct, or 30% of adjusted turnover. Interferences that are not serious carry penalties capped at 2,000 penalty units. In deciding whether an interference is serious, a court can weigh the sensitivity of the information, the number of individuals affected, whether children or vulnerable people were involved, and whether you had failed to implement proper privacy systems, a factor that lands squarely on a business that adopted AI without controls.

Separately, the Act now includes a statutory tort for serious invasion of privacy, which gives individuals a direct right to sue for damages. An AI mishap that exposes intimate or sensitive information is the kind of case that tort was designed for.

Finally, the notifiable data breach scheme applies. If there are reasonable grounds to believe an eligible data breach has occurred, for example customer data compromised through an AI tool, you must notify affected individuals and the OAIC. If you only suspect a breach, you must assess it expeditiously, and the assessment must be completed within 30 days. The OAIC can also direct you to notify. Getting the assessment and notification steps wrong adds a second regulatory problem on top of the breach itself.

A compliance checklist for using AI with personal information

Run through these steps before any AI tool touches customer data:

Map the tools: list every AI tool your business uses and what personal information flows into each.
Classify the use: for each flow, decide whether it is a primary purpose, a secondary purpose with consent, or covered by an exception.
De-identify by default: remove names, contact details and other identifiers before inputting where the task allows.
Check the vendor: where is the data stored, is it used for training, can you delete it, and what contractual protections apply? Document the assessment.
Update your privacy documents: make sure your privacy policy and collection notices state how you use AI.
Control access: restrict which staff can use AI tools with customer data, and train them on what can and cannot be input.
Plan for the breach: know who assesses a suspected eligible data breach and how you would notify within the statutory timeframe.

When a privacy lawyer is worth the cost

Much of this you can do yourself, but a privacy lawyer earns their fee in four situations:

You want to train a model on data you already hold: whether that is a permitted secondary use, and how to structure consent or de-identification, is a judgement call with real penalty exposure.
You are negotiating an AI vendor agreement: the contract is where APP 8 protection lives: data location, sub-processors, retention, deletion, security standards and liability.
You have had an incident: an eligible data breach assessment runs to statutory deadlines, and the quality of the assessment can determine whether the OAIC becomes involved.
You want to rely on an exception: whether an overseas recipient is protected in a way that is substantially similar to the APPs is a factual and legal assessment that is easy to get wrong.

A practitioner will review your data flows, draft or update your privacy documents, and give you a documented position on each AI use so that you are not defending an informal arrangement if something goes wrong.

Everything in this article condenses to one question to ask before anyone in your business inputs customer data into an AI tool: is this use consistent with the purpose for which you collected the information, or do you have consent for something else? If the answer is no, the data does not go in: not into a chatbot, not into a summariser, not into a training set. De-identify it first, obtain consent, or find another way to do the task.

That single discipline matters more than any policy document, because most AI privacy breaches do not start with a decision to misuse data. They start with convenience: a support agent pasting a customer's details into a tool to save time. The APPs do not care that it was convenient. If the use was a secondary purpose without consent, the disclosure is a breach, and you, as the APP entity, are accountable for what the tool does with the information from there.