Privacy and secrets in a research pipeline

An agent built a pipeline that pulls survey responses, sends the open-ended answers to a model for coding, and writes an export to a bucket so the dashboard can read it. Every step of that sentence touches other people’s personal data. This topic is about the engineering that makes handling it responsibly possible: knowing where the data is, keeping the route to it narrow, and keeping credentials out of your code.

To be clear about scope: this is an engineering primer, not legal advice. Lawful basis, retention periods, contracts with suppliers, international transfers and whether a specific incident is reportable are decisions for your organisation’s data protection lead or a qualified adviser. What follows is how to build systems that let them say yes.

What counts as personal data

Personal data is information relating to someone who can be identified, directly or indirectly, from it — on its own or combined with other data available to you or anyone else. That “indirectly” does most of the work. A table with the name column deleted is not anonymous if respondent id plus full postcode plus employer plus job title plus age singles out one person, and in business-to-business research it very often does.

The blind spot in research pipelines is free text. Respondents put their own name, their employer, their manager’s name and their medical history into a verbatim answer, whatever the question asked. Any system that stores, indexes, logs or forwards verbatims is handling personal data in an uncontrolled shape.

Special category data is a narrower set the UK GDPR treats as higher risk: race or ethnicity, political opinions, religious beliefs, trade union membership, genetics, biometrics used for identification, health, sex life and sexual orientation. Standard survey demographics reach into this more often than people assume. In engineering terms it means tighter access, shorter retention, and a stronger reason before it leaves your systems.

The UK GDPR ideas that change your code

Six of them turn into concrete engineering work.

Individual rights are the sharpest test of a pipeline’s design. A subject access request means someone asks for everything you hold about them. If your answer involves grepping a blob container, an SQLite database, three KV namespaces and a month of logs, the architecture has a problem, not the request.

# The function every pipeline holding personal data should have, and most don't.
async def locate_subject(respondent_id: str) -> dict[str, list[str]]:
    """Every place one person's data lives. Retention and erasure both need this."""
    return {
        "responses":  await db_rows("responses", respondent_id),
        "verbatims":  await db_rows("verbatims", respondent_id),
        "exports":    await bucket_objects_containing(respondent_id),  # the forgotten copy
        "llm_cache":  await kv_keys(f"coded:{respondent_id}:*"),       # and this one
    }

Minimisation is a code decision too: don’t request fields you never analyse, store a postcode district instead of a full postcode, truncate IP addresses in logs, and resist persisting the whole supplier payload “in case we need it”. Every field you keep is a field you must secure, retain, delete and disclose.

# Before: agent-written model, stores everything the supplier sent
class Respondent(BaseModel):
    email: str
    full_name: str
    ip_address: str
    postcode: str
    raw_payload: dict          # the entire supplier row, indefinitely

# After: the fields analysis actually uses
class Respondent(BaseModel):
    respondent_id: str         # supplier's id, or an HMAC of their email
    postcode_district: str     # "SW1A", not "SW1A 1AA"
    age_band: str              # "35-44", not a date of birth
    consented_at: datetime     # you will be asked to evidence this

Pseudonymisation is not anonymisation

Pseudonymisation replaces direct identifiers with a reference while a route back still exists — a key table, the hash input, or the remaining detail. It genuinely reduces risk, and the ICO encourages it, but the data stays personal data and stays in scope.

Anonymisation means re-identification is no longer reasonably likely by anyone. That is a property of the whole dataset, not of one column, and it usually requires coarsening the combinations that single people out: banding ages, truncating postcodes, suppressing small cells, removing employer names from verbatims.

The mistake to look for:

# Before: called "anonymised" in the commit message. It is not.
respondent_hash = hashlib.sha256(email.encode()).hexdigest()

Email addresses are guessable, so anyone with a candidate list can hash them all and match. A plain hash is a pseudonym. If you need a stable id that resists that, use a keyed hash with a secret pepper held somewhere separate from the data:

import hmac, hashlib

def respondent_id(email: str, pepper: bytes) -> str:
    # pepper comes from Key Vault, never from the same database as the rows
    return hmac.new(pepper, email.strip().lower().encode(), hashlib.sha256).hexdigest()

Stable across waves, useless to someone who steals only the data, and still pseudonymous — treat it as personal data.

Sending data to third parties

A vendor that processes your data on your instructions is a processor, and that needs a contract with data-protection terms before the first request. Before an LLM step goes live, five questions want written answers: is there a contract naming sub-processors; is training on your data off; where is it processed; how long is it retained; and is this use consistent with what respondents were told.

Then the engineering part, which is not sending identifiers you don’t need to send:

# Before: the whole verbatim, names and all, to a third-party API
payload = {"text": row["verbatim"], "respondent_id": row["respondent_id"]}

# After: redact first, and never send your own identifier to a vendor
import re

EMAIL = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+")
PHONE = re.compile(r"\b(?:0|\+44)\d[\d ]{7,12}\b")

def redact(text: str) -> str:
    text = EMAIL.sub("[email]", text)
    return PHONE.sub("[phone]", text)

payload = {"text": redact(row["verbatim"])}   # correlate by row position locally

Regular expressions catch the formatted identifiers, not names. Treat redaction as risk reduction, not a licence to skip the contract.

Encryption, and the gaps between

In transit encryption (TLS, so HTTPS everywhere) protects data crossing a network from interception. At rest encryption protects a stolen disk or database snapshot; on Azure and Cloudflare it is on by default and you mostly need to not turn it off.

Neither protects against the failures that actually happen. Encryption is irrelevant to an authorisation mistake: a public bucket object is served, decrypted, to whoever finds the URL. It is irrelevant to a leaked credential, which decrypts everything legitimately. And it does nothing for the copies nobody tracks — a CSV in someone’s Downloads folder, a temp file on a build agent, a database backup on a shared drive, a spreadsheet emailed to a client. The useful questions are where the copies are, and who can reach each one.

Least privilege and access control

The pattern an agent will hand you is one admin credential, used by every job, with full read and write across everything. The fix is boring and effective: each component gets the narrowest access it needs.

-- Before: the dashboard reads the same table the pipeline writes, as the same user.

-- After: an aggregated view with no identifiers, and a read-only role over it
CREATE VIEW v_csat_by_segment AS
SELECT segment, COUNT(*) AS base_unweighted, AVG(csat) AS mean_csat
FROM responses GROUP BY segment
HAVING COUNT(*) >= 30;              -- suppress small cells; no single-person rows

GRANT SELECT ON v_csat_by_segment TO dashboard_ro;   -- and nothing else

Better still, remove the long-lived credential entirely. On Azure, a managed identity lets the app authenticate as itself, so there is no connection string to leak:

# Give the container app an identity, then grant it exactly one role
az containerapp identity assign --name iff-pipeline --resource-group iff-prod --system-assigned
az role assignment create \
  --assignee-object-id "$PRINCIPAL_ID" --assignee-principal-type ServicePrincipal \
  --role "Key Vault Secrets User" \
  --scope "/subscriptions/$SUB/resourceGroups/iff-prod/providers/Microsoft.KeyVault/vaults/iff-kv"

Audit trails

An audit trail answers who or what read which data, when, and from where: append-only, stored separately from the data it describes, and reviewed rather than merely collected. The rule that gets broken is that it must not contain the personal data itself. Log the fact of access, not the payload — an audit table full of request bodies is a second copy of your database in a less protected place, usually with longer retention.

CREATE TABLE access_audit (
  id          INTEGER PRIMARY KEY,
  occurred_at TEXT NOT NULL,
  actor       TEXT NOT NULL,      -- user principal or service identity
  action      TEXT NOT NULL,      -- 'read' | 'export' | 'delete'
  dataset     TEXT NOT NULL,
  row_count   INTEGER,            -- how much, not what
  source_ip   TEXT,
  request_id  TEXT                -- ties back to the app log for the same call
);

And in the application log, redact by default rather than by remembering:

# Before: the whole validated model, verbatims included, into Application Insights
log.info("processed", extra={"row": response.model_dump()})

# After: ids and counts only
log.info("processed", extra={
    "respondent_id": response.respondent_id,   # pseudonym, not email
    "fields": len(response.model_fields_set),
})

Secrets hygiene

A secret is anything that grants access: connection strings, API keys, signing keys, the HMAC pepper. The rules, per OWASP’s guidance, are short:

• Never in source control, and never in a front-end bundle.
• Injected at runtime from a managed store — Azure Key Vault, or wrangler secret for Workers and Pages.
• Different per environment, so a staging leak cannot touch production data.
• Scoped to the minimum permission, and rotated on a schedule.
• Never printed, never in an error page, never in a log line.

# Before: the classic agent default
OPENAI_KEY = "sk-proj-..."                 # in the repo, forever, in every clone

# After: typed settings, loaded from the environment the platform injects
from pydantic_settings import BaseSettings, SettingsConfigDict
from pydantic import SecretStr

class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_file=".env", extra="ignore")
    database_url: SecretStr                # SecretStr keeps it out of repr and logs
    llm_api_key: SecretStr

settings = Settings()                      # fails loudly at startup if one is missing
client = OpenAI(api_key=settings.llm_api_key.get_secret_value())

az keyvault secret set --vault-name iff-kv --name llm-api-key --value "$NEW_KEY"
npx wrangler secret put LLM_API_KEY          # Workers/Pages: never in wrangler.toml

Two things people get wrong. First: a secret committed to git is compromised the moment it lands, and deleting the line does not help, because it remains in history, in every clone, fork and CI cache. Rotation is the only fix. Second: in a React app, anything the bundler includes reaches the browser. Vite only exposes variables prefixed VITE_, which is a warning label rather than protection. A key the dashboard needs belongs behind your own endpoint, with the browser calling you and your server calling the vendor.

Retention, and actually deleting things

A retention period that lives in a policy document and not in a scheduled job is not a retention period.

-- Runs nightly. Deletes, rather than flagging, once the period has passed.
DELETE FROM verbatims
WHERE collected_at < date('now', '-13 months');   -- period agreed, then automated

Remember the copies: exports in a bucket need a lifecycle rule, KV entries need a TTL at write time, and backups have their own clock. Deleting a row while the R2 export from last March still sits in a bucket has not deleted anything.

If something goes wrong

The first hour matters more than the write-up. Contain it: revoke the credential, make the bucket private, take the endpoint down. Preserve the logs and evidence before anyone tidies up. Then tell the person in your organisation responsible for data protection immediately — this is not a decision to make alone — and start recording what data was involved, roughly how many people, and what harm is plausible.

For context on the clock you are working against: under UK GDPR a reportable breach goes to the ICO as soon as possible and, where feasible, within 72 hours of becoming aware of it, where a risk to people’s rights and freedoms is likely. Where the risk is high, the affected individuals must be told without undue delay. Whether a given incident meets those thresholds is a judgement for your data protection lead, not for you at 11pm.

What to check in your own code

• Grep the repo for sk-, AccountKey=, password=, Bearer and .pem. Then check git history, not only the working tree.
• Grep the front end for VITE_ variables holding anything that looks like a credential.
• Search for model_dump(), dict(row) or json.dumps(payload) inside a log call. That is where verbatims leak into your telemetry.
• List every place a copy of personal data lands: tables, buckets, KV namespaces, exports, caches, logs, local CSVs. Can you name them all? Can you delete one person from all of them?
• Check bucket and container access: is any object with respondent data reachable without authentication?
• Look at the credential each scheduled job uses. Is it one shared admin identity, or scoped per job? Could it be a managed identity instead?
• Find the third-party API calls. Is a contract in place, is training off, and is anything redacted before the request?
• Is there a retention job that actually issues DELETE, and does it cover exports and KV as well as the database?
• Does the audit log record actor, action and counts — and does it record payloads it shouldn’t?
• Confirm any dataset described as “anonymised” really is. Try to re-identify five rows using the columns still present.