“Wait… are you sending journal entries to OpenAI?”
the very first thing my pal requested once I confirmed her Really feel-Write, an AI-powered journaling app I constructed throughout a hackathon in San Francisco.
I shrugged.
“It was an AI-themed hackathon, I needed to construct one thing quick.”
She didn’t miss a beat:
“Certain. However how do I belief what you constructed? Why not self-host your individual LLM?”
That stopped me chilly.
I used to be happy with how shortly the app got here collectively. However that single query, and those that adopted unraveled every little thing I believed I knew about constructing responsibly with AI. The hackathon judges flagged it too.
That second made me notice how casually we deal with belief when constructing with AI, particularly with instruments that deal with delicate knowledge.
I noticed one thing larger:
We don’t discuss sufficient about belief when constructing with AI.
Her reply caught with me. Georgia von Minden is an information scientist on the ACLU, the place she works intently with points round personally identifiable data in authorized and civil rights contexts. I’ve all the time valued her perception, however this dialog hit completely different.
So I requested her to elaborate extra what does belief actually imply on this context? particularly when AI techniques deal with private knowledge.
She informed me:
“Belief could be arduous to pin down, however knowledge governance is an effective place to start. Who has the info, the way it’s saved, and what it’s used for all matter. Ten years in the past, I might have answered this in a different way. However in the present day, with big computing energy and big knowledge shops, large-scale inference is an actual concern. OpenAI has important entry to each compute and knowledge, and their lack of transparency makes it cheap to be cautious.
With regards to personally identifiable data, laws and customary sense each level to the necessity for sturdy knowledge governance. Sending PII in API calls isn’t simply dangerous — it may additionally violate these guidelines and expose people to hurt.”
It jogged my memory that once we construct with AI, particularly techniques that contact delicate human knowledge, we aren’t simply writing code.
We’re making selections about privateness, energy, and belief.
The second you accumulate person knowledge, particularly one thing as private as journal entries, you’re entering into an area of accountability. It’s not nearly what your mannequin can do. It’s about what occurs to that knowledge, the place it goes, and who has entry to it.
The Phantasm of Simplicity
Immediately, it’s simpler than ever to spin up one thing that appears clever. With OpenAI or different LLMs, builders can construct AI instruments in hours. Startups can launch “AI-powered” options in a single day. And enterprises? They’re dashing to combine these brokers into their workflows.
However in all that pleasure, one factor typically will get missed: belief.
When individuals discuss AI Brokers, they’re typically referring to light-weight wrappers round LLMs. These brokers may reply questions, automate duties, and even make selections. However many are constructed unexpectedly, with little thought given to safety, compliance, or accountability.
Simply because a product makes use of OpenAI doesn’t imply it’s protected. What you’re actually trusting is the entire pipeline:
- Who constructed the wrapper?
- How is your knowledge being dealt with?
- Is your data saved, logged — or worse, leaked?
I’ve been utilizing the OpenAI API for shopper use instances myself. Just lately, I used to be provided free entry to the API — as much as 1 million tokens every day till the top of April — if I agreed to share my immediate knowledge.
(Picture by Writer)
I nearly opted in for a private aspect mission, however then it hit me: if an answer supplier accepted that very same deal to chop prices, their customers would don’t know their knowledge was being shared. On a private degree, which may appear innocent. However in an enterprise context? That’s a critical breach of privateness, and probably of contractual or regulatory obligations.
All it takes is one engineer saying “sure” to a deal like that, and your buyer knowledge is in another person’s fingers.
(Picture by Writer)
Enterprise AI Raises the Stakes
I’m seeing extra SaaS firms and devtool startups experiment with AI brokers. Some are getting it proper. Some AI Brokers allow you to convey their very own LLM, giving them management over the place the mannequin runs and the way knowledge is dealt with.
That’s a considerate strategy: you outline the belief boundaries.
However not everyone seems to be so cautious.
Many firms simply plug into OpenAI’s API, add a number of buttons, and name it “enterprise-ready.”
Spoiler: it’s not.
What Can Go Fallacious? A Lot.
When you’re integrating AI brokers into your stack with out asking arduous questions, right here’s what’s in danger:
- Knowledge leakage: Your prompts may embrace delicate buyer knowledge, API keys, or inside logic — and if that’s despatched to a third-party mannequin, it could possibly be uncovered.
In 2023, Samsung engineers unknowingly pasted inside supply code and notes into ChatGPT (Forbes). That knowledge may now be a part of future coaching units — a serious threat for mental property.
- Compliance violations: Sending personally identifiable data (PII) via a mannequin like OpenAI with out correct controls can violate GDPR, HIPAA, or your individual contracts.
Elon Musk’s firm X realized that the arduous manner. They launched their AI chatbot “Grok” by utilizing all person posts together with from EU customers to coach it, with out correct opt-in. Regulators stepped in shortly. Beneath strain, they paused Grok’s coaching within the EU (Politico).
- Opaque conduct: Non-deterministic brokers are arduous to debug or clarify. What occurs when a shopper asks why a chatbot gave a improper suggestion or uncovered one thing confidential? You want transparency to reply that — and plenty of brokers in the present day don’t provide it.
- Knowledge possession confusion: Who owns the output? Who logs the info? Does your supplier retrain in your inputs?
Zoom was caught doing precisely that in 2023. They quietly modified their Phrases of Service to permit buyer assembly knowledge for use for AI coaching (Quick Firm). After public backlash, they reversed the coverage but it surely was a reminder that belief could be misplaced in a single day.
- Safety oversights in wrappers: In 2024, Flowise — a preferred low-code LLM orchestration software — was discovered to have dozens of deployments left uncovered to the web, many with out authentication (Cybersecurity Information). Researchers found API keys, database credentials, and person knowledge sitting within the open. That’s not an OpenAI downside — that’s a builder downside. However finish customers nonetheless pay the value.
- AI options that go too far: Microsoft’s “Recall” characteristic — a part of their Copilot rollout — took automated screenshots of customers’ exercise to assist the AI assistant reply questions (DoublePulsar). It sounded useful… till safety professionals flagged it as a privateness nightmare. Microsoft needed to shortly backpedal and make the characteristic opt-in solely.
Not All the pieces Must Be OpenAI
OpenAI is extremely highly effective. But it surely’s not all the time the precise reply.
Generally a smaller, native mannequin is greater than sufficient. Generally rule-based logic does the job higher. And infrequently, essentially the most safe choice is one which runs fully inside your infrastructure, beneath your guidelines.
We shouldn’t blindly join an LLM and label it a “good assistant.”
Within the enterprise, belief, transparency, and management aren’t non-obligatory — they’re important.
There’s a rising variety of platforms enabling that form of management. Salesforce’s Einstein 1 Studio now helps bring-your-own-model, letting you join your individual LLM from AWS or Azure. IBM’s Watson lets enterprises deploy fashions internally with full audit trails. Databricks, with MosaicML, helps you to prepare non-public LLMs inside your individual cloud, so your delicate knowledge by no means leaves your infrastructure.
That’s what actual enterprise AI ought to seem like.
Backside Line
AI brokers are highly effective. They unlock workflows and automations we couldn’t do earlier than. However ease of improvement doesn’t imply it’s protected, particularly when dealing with delicate knowledge at scale.
Earlier than you roll out that shiny new agent, ask your self:
- Who controls the mannequin?
- The place is the info going?
- Are we compliant?
- Can we audit what it’s doing?
Within the age of AI, the largest threat isn’t unhealthy expertise.
It’s blind belief.
In regards to the Writer
I’m Ellen, a machine studying engineer with 6 years of expertise, at present working at a fintech startup in San Francisco. My background spans knowledge science roles in oil & gasoline consulting, in addition to main AI and knowledge coaching applications throughout APAC, the Center East, and Europe.
I’m at present finishing my Grasp’s in Knowledge Science (graduating Might 2025) and actively on the lookout for my subsequent alternative as a machine studying engineer. When you’re open to referring or connecting, I’d really recognize it!
I like creating real-world affect via AI and I’m all the time open to project-based collaborations as effectively.
Try my portfolio: liviaellen.com/portfolio
My Earlier AR Works: liviaellen.com/ar-profile
Assist my work with a espresso: https://ko-fi.com/liviaellen