Grok and Deepfakes: The Online Safety Act’s First Big AI Test

The controversy surrounding Grok AI, the chatbot integrated into Elon Musk’s platform X, has become one of the most significant early tests of UK online safety regulation in the age of generative AI. Grok is xAI’s generative AI system, embedded directly within X, which allows users to produce text and images in real time.

Allegations that Grok has been used to generate and circulate sexualised deepfakes, including non-consensual imagery of women and minors, have triggered ongoing regulatory investigations and renewed scrutiny of platform responsibility under the Online Safety Act 2023 (the “OSA”). This blog post considers the harm caused by deepfakes and the extent to which online platforms can be held responsible for the creation and dissemination of such material.

Deepfakes and the problems they cause

Deepfakes are images or videos generated or altered by AI that convincingly depict real people doing or saying things they never did. Whilst the technology has legitimate uses in entertainment and the creative industries, it has also become closely associated with image-based abuse.

The most concerning category is non-consensual sexual deepfakes. These involve generating explicit or sexualised content using someone’s likeness without permission. The harm is not just limited to the image itself. It includes reputational damage, psychological distress, and the loss of control over one’s identity online.

Women and girls are disproportionately targeted, and the ease with which such content can now be generated has turned this into a large-scale problem. What once required technical expertise can now be done through simple text prompts. The integration of AI tools into social media platforms has also made the sharing of such content as simple as clicking a button.

What are online platforms legally required to do?

The introduction of the OSA marked a shift in UK regulation from reactive enforcement to proactive safety duties. It places legal obligations on platforms to assess and mitigate risks associated with illegal and harmful content, rather than simply removing it after publication. Key duties relevant to the Grok case include:

• conducting illegal content risk assessments;
• taking proportionate steps to prevent users from encountering illegal content;
• implementing systems to remove illegal material rapidly once identified; and
• performing specific children’s risk assessments and applying enhanced protections for minors.

The OSA is enforced by Ofcom, which has investigatory and sanctioning powers, including the ability to issue fines of up to 10% of a company’s qualifying worldwide revenue or £18 million, whichever is greater, as well as the ability to seek court orders that can restrict platform operations in the UK. Importantly, the OSA explicitly includes categories of illegal content such as intimate image abuse and child sexual abuse material, both of which are central to concerns raised in the Grok investigations.

Ofcom and the Information Commissioner’s Office have launched formal investigations into whether X has failed to meet its legal obligations, including duties relating to illegal content risk assessments, platform safety design, and the protection of children from harmful material.

Does the OSA create a duty of care?

Although the OSA does not explicitly use the phrase “duty of care”, it effectively establishes a statutory safety regime that functions in a similar way. Instead of requiring proof of harm after the fact, the law focuses on whether platforms have taken reasonable and proportionate steps to prevent foreseeable risks. In practice, regulators are likely to assess whether:

• risks were properly identified in advance;
• the platform’s design contributed to foreseeable harm;
• mitigation measures were effective and timely; and
• systems were adequate for the scale and nature of the service.

This shifts the focus away from individual pieces of content and toward system-level governance, including algorithm design, product features, and AI tool integration.

Comment

This is a key moment for UK online safety law because it requires a practical interpretation of what a proactive duty consists of when AI can generate harmful content at scale in seconds. The Grok AI deepfake controversy is not an isolated incident, but an early test of how the OSA applies to generative AI systems embedded in social platforms.

The controversy also highlights a structural problem in digital regulation: generative AI is evolving faster than legal frameworks can adapt. Regulators are examining whether platforms like X should have anticipated the misuse of embedded AI tools and whether sufficient safeguards were in place before deployment. It highlights a shift in regulatory thinking: platforms are increasingly expected to prevent harm through system design, not just moderate it after it appears. The focus is therefore not only on content removal, but also on whether the system design itself created foreseeable risks of illegal content generation.

Whether the current legal framework is sufficient to keep up with rapidly advancing AI capabilities remains uncertain. What is clear is that the Grok controversy is likely to shape how UK regulators interpret systemic risk and platform responsibility going forward.