# Indonesia Grok Block: AI Image Moderation for Product Builders

> Indonesia switched Grok off on a Saturday and Malaysia followed on the Sunday. The regulators named the missing control, and it was the one that acts before the model runs.

URL: https://www.matthewswong.com/en/blog/indonesia-grok-block-ai-moderation-lesson/

- Author: Matthews Wong (https://www.matthewswong.com/en/about/)
- Published: 2026-09-17
- Updated: 2026-09-17
- Category: AI

> **Key takeaway:** On 10 January 2026 Indonesia blocked Grok nationwide over non-consensual sexualised deepfakes, and Malaysia followed a day later. For any product with an image generation feature, intake gates, tested refusal behaviour, content provenance and a takedown path that answers within hours are market-access requirements, not polish added after launch.

I read the alert on a Saturday, and a feature sitting in my own backlog changed category while I read it. Indonesia's Ministry of Communication and Digital Affairs had temporarily blocked Grok nationwide on 10 January 2026. Malaysia restricted it the next day. Two countries, one weekend, no notice period and no migration window.

I ship products to Indonesian users, so a decision taken in Jakarta is not an abstraction for me — it is the reason a feature does or does not exist. This is not a news recap. One section covers what happened; the rest is the control surface a builder owes when a feature turns a photograph into a new image: what you accept at intake, what refusal behaviour you test rather than assume, what provenance you attach, how fast a report resolves, and how a product survives a country switching part of it off. I describe none of the abuse and nothing that would help anyone reproduce it.

## The weekend two regulators switched a feature off

Indonesia acted first. The communication and digital affairs minister said the government treats non-consensual sexual deepfakes as a serious violation of human rights, dignity and the safety of citizens in the digital space, and that the measure was intended to protect women, children and the broader community. Malaysia's Communications and Multimedia Commission ordered a temporary restriction the following day, citing repeated misuse of the tool. Wire reporting carried by NPR describes them as the first two countries to block the service.

What matters to a builder is the part the regulators said was missing. Indonesia's director general of digital space supervision said initial findings showed the tool lacked effective safeguards to stop users creating and distributing pornographic content based on real photos of Indonesian residents. Malaysia's regulator said its notices demanding stronger safeguards drew responses that relied mainly on user reporting mechanisms, and that access would stay restricted until effective safeguards were in place.

Read those two statements as an engineering specification, because that is what they are. The named defect is at the input: real photographs of real people. The named inadequacy is a control that only acts once the file already exists. And the condition for restoring access is not an apology or a policy page — it is a control the regulator can see working. The OECD's AI Incidents Monitor record on the block notes the ministry warning of a potential permanent ban if the platform fails to comply. A feature you could have gated at intake cost access to an entire national market in a weekend.

## The input decides the risk class, not the prompt

If your feature accepts an uploaded photograph, classification has to happen at upload rather than on the prompt string. The prompt is the field the abuser controls and can rewrite indefinitely; the uploaded file is the thing that makes an output about one specific real person. The regulator's sentence names the input, not the wording. A banned-word list over prompts is the control most teams reach for first, because it is an afternoon of work, and it is the control that generalises worst.

Two things in that gate are worth defending. The face check runs before the model is invoked, so a refusal costs a detector call rather than an inference and cannot be argued around by rephrasing. And the refusal writes a record: six months later the question is not whether you had a policy but how many requests it stopped, and only a log answers that. The consent record is the harder half. For most products the honest version is a narrow allowance — the uploader's own face, a colleague with a signed release, an object with no person in it — rather than a general consent system, and narrowing the allowance is usually the better product decision anyway.

## Refusal is behaviour you measure, not a claim you cite

A vendor's safety claim is the vendor's evidence, not yours. A model card describes what a lab tested on its own suite, at a version you are probably not pinned to, in a request shape that is almost certainly not yours. The only statement you can honestly make — to a regulator, to a customer, or to yourself at two in the morning — is about the model you actually call, inside the pipeline you actually run. That means a refusal suite you own, committed to your repository, and run on every model change like any other regression gate.

The half of that file people skip is the allow half. A model that refuses everything scores perfectly on the refuse cases and ships a feature nobody can use, so over-refusal has to fail the gate too. Keep the two rates separate and never average them into one number; they move in opposite directions and a single figure hides both. Run the suite against the deployed configuration as well — your system prompt, your safety settings, your pre-filters — because a refusal observed in a vendor playground with vendor defaults says nothing about the request your backend sends.

> Pin the model version inside the suite file itself, not only in your deployment config, and fail the run when the two disagree. A refusal result is evidence about exactly one version of one model. The moment a vendor rolls a patch under an unpinned alias, yesterday's green run stops being evidence and nobody gets an alert.

## Provenance marks the honest file, not the malicious one

Provenance is worth shipping and worth being honest about. C2PA Content Credentials attach a signed manifest to a generated asset describing what made it. The specification's own worked example is a generative model producing an asset whose manifest carries a created action with digitalSourceType set to the IPTC term trainedAlgorithmicMedia — an interoperable statement any compliant reader can check without knowing a thing about your stack.

The limits are structural, and the specification does not hide them. A hard binding is a cryptographic hash over byte ranges of the asset, so it detects tampering and also breaks under the ordinary re-encode a social platform performs on upload. A soft binding, in the specification's words, is computed from the digital content of an asset rather than its raw bits, which is what lets a watermark or fingerprint survive that trip; a Durable Content Credential is defined as one with soft bindings that enable its discovery in a manifest repository. Even then the manifest can be separated from the asset, as C2PA's own FAQ acknowledges, and a screenshot ends the argument for anything not soft-bound.

So place provenance correctly in the threat model. It is very good at letting an honest platform label AI-generated media, and at letting you answer whether a reported file came from you at all. It is not a preventive control: no manifest has ever stopped a generation. Pair it with a visible mark — the specification has a term for that too, a perceptible component carrying human-consumable provenance information — because the visible one is the only part most viewers will ever see. If your safety story rests on provenance alone, you have a detection story with a prevention-shaped hole, which is exactly the shape the regulators objected to.

*The bottom band is the defence that was offered when regulators asked, and the restriction stayed in place. Reporting is a correct last line and a poor first one, because it cannot carry the four layers above it.*

## A report queue is a safeguard, and it is the last one

Reporting still has to exist, and it has to be good, because it is the layer a regulator can test from the outside without your cooperation. Indonesia's regulatory frame is explicit about clocks: under Ministerial Regulation 5 of 2020, an operator has four hours to remove content on an urgent request and 24 hours for other prohibited content once the ministry has given notice, with enforcement escalating from a warning through temporary blocking to full blocking and revocation of registration. Four hours is not a sprint item. It is an on-call rotation, and that changes what the reporting path has to be.

1. Make it reachable without an account. Someone harmed by a non-consensual image is usually not your user, so a report form behind a sign-in wall is a form that measures nothing.
2. Resolve a file to a record. Given only the image a stranger sends you, you need to answer which generation produced it, from which account, from which uploaded source. This is where the provenance work pays for itself, and where a soft binding beats a stripped manifest.
3. Separate takedown from investigation. Removing the asset and killing its URL should take minutes and need no verdict; deciding what happens to the account can take longer. Coupling them means every hard case leaves a live file online while you deliberate.
4. Instrument the clock, not the queue. Measure time from receipt to removal, publish the median and the worst case internally, and alert on the four-hour boundary rather than discovering it in a letter from a ministry.

Notice what this section does not claim. None of it prevents anything. That is precisely why it sits fifth: a report path is the correct last line and a poor first one, and a regulator telling you that the responses it received relied mainly on user reporting is what it looks like when a company presents its last line as the whole defence.

> Restricting a risky feature to paying users is a monetisation change, not a safety control. Grok limited image generation and editing to paying users after the global backlash and critics said it did not address the problem. A payment method is not a consent record, and it makes nobody in a photograph any safer.

## Make a regional block survivable, not fatal

Assume you will be wrong about something and a regulator will act before you have finished arguing. The architectural question is then narrow: when a country tells you to switch a capability off, how much of your product goes with it? If the answer is the whole product, after a release, you have converted a feature problem into a business continuity event. Country-scoped capability flags are the cheapest insurance in this entire post — one capability rather than one application, a config write rather than a deploy, and a remediation you can demonstrate the same afternoon a ministry asks for one.

Two details only appear once you have lived through a restriction. Recurring billing does not stop because a network does: subscribers in the blocked region keep being charged for something they cannot open, and that turns a regulatory problem into refunds, chargebacks and a support queue inside one billing cycle, so pausing collection by region belongs in the same runbook as the feature flag. And your obligations do not lapse with your reachability. You still hold those users' personal data, and under Indonesia's personal data protection law the access, correction and deletion requests keep arriving whether or not your product answers on that network. A block removes your traffic, not your duties.

*Payments is the panel teams forget, and it is the one that converts a regulatory restriction into refunds and chargebacks inside a single billing cycle.*

## The checklist I run before a generation feature ships

All of the above collapses into five questions I now ask before an image generation or editing feature goes anywhere near production. None of them are about model quality, and every one can be answered in a design review rather than during an incident.

- Does this feature accept a photograph of a person? If it does, it is a different product with a different review, whatever the prompt is allowed to say.
- Is there a refusal suite in the repository, pinned to a model version, carrying allow cases as well as refuse cases, running on every model change?
- Given only an image a stranger sends me, can I name the account and the request that produced it, without that account's cooperation?
- Can an unauthenticated stranger report that image, and can we take it down inside four hours on a Sunday?
- Can I switch this one capability off in one country with a config change, and pause that country's billing from the same runbook?

The list is short on purpose. Every item is something you either have or do not have on the day a screenshot goes viral, and none of them can be built during that day. The uncomfortable part is that a feature can pass a normal product review, ship, work well and still fail every question here, because nothing in a normal product review asks about the person who is not your user.

The lesson I took from January is not that generation features are too dangerous to build. It is that a control acting before the model runs is worth more than any number of controls acting after it, and that regulators are now scoring products on exactly that ordering. Build the intake gate first, own your refusal evidence, attach provenance without believing it prevents anything, staff the report queue to a clock, and keep the blast radius of any single feature down to one capability in one country. That is the price of shipping a generation feature into a market that can, and now does, take it away.

## Sources

- Al Jazeera — Indonesia blocks access to Musk's AI chatbot Grok over deepfake images, 10 January 2026
- NPR and the Associated Press — Malaysia and Indonesia become first to block Grok over AI deepfakes, 12 January 2026
- OECD.AI Incidents Monitor — Indonesia blocks Grok AI over deepfake sexual content concerns
- C2PA Technical Specification 2.4 — hard bindings, soft bindings and Durable Content Credentials
- Human Rights Watch — joint statement on Indonesia's Ministerial Regulation 5 of 2020 and its four-hour and 24-hour takedown windows

## Source links

- https://www.aljazeera.com/news/2026/1/10/indonesia-blocks-access-to-musks-ai-chatbot-grok-over-deepfake-images
- https://www.npr.org/2026/01/12/nx-s1-5674660/malaysia-indonesia-block-grok-ai-deepfakes
- https://oecd.ai/en/incidents/2026-01-27-7f99
- https://spec.c2pa.org/specifications/specifications/2.4/specs/C2PA_Specification.html
- https://www.hrw.org/news/2021/06/01/joint-statement-global-coalition-urge-indonesia-minister-ict-repeal-mr5

## FAQ

### Why did Indonesia block Grok in January 2026?

Indonesia's Ministry of Communication and Digital Affairs temporarily blocked Grok on 10 January 2026 after it was used to generate non-consensual sexualised deepfake images. The ministry's director general of digital space supervision said initial findings showed the tool lacked effective safeguards against creating such content from real photographs of Indonesian residents. Malaysia's regulator ordered a temporary restriction the following day.

### What safety controls should an AI image feature have before launch?

Four layers, in order of value. An intake gate that classifies by what was uploaded rather than by the prompt text, a refusal suite you run yourself against a pinned model version, content provenance such as C2PA credentials on every generated file, and a reporting path that removes content within hours. Reporting on its own is the layer regulators judged insufficient in this case.

### Does C2PA stop deepfakes from being made?

No. C2PA attaches a signed manifest describing what created a file, which makes it a transparency and detection mechanism rather than a preventive one. A hard binding is a hash over the asset's bytes and breaks on any re-encode, and the manifest can be separated from the file, which is why the specification adds soft bindings so a credential stays discoverable. Provenance helps you answer whether a file came from you; it does not stop it being generated.

### How fast must a platform take content down in Indonesia?

Under Ministerial Regulation 5 of 2020, an operator has four hours to remove content on an urgent request and 24 hours for other prohibited content after the ministry gives notice. Enforcement escalates from a warning through temporary blocking to full blocking and revocation of registration. A four-hour clock implies an on-call rotation, not a business-hours support queue.

### How do I make my product survive a country-level block?

Scope capabilities by region so a single feature can be switched off with a config change rather than a release, and keep sign-in and billing independent of that capability. Pause recurring collection for the blocked region in the same runbook, because charges continue by default and turn into refunds and chargebacks. Your data protection obligations continue even when your service is unreachable there.

---

- HTML version: https://www.matthewswong.com/en/blog/indonesia-grok-block-ai-moderation-lesson/
- Bahasa Indonesia: https://www.matthewswong.com/id/blog/indonesia-grok-block-ai-moderation-lesson/
- Site index for agents: https://www.matthewswong.com/llms.txt
- Sitemap: https://www.matthewswong.com/sitemap.xml
- API description (OpenAPI): https://www.matthewswong.com/openapi.json