AIBOM Hugging Face Study: 97,940 Models, Docs Missing

I generated an AIBOM for a model I was about to put behind an internal API, saw a valid CycloneDX file with every required field present, and nearly attached it to the review ticket as evidence. Then I opened the model card section. It was close to empty, and the file had still validated.

This post reads one paper closely: A Large-Scale Measurement of AI Bill of Materials Completeness in Hugging Face Models (arXiv 2607.17242, July 2026), which scored AIBOMs for public Hugging Face repositories. Every figure below is the paper's, taken from its HTML version, and I say where its method limits what it can claim. The scripts and snippets are mine, checked against the CycloneDX 1.6 schema, not against the paper's data.

What the paper measured, and how

The authors collected a snapshot of 2,942,466 public model records through the Hugging Face Hub API, kept the models with more than 100 downloads, and ended with 97,940 models for AIBOM generation. They generated one AIBOM per model with the OWASP GenAI Security Project AIBOM Generator, run through its command-line interface, and the output is CycloneDX JSON.

They did not invent a completeness metric. They reused the generator's own score: five weighted categories adding up to 100 points, with a penalty multiplier when critical fields are missing. Required fields, metadata and basic component information are worth 20 points each, model-card documentation 30, external references 10. Artifacts that failed JSON parsing, lacked the expected CycloneDX fields or had no score were excluded, and the authors compared 100 artifacts against the generator's score reports by hand to confirm extraction was accurate.

One detail worth knowing before you quote the paper: the abstract page says about 97.5K artifacts while the body text says 97,940 models. I use the body figure, since that is where the tables are. The arXiv abstract page itself gives only the headline conclusion, so the field-level numbers in this post come from the full text.

Structure is complete, and that is exactly the problem

Table I in the paper reports a mean overall score of 54.31 out of 100, a median of 54.10, a minimum of 46.60 and a maximum of 68.70 across the 97,940 artifacts. The spread is narrow: the standard deviation is 4.44. Almost every model lands in the same middle band regardless of how well it is documented.

Read the last column. Required fields are 100 percent in every artifact, so they cannot distinguish one model from another, and the paper drops them from the correlation. Metadata correlates 0.003 with the total, which is to say it also tells you nothing. The model-card category correlates 0.888 with the overall score: that single category, with the lowest mean of the five at 19.51 percent, is what separates a well-documented model from a poorly documented one. A validator that checks bomFormat, specVersion, serialNumber and version is checking the part that never varies.

The zeros: fields almost no generated AIBOM filled

Table III in the paper lists presence per field across all 97,940 artifacts. I have picked the rows that matter for a security or compliance review. The field names are the paper's, and several of them are the generator's names rather than CycloneDX schema names.

Notice the shape. A licence is present in roughly three artifacts out of four, a dataset in about two out of five, and then the fields a reviewer actually asks about collapse: known technical limitations in 16.74 percent, a safety-risk assessment in 9.76 percent, intended use in none. The meaningfulDescription row is the paper's own diagnostic for placeholder text, and it passes in 211 of 97,940 artifacts. A description field that is present 99.99 percent of the time is meaningful 0.22 percent of the time.

Where documentation does exist: papers, datasets and tasks

The paper's third research question asks how completeness varies by repository characteristics, and the answer is that documentation travels in clusters. Only 13.50 percent of artifacts reference a paper. Among models that do, 64.93 percent also declare datasets, against 35.35 percent among models without one. Technical-limitation coverage is 49.04 percent for models that have both a paper and a dataset, and 7.16 percent for models that have neither.

The task breakdown in Table IV points the same way. Text-generation is by far the largest group with 82,591 artifacts. Among those with both a paper and a dataset, version-control links reach 99.25 percent, technical limitations 61.59 percent and responsible-use or safety coverage 66.67 percent. For text-generation models with neither, the same three figures are 49.38, 6.66 and 9.56 percent. Environmental coverage is near zero across every task.

Table V adds licence to the mix. Models with no licence, no dataset and no paper show 2.87 percent technical limitations, 1.33 percent safety-risk assessment and 3.66 percent responsible-use coverage. The practical reading is that the AIBOM is not adding documentation; it is mirroring whatever the repository already had. If the author wrote a thorough model card, the AIBOM inherits it. If not, the generator has nothing to extract.

What the paper cannot tell you

Start with the best case. The top score in the whole population is 68.7, reached by 269 models, mostly Llama, Gemma, Qwen, Granite, Phi, Falcon, CodeLlama, BioMistral and StableLM variants, often in GGUF format. The paper still classes all of them as moderate. No public model in this sample clears the bar the scoring scheme itself would call good.

Now the limits, which the authors state in their threats to validity. Construct validity: field presence does not guarantee quality, which is why they added the meaningful-description check. Internal validity: the pipeline depends on the Hugging Face API, the generator and parsing scripts, so generator extraction logic could shape the results. External validity: the more-than-100-downloads filter excludes low-use, private, gated and non-Hugging-Face models, and practices and tools will change.

I would add one reading of my own, labelled as inference. Fields at exactly 0.00 percent across nearly 98,000 artifacts, such as intendedUse and ethicalConsiderations, look more like fields the generator does not populate than like 98,000 authors who all wrote nothing. The text I read does not settle which it is. Either way the consequence for you is the same: you cannot get these fields from a default run, so someone has to write them.

Hand-adding what the generator leaves empty

The generator's README documents the command-line flow and says output is CycloneDX 1.6 JSON. It does not document how its score is calculated or which source fields map to which AIBOM fields, so treat the file as a draft. The first block generates one; the second is the model-card patch you merge into it, using only property names that exist in the CycloneDX 1.6 schema.

# The generator is a Python CLI, per its README (CycloneDX 1.6 JSON output).
git clone https://github.com/GenAI-Security-Project/aibom-generator.git
cd aibom-generator
pip install -r requirements.txt

# One model in, one BOM out. Files land in sboms/ unless you pass --output.
python3 -m src.cli google-bert/bert-base-uncased --output sboms/bert.json

# Look at the model card, do not just validate the file. An empty result here
# is the paper's finding reproduced on your own model.
# (Assumes the standard CycloneDX 1.6 layout; check it on a real file first.)
jq '.components[] | select(.type == "machine-learning-model")
    | .modelCard.considerations' sboms/bert.json

The CycloneDX 1.6 modelCard object has modelParameters, quantitativeAnalysis and considerations. Under considerations the schema defines users, useCases, technicalLimitations, performanceTradeoffs, ethicalConsiderations, environmentalConsiderations and fairnessAssessments. Those are the slots the paper found empty, so they are where your hand-written content goes. An ethical consideration is an object with a name and a mitigationStrategy, and technicalLimitations is a plain list of strings.

// card-patch.json - written by the model owner, not by the generator.
// Property names are the CycloneDX 1.6 modelCard.considerations ones.
// The content below is an illustrative placeholder for a made-up classifier.
{
  "users": ["Support agents triaging inbound tickets"],
  "useCases": ["Route English-language tickets to one of six queues"],
  "technicalLimitations": [
    "Not evaluated on non-English text",
    "Accuracy degrades on tickets under ten words"
  ],
  "ethicalConsiderations": [
    {
      "name": "Mis-routing of urgent safety reports",
      "mitigationStrategy": "Keyword override sends safety terms to a human queue"
    }
  ]
}

// Merge it into the BOM. Leave environmentalConsiderations out until
// someone has measured energy: energyConsumptions requires activity,
// energyProviders and activityEnergyCost (kWh) in the 1.6 schema.
//
// jq --slurpfile card card-patch.json \
//   '(.components[] | select(.type == "machine-learning-model"))
//      .modelCard.considerations = $card[0]' \
//   sboms/bert.json > sboms/bert.patched.json

Energy is the field to be strict about. In the 1.6 schema an energyConsumptions entry requires an activity, energyProviders and an activityEnergyCost measured in kWh. If you have not measured training energy, leave the field out rather than inventing a figure: an absent field is honest and a fabricated one is worse than the 0.02 percent the paper found.

---
# Hugging Face model card header: the generator reads the card, so fix it here too.
license: apache-2.0
datasets:
- stanfordnlp/imdb
base_model: google-bert/bert-base-uncased
library_name: transformers
---
# Intended uses and limitations
Write them in the card text. Metadata alone does not carry them.

Gating on completeness instead of presence

The paper's advice to downstream users is not to treat the existence of an AIBOM as proof of completeness, and to check category and field scores before reuse, compliance or security review. This script turns that into a CI step. It assumes the standard CycloneDX 1.6 locations for each field, which I have not verified against the generator's actual output layout, so run it on one real file before you wire it up.

// aibom-audit.mjs  -  node aibom-audit.mjs sboms/bert.patched.json
import { readFileSync } from "node:fs";

const bom = JSON.parse(readFileSync(process.argv[2], "utf8"));
const model = (bom.components ?? []).find(
  (c) => c.type === "machine-learning-model",
);
if (!model) {
  console.error("No machine-learning-model component found");
  process.exit(2);
}

const card = model.modelCard ?? {};
const cons = card.considerations ?? {};
const nonEmpty = (a) => Array.isArray(a) && a.length > 0;

// Seven checks chosen to match the fields the paper found weakest.
const checks = {
  license: nonEmpty(model.licenses),
  datasets: nonEmpty(card.modelParameters?.datasets),
  useCases: nonEmpty(cons.useCases),
  technicalLimitations: nonEmpty(cons.technicalLimitations),
  ethicalConsiderations: nonEmpty(cons.ethicalConsiderations),
  energy: nonEmpty(cons.environmentalConsiderations?.energyConsumptions),
  paperReference: (model.externalReferences ?? []).some((r) =>
    /arxiv[.]org|doi[.]org/.test(r.url ?? ""),
  ),
};

const present = Object.values(checks).filter(Boolean).length;
for (const [name, ok] of Object.entries(checks)) {
  console.log((ok ? "ok      " : "MISSING ") + name);
}

// The threshold is a policy choice, not the paper's. Set your own.
const MIN_PRESENT = 5;
console.log(present + "/" + Object.keys(checks).length + " present");
process.exit(present >= MIN_PRESENT ? 0 : 1);

I picked the seven checks to match the fields the paper found weakest. The threshold is my choice, not the paper's: it fails the build when fewer than five of seven are present, which is a policy you should set for your own risk appetite. A model with a licence and datasets but no stated limitations should not pass an agent-facing review on structure alone.

What to do on Monday

Do these in order. The first takes ten minutes and tells you whether the rest is worth doing.

1. Pick one model that is already in production and generate its AIBOM with the generator's CLI, then open the file and look for modelCard content instead of validating it.
2. Run the audit script on it and record which of the seven fields are missing, so you have your own number instead of the paper's.
3. Write the missing considerations by hand, from the model card, the licence file and your own evaluation notes, and commit them next to the BOM so a reviewer can see who wrote them.
4. Add the audit as a CI step for any new model that enters your registry, with a threshold you chose and a named owner for exceptions.
5. Leave energy and fairness fields empty until someone measures them, and say so in the review ticket instead of letting a blank look like an oversight.

A complete AIBOM is a documentation exercise that happens to have a file format. The format is the cheap part; the paper shows the generator already does that part for everyone.

The rule I took from this paper: a valid AIBOM proves the generator ran, not that anyone documented the model. Required fields sit at 100 percent in 97,940 of 97,940 artifacts, while model-card completeness averages 19.51 percent and drives 0.888 of the score correlation. Gate on the model-card fields, write the missing ones by hand, and leave the fields you cannot measure empty.