Developer Publishes 5.6 Billion TikTok Records on Hugging Face

An open-source intelligence repository has become the unexpected home for one of the largest public caches of short-form video metadata in history. A developer known as hashfunction recently uploaded records detailing roughly 5.6 billion public TikTok videos to Hugging Face, making the massive trove freely available for download.

The collection, distributed through an account named datasocial, spans a timeline from July 2014 through October 2026. Rather than containing actual video footage, the 460 GB archive consists of Parquet files organized by month. Each individual row captures crucial metrics and descriptive elements, including captions, hashtags, sound identifiers, on-screen text, and TikTok Shop product markers.

The dataset also records granular engagement metrics such as view, like, comment, share, save, and download counts. Furthermore, it incorporates native platform labels, indicating whether specific content was flagged as artificial intelligence-generated or restricted from appearing on TikTok's primary recommendation feed, known as the For You page.

Behind the Extraction

According to an accompanying write-up by DataSocial, the information was gathered over a three-week period without requiring user accounts or logins. The collection method utilized TikTok's private mobile application programming interface, or API. Developers bypassed standard platform barriers by deploying generated device identities disguised as Android phones, reverse-engineered request signatures, and a spoofed Transport Layer Security handshake.

The operation yielded substantial results beyond the Hugging Face upload. Documentation indicates the scraper managed to harvest 3.23 billion creator profiles, 5.94 billion videos, and 2.8 billion comments in total. Other iterations of similar datasets, including a 4.5-billion-video collection, have also surfaced on Hugging Face.

While the data itself is accessible without a paywall—registering 1,181 downloads on Hugging Face shortly after release—the underlying infrastructure operates on a commercial model. The files are distributed under a creative commons license permitting only research and personal use. Commercial applications, daily updates, and creator profile access are directed to a dedicated web portal, and the source code for the scraper is sold separately.

Why it matters

Access to vast amounts of conversational and behavioral data remains a primary bottleneck for artificial intelligence developers. Technology firms and researchers prize the kind of granular metadata hosted on Hugging Face to train models that predict viral trends, analyze consumer preferences on e-commerce channels, or understand human phrasing in short-form video content.

Obtaining this information through official channels is exceptionally difficult. TikTok maintains strict limitations on data collection, restricting its formal Research Tools to qualifying academics at approved institutions in select regions like the United States, United Kingdom, European Union, Canada, and Switzerland. Section 3.4 of TikTok's United States terms of service explicitly prohibits extracting platform data using automated software without prior written authorization.

The release arrives amid a broader legal battle over data harvesting across the technology sector. Major platforms and content providers are increasingly pushing back against unauthorized scraping used for artificial intelligence model training, leading to high-profile lawsuits in federal courts.

What to watch

Observers will monitor whether TikTok takes legal or technical countermeasures against the hosts and creators of the dataset. The response from Hugging Face regarding the moderation of large-scale scraped archives will also set an important precedent for the open-source artificial intelligence community.