Skip to content

Site guides

Getting Reddit data: the Data API, OAuth, and the limits of public pages

Use the Reddit Data API with OAuth for public posts and comments, respect rate limits and terms, and understand why scraping the website is not a free substitute.

Last updated 3 min readBy CrawlerBros Editorial

Reddit data is available through the Reddit Data API for public posts and comments, with OAuth authentication and documented rate limits. The API is the supported path. Scraping the website to avoid the API’s limits or terms is a contractual and technical risk, and Reddit’s robots.txt and blocking behaviour reflect that.

This page is research material, not legal advice. Reddit’s terms and your jurisdiction govern your use.

Start with the Data API

The Data API exposes listings, submissions, comments, subreddits, and user data subject to the terms. Access requires:

  1. A Reddit account.
  2. An OAuth application (script, web, or installed app).
  3. A user agent that identifies your app.
  4. Compliance with the Data API Terms and the User Agreement.

The API uses OAuth 2.0. For a script app, the client credentials flow is common; for a user-context app, the authorization code flow. Read the current documentation for the exact endpoints, scopes, and rate limits.

Rate limits and user agent

The API publishes rate limits (commonly expressed in requests per minute) and requires a descriptive user agent. A generic python-requests/… user agent is a good way to get blocked. Use something like:

CrawlerBros-Research/1.0 by u/your_account

Respect the X-Ratelimit-Remaining and X-Ratelimit-Reset headers. On 429, wait until the reset time; do not rotate IPs or clients to evade the limit.

A request shape

GET https://oauth.reddit.com/r/example/new?limit=100
Authorization: Bearer <access_token>
User-Agent: CrawlerBros-Research/1.0 by u/your_account

The response is JSON with a data.children array and an after cursor. Paginate with after until it is null. This is a documented request shape, not a run against a live token.

What the API does not permit

  • Circumventing rate limits with multiple apps, IPs, or tokens.
  • Scraping the website to avoid the API terms.
  • Collecting deleted content after users delete it, where the terms prohibit it.
  • Building a commercial product without the required agreement.
  • Training a model without checking the current terms and rights.
  • Collecting personal data at scale without a lawful basis.

Reddit has changed its API terms and pricing before; check the current terms before you build a business on it. The API is not a free, unlimited firehose.

Public pages and robots.txt

Reddit’s robots.txt and anti-bot measures restrict automated access to many paths. A crawler that ignores them is taking on risk. If you have a permitted use case, use the API; if the API does not cover it, ask Reddit.

This site does not provide a Reddit page-scraping tutorial. The API is the supported route, and a page-scraping recipe would be a recipe for a block or a dispute.

Data quality notes

  • Deletes and edits: a comment can change or disappear. Store the observation time and do not present a stale copy as current.
  • Vote counts: Reddit fuzzes scores. Treat them as approximate.
  • Nested comments: the tree can be deep; store the parent ID and the depth.
  • Subreddit rules: each subreddit has its own rules; the API terms are not the only policy.
  • Personal data: usernames and comment histories can identify people. Minimise and set retention.
  • Contract: the Data API Terms and User Agreement govern the use.
  • Copyright: posts and comments are user content; the User Agreement grants Reddit a licence, not you.
  • Privacy: usernames and content can be personal data; the GDPR/CCPA analysis applies.
  • Database rights: a large Reddit dataset may attract database rights in the EU/UK.

The sessions and APIs lesson covers OAuth, and the GDPR guide covers personal data.

Sources

Sources checked