Skip to content
Back to top
View this page

# pyrate_limiter.buckets package

Concrete bucket implementations

class pyrate_limiter.buckets.InMemoryBucket(rates)

  Bases: `AbstractBucket`

  Simple In-memory Bucket using native list Clock can be either time.time or time.monotonic When leak, clock is required Pros: fast, safe, and precise Cons: since it resides in local memory, the data is not persistent, nor scalable Usecase: small applications, simple logic

  count()

    Count number of items in the bucket

    Return type:

      `int`

  flush()

    Flush the whole bucket - Must remove failing-rate after flushing

    Return type:

      `None`

  is_async = False
  items
  leak(current_timestamp=None)

    leaking bucket - removing items that are outdated

    Return type:

      `int`

  peek(index)

    Peek at the rate-item at a specific index in latest-to-earliest order NOTE: The reason we cannot peek from the start of the queue(earliest-to-latest) is we can’t really tell how many outdated items are still in the queue

    Return type:

      `RateItem` | `None`

  put(item)

    Put an item (typically the current time) in the bucket return true if successful, otherwise false

    Return type:

      `bool`

class pyrate_limiter.buckets.MultiprocessBucket(rates, items, mp_lock)

  Bases: `InMemoryBucket`

  classmethod init(rates)

    Creates a single ListProxy so that this bucket can be shared across multiple processes.

  items
  leak(current_timestamp=None)

    leaking bucket - removing items that are outdated

    Return type:

      `int`

  limiter_lock()

    An additional lock to be used by Limiter in-front of the thread lock. Intended for multiprocessing environments where a thread lock is insufficient.

  mp_lock
  put(item)

    Put an item (typically the current time) in the bucket return true if successful, otherwise false

    Return type:

      `bool`

pyrate_limiter.buckets.PgQueries

  alias of `Queries`

class pyrate_limiter.buckets.PostgresBucket(pool, table, rates)

  Bases: `AbstractBucket`

  close()

    Release any resources held by the bucket.

    Subclasses may override this method to perform any necessary cleanup (e.g., closing files, network connections, or releasing locks) when the bucket is no longer needed.

  count()

    Count number of items in the bucket

    Return type:

      `int` | `Awaitable`[`int`]

  flush()

    Flush the whole bucket - Must remove failing-rate after flushing

    Return type:

      `None` | `Awaitable`[`None`]

  is_async = False
  leak(current_timestamp=None)

    leaking bucket - removing items that are outdated

    Return type:

      `int` | `Awaitable`[`int`]

  peek(index)

    Peek at the rate-item at a specific index in latest-to-earliest order NOTE: The reason we cannot peek from the start of the queue(earliest-to-latest) is we can’t really tell how many outdated items are still in the queue

    Return type:

      `RateItem` | `None` | `Awaitable`[`RateItem` | `None`]

  pool
  put(item)

    Put an item (typically the current time) in the bucket return true if successful, otherwise false

    Return type:

      `bool` | `Awaitable`[`bool`]

  table

class pyrate_limiter.buckets.RedisBucket(rates, redis, bucket_key, script_hash)

  Bases: `AbstractBucket`

  A bucket using redis for storing data - We are not using redis’ built-in TIME since it is non-deterministic - In distributed context, use local server time or a remote time server - Each bucket instance use a dedicated connection to avoid race-condition - can be either sync or async

  bucket_key
  count()

    Count number of items in the bucket

  flush()

    Flush the whole bucket - Must remove failing-rate after flushing

  classmethod init(rates, redis, bucket_key)
  leak(current_timestamp=None)

    leaking bucket - removing items that are outdated

    Return type:

      `int` | `Awaitable`[`int`]

  now()

    Retrieve current timestamp from the clock backend.

  peek(index)

    Peek at the rate-item at a specific index in latest-to-earliest order NOTE: The reason we cannot peek from the start of the queue(earliest-to-latest) is we can’t really tell how many outdated items are still in the queue

    Return type:

      `RateItem` | `None` | `Awaitable`[`RateItem` | `None`]

  put(item)

    Add item to key

    Return type:

      `bool` | `Awaitable`[`bool`]

  redis
  script_hash

class pyrate_limiter.buckets.SQLiteBucket(rates, conn, table, lock=None)

  Bases: `AbstractBucket`

  For sqlite bucket, we are using the sql time function as the clock item’s timestamp wont matter here

  close()

    Release any resources held by the bucket.

    Subclasses may override this method to perform any necessary cleanup (e.g., closing files, network connections, or releasing locks) when the bucket is no longer needed.

  conn
  count()

    Count number of items in the bucket

    Return type:

      `int`

  flush()

    Flush the whole bucket - Must remove failing-rate after flushing

    Return type:

      `None`

  full_count_query
  classmethod init_from_file(rates, table='rate_bucket', db_path=None, create_new_table=True, use_file_lock=False)
    Return type:

      `SQLiteBucket`

  is_async = False
  leak(current_timestamp=None)

    Leaking/clean up bucket

    Return type:

      `int`

  limiter_lock()

    An additional lock to be used by Limiter in-front of the thread lock. Intended for multiprocessing environments where a thread lock is insufficient.

  lock
  now()

    Retrieve current timestamp from the clock backend.

  peek(index)

    Peek at the rate-item at a specific index in latest-to-earliest order NOTE: The reason we cannot peek from the start of the queue(earliest-to-latest) is we can’t really tell how many outdated items are still in the queue

    Return type:

      `RateItem` | `None`

  put(item)

    Put an item (typically the current time) in the bucket return true if successful, otherwise false

    Return type:

      `bool`

  table
  use_limiter_lock

class pyrate_limiter.buckets.SQLiteClock(conn)

  Bases: `AbstractClock`

  Get timestamp using SQLite as remote clock backend

  __init__(conn)

    In multiprocessing cases, use the bucket, so that a shared lock is used.

  classmethod default()
  lock
  now()

    Get time as of now, in milliseconds

    Return type:

      `int`

  time_query = "SELECT CAST(ROUND((julianday('now') - 2440587.5)*86400000) As INTEGER)"

pyrate_limiter.buckets.SQLiteQueries

  alias of `Queries`

## Submodules

- pyrate_limiter.buckets.in_memory_bucket module
  - `InMemoryBucket`
    - `InMemoryBucket.count()`
    - `InMemoryBucket.flush()`
    - `InMemoryBucket.is_async`
    - `InMemoryBucket.items`
    - `InMemoryBucket.leak()`
    - `InMemoryBucket.peek()`
    - `InMemoryBucket.put()`
- pyrate_limiter.buckets.mp_bucket module
  - `MultiprocessBucket`
    - `MultiprocessBucket.init()`
    - `MultiprocessBucket.items`
    - `MultiprocessBucket.leak()`
    - `MultiprocessBucket.limiter_lock()`
    - `MultiprocessBucket.mp_lock`
    - `MultiprocessBucket.put()`
- pyrate_limiter.buckets.postgres module
  - `PostgresBucket`
    - `PostgresBucket.close()`
    - `PostgresBucket.count()`
    - `PostgresBucket.flush()`
    - `PostgresBucket.is_async`
    - `PostgresBucket.leak()`
    - `PostgresBucket.peek()`
    - `PostgresBucket.pool`
    - `PostgresBucket.put()`
    - `PostgresBucket.table`
  - `Queries`
    - `Queries.COUNT`
    - `Queries.COUNT_WINDOW`
    - `Queries.CREATE_BUCKET_TABLE`
    - `Queries.CREATE_INDEX_ON_TIMESTAMP`
    - `Queries.FLUSH`
    - `Queries.LEAK`
    - `Queries.LEAK_COUNT`
    - `Queries.LOCK_TABLE`
    - `Queries.PEEK`
    - `Queries.PUT`
- pyrate_limiter.buckets.redis_bucket module
  - `LuaScript`
    - `LuaScript.PUT_ITEM`
  - `RedisBucket`
    - `RedisBucket.bucket_key`
    - `RedisBucket.count()`
    - `RedisBucket.flush()`
    - `RedisBucket.init()`
    - `RedisBucket.leak()`
    - `RedisBucket.now()`
    - `RedisBucket.peek()`
    - `RedisBucket.put()`
    - `RedisBucket.redis`
    - `RedisBucket.script_hash`
- pyrate_limiter.buckets.sqlite_bucket module
  - `Queries`
    - `Queries.COUNT_ALL`
    - `Queries.COUNT_BEFORE_INSERT`
    - `Queries.COUNT_BEFORE_LEAK`
    - `Queries.CREATE_BUCKET_TABLE`
    - `Queries.CREATE_INDEX_ON_TIMESTAMP`
    - `Queries.DROP_INDEX`
    - `Queries.DROP_TABLE`
    - `Queries.FLUSH`
    - `Queries.GET_ALL_ITEM`
    - `Queries.GET_FIRST_ITEM`
    - `Queries.GET_LAG`
    - `Queries.LEAK`
    - `Queries.PEEK`
    - `Queries.PUT_ITEM`
  - `SQLiteBucket`
    - `SQLiteBucket.close()`
    - `SQLiteBucket.conn`
    - `SQLiteBucket.count()`
    - `SQLiteBucket.flush()`
    - `SQLiteBucket.full_count_query`
    - `SQLiteBucket.init_from_file()`
    - `SQLiteBucket.is_async`
    - `SQLiteBucket.leak()`
    - `SQLiteBucket.limiter_lock()`
    - `SQLiteBucket.lock`
    - `SQLiteBucket.now()`
    - `SQLiteBucket.peek()`
    - `SQLiteBucket.put()`
    - `SQLiteBucket.table`
    - `SQLiteBucket.use_limiter_lock`
  - `SQLiteClock`
    - `SQLiteClock.__init__()`
    - `SQLiteClock.default()`
    - `SQLiteClock.lock`
    - `SQLiteClock.now()`
    - `SQLiteClock.time_query`