portalocker.redis module

Distributed locking through Redis pubsub instead of expiring keys.

The usual way to build a Redis lock is a key with a time to live: the holder writes SET <name> <token> NX PX <ttl> and keeps refreshing it. That design has an awkward failure mode. When a holder crashes, its network drops, or its machine is powered off, the key outlives it and every other process waits out the remaining TTL even though the holder is provably gone. Shortening the TTL narrows that window but risks a merely slow holder losing a lock it still believes it owns.

RedisLock keeps the lock in a subscription rather than in a key. A holder subscribes to the lock channel and a background thread keeps reading from it, so ownership is a property of a live connection. If the process dies the socket closes, Redis drops the subscriber, and the lock is released at once: there is no expiry to wait out and no heartbeat to refresh. The price is that nothing is stored anywhere, so a lock attempt has to ask the channel who is currently there.

Asking is a ping/pong on the channel itself:

acquire()
  |
  +- subscribe to the channel, naming the connection `client_name`
  +- count subscribers; being the only one means the lock is free
  +- otherwise publish a ping carrying a private response channel,
  |  which every subscriber answers with a record of its holder id
  |  and its current mode (see `RedisLockHolder`)
  +- decide (see `RedisLock._resolve_lock_holders`):
       compatible holders     -> join them, the lock is held
       elected, no readers    -> take the lock exclusively
       anything else          -> unsubscribe and retry

Shared readers hold the lock together, an exclusive writer holds it alone. There is no coordinator and no lock to take before taking the lock, so competing writers agree on a single winner by sorting the pending holder ids they all saw; see RedisLock._writer_is_elected. Subscribers that Redis still counts but that stopped answering are crashed processes, and their connections are killed so that the channel becomes consistent again.

Set health_check_interval on the connection (it is part of RedisLock.DEFAULT_REDIS_KWARGS) so that both sides notice a dead peer promptly.

Example

>>> import fakeredis
>>> import portalocker
>>> connection = fakeredis.FakeStrictRedis(
...     server=fakeredis.FakeServer(), decode_responses=True
... )
>>> lock = portalocker.RedisLock('some_channel', connection=connection)
>>> with lock:
...     print('do something here')
do something here

portalocker.redis.DEFAULT_THREAD_SLEEP_TIME = 0.1

Seconds the keep-alive thread sleeps between reads. It doubles as the fallback retry interval when check_interval is zero or negative.

portalocker.redis.DEFAULT_UNAVAILABLE_TIMEOUT = 1

Seconds a probe waits for the other subscribers to answer before the silent ones are treated as crashed and their connections are killed.

class portalocker.redis.PubSubWorkerThread(*args, **kwargs)

Bases: PubSubWorkerThread

redis-py’s pubsub reader thread, with failures escalated to main.

The subscription this thread services is the lock. While it runs, the holder answers liveness pings and Redis keeps counting it as a subscriber. If the thread were to die quietly - a dropped connection, a protocol error - the owning process would carry on believing it still holds a lock that every other process now considers released, which is exactly the split-brain this lock exists to avoid.

Interrupting the main thread turns that silent divergence into a loud failure the process cannot miss.

Parameters:

• args (Any)

• kwargs (Any)

Return type:

Any

run()

Read from the subscription, interrupting main on failure.

Raises:

Exception – Whatever the underlying reader raised, re-raised after _thread.interrupt_main has queued a KeyboardInterrupt in the main thread. Re-raising only ends this worker thread; the queued interrupt is what the rest of the process actually sees.

Return type:

None

portalocker.redis.REDIS_LOCK_PROTOCOL_VERSION = 1

Version stamped into every holder record. A reply that does not carry exactly this version is treated as coming from an older portalocker.

class portalocker.redis.RedisLock(channel, connection=None, timeout=None, check_interval=None, fail_when_locked=False, thread_sleep_time=0.1, unavailable_timeout=1, redis_kwargs=None, flags=<LockFlags.EXCLUSIVE: 2>)

Bases: LockBase[RedisLock]

An extremely reliable Redis lock based on pubsub.

The lock is held by a subscription kept open by a keep-alive thread.

As opposed to most Redis locking systems based on key/value pairs, this locking method is based on the pubsub system. The big advantage is that if the connection gets killed due to network issues, crashing processes or otherwise, it will still immediately unlock instead of waiting for a lock timeout.

To make sure both sides of the lock know about the connection state it is recommended to set the health_check_interval when creating the redis connection.

Parameters:

• channel (str) – the redis channel to use as locking key.

• connection (redis.client.Redis | None) – an optional redis connection if you already have one or if you need to specify the redis connection. A connection given here is never closed by the lock; one created by the lock itself is closed on release.

• timeout (float | None) – timeout when trying to acquire a lock

• check_interval (float | None) – check interval while waiting

• fail_when_locked (bool | None) – after the initial lock failed, return an error or lock the file. This does not wait for the timeout.

• thread_sleep_time (float) – sleep time between fetching messages from redis to prevent a busy/wait loop. In the case of lock conflicts this increases the time it takes to resolve the conflict. This should be smaller than the check_interval to be useful.

• unavailable_timeout (float) – If the conflicting lock is properly connected this should never exceed twice your redis latency. Note that this will increase the wait time possibly beyond your timeout and is always executed if a conflict arises.

• redis_kwargs (dict[str, Any] | None) – The redis connection arguments if no connection is given. The DEFAULT_REDIS_KWARGS are used as default, if you want to override these you need to explicitly specify a value (e.g. health_check_interval=0)

• flags (constants.LockFlags) – LockFlags.EXCLUSIVE (the default) or LockFlags.SHARED. Shared holders may coexist, while an exclusive holder waits for all shared holders to release. Other flag combinations are rejected; use fail_when_locked for non-blocking acquisition.

Example

Two readers can hold the same channel at the same time, while a writer would have to wait for both of them:

>>> import fakeredis
>>> import portalocker
>>> connection = fakeredis.FakeStrictRedis(
...     server=fakeredis.FakeServer(), decode_responses=True
... )
>>> reader = portalocker.RedisLock(
...     'shared_channel',
...     connection=connection,
...     flags=portalocker.LockFlags.SHARED,
... )
>>> other_reader = portalocker.RedisLock(
...     'shared_channel',
...     connection=connection,
...     flags=portalocker.LockFlags.SHARED,
... )
>>> with reader, other_reader:
...     print('both readers are in')
both readers are in

DEFAULT_REDIS_KWARGS: ClassVar[dict[str, Any]] = {'decode_responses': True, 'health_check_interval': 10}

acquire(timeout=None, check_interval=None, fail_when_locked=None)

Acquire the lock, retrying until it is free or time runs out.

Every attempt subscribes to the channel if this lock is not subscribed already, then either takes the lock outright or hands a probe result to _resolve_lock_holders:

1. Count the subscribers. Being the only one means nobody else is on the channel, so the lock is free and is taken immediately; an exclusive lock promotes itself from RedisLockMode.PENDING to RedisLockMode.EXCLUSIVE here. This is the uncontended fast path and costs one round trip.

2. Otherwise probe the channel with _collect_lock_holders and let _resolve_lock_holders decide.

A failed attempt normally unsubscribes again, so the next iteration subscribes from scratch; an elected writer is the exception and holds on to its subscription between attempts.

fail_when_locked turns the retry loop into a single attempt against a contended channel: rather than polling until the timeout expires against a holder that has demonstrably answered a ping, the first attempt that does not end in ownership raises AlreadyLocked. An elected writer always raises here too, even when no readers remain and the lock could have been taken outright.

If subscribing itself fails, _start_subscription rolls back before re-raising, so the original error propagates with the lock left inactive and the same object can be used again.

Parameters:

• timeout (float | None) – Seconds to keep retrying. Defaults to the instance’s timeout, itself defaulting to utils.DEFAULT_TIMEOUT. Zero still makes exactly one attempt.

• check_interval (float | None) – Base seconds to wait between attempts, jittered by _timeout_generator. Defaults to the instance’s check_interval.

• fail_when_locked (bool | None) – Raise AlreadyLocked on the first unsuccessful attempt instead of retrying. Defaults to the instance’s fail_when_locked.

Returns:

This lock, so with RedisLock(...) as lock binds the lock itself rather than a file handle.

Raises:

• AlreadyLocked – The timeout expired without acquiring the lock, or fail_when_locked was set and the first attempt did not succeed.

• AssertionError – This instance is already holding a lock. A RedisLock is not reentrant and holds at most one lock at a time; use a second instance, which gets its own holder_id.

Return type:

RedisLock

Example

>>> import fakeredis
>>> import portalocker
>>> connection = fakeredis.FakeStrictRedis(
...     server=fakeredis.FakeServer(), decode_responses=True
... )
>>> lock = portalocker.RedisLock(
...     'some_channel', connection=connection
... )
>>> lock.acquire(timeout=1) is lock
True
>>> lock.release()

channel: str

channel_handler(message)

Answer a liveness ping with this holder’s record.

Registered as the subscription callback in _start_subscription, so it runs on the PubSubWorkerThread for every message published to channel. Anything that is not a JSON object with a non-empty response_channel string is ignored, which keeps unrelated traffic on the channel harmless.

The reply is published on the private response channel the prober asked for and carries holder_id, the current mode and REDIS_LOCK_PROTOCOL_VERSION. Answering with the live mode rather than a stored one is what makes the protocol truthful: a writer that is still RedisLockMode.PENDING says so, and a probe therefore learns the state as it was at the moment it asked.

A probing lock is subscribed to its own channel, so it answers its own ping and appears in its own holder list.

Parameters:

message (dict[str, str]) – A redis-py pubsub message. Only messages of type message are answered; subscribe confirmations and other control frames are dropped.

Return type:

None

check_or_kill_lock(connection, timeout)

Ask whether anyone is alive on the channel, and reap if not.

The public liveness check from before 4.0.0. acquire no longer uses it: it probes with _collect_lock_holders and reaps with _kill_unavailable_locks, which understand individual holders and lock modes. This method predates holder ids and answers only the coarser question “is anybody answering on this channel?”.

The ping is published only after the subscription’s own confirmation frame has been consumed, or the wait for it times out. Redis queues a subscribe confirmation the moment a subscription is made; before 4.0.0 the reply poll accepted any message, so that confirmation was read as a reply, a crashed holder was reported as alive and stale locks were never reaped. The poll below now also requires type == 'message', and draining the confirmation first guarantees that the subscription is active before the ping goes out, so a real reply cannot be published into a subscription that is not listening yet.

Note

The reap step matches CLIENT LIST entries against client_name, which since 4.0.0 carries this instance’s own holder_id. On a live server that name identifies only this lock’s own connection, so another process’s crashed holder is not matched here; reaping across holders is what _kill_unavailable_locks does during acquire.

Parameters:

• connection (redis.client.Redis) – The connection to probe and to reap on.

• timeout (float) – Seconds to wait for the subscribe confirmation and, separately, for a reply, so a fully silent channel can take up to twice this long.

Returns:

True as soon as any reply arrives. None when nothing answered in time, after killing the matching pubsub connections. False is never returned.

Return type:

bool | None

property client_name: str

Name given to this holder’s subscriber connection.

_start_subscription sends CLIENT SETNAME over the pubsub connection itself, so the name lands on the connection that actually holds the subscription and shows up against it in CLIENT LIST. _kill_unavailable_locks reads it back the other way around: a listed connection whose name carries a holder_id that did not answer the last ping belongs to a crashed holder, and killing it releases the lock.

Returns:

legacy_client_name with this instance’s holder_id appended.

Example

>>> from portalocker import redis
>>> lock = redis.RedisLock('some_channel')
>>> lock.client_name == f'some_channel-lock-{lock.holder_id}'
True

close_connection: bool

connection: redis.client.Redis | None

flags: constants.LockFlags

get_connection()

Return the Redis connection, creating one on first use.

A connection handed to the constructor is returned unchanged and is never closed by this class; the caller owns it. A connection created here is owned by the lock, is built from redis_kwargs (with DEFAULT_REDIS_KWARGS filled in), and is closed by release, after which the next call creates a fresh one.

Returns:

The connection every command from this lock is issued on.

Return type:

redis.client.Redis

holder_id: str

property legacy_client_name: str

Connection name used by portalocker 3.2.0 and older.

Up to and including portalocker 3.2.0 every holder on a channel named its connection <channel>-lock, with no per-holder suffix, and answered a ping with a bare timestamp string instead of a record. Shared locks need holders to be individually identifiable, so 4.0.0 appended holder_id to the name (see client_name) and replaced the timestamp with a JSON record.

The old name is still recognised so that a 4.0.0 holder sharing a channel with an older one stays correct: an old reply is recorded as a single legacy RedisLockMode.EXCLUSIVE holder, which blocks readers and writers alike, and an old connection that stops answering is still reaped by name.

Returns:

The unsuffixed <channel>-lock name.

Example

>>> from portalocker import redis
>>> redis.RedisLock('some_channel').legacy_client_name
'some_channel-lock'

mode: RedisLockMode

pubsub: redis.client.PubSub | None = None

redis_kwargs: dict[str, Any]

release()

Give up the lock and undo everything acquire set up.

Stops and joins the keep-alive thread, unsubscribes and closes the pubsub connection, and forgets any election this lock had won. A connection the lock created itself is closed and cleared, so the next get_connection builds a fresh one; a connection supplied by the caller is left alone.

Dropping the subscription is not merely cleanup, it is the release: other processes learn the lock is free by no longer seeing this subscriber, with no key to delete and no expiry to wait for.

The same method doubles as the back-off between attempts. _resolve_lock_holders calls it after an unsuccessful probe so that a waiting lock stops being counted as a subscriber, and _start_subscription calls it to roll back a subscribe that failed halfway.

Calling this when nothing was acquired is harmless - it still closes a self-created connection if one exists - which is what makes both that rollback and __del__ safe.

Return type:

None

thread: PubSubWorkerThread | None

timeout: float

timeout when trying to acquire a lock

writer_elected: bool

class portalocker.redis.RedisLockHolder(holder_id, mode, legacy=False)

Bases: NamedTuple

One participant’s answer to a liveness ping.

A probe collects one of these per subscriber on the lock channel and the whole acquisition decision is made from the resulting list.

Example

>>> from portalocker import redis
>>> holder = redis.RedisLockHolder('a1b2', redis.RedisLockMode.SHARED)
>>> holder.holder_id, holder.mode.value, holder.legacy
('a1b2', 'shared', False)

Parameters:

• holder_id (str)

• mode (RedisLockMode)

• legacy (bool)

holder_id: str

The answering lock’s RedisLock.holder_id, or a synthetic legacy-<n> id for a reply that carried no identity.

legacy: bool

True when the reply could not be read as a protocol record, so its contents were assumed rather than parsed. Such a holder is always reported as RedisLockMode.EXCLUSIVE, because a portalocker release old enough not to speak the protocol has no notion of shared locks and must block everyone.

mode: RedisLockMode

The mode the holder advertised at the moment it answered.

class portalocker.redis.RedisLockMode(*values)

Bases: str, Enum

What a participant claims to be doing with the lock.

The member values are the strings that travel in the mode field of a holder record, which is why this is a str subclass: a member compares equal to its own wire value.

PENDING is the member that makes the protocol work. An exclusive waiter announces itself on the channel before it owns anything, which is what lets competing writers elect a single winner among themselves and what stops an endless stream of readers from starving a writer.

EXCLUSIVE = 'exclusive'

The holder owns the lock alone. Nobody else may join.

PENDING = 'pending'

The holder wants the lock exclusively but does not have it yet. It is competing in the election and blocking new shared holders.

SHARED = 'shared'

The holder is a reader. Any number of shared holders may hold the lock at the same time.