tbb::queuing_rw_mutex::scoped_lock Class Reference

The scoped locking pattern. More...

`#include <queuing_rw_mutex.h>`

Inheritance diagram for tbb::queuing_rw_mutex::scoped_lock:
Collaboration diagram for tbb::queuing_rw_mutex::scoped_lock:

## Public Member Functions

| scoped_lock ()
| Construct lock that has not acquired a mutex. More...
| scoped_lock (queuing_rw_mutex &m, bool write=true)
| Acquire lock on given mutex. More...
| ~scoped_lock ()
| Release lock (if lock is held). More...
void | acquire (queuing_rw_mutex &m, bool write=true)
| Acquire lock on given mutex. More...
bool | try_acquire (queuing_rw_mutex &m, bool write=true)
| Acquire lock on given mutex if free (i.e. non-blocking) More...
void | release ()
| Release lock. More...
bool | upgrade_to_writer ()
| Upgrade reader to become a writer. More...
bool | downgrade_to_reader ()
| Downgrade writer to become a reader. More...

## Private Types

typedef unsigned char | state_t

## Private Member Functions

void | initialize ()
| Initialize fields to mean "no lock held". More...
void | acquire_internal_lock ()
| Acquire the internal lock. More...
bool | try_acquire_internal_lock ()
| Try to acquire the internal lock. More...
void | release_internal_lock ()
| Release the internal lock. More...
void | wait_for_release_of_internal_lock ()
| Wait for internal lock to be released. More...
void | unblock_or_wait_on_internal_lock (uintptr_t)
| A helper function. More...
Private Member Functions inherited from tbb::internal::no_copy
| no_copy (const no_copy &)=delete
| no_copy ()=default

## Private Attributes

queuing_rw_mutex * | my_mutex
| The pointer to the mutex owned, or NULL if not holding a mutex. More...
scoped_lock *__TBB_atomic | my_prev
| The pointer to the previous and next competitors for a mutex. More...
scoped_lock *__TBB_atomic *__TBB_atomic | my_next
atomic< state_t > | my_state
| State of the request: reader, writer, active reader, other service states. More...
unsigned char __TBB_atomic | my_going
| The local spin-wait variable. More...
unsigned char | my_internal_lock
| A tiny internal lock. More...

## Detailed Description

The scoped locking pattern.

It helps to avoid the common problem of forgetting to release lock. It also nicely provides the "node" for queuing locks.

Definition at line 53 of file queuing_rw_mutex.h.

## Member Typedef Documentation

## state_t

typedef unsigned char tbb::queuing_rw_mutex::scoped_lock::state_t

private

Definition at line 105 of file queuing_rw_mutex.h.

## Constructor & Destructor Documentation

## scoped_lock() [1/2]

tbb::queuing_rw_mutex::scoped_lock::scoped_lock | ( |  | ) |

inline

Construct lock that has not acquired a mutex.

Equivalent to zero-initialization of *this.

Definition at line 69 of file queuing_rw_mutex.h.

{initialize();}
tbb::queuing_rw_mutex::scoped_lock::initialize
void initialize()
Initialize fields to mean "no lock held".
Definition: queuing_rw_mutex.h:55

References initialize().

Here is the call graph for this function:

## scoped_lock() [2/2]

tbb::queuing_rw_mutex::scoped_lock::scoped_lock | ( | queuing_rw_mutex & | m,
|  | bool | write = `true`
| ) |  |

inline

Acquire lock on given mutex.

Definition at line 72 of file queuing_rw_mutex.h.

{
initialize();
acquire(m,write);
}
tbb::queuing_rw_mutex::scoped_lock::acquire
void acquire(queuing_rw_mutex &m, bool write=true)
Acquire lock on given mutex.
Definition: queuing_rw_mutex.cpp:140

References acquire(), and initialize().

Here is the call graph for this function:

## ~scoped_lock()

tbb::queuing_rw_mutex::scoped_lock::~scoped_lock | ( |  | ) |

inline

Release lock (if lock is held).

Definition at line 78 of file queuing_rw_mutex.h.

{
if( my_mutex ) release();
}
tbb::queuing_rw_mutex::scoped_lock::release
void release()
Release lock.
Definition: queuing_rw_mutex.cpp:254
tbb::queuing_rw_mutex::scoped_lock::my_mutex
queuing_rw_mutex * my_mutex
The pointer to the mutex owned, or NULL if not holding a mutex.
Definition: queuing_rw_mutex.h:100

References my_mutex, and release().

Here is the call graph for this function:

## Member Function Documentation

## acquire()

void tbb::queuing_rw_mutex::scoped_lock::acquire | ( | queuing_rw_mutex & | m,
|  | bool | write = `true`
| ) |  |

Acquire lock on given mutex.

A method to acquire queuing_rw_mutex lock.

Definition at line 140 of file queuing_rw_mutex.cpp.

{
__TBB_ASSERT( !my_mutex, "scoped_lock is already holding a mutex");
// Must set all fields before the fetch_and_store, because once the
// fetch_and_store executes, *this becomes accessible to other threads.
my_mutex = &m;
__TBB_store_relaxed(my_prev , (scoped_lock*)0);
__TBB_store_relaxed(my_next , (scoped_lock*)0);
__TBB_store_relaxed(my_going, 0);
my_state = state_t(write ? STATE_WRITER : STATE_READER);
my_internal_lock = RELEASED;
queuing_rw_mutex::scoped_lock* pred = m.q_tail.fetch_and_store<tbb::release>(this);
if( write ) { // Acquiring for write
if( pred ) {
ITT_NOTIFY(sync_prepare, my_mutex);
pred = tricky_pointer(pred) & ~FLAG;
__TBB_ASSERT( !( uintptr_t(pred) & FLAG ), "use of corrupted pointer!" );
#if TBB_USE_ASSERT
__TBB_control_consistency_helper(); // on "m.q_tail"
__TBB_ASSERT( !__TBB_load_relaxed(pred->my_next), "the predecessor has another successor!");
#endif
__TBB_store_with_release(pred->my_next,this);
spin_wait_until_eq(my_going, 1);
}
} else { // Acquiring for read
#if DO_ITT_NOTIFY
bool sync_prepare_done = false;
#endif
if( pred ) {
unsigned short pred_state;
__TBB_ASSERT( !__TBB_load_relaxed(my_prev), "the predecessor is already set" );
if( uintptr_t(pred) & FLAG ) {
/* this is only possible if pred is an upgrading reader and it signals us to wait */
pred_state = STATE_UPGRADE_WAITING;
pred = tricky_pointer(pred) & ~FLAG;
} else {
// Load pred->my_state now, because once pred->my_next becomes
// non-NULL, we must assume that *pred might be destroyed.
pred_state = pred->my_state.compare_and_swap<tbb::acquire>(STATE_READER_UNBLOCKNEXT, STATE_READER);
}
__TBB_store_relaxed(my_prev, pred);
__TBB_ASSERT( !( uintptr_t(pred) & FLAG ), "use of corrupted pointer!" );
#if TBB_USE_ASSERT
__TBB_control_consistency_helper(); // on "m.q_tail"
__TBB_ASSERT( !__TBB_load_relaxed(pred->my_next), "the predecessor has another successor!");
#endif
__TBB_store_with_release(pred->my_next,this);
if( pred_state != STATE_ACTIVEREADER ) {
#if DO_ITT_NOTIFY
sync_prepare_done = true;
ITT_NOTIFY(sync_prepare, my_mutex);
#endif
spin_wait_until_eq(my_going, 1);
}
}
// The protected state must have been acquired here before it can be further released to any other reader(s):
unsigned short old_state = my_state.compare_and_swap<tbb::acquire>(STATE_ACTIVEREADER, STATE_READER);
if( old_state!=STATE_READER ) {
#if DO_ITT_NOTIFY
if( !sync_prepare_done )
ITT_NOTIFY(sync_prepare, my_mutex);
#endif
// Failed to become active reader -> need to unblock the next waiting reader first
__TBB_ASSERT( my_state==STATE_READER_UNBLOCKNEXT, "unexpected state" );
spin_wait_while_eq(my_next, (scoped_lock*)NULL);
/* my_state should be changed before unblocking the next otherwise it might finish
and another thread can get our old state and left blocked */
my_state = STATE_ACTIVEREADER;
__TBB_store_with_release(my_next->my_going,1);
}
}
ITT_NOTIFY(sync_acquired, my_mutex);
// Force acquire so that user's critical section receives correct values
// from processor that was previously in the user's critical section.
__TBB_load_with_acquire(my_going);
}
__TBB_control_consistency_helper
#define __TBB_control_consistency_helper()
Definition: gcc_generic.h:60
__TBB_ASSERT
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:165
ITT_NOTIFY
#define ITT_NOTIFY(name, obj)
Definition: itt_notify.h:112
tbb::FLAG
static const tricky_pointer::word FLAG
Mask for low order bit of a pointer.
Definition: queuing_rw_mutex.cpp:128
tbb::tricky_pointer
tricky_atomic_pointer< queuing_rw_mutex::scoped_lock > tricky_pointer
Definition: queuing_rw_mutex.cpp:120
tbb::STATE_UPGRADE_WAITING
@ STATE_UPGRADE_WAITING
Definition: queuing_rw_mutex.cpp:40
tbb::STATE_READER
@ STATE_READER
Definition: queuing_rw_mutex.cpp:36
tbb::STATE_WRITER
@ STATE_WRITER
Definition: queuing_rw_mutex.cpp:35
tbb::STATE_ACTIVEREADER
@ STATE_ACTIVEREADER
Definition: queuing_rw_mutex.cpp:38
tbb::STATE_READER_UNBLOCKNEXT
@ STATE_READER_UNBLOCKNEXT
Definition: queuing_rw_mutex.cpp:37
tbb::release
@ release
Release.
Definition: atomic.h:59
tbb::acquire
@ acquire
Acquire.
Definition: atomic.h:57
tbb::RELEASED
const unsigned char RELEASED
Definition: queuing_rw_mutex.cpp:47
tbb::internal::__TBB_load_with_acquire
T __TBB_load_with_acquire(const volatile T &location)
Definition: tbb_machine.h:712
tbb::internal::__TBB_store_relaxed
void __TBB_store_relaxed(volatile T &location, V value)
Definition: tbb_machine.h:742
tbb::internal::spin_wait_until_eq
void spin_wait_until_eq(const volatile T &location, const U value)
Spin UNTIL the value of the variable is equal to a given value.
Definition: tbb_machine.h:402
tbb::internal::__TBB_load_relaxed
T __TBB_load_relaxed(const volatile T &location)
Definition: tbb_machine.h:738
tbb::internal::__TBB_store_with_release
void __TBB_store_with_release(volatile T &location, V value)
Definition: tbb_machine.h:716
tbb::internal::spin_wait_while_eq
void spin_wait_while_eq(const volatile T &location, U value)
Spin WHILE the value of the variable is equal to a given value.
Definition: tbb_machine.h:394
tbb::queuing_rw_mutex::scoped_lock::scoped_lock
scoped_lock()
Construct lock that has not acquired a mutex.
Definition: queuing_rw_mutex.h:69
tbb::queuing_rw_mutex::scoped_lock::state_t
unsigned char state_t
Definition: queuing_rw_mutex.h:105
tbb::queuing_rw_mutex::scoped_lock::my_next
scoped_lock *__TBB_atomic *__TBB_atomic my_next
Definition: queuing_rw_mutex.h:103
tbb::queuing_rw_mutex::scoped_lock::my_state
atomic< state_t > my_state
State of the request: reader, writer, active reader, other service states.
Definition: queuing_rw_mutex.h:108
tbb::queuing_rw_mutex::scoped_lock::my_internal_lock
unsigned char my_internal_lock
A tiny internal lock.
Definition: queuing_rw_mutex.h:115
tbb::queuing_rw_mutex::scoped_lock::my_going
unsigned char __TBB_atomic my_going
The local spin-wait variable.
Definition: queuing_rw_mutex.h:112
tbb::queuing_rw_mutex::scoped_lock::my_prev
scoped_lock *__TBB_atomic my_prev
The pointer to the previous and next competitors for a mutex.
Definition: queuing_rw_mutex.h:103

References __TBB_ASSERT, __TBB_control_consistency_helper, tbb::internal::__TBB_load_relaxed(), tbb::internal::__TBB_load_with_acquire(), tbb::internal::__TBB_store_relaxed(), tbb::internal::__TBB_store_with_release(), tbb::acquire, tbb::FLAG, ITT_NOTIFY, my_next, my_state, tbb::queuing_rw_mutex::q_tail, tbb::release, tbb::RELEASED, tbb::internal::spin_wait_until_eq(), tbb::internal::spin_wait_while_eq(), tbb::STATE_ACTIVEREADER, tbb::STATE_READER, tbb::STATE_READER_UNBLOCKNEXT, tbb::STATE_UPGRADE_WAITING, and tbb::STATE_WRITER.

Referenced by scoped_lock().

Here is the call graph for this function:
Here is the caller graph for this function:

## acquire_internal_lock()

void tbb::queuing_rw_mutex::scoped_lock::acquire_internal_lock | ( |  | ) |

inline private

Acquire the internal lock.

Definition at line 55 of file queuing_rw_mutex.cpp.

{
// Usually, we would use the test-test-and-set idiom here, with exponential backoff.
// But so far, experiments indicate there is no value in doing so here.
while( !try_acquire_internal_lock() ) {
__TBB_Pause(1);
}
}
__TBB_Pause
void __TBB_Pause(int32_t)
Definition: tbb_machine.h:334
tbb::queuing_rw_mutex::scoped_lock::try_acquire_internal_lock
bool try_acquire_internal_lock()
Try to acquire the internal lock.
Definition: queuing_rw_mutex.cpp:50

References __TBB_Pause().

Here is the call graph for this function:

## downgrade_to_reader()

bool tbb::queuing_rw_mutex::scoped_lock::downgrade_to_reader | ( |  | ) |

Downgrade writer to become a reader.

Definition at line 360 of file queuing_rw_mutex.cpp.

{
if ( my_state == STATE_ACTIVEREADER ) return true; // Already a reader
ITT_NOTIFY(sync_releasing, my_mutex);
my_state = STATE_READER;
if( ! __TBB_load_relaxed(my_next) ) {
// the following load of q_tail must not be reordered with setting STATE_READER above
if( this==my_mutex->q_tail.load<full_fence>() ) {
unsigned short old_state = my_state.compare_and_swap<tbb::release>(STATE_ACTIVEREADER, STATE_READER);
if( old_state==STATE_READER )
return true; // Downgrade completed
}
/* wait for the next to register */
spin_wait_while_eq( my_next, (void*)NULL );
}
scoped_lock *const n = __TBB_load_with_acquire(my_next);
__TBB_ASSERT( n, "still no successor at this point!" );
if( n->my_state & STATE_COMBINED_WAITINGREADER )
__TBB_store_with_release(n->my_going,1);
else if( n->my_state==STATE_UPGRADE_WAITING )
// the next waiting for upgrade means this writer was upgraded before.
n->my_state = STATE_UPGRADE_LOSER;
my_state = STATE_ACTIVEREADER;
return true;
}
sync_releasing
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p sync_releasing
Definition: ittnotify_static.h:104
tbb::STATE_COMBINED_WAITINGREADER
@ STATE_COMBINED_WAITINGREADER
Definition: queuing_rw_mutex.cpp:42
tbb::STATE_UPGRADE_LOSER
@ STATE_UPGRADE_LOSER
Definition: queuing_rw_mutex.cpp:41
tbb::full_fence
@ full_fence
Sequential consistency.
Definition: atomic.h:55
tbb::queuing_rw_mutex::q_tail
atomic< scoped_lock * > q_tail
The last competitor requesting the lock.
Definition: queuing_rw_mutex.h:143

References __TBB_ASSERT, tbb::internal::__TBB_load_relaxed(), tbb::internal::__TBB_load_with_acquire(), tbb::internal::__TBB_store_with_release(), tbb::full_fence, ITT_NOTIFY, my_going, my_state, tbb::release, tbb::internal::spin_wait_while_eq(), tbb::STATE_ACTIVEREADER, tbb::STATE_COMBINED_WAITINGREADER, tbb::STATE_READER, tbb::STATE_UPGRADE_LOSER, tbb::STATE_UPGRADE_WAITING, and sync_releasing.

Here is the call graph for this function:

## initialize()

void tbb::queuing_rw_mutex::scoped_lock::initialize | ( |  | ) |

inline private

Initialize fields to mean "no lock held".

Definition at line 55 of file queuing_rw_mutex.h.

{
my_mutex = NULL;
my_internal_lock = 0;
my_going = 0;
#if TBB_USE_ASSERT
my_state = 0xFF; // Set to invalid state
internal::poison_pointer(my_next);
internal::poison_pointer(my_prev);
#endif /* TBB_USE_ASSERT */
}
tbb::internal::poison_pointer
void poison_pointer(T *__TBB_atomic &)
Definition: tbb_stddef.h:305

References my_going, my_internal_lock, my_mutex, my_next, my_prev, my_state, and tbb::internal::poison_pointer().

Referenced by scoped_lock().

Here is the call graph for this function:
Here is the caller graph for this function:

## release()

void tbb::queuing_rw_mutex::scoped_lock::release | ( |  | ) |

Release lock.

A method to release queuing_rw_mutex lock.

Definition at line 254 of file queuing_rw_mutex.cpp.

{
__TBB_ASSERT(my_mutex!=NULL, "no lock acquired");
ITT_NOTIFY(sync_releasing, my_mutex);
if( my_state == STATE_WRITER ) { // Acquired for write
// The logic below is the same as "writerUnlock", but elides
// "return" from the middle of the routine.
// In the statement below, acquire semantics of reading my_next is required
// so that following operations with fields of my_next are safe.
scoped_lock* n = __TBB_load_with_acquire(my_next);
if( !n ) {
if( this == my_mutex->q_tail.compare_and_swap<tbb::release>(NULL, this) ) {
// this was the only item in the queue, and the queue is now empty.
goto done;
}
spin_wait_while_eq( my_next, (scoped_lock*)NULL );
n = __TBB_load_with_acquire(my_next);
}
__TBB_store_relaxed(n->my_going, 2); // protect next queue node from being destroyed too early
if( n->my_state==STATE_UPGRADE_WAITING ) {
// the next waiting for upgrade means this writer was upgraded before.
acquire_internal_lock();
queuing_rw_mutex::scoped_lock* tmp = tricky_pointer::fetch_and_store<tbb::release>(&(n->my_prev), NULL);
n->my_state = STATE_UPGRADE_LOSER;
__TBB_store_with_release(n->my_going,1);
unblock_or_wait_on_internal_lock(get_flag(tmp));
} else {
__TBB_ASSERT( my_state & (STATE_COMBINED_WAITINGREADER | STATE_WRITER), "unexpected state" );
__TBB_ASSERT( !( uintptr_t(__TBB_load_relaxed(n->my_prev)) & FLAG ), "use of corrupted pointer!" );
__TBB_store_relaxed(n->my_prev, (scoped_lock*)0);
__TBB_store_with_release(n->my_going,1);
}
} else { // Acquired for read
queuing_rw_mutex::scoped_lock *tmp = NULL;
retry:
// Addition to the original paper: Mark my_prev as in use
queuing_rw_mutex::scoped_lock *pred = tricky_pointer::fetch_and_add<tbb::acquire>(&my_prev, FLAG);
if( pred ) {
if( !(pred->try_acquire_internal_lock()) )
{
// Failed to acquire the lock on pred. The predecessor either unlinks or upgrades.
// In the second case, it could or could not know my "in use" flag - need to check
tmp = tricky_pointer::compare_and_swap<tbb::release>(&my_prev, pred, tricky_pointer(pred) | FLAG );
if( !(uintptr_t(tmp) & FLAG) ) {
// Wait for the predecessor to change my_prev (e.g. during unlink)
spin_wait_while_eq( my_prev, tricky_pointer(pred)|FLAG );
// Now owner of pred is waiting for _us_ to release its lock
pred->release_internal_lock();
}
// else the "in use" flag is back -> the predecessor didn't get it and will release itself; nothing to do
tmp = NULL;
goto retry;
}
__TBB_ASSERT(pred && pred->my_internal_lock==ACQUIRED, "predecessor's lock is not acquired");
__TBB_store_relaxed(my_prev, pred);
acquire_internal_lock();
__TBB_store_with_release(pred->my_next,static_cast<scoped_lock *>(NULL));
if( !__TBB_load_relaxed(my_next) && this != my_mutex->q_tail.compare_and_swap<tbb::release>(pred, this) ) {
spin_wait_while_eq( my_next, (void*)NULL );
}
__TBB_ASSERT( !get_flag(__TBB_load_relaxed(my_next)), "use of corrupted pointer" );
// ensure acquire semantics of reading 'my_next'
if( scoped_lock *const l_next = __TBB_load_with_acquire(my_next) ) { // I->next != nil, TODO: rename to n after clearing up and adapting the n in the comment two lines below
// Equivalent to I->next->prev = I->prev but protected against (prev[n]&FLAG)!=0
tmp = tricky_pointer::fetch_and_store<tbb::release>(&(l_next->my_prev), pred);
// I->prev->next = I->next;
__TBB_ASSERT(__TBB_load_relaxed(my_prev)==pred, NULL);
__TBB_store_with_release(pred->my_next, my_next);
}
// Safe to release in the order opposite to acquiring which makes the code simpler
pred->release_internal_lock();
} else { // No predecessor when we looked
acquire_internal_lock(); // "exclusiveLock(&I->EL)"
scoped_lock* n = __TBB_load_with_acquire(my_next);
if( !n ) {
if( this != my_mutex->q_tail.compare_and_swap<tbb::release>(NULL, this) ) {
spin_wait_while_eq( my_next, (scoped_lock*)NULL );
n = __TBB_load_relaxed(my_next);
} else {
goto unlock_self;
}
}
__TBB_store_relaxed(n->my_going, 2); // protect next queue node from being destroyed too early
tmp = tricky_pointer::fetch_and_store<tbb::release>(&(n->my_prev), NULL);
__TBB_store_with_release(n->my_going,1);
}
unlock_self:
unblock_or_wait_on_internal_lock(get_flag(tmp));
}
done:
spin_wait_while_eq( my_going, 2 );
initialize();
}
tbb::ACQUIRED
const unsigned char ACQUIRED
Definition: queuing_rw_mutex.cpp:48
tbb::get_flag
uintptr_t get_flag(queuing_rw_mutex::scoped_lock *ptr)
Definition: queuing_rw_mutex.cpp:131
tbb::queuing_rw_mutex::scoped_lock::acquire_internal_lock
void acquire_internal_lock()
Acquire the internal lock.
Definition: queuing_rw_mutex.cpp:55
tbb::queuing_rw_mutex::scoped_lock::unblock_or_wait_on_internal_lock
void unblock_or_wait_on_internal_lock(uintptr_t)
A helper function.
Definition: queuing_rw_mutex.cpp:74

References __TBB_ASSERT, tbb::internal::__TBB_load_relaxed(), tbb::internal::__TBB_load_with_acquire(), tbb::internal::__TBB_store_relaxed(), tbb::internal::__TBB_store_with_release(), tbb::ACQUIRED, tbb::FLAG, tbb::get_flag(), ITT_NOTIFY, my_going, my_internal_lock, my_next, my_prev, my_state, tbb::release, release_internal_lock(), tbb::internal::spin_wait_while_eq(), tbb::STATE_COMBINED_WAITINGREADER, tbb::STATE_UPGRADE_LOSER, tbb::STATE_UPGRADE_WAITING, tbb::STATE_WRITER, sync_releasing, and try_acquire_internal_lock().

Referenced by ~scoped_lock().

Here is the call graph for this function:
Here is the caller graph for this function:

## release_internal_lock()

void tbb::queuing_rw_mutex::scoped_lock::release_internal_lock | ( |  | ) |

inline private

Release the internal lock.

Definition at line 64 of file queuing_rw_mutex.cpp.

{
__TBB_store_with_release(my_internal_lock,RELEASED);
}

References tbb::internal::__TBB_store_with_release(), and tbb::RELEASED.

Referenced by release().

Here is the call graph for this function:
Here is the caller graph for this function:

## try_acquire()

bool tbb::queuing_rw_mutex::scoped_lock::try_acquire | ( | queuing_rw_mutex & | m,
|  | bool | write = `true`
| ) |  |

Acquire lock on given mutex if free (i.e. non-blocking)

A method to acquire queuing_rw_mutex if it is free.

Definition at line 226 of file queuing_rw_mutex.cpp.

{
__TBB_ASSERT( !my_mutex, "scoped_lock is already holding a mutex");
if( load<relaxed>(m.q_tail) )
return false; // Someone already took the lock
// Must set all fields before the fetch_and_store, because once the
// fetch_and_store executes, *this becomes accessible to other threads.
__TBB_store_relaxed(my_prev, (scoped_lock*)0);
__TBB_store_relaxed(my_next, (scoped_lock*)0);
__TBB_store_relaxed(my_going, 0); // TODO: remove dead assignment?
my_state = state_t(write ? STATE_WRITER : STATE_ACTIVEREADER);
my_internal_lock = RELEASED;
// The CAS must have release semantics, because we are
// "sending" the fields initialized above to other processors.
if( m.q_tail.compare_and_swap<tbb::release>(this, NULL) )
return false; // Someone already took the lock
// Force acquire so that user's critical section receives correct values
// from processor that was previously in the user's critical section.
__TBB_load_with_acquire(my_going);
my_mutex = &m;
ITT_NOTIFY(sync_acquired, my_mutex);
return true;
}

References __TBB_ASSERT, tbb::internal::__TBB_load_with_acquire(), tbb::internal::__TBB_store_relaxed(), ITT_NOTIFY, tbb::queuing_rw_mutex::q_tail, tbb::release, tbb::RELEASED, tbb::STATE_ACTIVEREADER, and tbb::STATE_WRITER.

Here is the call graph for this function:

## try_acquire_internal_lock()

bool tbb::queuing_rw_mutex::scoped_lock::try_acquire_internal_lock | ( |  | ) |

inline private

Try to acquire the internal lock.

Returns true if lock was successfully acquired.

Definition at line 50 of file queuing_rw_mutex.cpp.

{
return as_atomic(my_internal_lock).compare_and_swap<tbb::acquire>(ACQUIRED,RELEASED) == RELEASED;
}
tbb::internal::as_atomic
atomic< T > & as_atomic(T &t)
Definition: atomic.h:572

References tbb::acquire, tbb::ACQUIRED, tbb::internal::as_atomic(), and tbb::RELEASED.

Referenced by release().

Here is the call graph for this function:
Here is the caller graph for this function:

## unblock_or_wait_on_internal_lock()

void tbb::queuing_rw_mutex::scoped_lock::unblock_or_wait_on_internal_lock | ( | uintptr_t | flag | ) |

inline private

A helper function.

Definition at line 74 of file queuing_rw_mutex.cpp.

{
if( flag )
wait_for_release_of_internal_lock();
else
release_internal_lock();
}
tbb::queuing_rw_mutex::scoped_lock::wait_for_release_of_internal_lock
void wait_for_release_of_internal_lock()
Wait for internal lock to be released.
Definition: queuing_rw_mutex.cpp:69
tbb::queuing_rw_mutex::scoped_lock::release_internal_lock
void release_internal_lock()
Release the internal lock.
Definition: queuing_rw_mutex.cpp:64

## upgrade_to_writer()

bool tbb::queuing_rw_mutex::scoped_lock::upgrade_to_writer | ( |  | ) |

Upgrade reader to become a writer.

Returns whether the upgrade happened without releasing and re-acquiring the lock

Definition at line 387 of file queuing_rw_mutex.cpp.

{
if ( my_state == STATE_WRITER ) return true; // Already a writer
queuing_rw_mutex::scoped_lock * tmp;
queuing_rw_mutex::scoped_lock * me = this;
ITT_NOTIFY(sync_releasing, my_mutex);
my_state = STATE_UPGRADE_REQUESTED;
requested:
__TBB_ASSERT( !(uintptr_t(__TBB_load_relaxed(my_next)) & FLAG), "use of corrupted pointer!" );
acquire_internal_lock();
if( this != my_mutex->q_tail.compare_and_swap<tbb::release>(tricky_pointer(me)|FLAG, this) ) {
spin_wait_while_eq( my_next, (void*)NULL );
queuing_rw_mutex::scoped_lock * n;
n = tricky_pointer::fetch_and_add<tbb::acquire>(&my_next, FLAG);
unsigned short n_state = n->my_state;
/* the next reader can be blocked by our state. the best thing to do is to unblock it */
if( n_state & STATE_COMBINED_WAITINGREADER )
__TBB_store_with_release(n->my_going,1);
tmp = tricky_pointer::fetch_and_store<tbb::release>(&(n->my_prev), this);
unblock_or_wait_on_internal_lock(get_flag(tmp));
if( n_state & (STATE_COMBINED_READER | STATE_UPGRADE_REQUESTED) ) {
// save n|FLAG for simplicity of following comparisons
tmp = tricky_pointer(n)|FLAG;
for( atomic_backoff b; __TBB_load_relaxed(my_next)==tmp; b.pause() ) {
if( my_state & STATE_COMBINED_UPGRADING ) {
if( __TBB_load_with_acquire(my_next)==tmp )
__TBB_store_relaxed(my_next, n);
goto waiting;
}
}
__TBB_ASSERT(__TBB_load_relaxed(my_next) != (tricky_pointer(n)|FLAG), NULL);
goto requested;
} else {
__TBB_ASSERT( n_state & (STATE_WRITER | STATE_UPGRADE_WAITING), "unexpected state");
__TBB_ASSERT( (tricky_pointer(n)|FLAG) == __TBB_load_relaxed(my_next), NULL);
__TBB_store_relaxed(my_next, n);
}
} else {
/* We are in the tail; whoever comes next is blocked by q_tail&FLAG */
release_internal_lock();
} // if( this != my_mutex->q_tail... )
my_state.compare_and_swap<tbb::acquire>(STATE_UPGRADE_WAITING, STATE_UPGRADE_REQUESTED);
waiting:
__TBB_ASSERT( !( intptr_t(__TBB_load_relaxed(my_next)) & FLAG ), "use of corrupted pointer!" );
__TBB_ASSERT( my_state & STATE_COMBINED_UPGRADING, "wrong state at upgrade waiting_retry" );
__TBB_ASSERT( me==this, NULL );
ITT_NOTIFY(sync_prepare, my_mutex);
/* if no one was blocked by the "corrupted" q_tail, turn it back */
my_mutex->q_tail.compare_and_swap<tbb::release>( this, tricky_pointer(me)|FLAG );
queuing_rw_mutex::scoped_lock * pred;
pred = tricky_pointer::fetch_and_add<tbb::acquire>(&my_prev, FLAG);
if( pred ) {
bool success = pred->try_acquire_internal_lock();
pred->my_state.compare_and_swap<tbb::release>(STATE_UPGRADE_WAITING, STATE_UPGRADE_REQUESTED);
if( !success ) {
tmp = tricky_pointer::compare_and_swap<tbb::release>(&my_prev, pred, tricky_pointer(pred)|FLAG );
if( uintptr_t(tmp) & FLAG ) {
spin_wait_while_eq(my_prev, pred);
pred = __TBB_load_relaxed(my_prev);
} else {
spin_wait_while_eq( my_prev, tricky_pointer(pred)|FLAG );
pred->release_internal_lock();
}
} else {
__TBB_store_relaxed(my_prev, pred);
pred->release_internal_lock();
spin_wait_while_eq(my_prev, pred);
pred = __TBB_load_relaxed(my_prev);
}
if( pred )
goto waiting;
} else {
// restore the corrupted my_prev field for possible further use (e.g. if downgrade back to reader)
__TBB_store_relaxed(my_prev, pred);
}
__TBB_ASSERT( !pred && !__TBB_load_relaxed(my_prev), NULL );
// additional lifetime issue prevention checks
// wait for the successor to finish working with my fields
wait_for_release_of_internal_lock();
// now wait for the predecessor to finish working with my fields
spin_wait_while_eq( my_going, 2 );
// Acquire critical section indirectly from previous owner or directly from predecessor (TODO: not clear).
__TBB_control_consistency_helper(); // on either "my_mutex->q_tail" or "my_going" (TODO: not clear)
bool result = ( my_state != STATE_UPGRADE_LOSER );
my_state = STATE_WRITER;
__TBB_store_relaxed(my_going, 1);
ITT_NOTIFY(sync_acquired, my_mutex);
return result;
}
tbb::STATE_COMBINED_READER
@ STATE_COMBINED_READER
Definition: queuing_rw_mutex.cpp:43
tbb::STATE_UPGRADE_REQUESTED
@ STATE_UPGRADE_REQUESTED
Definition: queuing_rw_mutex.cpp:39
tbb::STATE_COMBINED_UPGRADING
@ STATE_COMBINED_UPGRADING
Definition: queuing_rw_mutex.cpp:44

References __TBB_ASSERT, __TBB_control_consistency_helper, tbb::internal::__TBB_load_relaxed(), tbb::internal::__TBB_load_with_acquire(), tbb::internal::__TBB_store_relaxed(), tbb::internal::__TBB_store_with_release(), tbb::acquire, tbb::FLAG, tbb::get_flag(), ITT_NOTIFY, my_going, my_prev, my_state, tbb::release, tbb::internal::spin_wait_while_eq(), tbb::STATE_COMBINED_READER, tbb::STATE_COMBINED_UPGRADING, tbb::STATE_COMBINED_WAITINGREADER, tbb::STATE_UPGRADE_LOSER, tbb::STATE_UPGRADE_REQUESTED, tbb::STATE_UPGRADE_WAITING, tbb::STATE_WRITER, and sync_releasing.

Here is the call graph for this function:

## wait_for_release_of_internal_lock()

void tbb::queuing_rw_mutex::scoped_lock::wait_for_release_of_internal_lock | ( |  | ) |

inline private

Wait for internal lock to be released.

Definition at line 69 of file queuing_rw_mutex.cpp.

{
spin_wait_until_eq(my_internal_lock, RELEASED);
}

References tbb::RELEASED, and tbb::internal::spin_wait_until_eq().

Here is the call graph for this function:

## Member Data Documentation

## my_going

unsigned char __TBB_atomic tbb::queuing_rw_mutex::scoped_lock::my_going

private

The local spin-wait variable.

Corresponds to "spin" in the pseudocode but inverted for the sake of zero-initialization

Definition at line 112 of file queuing_rw_mutex.h.

Referenced by downgrade_to_reader(), initialize(), release(), and upgrade_to_writer().

## my_internal_lock

unsigned char tbb::queuing_rw_mutex::scoped_lock::my_internal_lock

private

A tiny internal lock.

Definition at line 115 of file queuing_rw_mutex.h.

Referenced by initialize(), and release().

## my_mutex

queuing_rw_mutex* tbb::queuing_rw_mutex::scoped_lock::my_mutex

private

The pointer to the mutex owned, or NULL if not holding a mutex.

Definition at line 100 of file queuing_rw_mutex.h.

Referenced by initialize(), and ~scoped_lock().

## my_next

scoped_lock* __TBB_atomic * __TBB_atomic tbb::queuing_rw_mutex::scoped_lock::my_next

private

Definition at line 103 of file queuing_rw_mutex.h.

Referenced by acquire(), initialize(), and release().

## my_prev

scoped_lock* __TBB_atomic tbb::queuing_rw_mutex::scoped_lock::my_prev

private

The pointer to the previous and next competitors for a mutex.

Definition at line 103 of file queuing_rw_mutex.h.

Referenced by initialize(), release(), and upgrade_to_writer().

## my_state

atomic<state_t> tbb::queuing_rw_mutex::scoped_lock::my_state

private

State of the request: reader, writer, active reader, other service states.

Definition at line 108 of file queuing_rw_mutex.h.

Referenced by acquire(), downgrade_to_reader(), initialize(), release(), and upgrade_to_writer().

The documentation for this class was generated from the following files:

- queuing_rw_mutex.h
- queuing_rw_mutex.cpp

Copyright © 2005-2020 Intel Corporation. All Rights Reserved.

Intel, Pentium, Intel Xeon, Itanium, Intel XScale and VTune are registered trademarks or trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

* Other names and brands may be claimed as the property of others.