tbb::internal::generic_scheduler Class Referenceabstract

Work stealing task scheduler. More...

`#include <scheduler.h>`

Inheritance diagram for tbb::internal::generic_scheduler:
Collaboration diagram for tbb::internal::generic_scheduler:

## Public Member Functions

bool | is_task_pool_published () const
bool | is_local_task_pool_quiescent () const
bool | is_quiescent_local_task_pool_empty () const
bool | is_quiescent_local_task_pool_reset () const
void | attach_mailbox (affinity_id id)
void | init_stack_info ()
| Sets up the data necessary for the stealing limiting heuristics. More...
bool | can_steal ()
| Returns true if stealing is allowed. More...
void | publish_task_pool ()
| Used by workers to enter the task pool. More...
void | leave_task_pool ()
| Leave the task pool. More...
void | reset_task_pool_and_leave ()
| Resets head and tail indices to 0, and leaves task pool. More...
task ** | lock_task_pool (arena_slot *victim_arena_slot) const
| Locks victim's task pool, and returns pointer to it. The pointer can be NULL. More...
void | unlock_task_pool (arena_slot *victim_arena_slot, task **victim_task_pool) const
| Unlocks victim's task pool. More...
void | acquire_task_pool () const
| Locks the local task pool. More...
void | release_task_pool () const
| Unlocks the local task pool. More...
task * | prepare_for_spawning (task *t)
| Checks if t is affinitized to another thread, and if so, bundles it as proxy. More...
void | commit_spawned_tasks (size_t new_tail)
| Makes newly spawned tasks visible to thieves. More...
void | commit_relocated_tasks (size_t new_tail)
| Makes relocated tasks visible to thieves and releases the local task pool. More...
task * | get_task (__TBB_ISOLATION_EXPR(isolation_tag isolation))
| Get a task from the local pool. More...
task * | get_task (size_t T)
| Get a task from the local pool at specified location T. More...
task * | get_mailbox_task (__TBB_ISOLATION_EXPR(isolation_tag isolation))
| Attempt to get a task from the mailbox. More...
task * | steal_task (__TBB_ISOLATION_EXPR(isolation_tag isolation))
| Attempts to steal a task from a randomly chosen thread/scheduler. More...
task * | steal_task_from (__TBB_ISOLATION_ARG(arena_slot &victim_arena_slot, isolation_tag isolation))
| Steal task from another scheduler's ready pool. More...
size_t | prepare_task_pool (size_t n)
| Makes sure that the task pool can accommodate at least n more elements. More...
bool | cleanup_master (bool blocking_terminate)
| Perform necessary cleanup when a master thread stops using TBB. More...
void | assert_task_pool_valid () const
void | attach_arena (arena *, size_t index, bool is_master)
void | nested_arena_entry (arena *, size_t)
void | nested_arena_exit ()
void | wait_until_empty ()
void | spawn (task &first, task *&next) __TBB_override
| For internal use only. More...
void | spawn_root_and_wait (task &first, task *&next) __TBB_override
| For internal use only. More...
void | enqueue (task &, void *reserved) __TBB_override
| For internal use only. More...
void | local_spawn (task *first, task *&next)
void | local_spawn_root_and_wait (task *first, task *&next)
virtual void | local_wait_for_all (task &parent, task *child)=0
void | destroy ()
| Destroy and deallocate this scheduler object. More...
void | cleanup_scheduler ()
| Cleans up this scheduler (the scheduler might be destroyed). More...
task & | allocate_task (size_t number_of_bytes, __TBB_CONTEXT_ARG(task *parent, task_group_context *context))
| Allocate task object, either from the heap or a free list. More...
template<free_task_hint h>
void | free_task (task &t)
| Put task on free list. More...
void | deallocate_task (task &t)
| Return task object to the memory allocator. More...
bool | is_worker () const
| True if running on a worker thread, false otherwise. More...
bool | outermost_level () const
| True if the scheduler is on the outermost dispatch level. More...
bool | master_outermost_level () const
| True if the scheduler is on the outermost dispatch level in a master thread. More...
bool | worker_outermost_level () const
| True if the scheduler is on the outermost dispatch level in a worker thread. More...
unsigned | max_threads_in_arena ()
| Returns the concurrency limit of the current arena. More...
virtual task * | receive_or_steal_task (__TBB_ISOLATION_ARG(__TBB_atomic reference_count &completion_ref_count, isolation_tag isolation))=0
| Try getting a task from other threads (via mailbox, stealing, FIFO queue, orphans adoption). More...
void | free_nonlocal_small_task (task &t)
| Free a small task t that that was allocated by a different scheduler. More...
Public Member Functions inherited from tbb::internal::scheduler
virtual void | wait_for_all (task &parent, task *child)=0
| For internal use only. More...
virtual | ~scheduler ()=0
| Pure virtual destructor;. More...

## Static Public Member Functions

static bool | is_version_3_task (task &t)
static bool | is_proxy (const task &t)
| True if t is a task_proxy. More...
static generic_scheduler * | create_master (arena *a)
| Initialize a scheduler for a master thread. More...
static generic_scheduler * | create_worker (market &m, size_t index, bool geniune)
| Initialize a scheduler for a worker thread. More...
static void | cleanup_worker (void *arg, bool worker)
| Perform necessary cleanup when a worker thread finishes. More...
static task * | plugged_return_list ()
| Special value used to mark my_return_list as not taking any more entries. More...

## Public Attributes

uintptr_t | my_stealing_threshold
| Position in the call stack specifying its maximal filling when stealing is still allowed. More...
market * | my_market
| The market I am in. More...
FastRandom | my_random
| Random number generator used for picking a random victim from which to steal. More...
task * | my_free_list
| Free list of small tasks that can be reused. More...
task * | my_dummy_task
| Fake root task created by slave threads. More...
long | my_ref_count
| Reference count for scheduler. More...
bool | my_auto_initialized
| True if *this was created by automatic TBB initialization. More...
__TBB_atomic intptr_t | my_small_task_count
| Number of small tasks that have been allocated by this scheduler. More...
task * | my_return_list
| List of small tasks that have been returned to this scheduler by other schedulers. More...
Public Attributes inherited from tbb::internal::intrusive_list_node
intrusive_list_node * | my_prev_node
intrusive_list_node * | my_next_node
Public Attributes inherited from tbb::internal::scheduler_state
size_t | my_arena_index
| Index of the arena slot the scheduler occupies now, or occupied last time. More...
arena_slot * | my_arena_slot
| Pointer to the slot in the arena we own at the moment. More...
arena * | my_arena
| The arena that I own (if master) or am servicing at the moment (if worker) More...
task * | my_innermost_running_task
| Innermost task whose task::execute() is running. A dummy task on the outermost level. More...
mail_inbox | my_inbox
affinity_id | my_affinity_id
| The mailbox id assigned to this scheduler. More...
scheduler_properties | my_properties

## Static Public Attributes

static const size_t | quick_task_size = 256-task_prefix_reservation_size
| If sizeof(task) is <=quick_task_size, it is handled on a free list instead of malloc'd. More...
static const size_t | null_arena_index = ~size_t(0)
static const size_t | min_task_pool_size = 64

## Protected Member Functions

| generic_scheduler (market &, bool)

## Friends

template<typename SchedulerTraits >
class | custom_scheduler

## Detailed Description

Work stealing task scheduler.

None of the fields here are ever read or written by threads other than the thread that creates the instance.

Class generic_scheduler is an abstract base class that contains most of the scheduler, except for tweaks specific to processors and tools (e.g. VTune(TM) Performance Tools). The derived template class custom_scheduler<SchedulerTraits> fills in the tweaks.

Definition at line 137 of file scheduler.h.

## Constructor & Destructor Documentation

## generic_scheduler()

tbb::internal::generic_scheduler::generic_scheduler | ( | market & | m,
|  | bool | genuine
| ) |  |

protected

Definition at line 84 of file scheduler.cpp.

: my_market(&m)
, my_random(this)
, my_ref_count(1)
#if __TBB_PREVIEW_RESUMABLE_TASKS
, my_co_context(m.worker_stack_size(), genuine ? NULL : this)
#endif
, my_small_task_count(1) // Extra 1 is a guard reference
#if __TBB_SURVIVE_THREAD_SWITCH && TBB_USE_ASSERT
, my_cilk_state(cs_none)
#endif /* __TBB_SURVIVE_THREAD_SWITCH && TBB_USE_ASSERT */
{
__TBB_ASSERT( !my_arena_index, "constructor expects the memory being zero-initialized" );
__TBB_ASSERT( governor::is_set(NULL), "scheduler is already initialized for this thread" );
my_innermost_running_task = my_dummy_task = &allocate_task( sizeof(task), __TBB_CONTEXT_ARG(NULL, &the_dummy_context) );
#if __TBB_PREVIEW_CRITICAL_TASKS
my_properties.has_taken_critical_task = false;
#endif
#if __TBB_PREVIEW_RESUMABLE_TASKS
my_properties.genuine = genuine;
my_current_is_recalled = NULL;
my_post_resume_action = PRA_NONE;
my_post_resume_arg = NULL;
my_wait_task = NULL;
#else
suppress_unused_warning(genuine);
#endif
my_properties.outermost = true;
#if __TBB_TASK_PRIORITY
my_ref_top_priority = &m.my_global_top_priority;
my_ref_reload_epoch = &m.my_global_reload_epoch;
#endif /* __TBB_TASK_PRIORITY */
#if __TBB_TASK_GROUP_CONTEXT
// Sync up the local cancellation state with the global one. No need for fence here.
my_context_state_propagation_epoch = the_context_state_propagation_epoch;
my_context_list_head.my_prev = &my_context_list_head;
my_context_list_head.my_next = &my_context_list_head;
ITT_SYNC_CREATE(&my_context_list_mutex, SyncType_Scheduler, SyncObj_ContextsList);
#endif /* __TBB_TASK_GROUP_CONTEXT */
ITT_SYNC_CREATE(&my_dummy_task->prefix().ref_count, SyncType_Scheduler, SyncObj_WorkerLifeCycleMgmt);
ITT_SYNC_CREATE(&my_return_list, SyncType_Scheduler, SyncObj_TaskReturnList);
}
__TBB_ASSERT
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:165
ITT_SYNC_CREATE
#define ITT_SYNC_CREATE(obj, type, name)
Definition: itt_notify.h:115
__TBB_CONTEXT_ARG
#define __TBB_CONTEXT_ARG(arg1, context)
Definition: scheduler_common.h:60
task
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task * task
Definition: ittnotify_static.h:119
tbb::internal::suppress_unused_warning
void suppress_unused_warning(const T1 &)
Utility template function to prevent "unused" warnings by various compilers.
Definition: tbb_stddef.h:398
tbb::internal::task_prefix::ref_count
__TBB_atomic reference_count ref_count
Reference count used for synchronization.
Definition: task.h:274
tbb::task::prefix
internal::task_prefix & prefix(internal::version_tag *=NULL) const
Get reference to corresponding task_prefix.
Definition: task.h:1002
tbb::internal::governor::is_set
static bool is_set(generic_scheduler *s)
Used to check validity of the local scheduler TLS contents.
Definition: governor.cpp:120
tbb::internal::scheduler_properties::outermost
bool outermost
Indicates that a scheduler is on outermost level.
Definition: scheduler.h:57
tbb::internal::scheduler_state::my_arena_index
size_t my_arena_index
Index of the arena slot the scheduler occupies now, or occupied last time.
Definition: scheduler.h:79
tbb::internal::scheduler_state::my_innermost_running_task
task * my_innermost_running_task
Innermost task whose task::execute() is running. A dummy task on the outermost level.
Definition: scheduler.h:88
tbb::internal::scheduler_state::my_properties
scheduler_properties my_properties
Definition: scheduler.h:101
tbb::internal::generic_scheduler::my_small_task_count
__TBB_atomic intptr_t my_small_task_count
Number of small tasks that have been allocated by this scheduler.
Definition: scheduler.h:461
tbb::internal::generic_scheduler::my_ref_count
long my_ref_count
Reference count for scheduler.
Definition: scheduler.h:190
tbb::internal::generic_scheduler::allocate_task
task & allocate_task(size_t number_of_bytes, __TBB_CONTEXT_ARG(task *parent, task_group_context *context))
Allocate task object, either from the heap or a free list.
Definition: scheduler.cpp:337
tbb::internal::generic_scheduler::my_return_list
task * my_return_list
List of small tasks that have been returned to this scheduler by other schedulers.
Definition: scheduler.h:465
tbb::internal::generic_scheduler::my_dummy_task
task * my_dummy_task
Fake root task created by slave threads.
Definition: scheduler.h:186
tbb::internal::generic_scheduler::my_market
market * my_market
The market I am in.
Definition: scheduler.h:172
tbb::internal::generic_scheduler::my_random
FastRandom my_random
Random number generator used for picking a random victim from which to steal.
Definition: scheduler.h:175

References __TBB_ASSERT, __TBB_CONTEXT_ARG, allocate_task(), tbb::internal::governor::is_set(), ITT_SYNC_CREATE, tbb::internal::scheduler_state::my_arena_index, my_dummy_task, tbb::internal::scheduler_state::my_innermost_running_task, tbb::internal::scheduler_state::my_properties, my_return_list, tbb::internal::scheduler_properties::outermost, tbb::task::prefix(), tbb::internal::task_prefix::ref_count, and tbb::internal::suppress_unused_warning().

Here is the call graph for this function:

## Member Function Documentation

## acquire_task_pool()

void tbb::internal::generic_scheduler::acquire_task_pool | ( |  | ) | const

inline

Locks the local task pool.

Garbles my_arena_slot->task_pool for the duration of the lock. Requires correctly set my_arena_slot->task_pool_ptr.

ATTENTION: This method is mostly the same as generic_scheduler::lock_task_pool(), with a little different logic of slot state checks (slot is either locked or points to our task pool). Thus if either of them is changed, consider changing the counterpart as well.

Definition at line 493 of file scheduler.cpp.

{
if ( !is_task_pool_published() )
return; // we are not in arena - nothing to lock
bool sync_prepare_done = false;
for( atomic_backoff b;;b.pause() ) {
#if TBB_USE_ASSERT
__TBB_ASSERT( my_arena_slot == my_arena->my_slots + my_arena_index, "invalid arena slot index" );
// Local copy of the arena slot task pool pointer is necessary for the next
// assertion to work correctly to exclude asynchronous state transition effect.
task** tp = my_arena_slot->task_pool;
__TBB_ASSERT( tp == LockedTaskPool || tp == my_arena_slot->task_pool_ptr, "slot ownership corrupt?" );
#endif
if( my_arena_slot->task_pool != LockedTaskPool &&
as_atomic(my_arena_slot->task_pool).compare_and_swap(LockedTaskPool, my_arena_slot->task_pool_ptr ) == my_arena_slot->task_pool_ptr )
{
// We acquired our own slot
ITT_NOTIFY(sync_acquired, my_arena_slot);
break;
}
else if( !sync_prepare_done ) {
// Start waiting
ITT_NOTIFY(sync_prepare, my_arena_slot);
sync_prepare_done = true;
}
// Someone else acquired a lock, so pause and do exponential backoff.
}
__TBB_ASSERT( my_arena_slot->task_pool == LockedTaskPool, "not really acquired task pool" );
} // generic_scheduler::acquire_task_pool
ITT_NOTIFY
#define ITT_NOTIFY(name, obj)
Definition: itt_notify.h:112
LockedTaskPool
#define LockedTaskPool
Definition: scheduler.h:47
tbb::internal::as_atomic
atomic< T > & as_atomic(T &t)
Definition: atomic.h:572
tbb::internal::arena::my_slots
arena_slot my_slots[1]
Definition: arena.h:390
tbb::internal::scheduler_state::my_arena
arena * my_arena
The arena that I own (if master) or am servicing at the moment (if worker)
Definition: scheduler.h:85
tbb::internal::scheduler_state::my_arena_slot
arena_slot * my_arena_slot
Pointer to the slot in the arena we own at the moment.
Definition: scheduler.h:82
tbb::internal::generic_scheduler::is_task_pool_published
bool is_task_pool_published() const
Definition: scheduler.h:628

References __TBB_ASSERT, tbb::internal::as_atomic(), is_task_pool_published(), ITT_NOTIFY, LockedTaskPool, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_index, tbb::internal::scheduler_state::my_arena_slot, tbb::internal::arena::my_slots, and tbb::internal::atomic_backoff::pause().

Referenced by cleanup_master(), get_task(), and prepare_task_pool().

Here is the call graph for this function:
Here is the caller graph for this function:

## allocate_task()

task & tbb::internal::generic_scheduler::allocate_task | ( | size_t | number_of_bytes,
|  | __TBB_CONTEXT_ARG(task *parent, task_group_context *context) |
| ) |  |

Allocate task object, either from the heap or a free list.

Returns uninitialized task object with initialized prefix.

Definition at line 337 of file scheduler.cpp.

{
GATHER_STATISTIC(++my_counters.active_tasks);
task *t;
if( number_of_bytes<=quick_task_size ) {
#if __TBB_HOARD_NONLOCAL_TASKS
if( (t = my_nonlocal_free_list) ) {
GATHER_STATISTIC(--my_counters.free_list_length);
__TBB_ASSERT( t->state()==task::freed, "free list of tasks is corrupted" );
my_nonlocal_free_list = t->prefix().next;
} else
#endif
if( (t = my_free_list) ) {
GATHER_STATISTIC(--my_counters.free_list_length);
__TBB_ASSERT( t->state()==task::freed, "free list of tasks is corrupted" );
my_free_list = t->prefix().next;
} else if( my_return_list ) {
// No fence required for read of my_return_list above, because __TBB_FetchAndStoreW has a fence.
t = (task*)__TBB_FetchAndStoreW( &my_return_list, 0 ); // with acquire
__TBB_ASSERT( t, "another thread emptied the my_return_list" );
__TBB_ASSERT( t->prefix().origin==this, "task returned to wrong my_return_list" );
ITT_NOTIFY( sync_acquired, &my_return_list );
my_free_list = t->prefix().next;
} else {
t = (task*)((char*)NFS_Allocate( 1, task_prefix_reservation_size+quick_task_size, NULL ) + task_prefix_reservation_size );
#if __TBB_COUNT_TASK_NODES
++my_task_node_count;
#endif /* __TBB_COUNT_TASK_NODES */
t->prefix().origin = this;
t->prefix().next = 0;
++my_small_task_count;
}
#if __TBB_PREFETCHING
task *t_next = t->prefix().next;
if( !t_next ) { // the task was last in the list
#if __TBB_HOARD_NONLOCAL_TASKS
if( my_free_list )
t_next = my_free_list;
else
#endif
if( my_return_list ) // enable prefetching, gives speedup
t_next = my_free_list = (task*)__TBB_FetchAndStoreW( &my_return_list, 0 );
}
if( t_next ) { // gives speedup for both cache lines
__TBB_cl_prefetch(t_next);
__TBB_cl_prefetch(&t_next->prefix());
}
#endif /* __TBB_PREFETCHING */
} else {
GATHER_STATISTIC(++my_counters.big_tasks);
t = (task*)((char*)NFS_Allocate( 1, task_prefix_reservation_size+number_of_bytes, NULL ) + task_prefix_reservation_size );
#if __TBB_COUNT_TASK_NODES
++my_task_node_count;
#endif /* __TBB_COUNT_TASK_NODES */
t->prefix().origin = NULL;
}
task_prefix& p = t->prefix();
#if __TBB_TASK_GROUP_CONTEXT
p.context = context;
#endif /* __TBB_TASK_GROUP_CONTEXT */
// Obsolete. But still in use, so has to be assigned correct value here.
p.owner = this;
p.ref_count = 0;
// Obsolete. Assign some not outrageously out-of-place value for a while.
p.depth = 0;
p.parent = parent;
// In TBB 2.1 and later, the constructor for task sets extra_state to indicate the version of the tbb/task.h header.
// In TBB 2.0 and earlier, the constructor leaves extra_state as zero.
p.extra_state = 0;
p.affinity = 0;
p.state = task::allocated;
__TBB_ISOLATION_EXPR( p.isolation = no_isolation );
return *t;
}
__TBB_cl_prefetch
#define __TBB_cl_prefetch(p)
Definition: mic_common.h:33
__TBB_ISOLATION_EXPR
#define __TBB_ISOLATION_EXPR(isolation)
Definition: scheduler_common.h:67
GATHER_STATISTIC
#define GATHER_STATISTIC(x)
Definition: tbb_statistics.h:232
parent
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task __itt_model_task_instance ITT_FORMAT p void ITT_FORMAT p void ITT_FORMAT p void size_t ITT_FORMAT d void ITT_FORMAT p const wchar_t ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s no args void ITT_FORMAT p size_t ITT_FORMAT d no args const wchar_t const wchar_t ITT_FORMAT s __itt_heap_function void size_t int ITT_FORMAT d __itt_heap_function void ITT_FORMAT p __itt_heap_function void void size_t int ITT_FORMAT d no args no args unsigned int ITT_FORMAT u const __itt_domain __itt_id ITT_FORMAT lu const __itt_domain __itt_id __itt_id parent
Definition: ittnotify_static.h:176
p
void const char const char int ITT_FORMAT __itt_group_sync p
Definition: ittnotify_static.h:91
tbb::internal::NFS_Allocate
void *__TBB_EXPORTED_FUNC NFS_Allocate(size_t n_element, size_t element_size, void *hint)
Allocate memory on cache/sector line boundary.
Definition: cache_aligned_allocator.cpp:176
tbb::internal::task_prefix_reservation_size
const size_t task_prefix_reservation_size
Number of bytes reserved for a task prefix.
Definition: scheduler_common.h:159
tbb::internal::no_isolation
const isolation_tag no_isolation
Definition: task.h:144
tbb::internal::task_prefix::next
tbb::task * next
"next" field for list of task
Definition: task.h:297
tbb::internal::task_prefix::origin
scheduler * origin
The scheduler that allocated the task, or NULL if the task is big.
Definition: task.h:239
tbb::task::allocated
@ allocated
task object is freshly allocated or recycled.
Definition: task.h:643
tbb::task::freed
@ freed
task object is on free list, or is going to be put there, or was just taken off.
Definition: task.h:645
tbb::internal::generic_scheduler::my_free_list
task * my_free_list
Free list of small tasks that can be reused.
Definition: scheduler.h:178
tbb::internal::generic_scheduler::quick_task_size
static const size_t quick_task_size
If sizeof(task) is <=quick_task_size, it is handled on a free list instead of malloc'd.
Definition: scheduler.h:144

References __TBB_ASSERT, __TBB_cl_prefetch, __TBB_ISOLATION_EXPR, tbb::task::allocated, tbb::task::freed, GATHER_STATISTIC, ITT_NOTIFY, my_free_list, my_return_list, my_small_task_count, tbb::internal::task_prefix::next, tbb::internal::NFS_Allocate(), tbb::internal::no_isolation, tbb::internal::task_prefix::origin, p, parent, tbb::task::prefix(), quick_task_size, tbb::task::state(), and tbb::internal::task_prefix_reservation_size.

Referenced by tbb::internal::allocate_root_proxy::allocate(), generic_scheduler(), and prepare_for_spawning().

Here is the call graph for this function:
Here is the caller graph for this function:

## assert_task_pool_valid()

void tbb::internal::generic_scheduler::assert_task_pool_valid | ( |  | ) | const

inline

Definition at line 398 of file scheduler.h.

{}

Referenced by local_spawn(), prepare_task_pool(), and tbb::task::self().

Here is the caller graph for this function:

## attach_arena()

void tbb::internal::generic_scheduler::attach_arena | ( | arena * | a,
|  | size_t | index,
|  | bool | is_master
| ) |  |

Definition at line 80 of file arena.cpp.

{
__TBB_ASSERT( a->my_market == my_market, NULL );
my_arena = a;
my_arena_index = index;
my_arena_slot = a->my_slots + index;
attach_mailbox( affinity_id(index+1) );
if ( is_master && my_inbox.is_idle_state( true ) ) {
// Master enters an arena with its own task to be executed. It means that master is not
// going to enter stealing loop and take affinity tasks.
my_inbox.set_is_idle( false );
}
#if __TBB_TASK_GROUP_CONTEXT
// Context to be used by root tasks by default (if the user has not specified one).
if( !is_master )
my_dummy_task->prefix().context = a->my_default_ctx;
#endif /* __TBB_TASK_GROUP_CONTEXT */
#if __TBB_TASK_PRIORITY
// In the current implementation master threads continue processing even when
// there are other masters with higher priority. Only TBB worker threads are
// redistributed between arenas based on the latters' priority. Thus master
// threads use arena's top priority as a reference point (in contrast to workers
// that use my_market->my_global_top_priority).
if( is_master ) {
my_ref_top_priority = &a->my_top_priority;
my_ref_reload_epoch = &a->my_reload_epoch;
}
my_local_reload_epoch = *my_ref_reload_epoch;
__TBB_ASSERT( !my_offloaded_tasks, NULL );
#endif /* __TBB_TASK_PRIORITY */
}
tbb::internal::affinity_id
unsigned short affinity_id
An id as used for specifying affinity.
Definition: task.h:139
tbb::internal::task_prefix::context
task_group_context * context
Shared context that is used to communicate asynchronous state changes.
Definition: task.h:230
tbb::internal::mail_inbox::set_is_idle
void set_is_idle(bool value)
Indicate whether thread that reads this mailbox is idle.
Definition: mailbox.h:222
tbb::internal::mail_inbox::is_idle_state
bool is_idle_state(bool value) const
Indicate whether thread that reads this mailbox is idle.
Definition: mailbox.h:229
tbb::internal::scheduler_state::my_inbox
mail_inbox my_inbox
Definition: scheduler.h:90
tbb::internal::generic_scheduler::attach_mailbox
void attach_mailbox(affinity_id id)
Definition: scheduler.h:667

References __TBB_ASSERT, attach_mailbox(), tbb::internal::task_prefix::context, tbb::internal::mail_inbox::is_idle_state(), tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_index, tbb::internal::scheduler_state::my_arena_slot, my_dummy_task, tbb::internal::scheduler_state::my_inbox, my_market, tbb::internal::arena::my_slots, tbb::task::prefix(), and tbb::internal::mail_inbox::set_is_idle().

Referenced by nested_arena_entry().

Here is the call graph for this function:
Here is the caller graph for this function:

## attach_mailbox()

void tbb::internal::generic_scheduler::attach_mailbox | ( | affinity_id | id | ) |

inline

Definition at line 667 of file scheduler.h.

{
__TBB_ASSERT(id>0,NULL);
my_inbox.attach( my_arena->mailbox(id) );
my_affinity_id = id;
}
id
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task __itt_model_task_instance ITT_FORMAT p void ITT_FORMAT p void ITT_FORMAT p void size_t ITT_FORMAT d void ITT_FORMAT p const wchar_t ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s no args void ITT_FORMAT p size_t ITT_FORMAT d no args const wchar_t const wchar_t ITT_FORMAT s __itt_heap_function void size_t int ITT_FORMAT d __itt_heap_function void ITT_FORMAT p __itt_heap_function void void size_t int ITT_FORMAT d no args no args unsigned int ITT_FORMAT u const __itt_domain __itt_id id
Definition: ittnotify_static.h:172
tbb::internal::arena::mailbox
mail_outbox & mailbox(affinity_id id)
Get reference to mailbox corresponding to given affinity_id.
Definition: arena.h:305
tbb::internal::mail_inbox::attach
void attach(mail_outbox &putter)
Attach inbox to a corresponding outbox.
Definition: mailbox.h:204
tbb::internal::scheduler_state::my_affinity_id
affinity_id my_affinity_id
The mailbox id assigned to this scheduler.
Definition: scheduler.h:99

References __TBB_ASSERT, tbb::internal::mail_inbox::attach(), id, tbb::internal::arena::mailbox(), tbb::internal::scheduler_state::my_affinity_id, tbb::internal::scheduler_state::my_arena, and tbb::internal::scheduler_state::my_inbox.

Referenced by attach_arena().

Here is the call graph for this function:
Here is the caller graph for this function:

## can_steal()

bool tbb::internal::generic_scheduler::can_steal | ( |  | ) |

inline

Returns true if stealing is allowed.

Definition at line 270 of file scheduler.h.

{
int anchor;
// TODO IDEA: Add performance warning?
#if __TBB_ipf
return my_stealing_threshold < (uintptr_t)&anchor && (uintptr_t)__TBB_get_bsp() < my_rsb_stealing_threshold;
#else
return my_stealing_threshold < (uintptr_t)&anchor;
#endif
}
__TBB_get_bsp
void * __TBB_get_bsp()
Retrieves the current RSE backing store pointer. IA64 specific.
tbb::internal::generic_scheduler::my_stealing_threshold
uintptr_t my_stealing_threshold
Position in the call stack specifying its maximal filling when stealing is still allowed.
Definition: scheduler.h:155

References __TBB_get_bsp(), and my_stealing_threshold.

Here is the call graph for this function:

## cleanup_master()

bool tbb::internal::generic_scheduler::cleanup_master | ( | bool | blocking_terminate | ) |

Perform necessary cleanup when a master thread stops using TBB.

Definition at line 1341 of file scheduler.cpp.

{
arena* const a = my_arena;
market * const m = my_market;
__TBB_ASSERT( my_market, NULL );
if( a && is_task_pool_published() ) {
acquire_task_pool();
if ( my_arena_slot->task_pool == EmptyTaskPool ||
__TBB_load_relaxed(my_arena_slot->head) >= __TBB_load_relaxed(my_arena_slot->tail) )
{
// Local task pool is empty
leave_task_pool();
}
else {
// Master's local task pool may e.g. contain proxies of affinitized tasks.
release_task_pool();
__TBB_ASSERT ( governor::is_set(this), "TLS slot is cleared before the task pool cleanup" );
// Set refcount to make the following dispach loop infinite (it is interrupted by the cleanup logic).
my_dummy_task->set_ref_count(2);
local_wait_for_all( *my_dummy_task, NULL );
__TBB_ASSERT( !is_task_pool_published(), NULL );
__TBB_ASSERT ( governor::is_set(this), "Other thread reused our TLS key during the task pool cleanup" );
}
}
#if __TBB_ARENA_OBSERVER
if( a )
a->my_observers.notify_exit_observers( my_last_local_observer, /*worker=*/false );
#endif
#if __TBB_SCHEDULER_OBSERVER
the_global_observer_list.notify_exit_observers( my_last_global_observer, /*worker=*/false );
#endif /* __TBB_SCHEDULER_OBSERVER */
#if _WIN32||_WIN64
m->unregister_master( master_exec_resource );
#endif /* _WIN32||_WIN64 */
if( a ) {
__TBB_ASSERT(a->my_slots+0 == my_arena_slot, NULL);
#if __TBB_STATISTICS
*my_arena_slot->my_counters += my_counters;
#endif /* __TBB_STATISTICS */
__TBB_store_with_release(my_arena_slot->my_scheduler, (generic_scheduler*)NULL);
}
#if __TBB_TASK_GROUP_CONTEXT
else { // task_group_context ownership was not transferred to arena
default_context()->~task_group_context();
NFS_Free(default_context());
}
context_state_propagation_mutex_type::scoped_lock lock(the_context_state_propagation_mutex);
my_market->my_masters.remove( *this );
lock.release();
#endif /* __TBB_TASK_GROUP_CONTEXT */
my_arena_slot = NULL; // detached from slot
cleanup_scheduler(); // do not use scheduler state after this point
if( a )
a->on_thread_leaving<arena::ref_external>();
// If there was an associated arena, it added a public market reference
return m->release( /*is_public*/ a != NULL, blocking_terminate );
}
EmptyTaskPool
#define EmptyTaskPool
Definition: scheduler.h:46
lock
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task __itt_model_task_instance ITT_FORMAT p void * lock
Definition: ittnotify_static.h:121
tbb::internal::NFS_Free
void __TBB_EXPORTED_FUNC NFS_Free(void *)
Free memory allocated by NFS_Allocate.
Definition: cache_aligned_allocator.cpp:198
tbb::internal::__TBB_load_relaxed
T __TBB_load_relaxed(const volatile T &location)
Definition: tbb_machine.h:738
tbb::internal::__TBB_store_with_release
void __TBB_store_with_release(volatile T &location, V value)
Definition: tbb_machine.h:716
tbb::task::set_ref_count
void set_ref_count(int count)
Set reference count.
Definition: task.h:761
tbb::internal::arena::ref_external
static const unsigned ref_external
Reference increment values for externals and workers.
Definition: arena.h:327
tbb::internal::generic_scheduler::local_wait_for_all
virtual void local_wait_for_all(task &parent, task *child)=0
tbb::internal::generic_scheduler::release_task_pool
void release_task_pool() const
Unlocks the local task pool.
Definition: scheduler.cpp:522
tbb::internal::generic_scheduler::generic_scheduler
generic_scheduler(market &, bool)
Definition: scheduler.cpp:84
tbb::internal::generic_scheduler::cleanup_scheduler
void cleanup_scheduler()
Cleans up this scheduler (the scheduler might be destroyed).
Definition: scheduler.cpp:294
tbb::internal::generic_scheduler::leave_task_pool
void leave_task_pool()
Leave the task pool.
Definition: scheduler.cpp:1260
tbb::internal::generic_scheduler::acquire_task_pool
void acquire_task_pool() const
Locks the local task pool.
Definition: scheduler.cpp:493

References __TBB_ASSERT, tbb::internal::__TBB_load_relaxed(), tbb::internal::__TBB_store_with_release(), acquire_task_pool(), cleanup_scheduler(), EmptyTaskPool, tbb::internal::governor::is_set(), is_task_pool_published(), leave_task_pool(), local_wait_for_all(), lock, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_slot, my_dummy_task, my_market, tbb::internal::arena::my_slots, tbb::internal::NFS_Free(), tbb::internal::arena::on_thread_leaving(), tbb::internal::arena::ref_external, tbb::internal::market::release(), release_task_pool(), and tbb::task::set_ref_count().

Here is the call graph for this function:

## cleanup_scheduler()

void tbb::internal::generic_scheduler::cleanup_scheduler | ( |  | ) |

Cleans up this scheduler (the scheduler might be destroyed).

Definition at line 294 of file scheduler.cpp.

{
__TBB_ASSERT( !my_arena_slot, NULL );
#if __TBB_TASK_PRIORITY
__TBB_ASSERT( my_offloaded_tasks == NULL, NULL );
#endif
#if __TBB_PREVIEW_CRITICAL_TASKS
__TBB_ASSERT( !my_properties.has_taken_critical_task, "Critical tasks miscount." );
#endif
#if __TBB_TASK_GROUP_CONTEXT
cleanup_local_context_list();
#endif /* __TBB_TASK_GROUP_CONTEXT */
free_task<small_local_task>( *my_dummy_task );
#if __TBB_HOARD_NONLOCAL_TASKS
while( task* t = my_nonlocal_free_list ) {
task_prefix& p = t->prefix();
my_nonlocal_free_list = p.next;
__TBB_ASSERT( p.origin && p.origin!=this, NULL );
free_nonlocal_small_task(*t);
}
#endif
// k accounts for a guard reference and each task that we deallocate.
intptr_t k = 1;
for(;;) {
while( task* t = my_free_list ) {
my_free_list = t->prefix().next;
deallocate_task(*t);
++k;
}
if( my_return_list==plugged_return_list() )
break;
my_free_list = (task*)__TBB_FetchAndStoreW( &my_return_list, (intptr_t)plugged_return_list() );
}
#if __TBB_COUNT_TASK_NODES
my_market->update_task_node_count( my_task_node_count );
#endif /* __TBB_COUNT_TASK_NODES */
// Update my_small_task_count last. Doing so sooner might cause another thread to free *this.
__TBB_ASSERT( my_small_task_count>=k, "my_small_task_count corrupted" );
governor::sign_off(this);
if( __TBB_FetchAndAddW( &my_small_task_count, -k )==k )
destroy();
}
tbb::internal::governor::sign_off
static void sign_off(generic_scheduler *s)
Unregister TBB scheduler instance from thread-local storage.
Definition: governor.cpp:145
tbb::internal::generic_scheduler::free_nonlocal_small_task
void free_nonlocal_small_task(task &t)
Free a small task t that that was allocated by a different scheduler.
Definition: scheduler.cpp:412
tbb::internal::generic_scheduler::destroy
void destroy()
Destroy and deallocate this scheduler object.
Definition: scheduler.cpp:285
tbb::internal::generic_scheduler::plugged_return_list
static task * plugged_return_list()
Special value used to mark my_return_list as not taking any more entries.
Definition: scheduler.h:458
tbb::internal::generic_scheduler::deallocate_task
void deallocate_task(task &t)
Return task object to the memory allocator.
Definition: scheduler.h:683

References __TBB_ASSERT, deallocate_task(), destroy(), free_nonlocal_small_task(), tbb::internal::scheduler_state::my_arena_slot, my_dummy_task, my_free_list, my_market, tbb::internal::scheduler_state::my_properties, my_return_list, my_small_task_count, tbb::internal::task_prefix::next, p, plugged_return_list(), tbb::task::prefix(), and tbb::internal::governor::sign_off().

Referenced by cleanup_master().

Here is the call graph for this function:
Here is the caller graph for this function:

## cleanup_worker()

void tbb::internal::generic_scheduler::cleanup_worker | ( | void * | arg,
|  | bool | worker
| ) |  |

static

Perform necessary cleanup when a worker thread finishes.

Definition at line 1331 of file scheduler.cpp.

{
generic_scheduler& s = *(generic_scheduler*)arg;
__TBB_ASSERT( !s.my_arena_slot, "cleaning up attached worker" );
#if __TBB_SCHEDULER_OBSERVER
if ( worker ) // can be called by master for worker, do not notify master twice
the_global_observer_list.notify_exit_observers( s.my_last_global_observer, /*worker=*/true );
#endif /* __TBB_SCHEDULER_OBSERVER */
s.cleanup_scheduler();
}
s
void const char const char int ITT_FORMAT __itt_group_sync s
Definition: ittnotify_static.h:91

References __TBB_ASSERT, and s.

Referenced by tbb::internal::market::cleanup().

Here is the caller graph for this function:

## commit_relocated_tasks()

void tbb::internal::generic_scheduler::commit_relocated_tasks | ( | size_t | new_tail | ) |

inline

Makes relocated tasks visible to thieves and releases the local task pool.

Obviously, the task pool must be locked when calling this method.

Definition at line 719 of file scheduler.h.

{
__TBB_ASSERT( is_local_task_pool_quiescent(),
"Task pool must be locked when calling commit_relocated_tasks()" );
__TBB_store_relaxed( my_arena_slot->head, 0 );
// Tail is updated last to minimize probability of a thread making arena
// snapshot being misguided into thinking that this task pool is empty.
__TBB_store_release( my_arena_slot->tail, new_tail );
release_task_pool();
}
__TBB_store_release
#define __TBB_store_release
Definition: tbb_machine.h:860
tbb::internal::__TBB_store_relaxed
void __TBB_store_relaxed(volatile T &location, V value)
Definition: tbb_machine.h:742
tbb::internal::generic_scheduler::is_local_task_pool_quiescent
bool is_local_task_pool_quiescent() const
Definition: scheduler.h:633

References __TBB_ASSERT, tbb::internal::__TBB_store_relaxed(), __TBB_store_release, is_local_task_pool_quiescent(), tbb::internal::scheduler_state::my_arena_slot, and release_task_pool().

Referenced by prepare_task_pool().

Here is the call graph for this function:
Here is the caller graph for this function:

## commit_spawned_tasks()

void tbb::internal::generic_scheduler::commit_spawned_tasks | ( | size_t | new_tail | ) |

inline

Makes newly spawned tasks visible to thieves.

Definition at line 710 of file scheduler.h.

{
__TBB_ASSERT ( new_tail <= my_arena_slot->my_task_pool_size, "task deque end was overwritten" );
// emit "task was released" signal
ITT_NOTIFY(sync_releasing, (void*)((uintptr_t)my_arena_slot+sizeof(uintptr_t)));
// Release fence is necessary to make sure that previously stored task pointers
// are visible to thieves.
__TBB_store_with_release( my_arena_slot->tail, new_tail );
}
sync_releasing
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p sync_releasing
Definition: ittnotify_static.h:104

References __TBB_ASSERT, tbb::internal::__TBB_store_with_release(), ITT_NOTIFY, tbb::internal::scheduler_state::my_arena_slot, and sync_releasing.

Referenced by local_spawn().

Here is the call graph for this function:
Here is the caller graph for this function:

## create_master()

generic_scheduler * tbb::internal::generic_scheduler::create_master | ( | arena * | a | ) |

static

Initialize a scheduler for a master thread.

Definition at line 1287 of file scheduler.cpp.

{
// add an internal market reference; the public reference is possibly added in create_arena
generic_scheduler* s = allocate_scheduler( market::global_market(/*is_public=*/false), /* genuine = */ true );
__TBB_ASSERT( !s->my_arena, NULL );
__TBB_ASSERT( s->my_market, NULL );
task& t = *s->my_dummy_task;
s->my_properties.type = scheduler_properties::master;
t.prefix().ref_count = 1;
#if __TBB_TASK_GROUP_CONTEXT
t.prefix().context = new ( NFS_Allocate(1, sizeof(task_group_context), NULL) )
task_group_context( task_group_context::isolated, task_group_context::default_traits );
#if __TBB_FP_CONTEXT
s->default_context()->capture_fp_settings();
#endif
// Do not call init_stack_info before the scheduler is set as master or worker.
s->init_stack_info();
context_state_propagation_mutex_type::scoped_lock lock(the_context_state_propagation_mutex);
s->my_market->my_masters.push_front( *s );
lock.release();
#endif /* __TBB_TASK_GROUP_CONTEXT */
if( a ) {
// Master thread always occupies the first slot
s->attach_arena( a, /*index*/0, /*is_master*/true );
s->my_arena_slot->my_scheduler = s;
#if __TBB_TASK_GROUP_CONTEXT
a->my_default_ctx = s->default_context(); // also transfers implied ownership
#endif
}
__TBB_ASSERT( s->my_arena_index == 0, "Master thread must occupy the first slot in its arena" );
governor::sign_on(s);
#if _WIN32||_WIN64
s->my_market->register_master( s->master_exec_resource );
#endif /* _WIN32||_WIN64 */
// Process any existing observers.
#if __TBB_ARENA_OBSERVER
__TBB_ASSERT( !a || a->my_observers.empty(), "Just created arena cannot have any observers associated with it" );
#endif
#if __TBB_SCHEDULER_OBSERVER
the_global_observer_list.notify_entry_observers( s->my_last_global_observer, /*worker=*/false );
#endif /* __TBB_SCHEDULER_OBSERVER */
return s;
}
tbb::internal::allocate_scheduler
generic_scheduler * allocate_scheduler(market &m, bool genuine)
Definition: scheduler.cpp:37
tbb::task_group_context::default_traits
@ default_traits
Definition: task.h:391
tbb::task_group_context::isolated
@ isolated
Definition: task.h:378
tbb::internal::governor::sign_on
static void sign_on(generic_scheduler *s)
Register TBB scheduler instance in thread-local storage.
Definition: governor.cpp:124
tbb::internal::market::global_market
static market & global_market(bool is_public, unsigned max_num_workers=0, size_t stack_size=0)
Factory method creating new market object.
Definition: market.cpp:96
tbb::internal::scheduler_properties::master
static const bool master
Definition: scheduler.h:52

References __TBB_ASSERT, tbb::internal::allocate_scheduler(), tbb::internal::task_prefix::context, tbb::task_group_context::default_traits, tbb::internal::market::global_market(), tbb::task_group_context::isolated, lock, tbb::internal::scheduler_properties::master, tbb::internal::NFS_Allocate(), tbb::task::prefix(), tbb::internal::task_prefix::ref_count, s, and tbb::internal::governor::sign_on().

Referenced by tbb::internal::governor::init_scheduler(), and tbb::internal::governor::init_scheduler_weak().

Here is the call graph for this function:
Here is the caller graph for this function:

## create_worker()

generic_scheduler * tbb::internal::generic_scheduler::create_worker | ( | market & | m,
|  | size_t | index,
|  | bool | geniune
| ) |  |

static

Initialize a scheduler for a worker thread.

Definition at line 1273 of file scheduler.cpp.

{
generic_scheduler* s = allocate_scheduler( m, genuine );
__TBB_ASSERT(!genuine || index, "workers should have index > 0");
s->my_arena_index = index; // index is not a real slot in arena yet
s->my_dummy_task->prefix().ref_count = 2;
s->my_properties.type = scheduler_properties::worker;
// Do not call init_stack_info before the scheduler is set as master or worker.
if (genuine)
s->init_stack_info();
governor::sign_on(s);
return s;
}
tbb::internal::scheduler_properties::worker
static const bool worker
Definition: scheduler.h:51

References __TBB_ASSERT, tbb::internal::allocate_scheduler(), s, tbb::internal::governor::sign_on(), and tbb::internal::scheduler_properties::worker.

Referenced by tbb::internal::market::create_one_job().

Here is the call graph for this function:
Here is the caller graph for this function:

## deallocate_task()

void tbb::internal::generic_scheduler::deallocate_task | ( | task & | t | ) |

inline

Return task object to the memory allocator.

Definition at line 683 of file scheduler.h.

{
#if TBB_USE_ASSERT
task_prefix& p = t.prefix();
p.state = 0xFF;
p.extra_state = 0xFF;
poison_pointer(p.next);
#endif /* TBB_USE_ASSERT */
NFS_Free((char*)&t-task_prefix_reservation_size);
#if __TBB_COUNT_TASK_NODES
--my_task_node_count;
#endif /* __TBB_COUNT_TASK_NODES */
}
tbb::internal::poison_pointer
void poison_pointer(T *__TBB_atomic &)
Definition: tbb_stddef.h:305

References tbb::internal::NFS_Free(), p, tbb::internal::poison_pointer(), tbb::task::prefix(), and tbb::internal::task_prefix_reservation_size.

Referenced by cleanup_scheduler(), free_nonlocal_small_task(), and free_task().

Here is the call graph for this function:
Here is the caller graph for this function:

## destroy()

void tbb::internal::generic_scheduler::destroy | ( |  | ) |

Destroy and deallocate this scheduler object.

Definition at line 285 of file scheduler.cpp.

{
__TBB_ASSERT(my_small_task_count == 0, "The scheduler is still in use.");
this->~generic_scheduler();
#if TBB_USE_DEBUG
memset((void*)this, -1, sizeof(generic_scheduler));
#endif
NFS_Free(this);
}

References __TBB_ASSERT, my_small_task_count, and tbb::internal::NFS_Free().

Referenced by cleanup_scheduler().

Here is the call graph for this function:
Here is the caller graph for this function:

## enqueue()

void tbb::internal::generic_scheduler::enqueue | ( | task & | t,
|  | void * | reserved
| ) |  |

virtual

For internal use only.

Implements tbb::internal::scheduler.

Definition at line 749 of file scheduler.cpp.

{
generic_scheduler *s = governor::local_scheduler();
// these redirections are due to bw-compatibility, consider reworking some day
__TBB_ASSERT( s->my_arena, "thread is not in any arena" );
s->my_arena->enqueue_task(t, (intptr_t)prio, s->my_random );
}
tbb::internal::governor::local_scheduler
static generic_scheduler * local_scheduler()
Obtain the thread-local instance of the TBB scheduler.
Definition: governor.h:129

References __TBB_ASSERT, tbb::internal::governor::local_scheduler(), and s.

Here is the call graph for this function:

## free_nonlocal_small_task()

void tbb::internal::generic_scheduler::free_nonlocal_small_task | ( | task & | t | ) |

Free a small task t that that was allocated by a different scheduler.

Definition at line 412 of file scheduler.cpp.

{
__TBB_ASSERT( t.state()==task::freed, NULL );
generic_scheduler& s = *static_cast<generic_scheduler*>(t.prefix().origin);
__TBB_ASSERT( &s!=this, NULL );
for(;;) {
task* old = s.my_return_list;
if( old==plugged_return_list() )
break;
// Atomically insert t at head of s.my_return_list
t.prefix().next = old;
ITT_NOTIFY( sync_releasing, &s.my_return_list );
if( as_atomic(s.my_return_list).compare_and_swap(&t, old )==old ) {
#if __TBB_PREFETCHING
__TBB_cl_evict(&t.prefix());
__TBB_cl_evict(&t);
#endif
return;
}
}
deallocate_task(t);
if( __TBB_FetchAndDecrementWrelease( &s.my_small_task_count )==1 ) {
// We freed the last task allocated by scheduler s, so it's our responsibility
// to free the scheduler.
s.destroy();
}
}
__TBB_cl_evict
#define __TBB_cl_evict(p)
Definition: mic_common.h:34
__TBB_FetchAndDecrementWrelease
#define __TBB_FetchAndDecrementWrelease(P)
Definition: tbb_machine.h:314

References __TBB_ASSERT, __TBB_cl_evict, __TBB_FetchAndDecrementWrelease, tbb::internal::as_atomic(), deallocate_task(), tbb::task::freed, ITT_NOTIFY, tbb::internal::task_prefix::next, tbb::internal::task_prefix::origin, plugged_return_list(), tbb::task::prefix(), s, tbb::task::state(), and sync_releasing.

Referenced by cleanup_scheduler(), and free_task().

Here is the call graph for this function:
Here is the caller graph for this function:

## free_task()

template<free_task_hint hint>

void tbb::internal::generic_scheduler::free_task | ( | task & | t | ) |

Put task on free list.

Does not call destructor.

Definition at line 730 of file scheduler.h.

{
#if __TBB_HOARD_NONLOCAL_TASKS
static const int h = hint&(~local_task);
#else
static const free_task_hint h = hint;
#endif
GATHER_STATISTIC(--my_counters.active_tasks);
task_prefix& p = t.prefix();
// Verify that optimization hints are correct.
__TBB_ASSERT( h!=small_local_task || p.origin==this, NULL );
__TBB_ASSERT( !(h&small_task) || p.origin, NULL );
__TBB_ASSERT( !(h&local_task) || (!p.origin || uintptr_t(p.origin) > uintptr_t(4096)), "local_task means allocated");
poison_value(p.depth);
poison_value(p.ref_count);
poison_pointer(p.owner);
#if __TBB_PREVIEW_RESUMABLE_TASKS
__TBB_ASSERT(1L << t.state() & (1L << task::executing | 1L << task::allocated | 1 << task::to_resume), NULL);
#else
__TBB_ASSERT(1L << t.state() & (1L << task::executing | 1L << task::allocated), NULL);
#endif
p.state = task::freed;
if( h==small_local_task || p.origin==this ) {
GATHER_STATISTIC(++my_counters.free_list_length);
p.next = my_free_list;
my_free_list = &t;
} else if( !(h&local_task) && p.origin && uintptr_t(p.origin) < uintptr_t(4096) ) {
// a special value reserved for future use, do nothing since
// origin is not pointing to a scheduler instance
} else if( !(h&local_task) && p.origin ) {
GATHER_STATISTIC(++my_counters.free_list_length);
#if __TBB_HOARD_NONLOCAL_TASKS
if( !(h&no_cache) ) {
p.next = my_nonlocal_free_list;
my_nonlocal_free_list = &t;
} else
#endif
free_nonlocal_small_task(t);
} else {
GATHER_STATISTIC(--my_counters.big_tasks);
deallocate_task(t);
}
}
poison_value
#define poison_value(g)
Definition: scheduler_common.h:235
h
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task __itt_model_task_instance ITT_FORMAT p void ITT_FORMAT p void ITT_FORMAT p void size_t ITT_FORMAT d void ITT_FORMAT p const wchar_t ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s no args void ITT_FORMAT p size_t ITT_FORMAT d no args const wchar_t const wchar_t ITT_FORMAT s __itt_heap_function h
Definition: ittnotify_static.h:159
tbb::internal::free_task_hint
free_task_hint
Optimization hint to free_task that enables it omit unnecessary tests and code.
Definition: scheduler_common.h:186
tbb::internal::no_cache
@ no_cache
Disable caching for a small task.
Definition: scheduler_common.h:198
tbb::internal::small_task
@ small_task
Task is known to be a small task.
Definition: scheduler_common.h:193
tbb::internal::local_task
@ local_task
Task is known to have been allocated by this scheduler.
Definition: scheduler_common.h:190
tbb::internal::small_local_task
@ small_local_task
Bitwise-OR of local_task and small_task.
Definition: scheduler_common.h:196
tbb::task::executing
@ executing
task is running, and will be destroyed after method execute() completes.
Definition: task.h:637

References __TBB_ASSERT, tbb::task::allocated, deallocate_task(), tbb::task::executing, free_nonlocal_small_task(), tbb::task::freed, GATHER_STATISTIC, h, tbb::internal::local_task, my_free_list, tbb::internal::no_cache, p, tbb::internal::poison_pointer(), poison_value, tbb::task::prefix(), tbb::internal::small_local_task, tbb::internal::small_task, and tbb::task::state().

Referenced by tbb::interface5::internal::task_base::destroy(), tbb::internal::allocate_root_proxy::free(), tbb::internal::allocate_additional_child_of_proxy::free(), tbb::internal::allocate_continuation_proxy::free(), tbb::internal::allocate_child_proxy::free(), and tbb::internal::auto_empty_task::~auto_empty_task().

Here is the call graph for this function:
Here is the caller graph for this function:

## get_mailbox_task()

task * tbb::internal::generic_scheduler::get_mailbox_task | ( | __TBB_ISOLATION_EXPR(isolation_tag isolation) |  | ) |

Attempt to get a task from the mailbox.

Gets a task only if it has not been executed by its sender or a thief that has stolen it from the sender's task pool. Otherwise returns NULL.

This method is intended to be used only by the thread extracting the proxy from its mailbox. (In contrast to local task pool, mailbox can be read only by its owner).

Definition at line 1234 of file scheduler.cpp.

{
__TBB_ASSERT( my_affinity_id>0, "not in arena" );
while ( task_proxy* const tp = my_inbox.pop( __TBB_ISOLATION_EXPR( isolation ) ) ) {
if ( task* result = tp->extract_task<task_proxy::mailbox_bit>() ) {
ITT_NOTIFY( sync_acquired, my_inbox.outbox() );
result->prefix().extra_state |= es_task_is_stolen;
return result;
}
// We have exclusive access to the proxy, and can destroy it.
free_task<no_cache_small_task>(*tp);
}
return NULL;
}
tbb::internal::es_task_is_stolen
@ es_task_is_stolen
Set if the task has been stolen.
Definition: scheduler_common.h:178
tbb::internal::task_proxy::mailbox_bit
static const intptr_t mailbox_bit
Definition: mailbox.h:31
tbb::internal::mail_inbox::pop
task_proxy * pop(__TBB_ISOLATION_EXPR(isolation_tag isolation))
Get next piece of mail, or NULL if mailbox is empty.
Definition: mailbox.h:213

References __TBB_ASSERT, __TBB_ISOLATION_EXPR, tbb::internal::es_task_is_stolen, ITT_NOTIFY, tbb::internal::task_proxy::mailbox_bit, tbb::internal::scheduler_state::my_affinity_id, tbb::internal::scheduler_state::my_inbox, and tbb::internal::mail_inbox::pop().

Here is the call graph for this function:

## get_task() [1/2]

task * tbb::internal::generic_scheduler::get_task | ( | __TBB_ISOLATION_EXPR(isolation_tag isolation) |  | ) |

inline

Get a task from the local pool.

Called only by the pool owner. Returns the pointer to the task or NULL if a suitable task is not found. Resets the pool if it is empty.

Definition at line 1012 of file scheduler.cpp.

{
__TBB_ASSERT( is_task_pool_published(), NULL );
// The current task position in the task pool.
size_t T0 = __TBB_load_relaxed( my_arena_slot->tail );
// The bounds of available tasks in the task pool. H0 is only used when the head bound is reached.
size_t H0 = (size_t)-1, T = T0;
task* result = NULL;
bool task_pool_empty = false;
__TBB_ISOLATION_EXPR( bool tasks_omitted = false );
do {
__TBB_ASSERT( !result, NULL );
__TBB_store_relaxed( my_arena_slot->tail, --T );
atomic_fence();
if ( (intptr_t)__TBB_load_relaxed( my_arena_slot->head ) > (intptr_t)T ) {
acquire_task_pool();
H0 = __TBB_load_relaxed( my_arena_slot->head );
if ( (intptr_t)H0 > (intptr_t)T ) {
// The thief has not backed off - nothing to grab.
__TBB_ASSERT( H0 == __TBB_load_relaxed( my_arena_slot->head )
&& T == __TBB_load_relaxed( my_arena_slot->tail )
&& H0 == T + 1, "victim/thief arbitration algorithm failure" );
reset_task_pool_and_leave();
// No tasks in the task pool.
task_pool_empty = true;
break;
} else if ( H0 == T ) {
// There is only one task in the task pool.
reset_task_pool_and_leave();
task_pool_empty = true;
} else {
// Release task pool if there are still some tasks.
// After the release, the tail will be less than T, thus a thief
// will not attempt to get a task at position T.
release_task_pool();
}
}
__TBB_control_consistency_helper(); // on my_arena_slot->head
#if __TBB_TASK_ISOLATION
result = get_task( T, isolation, tasks_omitted );
if ( result ) {
poison_pointer( my_arena_slot->task_pool_ptr[T] );
break;
} else if ( !tasks_omitted ) {
poison_pointer( my_arena_slot->task_pool_ptr[T] );
__TBB_ASSERT( T0 == T+1, NULL );
T0 = T;
}
#else
result = get_task( T );
#endif /* __TBB_TASK_ISOLATION */
} while ( !result && !task_pool_empty );
#if __TBB_TASK_ISOLATION
if ( tasks_omitted ) {
if ( task_pool_empty ) {
// All tasks have been checked. The task pool should be in reset state.
// We just restore the bounds for the available tasks.
// TODO: Does it have sense to move them to the beginning of the task pool?
__TBB_ASSERT( is_quiescent_local_task_pool_reset(), NULL );
if ( result ) {
// If we have a task, it should be at H0 position.
__TBB_ASSERT( H0 == T, NULL );
++H0;
}
__TBB_ASSERT( H0 <= T0, NULL );
if ( H0 < T0 ) {
// Restore the task pool if there are some tasks.
__TBB_store_relaxed( my_arena_slot->head, H0 );
__TBB_store_relaxed( my_arena_slot->tail, T0 );
// The release fence is used in publish_task_pool.
publish_task_pool();
// Synchronize with snapshot as we published some tasks.
my_arena->advertise_new_work<arena::wakeup>();
}
} else {
// A task has been obtained. We need to make a hole in position T.
__TBB_ASSERT( is_task_pool_published(), NULL );
__TBB_ASSERT( result, NULL );
my_arena_slot->task_pool_ptr[T] = NULL;
__TBB_store_with_release( my_arena_slot->tail, T0 );
// Synchronize with snapshot as we published some tasks.
// TODO: consider some approach not to call wakeup for each time. E.g. check if the tail reached the head.
my_arena->advertise_new_work<arena::wakeup>();
}
// Now it is safe to call note_affinity because the task pool is restored.
if ( my_innermost_running_task == result ) {
assert_task_valid( result );
result->note_affinity( my_affinity_id );
}
}
#endif /* __TBB_TASK_ISOLATION */
__TBB_ASSERT( (intptr_t)__TBB_load_relaxed( my_arena_slot->tail ) >= 0, NULL );
__TBB_ASSERT( result || __TBB_ISOLATION_EXPR( tasks_omitted || ) is_quiescent_local_task_pool_reset(), NULL );
return result;
} // generic_scheduler::get_task
__TBB_control_consistency_helper
#define __TBB_control_consistency_helper()
Definition: gcc_generic.h:60
tbb::atomic_fence
void atomic_fence()
Sequentially consistent full memory fence.
Definition: tbb_machine.h:342
tbb::internal::assert_task_valid
void assert_task_valid(const task *)
Definition: scheduler_common.h:237
tbb::internal::arena::wakeup
@ wakeup
Definition: arena.h:286
tbb::internal::arena::advertise_new_work
void advertise_new_work()
If necessary, raise a flag that there is new job in arena.
Definition: arena.h:484
tbb::internal::generic_scheduler::is_quiescent_local_task_pool_reset
bool is_quiescent_local_task_pool_reset() const
Definition: scheduler.h:644
tbb::internal::generic_scheduler::get_task
task * get_task(__TBB_ISOLATION_EXPR(isolation_tag isolation))
Get a task from the local pool.
Definition: scheduler.cpp:1012
tbb::internal::generic_scheduler::reset_task_pool_and_leave
void reset_task_pool_and_leave()
Resets head and tail indices to 0, and leaves task pool.
Definition: scheduler.h:702
tbb::internal::generic_scheduler::publish_task_pool
void publish_task_pool()
Used by workers to enter the task pool.
Definition: scheduler.cpp:1248

References __TBB_ASSERT, __TBB_control_consistency_helper, __TBB_ISOLATION_EXPR, tbb::internal::__TBB_load_relaxed(), tbb::internal::__TBB_store_relaxed(), tbb::internal::__TBB_store_with_release(), acquire_task_pool(), tbb::internal::arena::advertise_new_work(), tbb::internal::assert_task_valid(), tbb::atomic_fence(), is_quiescent_local_task_pool_reset(), is_task_pool_published(), tbb::internal::scheduler_state::my_affinity_id, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_slot, tbb::internal::scheduler_state::my_innermost_running_task, tbb::task::note_affinity(), tbb::internal::poison_pointer(), publish_task_pool(), release_task_pool(), reset_task_pool_and_leave(), and tbb::internal::arena::wakeup.

Here is the call graph for this function:

## get_task() [2/2]

task * tbb::internal::generic_scheduler::get_task | ( | size_t | T | ) |

inline

Get a task from the local pool at specified location T.

Returns the pointer to the task or NULL if the task cannot be executed, e.g. proxy has been deallocated or isolation constraint is not met. tasks_omitted tells if some tasks have been omitted. Called only by the pool owner. The caller should guarantee that the position T is not available for a thief.

Definition at line 961 of file scheduler.cpp.

{
__TBB_ASSERT( __TBB_load_relaxed( my_arena_slot->tail ) <= T
|| is_local_task_pool_quiescent(), "Is it safe to get a task at position T?" );
task* result = my_arena_slot->task_pool_ptr[T];
__TBB_ASSERT( !is_poisoned( result ), "The poisoned task is going to be processed" );
#if __TBB_TASK_ISOLATION
if ( !result )
return NULL;
bool omit = isolation != no_isolation && isolation != result->prefix().isolation;
if ( !omit && !is_proxy( *result ) )
return result;
else if ( omit ) {
tasks_omitted = true;
return NULL;
}
#else
poison_pointer( my_arena_slot->task_pool_ptr[T] );
if ( !result || !is_proxy( *result ) )
return result;
#endif /* __TBB_TASK_ISOLATION */
task_proxy& tp = static_cast<task_proxy&>(*result);
if ( task *t = tp.extract_task<task_proxy::pool_bit>() ) {
GATHER_STATISTIC( ++my_counters.proxies_executed );
// Following assertion should be true because TBB 2.0 tasks never specify affinity, and hence are not proxied.
__TBB_ASSERT( is_version_3_task( *t ), "backwards compatibility with TBB 2.0 broken" );
my_innermost_running_task = t; // prepare for calling note_affinity()
#if __TBB_TASK_ISOLATION
// Task affinity has changed. Postpone calling note_affinity because the task pool is in invalid state.
if ( !tasks_omitted )
#endif /* __TBB_TASK_ISOLATION */
{
poison_pointer( my_arena_slot->task_pool_ptr[T] );
t->note_affinity( my_affinity_id );
}
return t;
}
// Proxy was empty, so it's our responsibility to free it
free_task<small_task>( tp );
#if __TBB_TASK_ISOLATION
if ( tasks_omitted )
my_arena_slot->task_pool_ptr[T] = NULL;
#endif /* __TBB_TASK_ISOLATION */
return NULL;
}
tbb::internal::task_proxy::pool_bit
static const intptr_t pool_bit
Definition: mailbox.h:30
tbb::internal::generic_scheduler::is_version_3_task
static bool is_version_3_task(task &t)
Definition: scheduler.h:146
tbb::internal::generic_scheduler::is_proxy
static bool is_proxy(const task &t)
True if t is a task_proxy.
Definition: scheduler.h:348

References __TBB_ASSERT, tbb::internal::__TBB_load_relaxed(), tbb::internal::task_proxy::extract_task(), GATHER_STATISTIC, tbb::internal::task_prefix::isolation, tbb::internal::no_isolation, tbb::internal::poison_pointer(), tbb::internal::task_proxy::pool_bit, and tbb::task::prefix().

Here is the call graph for this function:

## init_stack_info()

void tbb::internal::generic_scheduler::init_stack_info | ( |  | ) |

Sets up the data necessary for the stealing limiting heuristics.

Definition at line 158 of file scheduler.cpp.

{
// Stacks are growing top-down. Highest address is called "stack base",
// and the lowest is "stack limit".
__TBB_ASSERT( !my_stealing_threshold, "Stealing threshold has already been calculated" );
size_t stack_size = my_market->worker_stack_size();
#if USE_WINTHREAD
#if defined(_MSC_VER)&&_MSC_VER<1400 && !_WIN64
NT_TIB *pteb;
__asm mov eax, fs:[0x18]
__asm mov pteb, eax
#else
NT_TIB *pteb = (NT_TIB*)NtCurrentTeb();
#endif
__TBB_ASSERT( &pteb < pteb->StackBase && &pteb > pteb->StackLimit, "invalid stack info in TEB" );
__TBB_ASSERT( stack_size >0, "stack_size not initialized?" );
// When a thread is created with the attribute STACK_SIZE_PARAM_IS_A_RESERVATION, stack limit
// in the TIB points to the committed part of the stack only. This renders the expression
// "(uintptr_t)pteb->StackBase / 2 + (uintptr_t)pteb->StackLimit / 2" virtually useless.
// Thus for worker threads we use the explicit stack size we used while creating them.
// And for master threads we rely on the following fact and assumption:
// - the default stack size of a master thread on Windows is 1M;
// - if it was explicitly set by the application it is at least as large as the size of a worker stack.
if ( is_worker() || stack_size < MByte )
my_stealing_threshold = (uintptr_t)pteb->StackBase - stack_size / 2;
else
my_stealing_threshold = (uintptr_t)pteb->StackBase - MByte / 2;
#else /* USE_PTHREAD */
// There is no portable way to get stack base address in Posix, so we use
// non-portable method (on all modern Linux) or the simplified approach
// based on the common sense assumptions. The most important assumption
// is that the main thread's stack size is not less than that of other threads.
// See also comment 3 at the end of this file
void *stack_base = &stack_size;
#if __linux__ && !__bg__
#if __TBB_ipf
void *rsb_base = __TBB_get_bsp();
#endif
size_t np_stack_size = 0;
// Points to the lowest addressable byte of a stack.
void *stack_limit = NULL;
#if __TBB_PREVIEW_RESUMABLE_TASKS
if ( !my_properties.genuine ) {
stack_limit = my_co_context.get_stack_limit();
__TBB_ASSERT( (uintptr_t)stack_base > (uintptr_t)stack_limit, "stack size must be positive" );
// Size of the stack free part
stack_size = size_t((char*)stack_base - (char*)stack_limit);
}
#endif
pthread_attr_t np_attr_stack;
if( !stack_limit && 0 == pthread_getattr_np(pthread_self(), &np_attr_stack) ) {
if ( 0 == pthread_attr_getstack(&np_attr_stack, &stack_limit, &np_stack_size) ) {
#if __TBB_ipf
pthread_attr_t attr_stack;
if ( 0 == pthread_attr_init(&attr_stack) ) {
if ( 0 == pthread_attr_getstacksize(&attr_stack, &stack_size) ) {
if ( np_stack_size < stack_size ) {
// We are in a secondary thread. Use reliable data.
// IA-64 architecture stack is split into RSE backup and memory parts
rsb_base = stack_limit;
stack_size = np_stack_size/2;
// Limit of the memory part of the stack
stack_limit = (char*)stack_limit + stack_size;
}
// We are either in the main thread or this thread stack
// is bigger that that of the main one. As we cannot discern
// these cases we fall back to the default (heuristic) values.
}
pthread_attr_destroy(&attr_stack);
}
// IA-64 architecture stack is split into RSE backup and memory parts
my_rsb_stealing_threshold = (uintptr_t)((char*)rsb_base + stack_size/2);
#endif /* __TBB_ipf */
// TODO: pthread_attr_getstack cannot be used with Intel(R) Cilk(TM) Plus
// __TBB_ASSERT( (uintptr_t)stack_base > (uintptr_t)stack_limit, "stack size must be positive" );
// Size of the stack free part
stack_size = size_t((char*)stack_base - (char*)stack_limit);
}
pthread_attr_destroy(&np_attr_stack);
}
#endif /* __linux__ */
__TBB_ASSERT( stack_size>0, "stack size must be positive" );
my_stealing_threshold = (uintptr_t)((char*)stack_base - stack_size/2);
#endif /* USE_PTHREAD */
}
tbb::internal::MByte
const size_t MByte
Definition: tbb_misc.h:45
tbb::internal::market::worker_stack_size
size_t worker_stack_size() const
Returns the requested stack size of worker threads.
Definition: market.h:314
tbb::internal::generic_scheduler::is_worker
bool is_worker() const
True if running on a worker thread, false otherwise.
Definition: scheduler.h:673

References __TBB_ASSERT, __TBB_get_bsp(), is_worker(), tbb::internal::MByte, my_market, tbb::internal::scheduler_state::my_properties, my_stealing_threshold, and tbb::internal::market::worker_stack_size().

Here is the call graph for this function:

## is_local_task_pool_quiescent()

bool tbb::internal::generic_scheduler::is_local_task_pool_quiescent | ( |  | ) | const

inline

Definition at line 633 of file scheduler.h.

{
__TBB_ASSERT(my_arena_slot, 0);
task** tp = my_arena_slot->task_pool;
return tp == EmptyTaskPool || tp == LockedTaskPool;
}

References __TBB_ASSERT, EmptyTaskPool, LockedTaskPool, and tbb::internal::scheduler_state::my_arena_slot.

Referenced by commit_relocated_tasks(), is_quiescent_local_task_pool_empty(), and is_quiescent_local_task_pool_reset().

Here is the caller graph for this function:

## is_proxy()

static bool tbb::internal::generic_scheduler::is_proxy | ( | const task & | t | ) |

inline static

True if t is a task_proxy.

Definition at line 348 of file scheduler.h.

{
return t.prefix().extra_state==es_task_proxy;
}
tbb::internal::es_task_proxy
@ es_task_proxy
Tag for v3 task_proxy.
Definition: scheduler_common.h:174

References tbb::internal::es_task_proxy, tbb::internal::task_prefix::extra_state, and tbb::task::prefix().

Referenced by steal_task(), and steal_task_from().

Here is the call graph for this function:
Here is the caller graph for this function:

## is_quiescent_local_task_pool_empty()

bool tbb::internal::generic_scheduler::is_quiescent_local_task_pool_empty | ( |  | ) | const

inline

Definition at line 639 of file scheduler.h.

{
__TBB_ASSERT( is_local_task_pool_quiescent(), "Task pool is not quiescent" );
return __TBB_load_relaxed(my_arena_slot->head) == __TBB_load_relaxed(my_arena_slot->tail);
}

References __TBB_ASSERT, tbb::internal::__TBB_load_relaxed(), is_local_task_pool_quiescent(), and tbb::internal::scheduler_state::my_arena_slot.

Referenced by leave_task_pool().

Here is the call graph for this function:
Here is the caller graph for this function:

## is_quiescent_local_task_pool_reset()

bool tbb::internal::generic_scheduler::is_quiescent_local_task_pool_reset | ( |  | ) | const

inline

Definition at line 644 of file scheduler.h.

{
__TBB_ASSERT( is_local_task_pool_quiescent(), "Task pool is not quiescent" );
return __TBB_load_relaxed(my_arena_slot->head) == 0 && __TBB_load_relaxed(my_arena_slot->tail) == 0;
}

References __TBB_ASSERT, tbb::internal::__TBB_load_relaxed(), is_local_task_pool_quiescent(), and tbb::internal::scheduler_state::my_arena_slot.

Referenced by get_task(), and prepare_task_pool().

Here is the call graph for this function:
Here is the caller graph for this function:

## is_task_pool_published()

bool tbb::internal::generic_scheduler::is_task_pool_published | ( |  | ) | const

inline

Definition at line 628 of file scheduler.h.

{
__TBB_ASSERT(my_arena_slot, 0);
return my_arena_slot->task_pool != EmptyTaskPool;
}

References __TBB_ASSERT, EmptyTaskPool, and tbb::internal::scheduler_state::my_arena_slot.

Referenced by acquire_task_pool(), cleanup_master(), get_task(), leave_task_pool(), local_spawn(), prepare_task_pool(), and release_task_pool().

Here is the caller graph for this function:

## is_version_3_task()

static bool tbb::internal::generic_scheduler::is_version_3_task | ( | task & | t | ) |

inline static

Definition at line 146 of file scheduler.h.

{
#if __TBB_PREVIEW_CRITICAL_TASKS
return (t.prefix().extra_state & 0x7)>=0x1;
#else
return (t.prefix().extra_state & 0x0F)>=0x1;
#endif
}

References tbb::internal::task_prefix::extra_state, and tbb::task::prefix().

Referenced by prepare_for_spawning(), and steal_task().

Here is the call graph for this function:
Here is the caller graph for this function:

## is_worker()

bool tbb::internal::generic_scheduler::is_worker | ( |  | ) | const

inline

True if running on a worker thread, false otherwise.

Definition at line 673 of file scheduler.h.

{
return my_properties.type == scheduler_properties::worker;
}
tbb::internal::scheduler_properties::type
bool type
Indicates that a scheduler acts as a master or a worker.
Definition: scheduler.h:54

References tbb::internal::scheduler_state::my_properties, tbb::internal::scheduler_properties::type, and tbb::internal::scheduler_properties::worker.

Referenced by tbb::internal::market::cleanup(), init_stack_info(), master_outermost_level(), nested_arena_entry(), nested_arena_exit(), and worker_outermost_level().

Here is the caller graph for this function:

## leave_task_pool()

void tbb::internal::generic_scheduler::leave_task_pool | ( |  | ) |

inline

Leave the task pool.

Leaving task pool automatically releases the task pool if it is locked.

Definition at line 1260 of file scheduler.cpp.

{
__TBB_ASSERT( is_task_pool_published(), "Not in arena" );
// Do not reset my_arena_index. It will be used to (attempt to) re-acquire the slot next time
__TBB_ASSERT( &my_arena->my_slots[my_arena_index] == my_arena_slot, "arena slot and slot index mismatch" );
__TBB_ASSERT ( my_arena_slot->task_pool == LockedTaskPool, "Task pool must be locked when leaving arena" );
__TBB_ASSERT ( is_quiescent_local_task_pool_empty(), "Cannot leave arena when the task pool is not empty" );
ITT_NOTIFY(sync_releasing, &my_arena->my_slots[my_arena_index]);
// No release fence is necessary here as this assignment precludes external
// accesses to the local task pool when becomes visible. Thus it is harmless
// if it gets hoisted above preceding local bookkeeping manipulations.
__TBB_store_relaxed( my_arena_slot->task_pool, EmptyTaskPool );
}
tbb::internal::generic_scheduler::is_quiescent_local_task_pool_empty
bool is_quiescent_local_task_pool_empty() const
Definition: scheduler.h:639

References __TBB_ASSERT, tbb::internal::__TBB_store_relaxed(), EmptyTaskPool, is_quiescent_local_task_pool_empty(), is_task_pool_published(), ITT_NOTIFY, LockedTaskPool, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_index, tbb::internal::scheduler_state::my_arena_slot, tbb::internal::arena::my_slots, and sync_releasing.

Referenced by cleanup_master(), and reset_task_pool_and_leave().

Here is the call graph for this function:
Here is the caller graph for this function:

## local_spawn()

void tbb::internal::generic_scheduler::local_spawn | ( | task * | first,
|  | task *& | next
| ) |  |

Conceptually, this method should be a member of class scheduler. But doing so would force us to publish class scheduler in the headers.

Definition at line 653 of file scheduler.cpp.

{
__TBB_ASSERT( first, NULL );
__TBB_ASSERT( governor::is_set(this), NULL );
#if __TBB_TODO
// We need to consider capping the max task pool size and switching
// to in-place task execution whenever it is reached.
#endif
if ( &first->prefix().next == &next ) {
// Single task is being spawned
#if __TBB_TODO
// TODO:
// In the future we need to add overloaded spawn method for a single task,
// and a method accepting an array of task pointers (we may also want to
// change the implementation of the task_list class). But since such changes
// may affect the binary compatibility, we postpone them for a while.
#endif
#if __TBB_PREVIEW_CRITICAL_TASKS
if( !handled_as_critical( *first ) )
#endif
{
size_t T = prepare_task_pool( 1 );
my_arena_slot->task_pool_ptr[T] = prepare_for_spawning( first );
commit_spawned_tasks( T + 1 );
if ( !is_task_pool_published() )
publish_task_pool();
}
}
else {
// Task list is being spawned
#if __TBB_TODO
// TODO: add task_list::front() and implement&document the local execution ordering which is
// opposite to the current implementation. The idea is to remove hackish fast_reverse_vector
// and use push_back/push_front when accordingly LIFO and FIFO order of local execution is
// desired. It also requires refactoring of the reload_tasks method and my_offloaded_tasks list.
// Additional benefit may come from adding counter to the task_list so that it can reserve enough
// space in the task pool in advance and move all the tasks directly without any intermediate
// storages. But it requires dealing with backward compatibility issues and still supporting
// counter-less variant (though not necessarily fast implementation).
#endif
task *arr[min_task_pool_size];
fast_reverse_vector<task*> tasks(arr, min_task_pool_size);
task *t_next = NULL;
for( task* t = first; ; t = t_next ) {
// If t is affinitized to another thread, it may already be executed
// and destroyed by the time prepare_for_spawning returns.
// So milk it while it is alive.
bool end = &t->prefix().next == &next;
t_next = t->prefix().next;
#if __TBB_PREVIEW_CRITICAL_TASKS
if( !handled_as_critical( *t ) )
#endif
tasks.push_back( prepare_for_spawning(t) );
if( end )
break;
}
if( size_t num_tasks = tasks.size() ) {
size_t T = prepare_task_pool( num_tasks );
tasks.copy_memory( my_arena_slot->task_pool_ptr + T );
commit_spawned_tasks( T + num_tasks );
if ( !is_task_pool_published() )
publish_task_pool();
}
}
my_arena->advertise_new_work<arena::work_spawned>();
assert_task_pool_valid();
}
end
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task __itt_model_task_instance ITT_FORMAT p void ITT_FORMAT p void ITT_FORMAT p void size_t ITT_FORMAT d void ITT_FORMAT p const wchar_t ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s no args void ITT_FORMAT p size_t ITT_FORMAT d no args const wchar_t const wchar_t ITT_FORMAT s __itt_heap_function void size_t int ITT_FORMAT d __itt_heap_function void ITT_FORMAT p __itt_heap_function void void size_t int ITT_FORMAT d no args no args unsigned int ITT_FORMAT u const __itt_domain __itt_id ITT_FORMAT lu const __itt_domain __itt_id __itt_id __itt_string_handle ITT_FORMAT p const __itt_domain __itt_id ITT_FORMAT p const __itt_domain __itt_id __itt_timestamp __itt_timestamp end
Definition: ittnotify_static.h:182
tbb::internal::first
auto first(Container &c) -> decltype(begin(c))
Definition: _range_iterator.h:34
tbb::internal::arena::work_spawned
@ work_spawned
Definition: arena.h:285
tbb::internal::generic_scheduler::prepare_task_pool
size_t prepare_task_pool(size_t n)
Makes sure that the task pool can accommodate at least n more elements.
Definition: scheduler.cpp:439
tbb::internal::generic_scheduler::min_task_pool_size
static const size_t min_task_pool_size
Definition: scheduler.h:369
tbb::internal::generic_scheduler::commit_spawned_tasks
void commit_spawned_tasks(size_t new_tail)
Makes newly spawned tasks visible to thieves.
Definition: scheduler.h:710
tbb::internal::generic_scheduler::prepare_for_spawning
task * prepare_for_spawning(task *t)
Checks if t is affinitized to another thread, and if so, bundles it as proxy.
Definition: scheduler.cpp:595
tbb::internal::generic_scheduler::assert_task_pool_valid
void assert_task_pool_valid() const
Definition: scheduler.h:398

References __TBB_ASSERT, tbb::internal::arena::advertise_new_work(), assert_task_pool_valid(), commit_spawned_tasks(), tbb::internal::fast_reverse_vector< T, max_segments >::copy_memory(), end, tbb::internal::first(), tbb::internal::governor::is_set(), is_task_pool_published(), min_task_pool_size, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_slot, tbb::internal::task_prefix::next, tbb::task::prefix(), prepare_for_spawning(), prepare_task_pool(), publish_task_pool(), tbb::internal::fast_reverse_vector< T, max_segments >::push_back(), tbb::internal::fast_reverse_vector< T, max_segments >::size(), and tbb::internal::arena::work_spawned.

Referenced by local_spawn_root_and_wait(), spawn(), and tbb::internal::custom_scheduler< SchedulerTraits >::tally_completion_of_predecessor().

Here is the call graph for this function:
Here is the caller graph for this function:

## local_spawn_root_and_wait()

void tbb::internal::generic_scheduler::local_spawn_root_and_wait | ( | task * | first,
|  | task *& | next
| ) |  |

Definition at line 720 of file scheduler.cpp.

{
__TBB_ASSERT( governor::is_set(this), NULL );
__TBB_ASSERT( first, NULL );
auto_empty_task dummy( __TBB_CONTEXT_ARG(this, first->prefix().context) );
internal::reference_count n = 0;
for( task* t=first; ; t=t->prefix().next ) {
++n;
__TBB_ASSERT( !t->prefix().parent, "not a root task, or already running" );
t->prefix().parent = &dummy;
if( &t->prefix().next==&next ) break;
#if __TBB_TASK_GROUP_CONTEXT
__TBB_ASSERT( t->prefix().context == t->prefix().next->prefix().context,
"all the root tasks in list must share the same context");
#endif /* __TBB_TASK_GROUP_CONTEXT */
}
dummy.prefix().ref_count = n+1;
if( n>1 )
local_spawn( first->prefix().next, next );
local_wait_for_all( dummy, first );
}
tbb::internal::reference_count
intptr_t reference_count
A reference count.
Definition: task.h:131
tbb::internal::generic_scheduler::local_spawn
void local_spawn(task *first, task *&next)
Definition: scheduler.cpp:653

References __TBB_ASSERT, __TBB_CONTEXT_ARG, tbb::internal::task_prefix::context, tbb::internal::first(), tbb::internal::governor::is_set(), local_spawn(), local_wait_for_all(), tbb::internal::task_prefix::next, tbb::internal::task_prefix::parent, tbb::internal::auto_empty_task::prefix(), tbb::task::prefix(), and tbb::internal::task_prefix::ref_count.

Referenced by spawn_root_and_wait().

Here is the call graph for this function:
Here is the caller graph for this function:

## local_wait_for_all()

virtual void tbb::internal::generic_scheduler::local_wait_for_all | ( | task & | parent,
|  | task * | child
| ) |  |

pure virtual

Implemented in tbb::internal::custom_scheduler< SchedulerTraits >.

Referenced by cleanup_master(), local_spawn_root_and_wait(), and wait_until_empty().

Here is the caller graph for this function:

## lock_task_pool()

task ** tbb::internal::generic_scheduler::lock_task_pool | ( | arena_slot * | victim_arena_slot | ) | const

inline

Locks victim's task pool, and returns pointer to it. The pointer can be NULL.

Garbles victim_arena_slot->task_pool for the duration of the lock.

ATTENTION: This method is mostly the same as generic_scheduler::acquire_task_pool(), with a little different logic of slot state checks (slot can be empty, locked or point to any task pool other than ours, and asynchronous transitions between all these states are possible). Thus if any of them is changed, consider changing the counterpart as well

Definition at line 537 of file scheduler.cpp.

{
task** victim_task_pool;
bool sync_prepare_done = false;
for( atomic_backoff backoff;; /*backoff pause embedded in the loop*/) {
victim_task_pool = victim_arena_slot->task_pool;
// NOTE: Do not use comparison of head and tail indices to check for
// the presence of work in the victim's task pool, as they may give
// incorrect indication because of task pool relocations and resizes.
if ( victim_task_pool == EmptyTaskPool ) {
// The victim thread emptied its task pool - nothing to lock
if( sync_prepare_done )
ITT_NOTIFY(sync_cancel, victim_arena_slot);
break;
}
if( victim_task_pool != LockedTaskPool &&
as_atomic(victim_arena_slot->task_pool).compare_and_swap(LockedTaskPool, victim_task_pool ) == victim_task_pool )
{
// We've locked victim's task pool
ITT_NOTIFY(sync_acquired, victim_arena_slot);
break;
}
else if( !sync_prepare_done ) {
// Start waiting
ITT_NOTIFY(sync_prepare, victim_arena_slot);
sync_prepare_done = true;
}
GATHER_STATISTIC( ++my_counters.thieves_conflicts );
// Someone else acquired a lock, so pause and do exponential backoff.
#if __TBB_STEALING_ABORT_ON_CONTENTION
if(!backoff.bounded_pause()) {
// the 16 was acquired empirically and a theory behind it supposes
// that number of threads becomes much bigger than number of
// tasks which can be spawned by one thread causing excessive contention.
// TODO: However even small arenas can benefit from the abort on contention
// if preemption of a thief is a problem
if(my_arena->my_limit >= 16)
return EmptyTaskPool;
__TBB_Yield();
}
#else
backoff.pause();
#endif
}
__TBB_ASSERT( victim_task_pool == EmptyTaskPool ||
(victim_arena_slot->task_pool == LockedTaskPool && victim_task_pool != LockedTaskPool),
"not really locked victim's task pool?" );
return victim_task_pool;
} // generic_scheduler::lock_task_pool
__TBB_Yield
#define __TBB_Yield()
Definition: ibm_aix51.h:44
sync_cancel
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p sync_cancel
Definition: ittnotify_static.h:102

References __TBB_ASSERT, __TBB_Yield, tbb::internal::as_atomic(), EmptyTaskPool, GATHER_STATISTIC, ITT_NOTIFY, LockedTaskPool, tbb::internal::scheduler_state::my_arena, and sync_cancel.

Referenced by steal_task_from().

Here is the call graph for this function:
Here is the caller graph for this function:

## master_outermost_level()

bool tbb::internal::generic_scheduler::master_outermost_level | ( |  | ) | const

inline

True if the scheduler is on the outermost dispatch level in a master thread.

Returns true when this scheduler instance is associated with an application thread, and is not executing any TBB task. This includes being in a TBB dispatch loop (one of wait_for_all methods) invoked directly from that thread.

Definition at line 653 of file scheduler.h.

{
return !is_worker() && outermost_level();
}
tbb::internal::generic_scheduler::outermost_level
bool outermost_level() const
True if the scheduler is on the outermost dispatch level.
Definition: scheduler.h:649

References is_worker(), and outermost_level().

Here is the call graph for this function:

## max_threads_in_arena()

unsigned tbb::internal::generic_scheduler::max_threads_in_arena | ( |  | ) |

inline

Returns the concurrency limit of the current arena.

Definition at line 677 of file scheduler.h.

{
__TBB_ASSERT(my_arena, NULL);
return my_arena->my_num_slots;
}

References __TBB_ASSERT, and tbb::internal::scheduler_state::my_arena.

Referenced by tbb::internal::get_initial_auto_partitioner_divisor(), and tbb::internal::affinity_partitioner_base_v3::resize().

Here is the caller graph for this function:

## nested_arena_entry()

void tbb::internal::generic_scheduler::nested_arena_entry | ( | arena * | a,
|  | size_t | slot_index
| ) |  |

Definition at line 729 of file arena.cpp.

{
__TBB_ASSERT( is_alive(a->my_guard), NULL );
__TBB_ASSERT( a!=my_arena, NULL);
// overwrite arena settings
#if __TBB_TASK_PRIORITY
if ( my_offloaded_tasks )
my_arena->orphan_offloaded_tasks( *this );
my_offloaded_tasks = NULL;
#endif /* __TBB_TASK_PRIORITY */
attach_arena( a, slot_index, /*is_master*/true );
__TBB_ASSERT( my_arena == a, NULL );
governor::assume_scheduler( this );
// TODO? ITT_NOTIFY(sync_acquired, a->my_slots + index);
// TODO: it requires market to have P workers (not P-1)
// TODO: a preempted worker should be excluded from assignment to other arenas e.g. my_slack--
if( !is_worker() && slot_index >= my_arena->my_num_reserved_slots )
my_arena->my_market->adjust_demand(*my_arena, -1);
#if __TBB_ARENA_OBSERVER
my_last_local_observer = 0; // TODO: try optimize number of calls
my_arena->my_observers.notify_entry_observers( my_last_local_observer, /*worker=*/false );
#endif
#if __TBB_PREVIEW_RESUMABLE_TASKS
my_wait_task = NULL;
#endif
}
tbb::internal::governor::assume_scheduler
static void assume_scheduler(generic_scheduler *s)
Temporarily set TLS slot to the given scheduler.
Definition: governor.cpp:116
tbb::internal::generic_scheduler::attach_arena
void attach_arena(arena *, size_t index, bool is_master)
Definition: arena.cpp:80

References __TBB_ASSERT, tbb::internal::governor::assume_scheduler(), attach_arena(), is_worker(), and tbb::internal::scheduler_state::my_arena.

Here is the call graph for this function:

## nested_arena_exit()

void tbb::internal::generic_scheduler::nested_arena_exit | ( |  | ) |

Definition at line 756 of file arena.cpp.

{
#if __TBB_ARENA_OBSERVER
my_arena->my_observers.notify_exit_observers( my_last_local_observer, /*worker=*/false );
#endif /* __TBB_ARENA_OBSERVER */
#if __TBB_TASK_PRIORITY
if ( my_offloaded_tasks )
my_arena->orphan_offloaded_tasks( *this );
#endif
if( !is_worker() && my_arena_index >= my_arena->my_num_reserved_slots )
my_arena->my_market->adjust_demand(*my_arena, 1);
// Free the master slot.
__TBB_ASSERT(my_arena->my_slots[my_arena_index].my_scheduler, "A slot is already empty");
__TBB_store_with_release(my_arena->my_slots[my_arena_index].my_scheduler, (generic_scheduler*)NULL);
my_arena->my_exit_monitors.notify_one(); // do not relax!
}

References __TBB_ASSERT, tbb::internal::__TBB_store_with_release(), is_worker(), tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_index, and tbb::internal::arena::my_slots.

Referenced by tbb::internal::nested_arena_context::~nested_arena_context().

Here is the call graph for this function:
Here is the caller graph for this function:

## outermost_level()

bool tbb::internal::generic_scheduler::outermost_level | ( |  | ) | const

inline

True if the scheduler is on the outermost dispatch level.

Definition at line 649 of file scheduler.h.

{
return my_properties.outermost;
}

References tbb::internal::scheduler_state::my_properties, and tbb::internal::scheduler_properties::outermost.

Referenced by master_outermost_level(), and worker_outermost_level().

Here is the caller graph for this function:

## plugged_return_list()

static task* tbb::internal::generic_scheduler::plugged_return_list | ( |  | ) |

inline static

Special value used to mark my_return_list as not taking any more entries.

Definition at line 458 of file scheduler.h.

{return (task*)(intptr_t)(-1);}

Referenced by cleanup_scheduler(), and free_nonlocal_small_task().

Here is the caller graph for this function:

## prepare_for_spawning()

task * tbb::internal::generic_scheduler::prepare_for_spawning | ( | task * | t | ) |

inline

Checks if t is affinitized to another thread, and if so, bundles it as proxy.

Returns either t or proxy containing t.

Definition at line 595 of file scheduler.cpp.

{
__TBB_ASSERT( t->state()==task::allocated, "attempt to spawn task that is not in 'allocated' state" );
t->prefix().state = task::ready;
#if TBB_USE_ASSERT
if( task* parent = t->parent() ) {
internal::reference_count ref_count = parent->prefix().ref_count;
__TBB_ASSERT( ref_count>=0, "attempt to spawn task whose parent has a ref_count<0" );
__TBB_ASSERT( ref_count!=0, "attempt to spawn task whose parent has a ref_count==0 (forgot to set_ref_count?)" );
parent->prefix().extra_state |= es_ref_count_active;
}
#endif /* TBB_USE_ASSERT */
affinity_id dst_thread = t->prefix().affinity;
__TBB_ASSERT( dst_thread == 0 || is_version_3_task(*t),
"backwards compatibility to TBB 2.0 tasks is broken" );
#if __TBB_TASK_ISOLATION
isolation_tag isolation = my_innermost_running_task->prefix().isolation;
t->prefix().isolation = isolation;
#endif /* __TBB_TASK_ISOLATION */
if( dst_thread != 0 && dst_thread != my_affinity_id ) {
task_proxy& proxy = (task_proxy&)allocate_task( sizeof(task_proxy),
__TBB_CONTEXT_ARG(NULL, NULL) );
// Mark as a proxy
proxy.prefix().extra_state = es_task_proxy;
proxy.outbox = &my_arena->mailbox(dst_thread);
// Mark proxy as present in both locations (sender's task pool and destination mailbox)
proxy.task_and_tag = intptr_t(t) | task_proxy::location_mask;
#if __TBB_TASK_PRIORITY
poison_pointer( proxy.prefix().context );
#endif /* __TBB_TASK_PRIORITY */
__TBB_ISOLATION_EXPR( proxy.prefix().isolation = isolation );
ITT_NOTIFY( sync_releasing, proxy.outbox );
// Mail the proxy, if success, it may be destroyed by another thread at any moment after this point.
if ( proxy.outbox->push(&proxy) )
return &proxy;
// The mailbox is overfilled, deallocate the proxy and return the initial task.
free_task<small_task>(proxy);
}
return t;
}
tbb::internal::isolation_tag
intptr_t isolation_tag
A tag for task isolation.
Definition: task.h:143
tbb::internal::es_ref_count_active
@ es_ref_count_active
Set if ref_count might be changed by another thread. Used for debugging.
Definition: scheduler_common.h:176
tbb::internal::task_prefix::extra_state
unsigned char extra_state
Miscellaneous state that is not directly visible to users, stored as a byte for compactness.
Definition: task.h:292
tbb::internal::task_prefix::isolation
isolation_tag isolation
The tag used for task isolation.
Definition: task.h:220
tbb::task::ready
@ ready
task is in ready pool, or is going to be put there, or was just taken off.
Definition: task.h:641
tbb::internal::task_proxy::location_mask
static const intptr_t location_mask
Definition: mailbox.h:32

References __TBB_ASSERT, __TBB_CONTEXT_ARG, __TBB_ISOLATION_EXPR, tbb::internal::task_prefix::affinity, allocate_task(), tbb::task::allocated, tbb::internal::task_prefix::context, tbb::internal::es_ref_count_active, tbb::internal::es_task_proxy, tbb::internal::task_prefix::extra_state, is_version_3_task(), tbb::internal::task_prefix::isolation, ITT_NOTIFY, tbb::internal::task_proxy::location_mask, tbb::internal::arena::mailbox(), tbb::internal::scheduler_state::my_affinity_id, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_innermost_running_task, tbb::internal::task_proxy::outbox, tbb::task::parent(), parent, tbb::internal::poison_pointer(), tbb::task::prefix(), tbb::internal::mail_outbox::push(), tbb::task::ready, tbb::internal::task_prefix::state, tbb::task::state(), sync_releasing, and tbb::internal::task_proxy::task_and_tag.

Referenced by local_spawn().

Here is the call graph for this function:
Here is the caller graph for this function:

## prepare_task_pool()

size_t tbb::internal::generic_scheduler::prepare_task_pool | ( | size_t | n | ) |

inline

Makes sure that the task pool can accommodate at least n more elements.

If necessary relocates existing task pointers or grows the ready task deque. Returns (possible updated) tail index (not accounting for n).

Definition at line 439 of file scheduler.cpp.

{
size_t T = __TBB_load_relaxed(my_arena_slot->tail); // mirror
if ( T + num_tasks <= my_arena_slot->my_task_pool_size )
return T;
size_t new_size = num_tasks;
if ( !my_arena_slot->my_task_pool_size ) {
__TBB_ASSERT( !is_task_pool_published() && is_quiescent_local_task_pool_reset(), NULL );
__TBB_ASSERT( !my_arena_slot->task_pool_ptr, NULL );
if ( num_tasks < min_task_pool_size ) new_size = min_task_pool_size;
my_arena_slot->allocate_task_pool( new_size );
return 0;
}
acquire_task_pool();
size_t H = __TBB_load_relaxed( my_arena_slot->head ); // mirror
task** task_pool = my_arena_slot->task_pool_ptr;;
__TBB_ASSERT( my_arena_slot->my_task_pool_size >= min_task_pool_size, NULL );
// Count not skipped tasks. Consider using std::count_if.
for ( size_t i = H; i < T; ++i )
if ( task_pool[i] ) ++new_size;
// If the free space at the beginning of the task pool is too short, we
// are likely facing a pathological single-producer-multiple-consumers
// scenario, and thus it's better to expand the task pool
bool allocate = new_size > my_arena_slot->my_task_pool_size - min_task_pool_size/4;
if ( allocate ) {
// Grow task pool. As this operation is rare, and its cost is asymptotically
// amortizable, we can tolerate new task pool allocation done under the lock.
if ( new_size < 2 * my_arena_slot->my_task_pool_size )
new_size = 2 * my_arena_slot->my_task_pool_size;
my_arena_slot->allocate_task_pool( new_size ); // updates my_task_pool_size
}
// Filter out skipped tasks. Consider using std::copy_if.
size_t T1 = 0;
for ( size_t i = H; i < T; ++i )
if ( task_pool[i] )
my_arena_slot->task_pool_ptr[T1++] = task_pool[i];
// Deallocate the previous task pool if a new one has been allocated.
if ( allocate )
NFS_Free( task_pool );
else
my_arena_slot->fill_with_canary_pattern( T1, my_arena_slot->tail );
// Publish the new state.
commit_relocated_tasks( T1 );
assert_task_pool_valid();
return T1;
}
new_size
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task __itt_model_task_instance ITT_FORMAT p void ITT_FORMAT p void ITT_FORMAT p void size_t ITT_FORMAT d void ITT_FORMAT p const wchar_t ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s no args void ITT_FORMAT p size_t ITT_FORMAT d no args const wchar_t const wchar_t ITT_FORMAT s __itt_heap_function void size_t int ITT_FORMAT d __itt_heap_function void ITT_FORMAT p __itt_heap_function void void size_t new_size
Definition: ittnotify_static.h:163
tbb::internal::generic_scheduler::commit_relocated_tasks
void commit_relocated_tasks(size_t new_tail)
Makes relocated tasks visible to thieves and releases the local task pool.
Definition: scheduler.h:719
tbb::internal::arena_slot::allocate_task_pool
void allocate_task_pool(size_t n)
Definition: scheduler_common.h:387
tbb::internal::arena_slot::fill_with_canary_pattern
void fill_with_canary_pattern(size_t, size_t)
Definition: scheduler_common.h:384

References __TBB_ASSERT, tbb::internal::__TBB_load_relaxed(), acquire_task_pool(), tbb::internal::arena_slot::allocate_task_pool(), assert_task_pool_valid(), commit_relocated_tasks(), tbb::internal::arena_slot::fill_with_canary_pattern(), is_quiescent_local_task_pool_reset(), is_task_pool_published(), min_task_pool_size, tbb::internal::scheduler_state::my_arena_slot, new_size, and tbb::internal::NFS_Free().

Referenced by local_spawn().

Here is the call graph for this function:
Here is the caller graph for this function:

## publish_task_pool()

void tbb::internal::generic_scheduler::publish_task_pool | ( |  | ) |

inline

Used by workers to enter the task pool.

Does not lock the task pool in case if arena slot has been successfully grabbed.

Definition at line 1248 of file scheduler.cpp.

{
__TBB_ASSERT ( my_arena, "no arena: initialization not completed?" );
__TBB_ASSERT ( my_arena_index < my_arena->my_num_slots, "arena slot index is out-of-bound" );
__TBB_ASSERT ( my_arena_slot == &my_arena->my_slots[my_arena_index], NULL);
__TBB_ASSERT ( my_arena_slot->task_pool == EmptyTaskPool, "someone else grabbed my arena slot?" );
__TBB_ASSERT ( __TBB_load_relaxed(my_arena_slot->head) < __TBB_load_relaxed(my_arena_slot->tail),
"entering arena without tasks to share" );
// Release signal on behalf of previously spawned tasks (when this thread was not in arena yet)
ITT_NOTIFY(sync_releasing, my_arena_slot);
__TBB_store_with_release( my_arena_slot->task_pool, my_arena_slot->task_pool_ptr );
}

References __TBB_ASSERT, tbb::internal::__TBB_load_relaxed(), tbb::internal::__TBB_store_with_release(), EmptyTaskPool, ITT_NOTIFY, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_index, tbb::internal::scheduler_state::my_arena_slot, tbb::internal::arena::my_slots, and sync_releasing.

Referenced by get_task(), and local_spawn().

Here is the call graph for this function:
Here is the caller graph for this function:

## receive_or_steal_task()

virtual task* tbb::internal::generic_scheduler::receive_or_steal_task | ( | __TBB_ISOLATION_ARG(__TBB_atomic reference_count &completion_ref_count, isolation_tag isolation) |  | ) |

pure virtual

Try getting a task from other threads (via mailbox, stealing, FIFO queue, orphans adoption).

Returns obtained task or NULL if all attempts fail.

Implemented in tbb::internal::custom_scheduler< SchedulerTraits >.

## release_task_pool()

void tbb::internal::generic_scheduler::release_task_pool | ( |  | ) | const

inline

Unlocks the local task pool.

Restores my_arena_slot->task_pool munged by acquire_task_pool. Requires correctly set my_arena_slot->task_pool_ptr.

Definition at line 522 of file scheduler.cpp.

{
if ( !is_task_pool_published() )
return; // we are not in arena - nothing to unlock
__TBB_ASSERT( my_arena_slot, "we are not in arena" );
__TBB_ASSERT( my_arena_slot->task_pool == LockedTaskPool, "arena slot is not locked" );
ITT_NOTIFY(sync_releasing, my_arena_slot);
__TBB_store_with_release( my_arena_slot->task_pool, my_arena_slot->task_pool_ptr );
}

References __TBB_ASSERT, tbb::internal::__TBB_store_with_release(), is_task_pool_published(), ITT_NOTIFY, LockedTaskPool, tbb::internal::scheduler_state::my_arena_slot, and sync_releasing.

Referenced by cleanup_master(), commit_relocated_tasks(), and get_task().

Here is the call graph for this function:
Here is the caller graph for this function:

## reset_task_pool_and_leave()

void tbb::internal::generic_scheduler::reset_task_pool_and_leave | ( |  | ) |

inline

Resets head and tail indices to 0, and leaves task pool.

The task pool must be locked by the owner (via acquire_task_pool).

Definition at line 702 of file scheduler.h.

{
__TBB_ASSERT( my_arena_slot->task_pool == LockedTaskPool, "Task pool must be locked when resetting task pool" );
__TBB_store_relaxed( my_arena_slot->tail, 0 );
__TBB_store_relaxed( my_arena_slot->head, 0 );
leave_task_pool();
}

References __TBB_ASSERT, tbb::internal::__TBB_store_relaxed(), leave_task_pool(), LockedTaskPool, and tbb::internal::scheduler_state::my_arena_slot.

Referenced by get_task().

Here is the call graph for this function:
Here is the caller graph for this function:

## spawn()

void tbb::internal::generic_scheduler::spawn | ( | task & | first,
|  | task *& | next
| ) |  |

virtual

For internal use only.

Implements tbb::internal::scheduler.

Definition at line 741 of file scheduler.cpp.

{
governor::local_scheduler()->local_spawn( &first, next );
}

References tbb::internal::first(), tbb::internal::governor::local_scheduler(), and local_spawn().

Here is the call graph for this function:

## spawn_root_and_wait()

void tbb::internal::generic_scheduler::spawn_root_and_wait | ( | task & | first,
|  | task *& | next
| ) |  |

virtual

For internal use only.

Implements tbb::internal::scheduler.

Definition at line 745 of file scheduler.cpp.

{
governor::local_scheduler()->local_spawn_root_and_wait( &first, next );
}
tbb::internal::generic_scheduler::local_spawn_root_and_wait
void local_spawn_root_and_wait(task *first, task *&next)
Definition: scheduler.cpp:720

References tbb::internal::first(), tbb::internal::governor::local_scheduler(), and local_spawn_root_and_wait().

Here is the call graph for this function:

## steal_task()

task * tbb::internal::generic_scheduler::steal_task | ( | __TBB_ISOLATION_EXPR(isolation_tag isolation) |  | ) |

Attempts to steal a task from a randomly chosen thread/scheduler.

Definition at line 1109 of file scheduler.cpp.

{
// Try to steal a task from a random victim.
size_t k = my_random.get() % (my_arena->my_limit-1);
arena_slot* victim = &my_arena->my_slots[k];
// The following condition excludes the master that might have
// already taken our previous place in the arena from the list .
// of potential victims. But since such a situation can take
// place only in case of significant oversubscription, keeping
// the checks simple seems to be preferable to complicating the code.
if( k >= my_arena_index )
++victim; // Adjusts random distribution to exclude self
task **pool = victim->task_pool;
task *t = NULL;
if( pool == EmptyTaskPool || !(t = steal_task_from( __TBB_ISOLATION_ARG(*victim, isolation) )) )
return NULL;
if( is_proxy(*t) ) {
task_proxy &tp = *(task_proxy*)t;
t = tp.extract_task<task_proxy::pool_bit>();
if ( !t ) {
// Proxy was empty, so it's our responsibility to free it
free_task<no_cache_small_task>(tp);
return NULL;
}
GATHER_STATISTIC( ++my_counters.proxies_stolen );
}
t->prefix().extra_state |= es_task_is_stolen;
if( is_version_3_task(*t) ) {
my_innermost_running_task = t;
t->prefix().owner = this;
t->note_affinity( my_affinity_id );
}
GATHER_STATISTIC( ++my_counters.steals_committed );
return t;
}
__TBB_ISOLATION_ARG
#define __TBB_ISOLATION_ARG(arg1, isolation)
Definition: scheduler_common.h:68
tbb::internal::task_prefix::owner
scheduler * owner
Obsolete. The scheduler that owns the task.
Definition: task.h:247
tbb::internal::generic_scheduler::steal_task_from
task * steal_task_from(__TBB_ISOLATION_ARG(arena_slot &victim_arena_slot, isolation_tag isolation))
Steal task from another scheduler's ready pool.
Definition: scheduler.cpp:1144
tbb::internal::FastRandom::get
unsigned short get()
Get a random number.
Definition: tbb_misc.h:146

References __TBB_ISOLATION_ARG, EmptyTaskPool, tbb::internal::es_task_is_stolen, tbb::internal::task_prefix::extra_state, tbb::internal::task_proxy::extract_task(), GATHER_STATISTIC, tbb::internal::FastRandom::get(), is_proxy(), is_version_3_task(), tbb::internal::scheduler_state::my_affinity_id, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_index, tbb::internal::scheduler_state::my_innermost_running_task, my_random, tbb::internal::arena::my_slots, tbb::task::note_affinity(), tbb::internal::task_prefix::owner, tbb::internal::task_proxy::pool_bit, tbb::task::prefix(), and steal_task_from().

Here is the call graph for this function:

## steal_task_from()

task * tbb::internal::generic_scheduler::steal_task_from | ( | __TBB_ISOLATION_ARG(arena_slot &victim_arena_slot, isolation_tag isolation) |  | ) |

Steal task from another scheduler's ready pool.

Definition at line 1144 of file scheduler.cpp.

{
task** victim_pool = lock_task_pool( &victim_slot );
if ( !victim_pool )
return NULL;
task* result = NULL;
size_t H = __TBB_load_relaxed(victim_slot.head); // mirror
size_t H0 = H;
bool tasks_omitted = false;
do {
__TBB_store_relaxed( victim_slot.head, ++H );
atomic_fence();
if ( (intptr_t)H > (intptr_t)__TBB_load_relaxed( victim_slot.tail ) ) {
// Stealing attempt failed, deque contents has not been changed by us
GATHER_STATISTIC( ++my_counters.thief_backoffs );
__TBB_store_relaxed( victim_slot.head, /*dead: H = */ H0 );
__TBB_ASSERT( !result, NULL );
goto unlock;
}
__TBB_control_consistency_helper(); // on victim_slot.tail
result = victim_pool[H-1];
__TBB_ASSERT( !is_poisoned( result ), NULL );
if ( result ) {
__TBB_ISOLATION_EXPR( if ( isolation == no_isolation || isolation == result->prefix().isolation ) )
{
if ( !is_proxy( *result ) )
break;
task_proxy& tp = *static_cast<task_proxy*>(result);
// If mailed task is likely to be grabbed by its destination thread, skip it.
if ( !(task_proxy::is_shared( tp.task_and_tag ) && tp.outbox->recipient_is_idle()) )
break;
GATHER_STATISTIC( ++my_counters.proxies_bypassed );
}
// The task cannot be executed either due to isolation or proxy constraints.
result = NULL;
tasks_omitted = true;
} else if ( !tasks_omitted ) {
// Cleanup the task pool from holes until a task is skipped.
__TBB_ASSERT( H0 == H-1, NULL );
poison_pointer( victim_pool[H0] );
H0 = H;
}
} while ( !result );
__TBB_ASSERT( result, NULL );
// emit "task was consumed" signal
ITT_NOTIFY( sync_acquired, (void*)((uintptr_t)&victim_slot+sizeof( uintptr_t )) );
poison_pointer( victim_pool[H-1] );
if ( tasks_omitted ) {
// Some proxies in the task pool have been omitted. Set the stolen task to NULL.
victim_pool[H-1] = NULL;
__TBB_store_relaxed( victim_slot.head, /*dead: H = */ H0 );
}
unlock:
unlock_task_pool( &victim_slot, victim_pool );
#if __TBB_PREFETCHING
__TBB_cl_evict(&victim_slot.head);
__TBB_cl_evict(&victim_slot.tail);
#endif
if ( tasks_omitted )
// Synchronize with snapshot as the head and tail can be bumped which can falsely trigger EMPTY state
my_arena->advertise_new_work<arena::wakeup>();
return result;
}
tbb::internal::task_proxy::is_shared
static bool is_shared(intptr_t tat)
True if the proxy is stored both in its sender's pool and in the destination mailbox.
Definition: mailbox.h:46
tbb::internal::generic_scheduler::lock_task_pool
task ** lock_task_pool(arena_slot *victim_arena_slot) const
Locks victim's task pool, and returns pointer to it. The pointer can be NULL.
Definition: scheduler.cpp:537
tbb::internal::generic_scheduler::unlock_task_pool
void unlock_task_pool(arena_slot *victim_arena_slot, task **victim_task_pool) const
Unlocks victim's task pool.
Definition: scheduler.cpp:586

References __TBB_ASSERT, __TBB_cl_evict, __TBB_control_consistency_helper, __TBB_ISOLATION_EXPR, tbb::internal::__TBB_load_relaxed(), tbb::internal::__TBB_store_relaxed(), tbb::internal::arena::advertise_new_work(), tbb::atomic_fence(), GATHER_STATISTIC, is_proxy(), tbb::internal::task_proxy::is_shared(), tbb::internal::task_prefix::isolation, ITT_NOTIFY, lock_task_pool(), tbb::internal::scheduler_state::my_arena, tbb::internal::no_isolation, tbb::internal::task_proxy::outbox, tbb::internal::poison_pointer(), tbb::task::prefix(), tbb::internal::mail_outbox::recipient_is_idle(), tbb::internal::task_proxy::task_and_tag, unlock_task_pool(), and tbb::internal::arena::wakeup.

Referenced by steal_task().

Here is the call graph for this function:
Here is the caller graph for this function:

## unlock_task_pool()

void tbb::internal::generic_scheduler::unlock_task_pool | ( | arena_slot * | victim_arena_slot,
|  | task ** | victim_task_pool
| ) |  | const

inline

Unlocks victim's task pool.

Restores victim_arena_slot->task_pool munged by lock_task_pool.

Definition at line 586 of file scheduler.cpp.

{
__TBB_ASSERT( victim_arena_slot, "empty victim arena slot pointer" );
__TBB_ASSERT( victim_arena_slot->task_pool == LockedTaskPool, "victim arena slot is not locked" );
ITT_NOTIFY(sync_releasing, victim_arena_slot);
__TBB_store_with_release( victim_arena_slot->task_pool, victim_task_pool );
}

References __TBB_ASSERT, tbb::internal::__TBB_store_with_release(), ITT_NOTIFY, LockedTaskPool, and sync_releasing.

Referenced by steal_task_from().

Here is the call graph for this function:
Here is the caller graph for this function:

## wait_until_empty()

void tbb::internal::generic_scheduler::wait_until_empty | ( |  | ) |

Definition at line 772 of file arena.cpp.

{
my_dummy_task->prefix().ref_count++; // prevents exit from local_wait_for_all when local work is done enforcing the stealing
while( my_arena->my_pool_state != arena::SNAPSHOT_EMPTY )
local_wait_for_all(*my_dummy_task, NULL);
my_dummy_task->prefix().ref_count--;
}
tbb::internal::arena::SNAPSHOT_EMPTY
static const pool_state_t SNAPSHOT_EMPTY
No tasks to steal since last snapshot was taken.
Definition: arena.h:318

References local_wait_for_all(), tbb::internal::scheduler_state::my_arena, my_dummy_task, tbb::task::prefix(), tbb::internal::task_prefix::ref_count, and tbb::internal::arena::SNAPSHOT_EMPTY.

Here is the call graph for this function:

## worker_outermost_level()

bool tbb::internal::generic_scheduler::worker_outermost_level | ( |  | ) | const

inline

True if the scheduler is on the outermost dispatch level in a worker thread.

Definition at line 657 of file scheduler.h.

{
return is_worker() && outermost_level();
}

References is_worker(), and outermost_level().

Here is the call graph for this function:

## Friends And Related Function Documentation

## custom_scheduler

template<typename SchedulerTraits >

friend class custom_scheduler

friend

Definition at line 389 of file scheduler.h.

## Member Data Documentation

## min_task_pool_size

const size_t tbb::internal::generic_scheduler::min_task_pool_size = 64

static

Initial size of the task deque sufficient to serve without reallocation 4 nested parallel_for calls with iteration space of 65535 grains each.

Definition at line 369 of file scheduler.h.

Referenced by local_spawn(), and prepare_task_pool().

## my_auto_initialized

bool tbb::internal::generic_scheduler::my_auto_initialized

True if *this was created by automatic TBB initialization.

Definition at line 197 of file scheduler.h.

## my_dummy_task

task* tbb::internal::generic_scheduler::my_dummy_task

Fake root task created by slave threads.

The task is used as the "parent" argument to method wait_for_all.

Definition at line 186 of file scheduler.h.

Referenced by attach_arena(), cleanup_master(), cleanup_scheduler(), generic_scheduler(), tbb::internal::nested_arena_context::mimic_outermost_level(), wait_until_empty(), and tbb::internal::nested_arena_context::~nested_arena_context().

## my_free_list

task* tbb::internal::generic_scheduler::my_free_list

Free list of small tasks that can be reused.

Definition at line 178 of file scheduler.h.

Referenced by allocate_task(), cleanup_scheduler(), and free_task().

## my_market

market* tbb::internal::generic_scheduler::my_market

The market I am in.

Definition at line 172 of file scheduler.h.

Referenced by attach_arena(), cleanup_master(), cleanup_scheduler(), and init_stack_info().

## my_random

FastRandom tbb::internal::generic_scheduler::my_random

Random number generator used for picking a random victim from which to steal.

Definition at line 175 of file scheduler.h.

Referenced by steal_task(), and tbb::internal::custom_scheduler< SchedulerTraits >::tally_completion_of_predecessor().

## my_ref_count

long tbb::internal::generic_scheduler::my_ref_count

Reference count for scheduler.

Number of task_scheduler_init objects that point to this scheduler

Definition at line 190 of file scheduler.h.

## my_return_list

task* tbb::internal::generic_scheduler::my_return_list

List of small tasks that have been returned to this scheduler by other schedulers.

Definition at line 465 of file scheduler.h.

Referenced by allocate_task(), cleanup_scheduler(), and generic_scheduler().

## my_small_task_count

__TBB_atomic intptr_t tbb::internal::generic_scheduler::my_small_task_count

Number of small tasks that have been allocated by this scheduler.

Definition at line 461 of file scheduler.h.

Referenced by allocate_task(), cleanup_scheduler(), and destroy().

## my_stealing_threshold

uintptr_t tbb::internal::generic_scheduler::my_stealing_threshold

Position in the call stack specifying its maximal filling when stealing is still allowed.

Definition at line 155 of file scheduler.h.

Referenced by can_steal(), and init_stack_info().

## null_arena_index

const size_t tbb::internal::generic_scheduler::null_arena_index = ~size_t(0)

static

Definition at line 161 of file scheduler.h.

## quick_task_size

const size_t tbb::internal::generic_scheduler::quick_task_size = 256-task_prefix_reservation_size

static

If sizeof(task) is <=quick_task_size, it is handled on a free list instead of malloc'd.

Definition at line 144 of file scheduler.h.

Referenced by allocate_task().

The documentation for this class was generated from the following files:

- scheduler.h
- arena.cpp
- scheduler.cpp

Copyright © 2005-2020 Intel Corporation. All Rights Reserved.

Intel, Pentium, Intel Xeon, Itanium, Intel XScale and VTune are registered trademarks or trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

* Other names and brands may be claimed as the property of others.