SparseFileManager Class Reference
File I/O

`#include <SparseFile.h>`

## Public Types

typedef std::list< SparseFile::CacheBlock > | CacheList

## Public Member Functions

template<class Data_T>
void | activateBlock (int fileId, int blockIdx)
| Called by SparseField when it's about to read from a block. This should not be called by the user, and may be removed from the public interface later.
float | cacheEfficiency ()
| Computes the efficiency, the ratio of the number of blocks ever loaded to the number of loads. If this is <1, then there were reloads.
float | cacheFractionLoaded ()
| Computes the ratio of blocks in the cache to the total number of blocks that have been loaded (including unloaded blocks).
float | cacheLoadsPerBlock ()
| Computes the overall loaded-blocks-to-load ratio for cached files.
template<class Data_T>
void | decBlockRef (int fileId, int blockIdx)
| Decrements the usage reference count on the specified block, after its value is no longer being used This should not be called by the user, and may be removed from the public interface later.
bool | doLimitMemUse () const
| Returns whether to limit memory usage and do dynamic loading for sparse fields.
void | flushCache ()
| Flushes the entire block cache for all files, should probably only be used for debugging.
template<class Data_T>
void | incBlockRef (int fileId, int blockIdx)
| Increments the usage reference count on the specified block, to prevent it from getting unloaded while it's still in use. This should not be called by the user, and may be removed from the public interface later.
long long int | memSize () const
| Returns the number of bytes used by the SparseFileManager itself.
long long | numLoadedBlocks ()
| Returns the total number of blocks currently loaded into cache.
void | resetCacheStatistics ()
| Resets block load.
void | setLimitMemUse (bool enabled)
| Sets whether to limit memory usage and do dynamic loading for sparse fields.
void | setMaxMemUse (float maxMemUse)
| Sets the maximum memory usage, in MB, by dynamically loaded sparse fields.
long long | totalLoadedBlocks ()
| Returns the total number of blocks loaded (max 1 per block) into cache.
long long | totalLoads ()
| Returns the total number of block loads in the cache.

## Static Public Member Functions

static SparseFileManager & | singleton ()
| Returns a reference to the singleton instance.

## Protected Member Functions

template<class Data_T>
int | getNextId (const std::string filename, const std::string layerPath)
| Returns the id of the next cache item. This is stored in the SparseField in order to reference its fields at a later time.
template<class Data_T>
SparseFile::Reference< Data_T > * | reference (int index)
| Returns a reference to the Reference object with the given index.
template<class Data_T>
void | removeFieldFromCache (int refIdx)

## Private Member Functions

void | addBlockToCache (DataTypeEnum blockType, int fileId, int blockIdx)
| Adds the newly loaded block to the cache, managed by the paging algorithm.
template<class Data_T>
void | deallocateBlock (CacheList::iterator &it)
| Utility function to deallocate a single block.
template<class Data_T>
int64_t | deallocateBlock (const SparseFile::CacheBlock &cb)
| Utility function to attempt to deallocate a single block and advance the "hand".
void | deallocateBlocks (int64_t bytesNeeded)
| Utility function to reclaim the specified number of bytes by deallocating unneeded blocks.
| SparseFileManager ()
| Private to prevent instantiation.

## Private Attributes

CacheList | m_blockCacheList
| List of dynamically loaded blocks to be considered for unloading when the cache is full. Currently using Second-chance/Clock paging algorithm. For a description of the algorithm, look at: http://en.wikipedia.org/wiki/Page_replacement_algorithm#Second-chance.
SparseFile::FileReferences | m_fileData
| Vector containing information for each of the managed fields. The order matches the index stored in each SparseField::m_fileId.
bool | m_limitMemUse
| Whether to limit memory use of sparse fields from disk. Enables the cache and dynamic loading when true.
float | m_maxMemUse
| Max amount om memory to use in megabytes.
int64_t | m_maxMemUseInBytes
| Max amount om memory to use in bytes.
int64_t | m_memUse
| Current amount of memory in use in bytes.
boost::mutex | m_mutex
| Mutex to prevent multiple threads from deallocating blocks at the same time.
CacheList::iterator | m_nextBlock
| Pointer to the next block to test for unloading in the cache, the "hand" of the clock.

## Static Private Attributes

static boost::scoped_ptr< SparseFileManager > | ms_singleton
| Pointer to singleton.

## Friends

template<class Data_T>
class | SparseField

## Detailed Description

Handles specifics about reading sparse fields from disk. Its primary use is to control sparse fields read using memory limiting (dynamic loading).

To enable the dynamic cache for a file, call setLimitMemUse(true) before opening the file. If you want other files to be fully loaded, call setLimitMemUse(false).

Example of how to use the cache manager to automatically unload sparse blocks from a f3d file:

    SparseFileManager &sparseManager = SparseFileManager::singleton();
    sparseManager.setLimitMemUse(true);  // enables cache for files to be opened
    sparseManager.setMaxMemUse(1000.0);  // sets cache to 1 GB
    Field3DInputFile cacheManagedFile;
    if (!cacheManagedFile.open(filename)) {
      Msg::print( "Couldn't open file: " + filename);
      return 1;
    }
    sparseManager.setLimitMemUse(false);  // disables cache for other files

    ... // You can use the file normally, loading layers and then accessing
    ... // with const_iterator, value(), or empty block functions like
    ... // getBlockEmptyValue().
    ... // Layers loaded from cacheManagedFile will be managed by the cache,
    ... // but other files you open will be fully loaded when opened because of
    ... // the setLimitMemUse(false) call.

    Msg::print("blocks in cache: " +
      boost::lexical_cast<std::string>(sparseManager.numLoadedBlocks()));
    Msg::print("cache blocks ever loaded: " +
      boost::lexical_cast<std::string>(sparseManager.totalLoadedBlocks()));
    Msg::print("cache loads: " +
      boost::lexical_cast<std::string>(sparseManager.totalLoads()));
    Msg::print("cache fraction loaded: " +
      boost::lexical_cast<std::string>(sparseManager.cacheFractionLoaded()));
    Msg::print("cache loads per block: " +
      boost::lexical_cast<std::string>(sparseManager.cacheLoadsPerBlock()));
    Msg::print("cache efficiency: " +
      boost::lexical_cast<std::string>(sparseManager.cacheEfficiency()));

If you want to flush the cache manually instead of waiting for the process to end and clean up its memory:

    sparseManager.flushCache();
    sparseManager.resetCacheStatistics();

Definition at line 397 of file SparseFile.h.

## Member Typedef Documentation

## CacheList

typedef std::list<SparseFile::CacheBlock> SparseFileManager::CacheList

Definition at line 407 of file SparseFile.h.

## Constructor & Destructor Documentation

## SparseFileManager()

SparseFileManager::SparseFileManager | ( |  | ) |

private

Private to prevent instantiation.

Definition at line 282 of file SparseFile.cpp.

: m_memUse(0),
m_limitMemUse(false)
{
setMaxMemUse(1000.0);
m_nextBlock = m_blockCacheList.begin();
}
SparseFileManager::setMaxMemUse
void setMaxMemUse(float maxMemUse)
Sets the maximum memory usage, in MB, by dynamically loaded sparse fields.
Definition SparseFile.cpp:90
SparseFileManager::m_limitMemUse
bool m_limitMemUse
Whether to limit memory use of sparse fields from disk. Enables the cache and dynamic loading when tr...
Definition SparseFile.h:527
SparseFileManager::m_blockCacheList
CacheList m_blockCacheList
List of dynamically loaded blocks to be considered for unloading when the cache is full....
Definition SparseFile.h:538
SparseFileManager::m_memUse
int64_t m_memUse
Current amount of memory in use in bytes.
Definition SparseFile.h:523
SparseFileManager::m_nextBlock
CacheList::iterator m_nextBlock
Pointer to the next block to test for unloading in the cache, the "hand" of the clock.
Definition SparseFile.h:542

References m_blockCacheList, m_limitMemUse, m_memUse, m_nextBlock, and setMaxMemUse().

Referenced by singleton().

## Member Function Documentation

## singleton()

SparseFileManager & SparseFileManager::singleton | ( |  | ) |

static

Returns a reference to the singleton instance.

Definition at line 66 of file SparseFile.cpp.

{
if (ms_singleton.get() == NULL) {
ms_singleton.reset(new SparseFileManager);
}
return *ms_singleton;
}
SparseFileManager::ms_singleton
static boost::scoped_ptr< SparseFileManager > ms_singleton
Pointer to singleton.
Definition SparseFile.h:498
SparseFileManager::SparseFileManager
SparseFileManager()
Private to prevent instantiation.
Definition SparseFile.cpp:282

References ms_singleton, and SparseFileManager().

Referenced by SparseField< Data_T >::addReference().

## setLimitMemUse()

void SparseFileManager::setLimitMemUse | ( | bool | enabled | ) |

Sets whether to limit memory usage and do dynamic loading for sparse fields.

Definition at line 76 of file SparseFile.cpp.

{
m_limitMemUse = enabled;
}

References m_limitMemUse.

## doLimitMemUse()

bool SparseFileManager::doLimitMemUse | ( |  | ) | const

Returns whether to limit memory usage and do dynamic loading for sparse fields.

Definition at line 83 of file SparseFile.cpp.

{
return m_limitMemUse;
}

References m_limitMemUse.

## setMaxMemUse()

void SparseFileManager::setMaxMemUse | ( | float | maxMemUse | ) |

Sets the maximum memory usage, in MB, by dynamically loaded sparse fields.

Definition at line 90 of file SparseFile.cpp.

{
m_maxMemUse = maxMemUse;
m_maxMemUseInBytes = static_cast<int64_t>(m_maxMemUse * 1024 * 1024);
}
SparseFileManager::m_maxMemUse
float m_maxMemUse
Max amount om memory to use in megabytes.
Definition SparseFile.h:517
SparseFileManager::m_maxMemUseInBytes
int64_t m_maxMemUseInBytes
Max amount om memory to use in bytes.
Definition SparseFile.h:520

References m_maxMemUse, and m_maxMemUseInBytes.

Referenced by SparseFileManager().

## flushCache()

void SparseFileManager::flushCache | ( |  | ) |

Flushes the entire block cache for all files, should probably only be used for debugging.

Definition at line 223 of file SparseFile.cpp.

{
boost::mutex::scoped_lock lock(m_mutex);
CacheList::iterator it = m_blockCacheList.begin();
while (it != m_blockCacheList.end()) {
SparseFile::CacheBlock &cb = *it;
switch(cb.blockType) {
case DataTypeHalf:
deallocateBlock<half>(it);
break;
case DataTypeFloat:
deallocateBlock<float>(it);
break;
case DataTypeDouble:
deallocateBlock<double>(it);
break;
case DataTypeVecHalf:
deallocateBlock<V3h>(it);
break;
case DataTypeVecFloat:
deallocateBlock<V3f>(it);
break;
case DataTypeVecDouble:
deallocateBlock<V3d>(it);
break;
case DataTypeUnknown:
default:
break;
}
}
m_nextBlock = m_blockCacheList.begin();
}
DataTypeFloat
@ DataTypeFloat
Definition Traits.h:112
DataTypeHalf
@ DataTypeHalf
Definition Traits.h:109
DataTypeVecHalf
@ DataTypeVecHalf
Definition Traits.h:114
DataTypeVecDouble
@ DataTypeVecDouble
Definition Traits.h:116
DataTypeUnknown
@ DataTypeUnknown
Definition Traits.h:117
DataTypeVecFloat
@ DataTypeVecFloat
Definition Traits.h:115
DataTypeDouble
@ DataTypeDouble
Definition Traits.h:113
SparseFileManager::deallocateBlock
int64_t deallocateBlock(const SparseFile::CacheBlock &cb)
Utility function to attempt to deallocate a single block and advance the "hand".
Definition SparseFile.cpp:99
SparseFileManager::m_mutex
boost::mutex m_mutex
Mutex to prevent multiple threads from deallocating blocks at the same time.
Definition SparseFile.h:546
SparseFile::CacheBlock::blockType
DataTypeEnum blockType
Definition SparseFile.h:324

References SparseFile::CacheBlock::blockType, DataTypeDouble, DataTypeFloat, DataTypeHalf, DataTypeUnknown, DataTypeVecDouble, DataTypeVecFloat, DataTypeVecHalf, deallocateBlock(), m_blockCacheList, m_mutex, and m_nextBlock.

## totalLoads()

long long SparseFileManager::totalLoads | ( |  | ) |

Returns the total number of block loads in the cache.

Definition at line 292 of file SparseFile.cpp.

{
long long int numLoads = 0;
for (size_t i=0; i<m_fileData.numRefs<half>(); i++) {
numLoads += m_fileData.ref<half>(i)->totalLoads();
}
for (size_t i=0; i<m_fileData.numRefs<V3h>(); i++) {
numLoads += m_fileData.ref<V3h>(i)->totalLoads();
}
for (size_t i=0; i<m_fileData.numRefs<float>(); i++) {
numLoads += m_fileData.ref<float>(i)->totalLoads();
}
for (size_t i=0; i<m_fileData.numRefs<V3f>(); i++) {
numLoads += m_fileData.ref<V3f>(i)->totalLoads();
}
for (size_t i=0; i<m_fileData.numRefs<double>(); i++) {
numLoads += m_fileData.ref<double>(i)->totalLoads();
}
for (size_t i=0; i<m_fileData.numRefs<V3d>(); i++) {
numLoads += m_fileData.ref<V3d>(i)->totalLoads();
}
return numLoads;
}
V3d
Imath::V3d V3d
Definition SpiMathLib.h:74
half
FIELD3D_NAMESPACE_OPENtypedef ::half half
Definition SpiMathLib.h:64
V3h
Imath::Vec3< half > V3h
Definition SpiMathLib.h:72
V3f
Imath::V3f V3f
Definition SpiMathLib.h:73
SparseFileManager::m_fileData
SparseFile::FileReferences m_fileData
Vector containing information for each of the managed fields. The order matches the index stored in e...
Definition SparseFile.h:531
SparseFileManager::totalLoads
long long totalLoads()
Returns the total number of block loads in the cache.
Definition SparseFile.cpp:292

References half, m_fileData, and totalLoads().

Referenced by cacheEfficiency(), cacheLoadsPerBlock(), and totalLoads().

## numLoadedBlocks()

long long SparseFileManager::numLoadedBlocks | ( |  | ) |

Returns the total number of blocks currently loaded into cache.

Definition at line 325 of file SparseFile.cpp.

{
long long int numBlocks = 0;
for (size_t i=0; i<m_fileData.numRefs<half>(); i++) {
numBlocks += m_fileData.ref<half>(i)->numLoadedBlocks();
}
for (size_t i=0; i<m_fileData.numRefs<V3h>(); i++) {
numBlocks += m_fileData.ref<V3h>(i)->numLoadedBlocks();
}
for (size_t i=0; i<m_fileData.numRefs<float>(); i++) {
numBlocks += m_fileData.ref<float>(i)->numLoadedBlocks();
}
for (size_t i=0; i<m_fileData.numRefs<V3f>(); i++) {
numBlocks += m_fileData.ref<V3f>(i)->numLoadedBlocks();
}
for (size_t i=0; i<m_fileData.numRefs<double>(); i++) {
numBlocks += m_fileData.ref<double>(i)->numLoadedBlocks();
}
for (size_t i=0; i<m_fileData.numRefs<V3d>(); i++) {
numBlocks += m_fileData.ref<V3d>(i)->numLoadedBlocks();
}
return numBlocks;
}
SparseFileManager::numLoadedBlocks
long long numLoadedBlocks()
Returns the total number of blocks currently loaded into cache.
Definition SparseFile.cpp:325

References half, m_fileData, and numLoadedBlocks().

Referenced by cacheFractionLoaded(), and numLoadedBlocks().

## totalLoadedBlocks()

long long SparseFileManager::totalLoadedBlocks | ( |  | ) |

Returns the total number of blocks loaded (max 1 per block) into cache.

Definition at line 358 of file SparseFile.cpp.

{
long long int numBlocks = 0;
for (size_t i=0; i<m_fileData.numRefs<half>(); i++) {
numBlocks += m_fileData.ref<half>(i)->totalLoadedBlocks();
}
for (size_t i=0; i<m_fileData.numRefs<V3h>(); i++) {
numBlocks += m_fileData.ref<V3h>(i)->totalLoadedBlocks();
}
for (size_t i=0; i<m_fileData.numRefs<float>(); i++) {
numBlocks += m_fileData.ref<float>(i)->totalLoadedBlocks();
}
for (size_t i=0; i<m_fileData.numRefs<V3f>(); i++) {
numBlocks += m_fileData.ref<V3f>(i)->totalLoadedBlocks();
}
for (size_t i=0; i<m_fileData.numRefs<double>(); i++) {
numBlocks += m_fileData.ref<double>(i)->totalLoadedBlocks();
}
for (size_t i=0; i<m_fileData.numRefs<V3d>(); i++) {
numBlocks += m_fileData.ref<V3d>(i)->totalLoadedBlocks();
}
return numBlocks;
}
SparseFileManager::totalLoadedBlocks
long long totalLoadedBlocks()
Returns the total number of blocks loaded (max 1 per block) into cache.
Definition SparseFile.cpp:358

References half, m_fileData, and totalLoadedBlocks().

Referenced by cacheEfficiency(), cacheFractionLoaded(), cacheLoadsPerBlock(), and totalLoadedBlocks().

## cacheFractionLoaded()

float SparseFileManager::cacheFractionLoaded | ( |  | ) |

Computes the ratio of blocks in the cache to the total number of blocks that have been loaded (including unloaded blocks).

Definition at line 391 of file SparseFile.cpp.

{
return ((double)numLoadedBlocks())/std::max(1.0, ((double)totalLoadedBlocks()));
}

References numLoadedBlocks(), and totalLoadedBlocks().

## cacheLoadsPerBlock()

float SparseFileManager::cacheLoadsPerBlock | ( |  | ) |

Computes the overall loaded-blocks-to-load ratio for cached files.

Definition at line 398 of file SparseFile.cpp.

{
return ((double)totalLoads())/std::max(1.0, ((double)totalLoadedBlocks()));
}

References totalLoadedBlocks(), and totalLoads().

## cacheEfficiency()

float SparseFileManager::cacheEfficiency | ( |  | ) |

Computes the efficiency, the ratio of the number of blocks ever loaded to the number of loads. If this is <1, then there were reloads.

Definition at line 405 of file SparseFile.cpp.

{
return ((double)totalLoadedBlocks())/std::max(1.0, ((double)totalLoads()));
}

References totalLoadedBlocks(), and totalLoads().

## resetCacheStatistics()

void SparseFileManager::resetCacheStatistics | ( |  | ) |

Resets block load.

Definition at line 412 of file SparseFile.cpp.

{
for (size_t i=0; i<m_fileData.numRefs<half>(); i++) {
m_fileData.ref<half>(i)->resetCacheStatistics();
}
for (size_t i=0; i<m_fileData.numRefs<V3h>(); i++) {
m_fileData.ref<V3h>(i)->resetCacheStatistics();
}
for (size_t i=0; i<m_fileData.numRefs<float>(); i++) {
m_fileData.ref<float>(i)->resetCacheStatistics();
}
for (size_t i=0; i<m_fileData.numRefs<V3f>(); i++) {
m_fileData.ref<V3f>(i)->resetCacheStatistics();
}
for (size_t i=0; i<m_fileData.numRefs<double>(); i++) {
m_fileData.ref<double>(i)->resetCacheStatistics();
}
for (size_t i=0; i<m_fileData.numRefs<V3d>(); i++) {
m_fileData.ref<V3d>(i)->resetCacheStatistics();
}
}
SparseFileManager::resetCacheStatistics
void resetCacheStatistics()
Resets block load.
Definition SparseFile.cpp:412

References half, m_fileData, and resetCacheStatistics().

Referenced by resetCacheStatistics().

## memSize()

long long int SparseFileManager::memSize | ( |  | ) | const

Returns the number of bytes used by the SparseFileManager itself.

Definition at line 442 of file SparseFile.cpp.

{
boost::mutex::scoped_lock lock(m_mutex);
return sizeof(*this) + m_fileData.memSize() +
m_blockCacheList.size() * sizeof(SparseFile::CacheBlock);
}

References m_blockCacheList, m_fileData, and m_mutex.

## incBlockRef()

template<class Data_T>

void SparseFileManager::incBlockRef | ( | int | fileId,
|  | int | blockIdx )

Increments the usage reference count on the specified block, to prevent it from getting unloaded while it's still in use. This should not be called by the user, and may be removed from the public interface later.

Definition at line 1298 of file SparseFile.h.

{
SparseFile::Reference<Data_T> *reference = m_fileData.ref<Data_T>(fileId);
if (reference->fileBlockIndices[blockIdx] >= 0) {
reference->incBlockRef(blockIdx);
}
}
SparseFileManager::reference
SparseFile::Reference< Data_T > * reference(int index)
Returns a reference to the Reference object with the given index.
Definition SparseFile.h:1248

References m_fileData, and reference().

## decBlockRef()

template<class Data_T>

void SparseFileManager::decBlockRef | ( | int | fileId,
|  | int | blockIdx )

Decrements the usage reference count on the specified block, after its value is no longer being used This should not be called by the user, and may be removed from the public interface later.

Definition at line 1311 of file SparseFile.h.

{
SparseFile::Reference<Data_T> *reference = m_fileData.ref<Data_T>(fileId);
if (reference->fileBlockIndices[blockIdx] >= 0) {
reference->decBlockRef(blockIdx);
}
}

References m_fileData, and reference().

## activateBlock()

template<class Data_T>

void SparseFileManager::activateBlock | ( | int | fileId,
|  | int | blockIdx )

Called by SparseField when it's about to read from a block. This should not be called by the user, and may be removed from the public interface later.

Definition at line 1257 of file SparseFile.h.

{
SparseFile::Reference<Data_T> *reference = m_fileData.ref<Data_T>(fileId);
if (reference->fileBlockIndices[blockIdx] >= 0) {
if (!reference->blockLoaded[blockIdx]) {
int blockSize = reference->blockSize(blockIdx);
if (m_limitMemUse) {
// if we already have enough free memory, deallocateBlocks()
// will just return
deallocateBlocks(blockSize);
}
if (!reference->fileIsOpen()) {
reference->openFile();
}
boost::mutex::scoped_lock lock_A(m_mutex);
#if F3D_SHORT_MUTEX_ARRAY
boost::mutex::scoped_lock
lock_B(reference->blockMutex[blockIdx % reference->blockMutexSize]);
#else
boost::mutex::scoped_lock lock_B(reference->blockMutex[blockIdx]);
#endif
// check to see if it was loaded between when the function
// started and we got the lock on the block
if (!reference->blockLoaded[blockIdx]) {
reference->loadBlock(blockIdx);
reference->loadCounts[blockIdx]++;
addBlockToCache(DataTypeTraits<Data_T>::typeEnum(), fileId, blockIdx);
m_memUse += blockSize;
}
}
}
reference->blockUsed[blockIdx] = true;
}
SparseFileManager::addBlockToCache
void addBlockToCache(DataTypeEnum blockType, int fileId, int blockIdx)
Adds the newly loaded block to the cache, managed by the paging algorithm.
Definition SparseFile.cpp:260
SparseFileManager::deallocateBlocks
void deallocateBlocks(int64_t bytesNeeded)
Utility function to reclaim the specified number of bytes by deallocating unneeded blocks.
Definition SparseFile.cpp:160
DataTypeTraits::typeEnum
static DataTypeEnum typeEnum()

References addBlockToCache(), deallocateBlocks(), m_fileData, m_limitMemUse, m_memUse, m_mutex, reference(), and DataTypeTraits< T >::typeEnum().

Referenced by SparseField< Data_T >::const_iterator::operator->().

## reference()

template<class Data_T>

SparseFile::Reference< Data_T > * SparseFileManager::reference | ( | int | index | ) |

protected

Returns a reference to the Reference object with the given index.

Definition at line 1248 of file SparseFile.h.

{
return m_fileData.ref<Data_T>(index);
}

References m_fileData.

Referenced by activateBlock(), deallocateBlock(), deallocateBlock(), decBlockRef(), incBlockRef(), and removeFieldFromCache().

## getNextId()

template<class Data_T>

int SparseFileManager::getNextId | ( | const std::string | filename,
|  | const std::string | layerPath )

protected

Returns the id of the next cache item. This is stored in the SparseField in order to reference its fields at a later time.

Definition at line 1182 of file SparseFile.h.

{
using namespace SparseFile;
// Must hold a mutex while appending to m_fileData
boost::mutex::scoped_lock lock(m_mutex);
int id = m_fileData.append<Data_T>(Reference<Data_T>::create(filename,
layerPath));
return id;
}

References SparseFile::Reference< Data_T >::create(), m_fileData, and m_mutex.

## removeFieldFromCache()

template<class Data_T>

void SparseFileManager::removeFieldFromCache | ( | int | refIdx | ) |

protected

Definition at line 1199 of file SparseFile.h.

{
boost::mutex::scoped_lock lock(m_mutex);
DataTypeEnum blockType = DataTypeTraits<Data_T>::typeEnum();
SparseFile::Reference<Data_T> *reference = m_fileData.ref<Data_T>(refIdx);
CacheList::iterator it = m_blockCacheList.begin();
CacheList::iterator end = m_blockCacheList.end();
CacheList::iterator next;
int64_t bytesFreed = 0;
while (it != end) {
if (it->blockType == blockType && it->refIdx == refIdx) {
if (it == m_nextBlock) {
++m_nextBlock;
}
next = it;
++next;
bytesFreed += reference->blockSize(it->blockIdx);
m_blockCacheList.erase(it);
it = next;
} else {
++it;
}
}
m_memUse -= bytesFreed;
std::vector<int>().swap(reference->fileBlockIndices);
#if F3D_NO_BLOCKS_ARRAY
reference->fileBlockIndices.resize(reference->numBlocks, -1);
#else
reference->fileBlockIndices.resize(reference->blocks.size(), -1);
typedef typename SparseFile::Reference<Data_T>::BlockPtrs BlockPtrs;
BlockPtrs().swap(reference->blocks);
#endif
std::vector<int>().swap(reference->blockLoaded);
std::vector<bool>().swap(reference->blockUsed);
std::vector<int>().swap(reference->loadCounts);
std::vector<int>().swap(reference->refCounts);
delete[] reference->blockMutex;
reference->blockMutex = NULL;
}
DataTypeEnum
DataTypeEnum
Definition Traits.h:108
SparseFile::Reference::BlockPtrs
Sparse::SparseBlock< Data_T > * BlockPtrs
Definition SparseFile.h:120

References m_blockCacheList, m_fileData, m_memUse, m_mutex, m_nextBlock, reference(), and DataTypeTraits< T >::typeEnum().

## addBlockToCache()

void SparseFileManager::addBlockToCache | ( | DataTypeEnum | blockType,
|  | int | fileId,
|  | int | blockIdx )

private

Adds the newly loaded block to the cache, managed by the paging algorithm.

Definition at line 260 of file SparseFile.cpp.

{
// Note: this lock is obtained while we also have a lock on the
// specific block (in activateBlock()), so we should make sure we
// never lock the SparseFileManager and *then* a block, to ensure we
// don't have a deadlock.
//
// Note: this was changed so the order was consistent w/ dealloc
// again, see activateBlock()
// boost::mutex::scoped_lock lock(m_mutex);
SparseFile::CacheBlock block(blockType, fileId, blockIdx);
if (m_nextBlock == m_blockCacheList.end()) {
m_blockCacheList.push_back(block);
} else {
m_blockCacheList.insert(m_nextBlock, block);
}
}

References m_blockCacheList, and m_nextBlock.

Referenced by activateBlock().

## deallocateBlocks()

void SparseFileManager::deallocateBlocks | ( | int64_t | bytesNeeded | ) |

private

Utility function to reclaim the specified number of bytes by deallocating unneeded blocks.

Definition at line 160 of file SparseFile.cpp.

{
boost::mutex::scoped_lock lock_A(m_mutex);
while (m_blockCacheList.begin() != m_blockCacheList.end() &&
m_maxMemUseInBytes-m_memUse < bytesNeeded) {
if (m_nextBlock == m_blockCacheList.end())
m_nextBlock = m_blockCacheList.begin();
SparseFile::CacheBlock &cb = *m_nextBlock;
// if bytesFreed is set to >0, then we've already freed a block
// and advanced the "clock hand" iterator
int64_t bytesFreed = 0;
switch(cb.blockType) {
case DataTypeHalf:
bytesFreed = deallocateBlock<half>(cb);
if (bytesFreed > 0) {
continue;
}
break;
case DataTypeFloat:
bytesFreed = deallocateBlock<float>(cb);
if (bytesFreed > 0) {
continue;
}
break;
case DataTypeDouble:
bytesFreed = deallocateBlock<double>(cb);
if (bytesFreed > 0) {
continue;
}
break;
case DataTypeVecHalf:
bytesFreed = deallocateBlock<V3h>(cb);
if (bytesFreed > 0) {
continue;
}
break;
case DataTypeVecFloat:
bytesFreed = deallocateBlock<V3f>(cb);
if (bytesFreed > 0) {
continue;
}
break;
case DataTypeVecDouble:
bytesFreed = deallocateBlock<V3d>(cb);
if (bytesFreed > 0) {
continue;
}
break;
case DataTypeUnknown:
default:
break;
}
++m_nextBlock;
}
}

References SparseFile::CacheBlock::blockType, DataTypeDouble, DataTypeFloat, DataTypeHalf, DataTypeUnknown, DataTypeVecDouble, DataTypeVecFloat, DataTypeVecHalf, deallocateBlock(), m_blockCacheList, m_maxMemUseInBytes, m_memUse, m_mutex, and m_nextBlock.

Referenced by activateBlock().

## deallocateBlock() [1/2]

template<class Data_T>

int64_t SparseFileManager::deallocateBlock | ( | const SparseFile::CacheBlock & | cb | ) |

private

Utility function to attempt to deallocate a single block and advance the "hand".

Definition at line 99 of file SparseFile.cpp.

{
int64_t bytesFreed = 0;
SparseFile::Reference<Data_T> *reference = m_fileData.ref<Data_T>(cb.refIdx);
// Note: we don't need to lock the block's mutex because
// deallocateBlock() is only called while the SparseFileManager's
// mutex is also locked (in flushCache() or deallocateBlocks()).
// Don't lock the block, to make sure we don't have a deadlock by
// holding two locks at the same time. (Because addBlockToCache()
// locks the manager but is also in a block-specific lock.)
// lock the current block to make sure its blockUsed flag and ref
// counts don't change
// Note: this lock order is made consistent w/ allocate to prevent
// deadlocks and crashes.
#if F3D_SHORT_MUTEX_ARRAY
boost::mutex::scoped_lock
lock_B(reference->blockMutex[cb.blockIdx % reference->blockMutexSize]);
#else
boost::mutex::scoped_lock lock_B(reference->blockMutex[cb.blockIdx]);
#endif
// check whether the block is still in use
if (reference->refCounts[cb.blockIdx] > 0)
return bytesFreed;
if (reference->blockUsed[cb.blockIdx]) {
// the block was recently used according to Second-chance paging
// algorithm, so skip it
reference->blockUsed[cb.blockIdx] = false;
}
else {
// the block wasn't in use, so free it
reference->unloadBlock(cb.blockIdx);
bytesFreed = reference->blockSize(cb.blockIdx);
m_memUse -= bytesFreed;
CacheList::iterator toRemove = m_nextBlock;
++m_nextBlock;
m_blockCacheList.erase(toRemove);
}
return bytesFreed;
}
SparseFile::CacheBlock::refIdx
int refIdx
Definition SparseFile.h:325
SparseFile::CacheBlock::blockIdx
int blockIdx
Definition SparseFile.h:326

References SparseFile::CacheBlock::blockIdx, m_blockCacheList, m_fileData, m_memUse, m_nextBlock, reference(), and SparseFile::CacheBlock::refIdx.

Referenced by deallocateBlocks(), and flushCache().

## deallocateBlock() [2/2]

template<class Data_T>

void SparseFileManager::deallocateBlock | ( | CacheList::iterator & | it | ) |

private

Utility function to deallocate a single block.

Definition at line 148 of file SparseFile.cpp.

{
SparseFile::CacheBlock &cb = *it;
SparseFile::Reference<Data_T> *reference = m_fileData.ref<Data_T>(cb.refIdx);
int64_t bytesFreed = reference->blockSize(cb.blockIdx);
m_memUse -= bytesFreed;
reference->unloadBlock(cb.blockIdx);
it = m_blockCacheList.erase(it);
}

References SparseFile::CacheBlock::blockIdx, m_blockCacheList, m_fileData, m_memUse, reference(), and SparseFile::CacheBlock::refIdx.

## Friends And Related Symbol Documentation

## SparseField

template<class Data_T>

friend class SparseField

friend

Definition at line 403 of file SparseFile.h.

References SparseField.

Referenced by SparseField.

## Member Data Documentation

## ms_singleton

FIELD3D_NAMESPACE_OPEN boost::scoped_ptr< SparseFileManager > SparseFileManager::ms_singleton

static private

Pointer to singleton.

Definition at line 498 of file SparseFile.h.

Referenced by singleton().

## m_maxMemUse

float SparseFileManager::m_maxMemUse

private

Max amount om memory to use in megabytes.

Definition at line 517 of file SparseFile.h.

Referenced by setMaxMemUse().

## m_maxMemUseInBytes

int64_t SparseFileManager::m_maxMemUseInBytes

private

Max amount om memory to use in bytes.

Definition at line 520 of file SparseFile.h.

Referenced by deallocateBlocks(), and setMaxMemUse().

## m_memUse

int64_t SparseFileManager::m_memUse

private

Current amount of memory in use in bytes.

Definition at line 523 of file SparseFile.h.

Referenced by activateBlock(), deallocateBlock(), deallocateBlock(), deallocateBlocks(), removeFieldFromCache(), and SparseFileManager().

## m_limitMemUse

bool SparseFileManager::m_limitMemUse

private

Whether to limit memory use of sparse fields from disk. Enables the cache and dynamic loading when true.

Definition at line 527 of file SparseFile.h.

Referenced by activateBlock(), doLimitMemUse(), setLimitMemUse(), and SparseFileManager().

## m_fileData

SparseFile::FileReferences SparseFileManager::m_fileData

private

Vector containing information for each of the managed fields. The order matches the index stored in each SparseField::m_fileId.

Definition at line 531 of file SparseFile.h.

Referenced by activateBlock(), deallocateBlock(), deallocateBlock(), decBlockRef(), getNextId(), incBlockRef(), memSize(), numLoadedBlocks(), reference(), removeFieldFromCache(), resetCacheStatistics(), totalLoadedBlocks(), and totalLoads().

## m_blockCacheList

CacheList SparseFileManager::m_blockCacheList

private

List of dynamically loaded blocks to be considered for unloading when the cache is full. Currently using Second-chance/Clock paging algorithm. For a description of the algorithm, look at: http://en.wikipedia.org/wiki/Page_replacement_algorithm#Second-chance.

Definition at line 538 of file SparseFile.h.

Referenced by addBlockToCache(), deallocateBlock(), deallocateBlock(), deallocateBlocks(), flushCache(), memSize(), removeFieldFromCache(), and SparseFileManager().

## m_nextBlock

CacheList::iterator SparseFileManager::m_nextBlock

private

Pointer to the next block to test for unloading in the cache, the "hand" of the clock.

Definition at line 542 of file SparseFile.h.

Referenced by addBlockToCache(), deallocateBlock(), deallocateBlocks(), flushCache(), removeFieldFromCache(), and SparseFileManager().

## m_mutex

boost::mutex SparseFileManager::m_mutex

mutable private

Mutex to prevent multiple threads from deallocating blocks at the same time.

Definition at line 546 of file SparseFile.h.

Referenced by activateBlock(), deallocateBlocks(), flushCache(), getNextId(), memSize(), and removeFieldFromCache().

The documentation for this class was generated from the following files:

- export/SparseFile.h
- src/SparseFile.cpp