SFSMASTER.CFG(5)

NAME
sfsmaster.cfg - main configuration file for sfsmaster

DESCRIPTION

The file sfsmaster.cfg contains configuration of SaunaFS metadata server process.

SYNTAX

Syntax is:

OPTION = VALUE

Lines starting with # character are ignored.

OPTIONS

Configuration options:

PERSONALITY
    Current personality of this instance of metadata server. Valid values are master, shadow and ha-cluster-managed. If installation is managed by an HA cluster the only valid value is ha-cluster-managed, otherwise the only valid values are master and shadow, in which case only one metadata server in SaunaFS shall have master personality.
    master means that this instance of metadata server acts as main metadata server govering all file system metadata modifications.
    shadow means that this instance of metadata server acts as backup metadata server ready for immediate deployment as new master in case of current master failure.
    Metadata server personality can be changed at any moment as long as one changes personality from shadow to master, changing personality the other way around is forbidden.
    ha-cluster-managed means that this instance is managed by HA cluster, server runs in shadow mode as long as its not remotely promoted to master.

CLUSTER_ID
    Uniquely identifies this namespace. Prevents accidental connections to wrong metadata or chunkservers. (default is default)

DATA_PATH
    where to store metadata files and lock file

WORKING_USER
    user to run daemon as

WORKING_GROUP
    group to run daemon as (optional - if empty then default user group will be used)

SYSLOG_IDENT
    name of process to place in syslog messages (default is sfsmaster)

LOCK_MEMORY
    whether to perform mlockall() to avoid swapping out sfsmaster process (default is 0, i.e. no)

LIMIT_GLIBC_MALLOC_ARENAS
    Linux only: limit glibc malloc arenas to given value - prevents from using huge amount of virtual memory. This can influence performance by reducing memory fragmentation and improving cache locality, but it may also lead to contention and reduced parallelism in multi-threaded applications. Use it in constrained memory environments, recommended values are 4 or 8. (default is 0: disabled or let glibc decide)

NICE_LEVEL
    nice level to run daemon with (default is -19 if possible; note: process must be started as root to increase priority)

EXPORTS_FILENAME
    alternative name of sfsexports.cfg file

TOPOLOGY_FILENAME
    alternative name of sfstopology.cfg file

CUSTOM_GOALS_FILENAME
    alternative name of sfsgoals.cfg file

PREFER_LOCAL_CHUNKSERVER
    If a client mountpoint has a local chunkserver, and a given chunk happens to reside locally, then sfsmaster will list the local chunkserver first. However, when the local client mount is issuing many read(s)/write(s), to many local chunks, these requests can overload the local chunkserver and disk subsystem. Setting this to 0(the default is 1) means that remote chunkservers will be considered as equivalent to the local chunkserver.
    This is useful when the network is faster than the disk, and when there is high-IO load on the client mountpoints.

BACK_LOGS
    number of metadata change log files (default is 50)

BACK_META_KEEP_PREVIOUS
    number of previous metadata files to be kept (default is 1)

AUTO_RECOVERY
    when this option is set (equals 1) master will try to recover metadata from changelog when it is being started after a crash; otherwise it will refuse to start and sfsmetarestore should be used to recover the metadata (default is 0)

OPERATIONS_DELAY_INIT
    initial delay in seconds before starting chunk operations (default is 300)

OPERATIONS_DELAY_DISCONNECT
    chunk operations delay in seconds after chunkserver disconnection (default is 3600)

MATOML_LISTEN_HOST
    IP address to listen on for metalogger connections (* means any)

MATOML_LISTEN_PORT
    port to listen on for metalogger connections (default is 9419)

MATOML_LOG_PRESERVE_SECONDS
    how many seconds of change logs have to be preserved in memory (default is 600; note: logs are stored in blocks of 5k lines, so sometimes real number of seconds may be little bigger; zero disables extra logs storage)

MATOCS_LISTEN_HOST
    IP address to listen on for chunkserver connections (* means any)

MATOCS_LISTEN_PORT
    port to listen on for chunkserver connections (default is 9420)

MATOCL_LISTEN_HOST
    IP address to listen on for client (mount) connections (* means any)

MATOCL_LISTEN_PORT
    port to listen on for client (mount) connections (default is 9421)

MATOTS_LISTEN_HOST
    IP address to listen on for tapeserver connections (* means any)

MATOTS_LISTEN_PORT
    Port to listen on for tapeserver connections (default is 9424)

CHUNKS_LOOP_MAX_CPS
    Chunks loop shouldn’t check more chunks per seconds than given number (default is 100000)

CHUNKS_LOOP_MIN_TIME
    Chunks loop will check all chunks in specified time (default is 300) unless CHUNKS_LOOP_MAX_CPS will force slower execution.

CHUNKS_LOOP_PERIOD
    Time in milliseconds between chunks loop execution (default is 1000).

CHUNKS_LOOP_MAX_CPU
    Hard limit on CPU usage by chunks loop (percentage value, default is 60).

CHUNKS_SOFT_DEL_LIMIT
    Soft maximum number of chunks to delete on one chunkserver (default is 10)

CHUNKS_HARD_DEL_LIMIT
    Hard maximum number of chunks to delete on one chunkserver (default is 25)

CHUNKS_WRITE_REP_LIMIT
    Maximum number of chunks to replicate to one chunkserver (default is 2)

CHUNKS_READ_REP_LIMIT
    Maximum number of chunks to replicate from one chunkserver (default is 10)

ENDANGERED_CHUNKS_PRIORITY
    Percentage of endangered chunks that should be replicated with high priority. Example: when set to 0.2, up to 20% of chunks served in one turn would be extracted from endangered priority queue. When set to 1 (max), no other chunks would be processed as long as there are any endangered chunks in the queue (not advised) (default is 0, i.e. there is no overhead for prioritizing endangered chunks).

ENDANGERED_CHUNKS_MAX_CAPACITY
    Max capacity of endangered chunks queue. This value can limit memory usage of master server if there are lots of endangered chunks in the system. This value is ignored if ENDANGERED_CHUNKS_PRIORITY is set to 0. (default is 1Mi, i.e. no more than 1Mi chunks will be kept in a queue).

ACCEPTABLE_DIFFERENCE
    A maximum difference between disk usage on chunkservers that doesn’t trigger chunk rebalancing (default is 0.1, i.e. 10%).

CHUNKS_REBALANCING_BETWEEN_LABELS
    When balancing disk usage, allow moving chunks between servers with different labels (default is 0, i.e. chunks will be moved only between servers with the same label).

GLOBALIOLIMITS_FILENAME
    Configuration of global I/O limits (default is no I/O limiting)

GLOBALIOLIMITS_RENEGOTIATION_PERIOD_SECONDS
    How often mountpoints will request bandwidth allocations under constant, predictable load (default is 0.1)

GLOBALIOLIMITS_ACCUMULATE_MS
    After inactivity, no waiting is required to transfer the amount of data equivalent to normal data flow over the period of that many milliseconds (default is 250)

METADATA_CHECKSUM_INTERVAL
    how often metadata checksum shall be sent to backup servers (default is: every 50 metadata updates)

METADATA_CHECKSUM_RECALCULATION_SPEED
    how fast should metadata be recalculated in background (default : 100 objects per function call)

DISABLE_METADATA_CHECKSUM_VERIFICATION
    should checksum verification be disabled while applying changelog

NO_ATIME
    when this option is set to 1 inode access time is not updated on every access, otherwise (when set to 0) it is updated (default is 0)

METADATA_SAVE_REQUEST_MIN_PERIOD
    minimal time in seconds between metadata dumps caused by requests from shadow masters (default is 1800)

SESSION_SUSTAIN_TIME
    Time in seconds for which client session data (e.g. list of open files) should be sustained in the master server after connection with the client was lost. Values between 60 and 604800 (one week) are accepted. (default is 86400)

USE_BDB_FOR_NAME_STORAGE
    When this option is set to 1 Berkeley DB is used for storing file/directory names in file (DATA_PATH/name_storage.db). By default all strings are kept in system memory. (default is 0)

BDB_NAME_STORAGE_CACHE_SIZE
    Size of memory cache (in MB) for file/directory names used by Berkeley DB storage. (default is 10)

AVOID_SAME_IP_CHUNKSERVERS
    When this option is set to 1, process of selecting chunkservers for chunks will try to avoid using those that share the same ip. (default is 0)

REDUNDANCY_LEVEL
    minimum number of required redundant chunk parts that can be lost before chunk becomes endangered (default is 0)

SNAPSHOT_INITIAL_BATCH_SIZE
    This option can be used to specify initial number of snapshotted nodes that will be atomically cloned before enqueuing the task for execution in fixed-sized batches. (default is 1000)

SNAPSHOT_INITIAL_BATCH_SIZE_LIMIT
    This option specifies the maximum initial batch size set for snapshot request. (default is 10000)

FILE_TEST_LOOP_MIN_TIME Test files loop will try to check all files in specified time in seconds (default is 3600). It’s possible for the loop to take more time if the master server is busy or the machine doesn’t have enough processing power to make all the needed calculations.

Options below are mandatory for all Shadow instances:

MASTER_HOST
    address of the host running SaunaFS metadata server that currently acts as master

MASTER_PORT
    port number where SaunaFS metadata server currently running as master listens for connections from 'shadow’s and metaloggers (default is 9420)

MASTER_RECONNECTION_DELAY
    delay in seconds before trying to reconnect to metadata server after disconnection (default is 1)

MASTER_TIMEOUT
    timeout (in seconds) for metadata server connections (default is 60)

LOAD_FACTOR_PENALTY
    When set, percentage of load will be added to chunkserver disk usage to determine most fitting chunkserver. Heavy loaded chunkservers will be picked for operations less frequently. (default is 0, correct values are in range from 0 to 0.5)

PRIORITIZE_DATA_PARTS
    When set, master server will prioritize data parts in EC goals to land in the chunkservers with higher percentage of available space. Could cause parities landing always in the same chunkservers if the cluster is not well balanced. (default: 1)

POLL_TIMEOUT_MS
    Maximum amount of time in milliseconds that the polling operation will wait for events. The value is applied for the polling in the events loop. Smaller values could reduce latency at the cost of CPU usage (default: 50)

ENABLE_PROMETHEUS (EXPERIMENTAL)
    Whether to enable Prometheus support and metric collection. Note that this requires compiling with Prometheus support. Set to either 1 to enable, or 0 to disable (default is 0)

PROMETHEUS_HOST (EXPERIMENTAL)
    Host address where Prometheus metric data can be collected, must be in the format of HOST:PORT (default 0.0.0.0:9499)

CREATE_EMPTY_FOLDERS_WHEN_SPACE_DEPLETED (EXPERIMENTAL) When enabled, this option allows the system to create metadata for empty folders when storage space is depleted. This helps preserving the intended file and directory structure. When this option is disabled, no metadata for folders will be created if there is insufficient storage space (default: 1).

LOG_LEVEL
    Setup logging. Uses the environment variable SAUNAFS_LOG_LEVEL or config value LOG_LEVEL to determine logging level. Valid log levels are

        • trace

        • debug

        • info

        • warn or warning

        • err or error

        • crit or critical

        • off

TLS_CERT_FILE (EXPERIMENTAL)
    Path to the TLS certificate file the master/shadow server will use for TLS connections (there is no default value).

TLS_KEY_FILE (EXPERIMENTAL)
    Path to the TLS private key file the master/shadow server will use for TLS connections (there is no default value).

TLS_CA_CERT_FILE (EXPERIMENTAL)
    Path to the trusted CA certificate which is used to authenticate the TLS connection (there is no default value).

USE_CHUNKSERVER_SIDE_CHUNK_LOCK (EXPERIMENTAL)
    When set to 1, enables sending chunk part lock messages to the chunkservers. This can be useful to track down which chunk parts are currently being written. Reloadable (default: 0).

EMPTY_RESERVED_FILES_PERIOD_MSECONDS (EXPERIMENTAL)
    Interval for periodic cleaning of reserved files, in milliseconds. If set to 0, the reserved files deletion is disabled. (Default: 0)

    Warning

    Administrator-only option. Enabling periodic deletion may remove master-side references to reserved files that are still held by other applications or client instances that have not released them yet. This can disrupt ongoing operations and lead to unexpected errors.

    Recommended approach: Do not enable automatic deletion unless you fully understand the impact on running workloads. Prefer manually freeing reserved files by locating each reference or process using them and stopping it before releasing the reservation.

NOTES

Chunks in master are tested in loop. Speed (or frequency) is regulated by two options CHUNKS_LOOP_MIN_TIME and CHUNKS_LOOP_MAX_CPS. First defines minimal time of the loop and second maximal number of chunk tests per second. Typically at the beginning, when number of chunks is small, time is constant, regulated by CHUNK_LOOP_MIN_TIME, but when number of chunks became bigger then time of loop can increase according to CHUNKS_LOOP_MAX_CPS.

Deletion limits are defined as soft and hard limit. When number of chunks to delete increases from loop to loop then current limit can be temporary increased above soft limit, but never above hard limit.

REPORTING BUGS

Report bugs to the GitHub repository as an issue.

COPYRIGHT

Copyright 2008-2009 Gemius SA

Copyright 2013-2019 Skytechnology sp. z o.o.

Copyright 2023-2024 Leil Storage OÜ

SaunaFS is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

SaunaFS is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with SaunaFS. If not, see

SEE ALSO

sfsmaster(8), sfsexports.cfg(5), sfstopology.cfg(5)