Loading...
Searching...
No Matches
User-Defined Formats

# Introduction

NetCDF-C supports user-defined formats (UDFs) that allow developers to extend the library to work with custom file formats. The library provides 10 UDF slots (UDF0 through UDF9) that can be registered either programmatically or via RC file configuration.

User-defined formats enable:

- Support for proprietary or specialized file formats
- Custom storage backends
- Format translation and adaptation layers
- Integration with domain-specific data formats

# Available UDF Slots

The netCDF-C library provides 10 user-defined format slots:

- UDF0 and UDF1: Original slots, mode flags in lower 16 bits
- UDF2 through UDF9: Extended slots, mode flags in upper 16 bits

Each slot can be independently configured with its own dispatch table, initialization function, and optional magic number for automatic format detection.

# Registering UDFs Programmatically

## Using nc_def_user_format()

The nc_def_user_format() function registers a user-defined format:

int nc_def_user_format(int mode_flag, NC_Dispatch *dispatch_table,
char *magic_number);
nc_def_user_format
EXTERNL int nc_def_user_format(int mode_flag, NC_Dispatch *dispatch_table, char *magic_number)
Add handling of user-defined format.
Definition dfile.c:166
NC_Dispatch
struct NC_Dispatch NC_Dispatch
Register a user-defined format.
Definition netcdf.h:652

Parameters:

- mode_flag: One of NC_UDF0 through NC_UDF9, optionally combined with other mode flags (e.g., NC_NETCDF4)
- dispatch_table: Pointer to your dispatch table structure
- magic_number: Optional magic number string (max NC_MAX_MAGIC_NUMBER_LEN bytes) for automatic format detection, or NULL

Example:

#include <netcdf.h>
/* Your dispatch table */
extern NC_Dispatch my_format_dispatcher;
int main() {
/* Register UDF in slot 0 with magic number */
if (nc_def_user_format(NC_UDF0 | NC_NETCDF4, &my_format_dispatcher,
"MYFORMAT")) {
fprintf(stderr, "Failed to register UDF\n");
return 1;
}
/* Now files with "MYFORMAT" magic number will use your dispatcher */
int ncid;
nc_open("myfile.dat", 0, &ncid);
/* ... */
nc_close(ncid);
return 0;
}
nc_close
EXTERNL int nc_close(int ncid)
Close an open netCDF dataset.
Definition dfile.c:1334
nc_open
EXTERNL int nc_open(const char *path, int mode, int *ncidp)
Open an existing netCDF file.
Definition dfile.c:698
netcdf.h
Main header file for the C API.
NC_NETCDF4
#define NC_NETCDF4
Use netCDF-4/HDF5 format.
Definition netcdf.h:179
NC_UDF0
#define NC_UDF0
User-defined format 0 (bit 6).
Definition netcdf.h:152

## Querying Registered UDFs

Use nc_inq_user_format() to query registered formats:

int nc_inq_user_format(int mode_flag, NC_Dispatch **dispatch_table,
char *magic_number);
nc_inq_user_format
EXTERNL int nc_inq_user_format(int mode_flag, NC_Dispatch **dispatch_table, char *magic_number)
Query a registered user-defined format.
Definition dfile.c:220

# Configuring UDFs via RC Files

UDFs can be automatically loaded from RC file configuration, eliminating the need to modify application code.

## RC File Location

RC files are searched in this order:

1. $HOME/.ncrc
2. $HOME/.daprc
3. $HOME/.dodsrc
4. $CWD/.ncrc
5. $CWD/.daprc
6. $CWD/.dodsrc

Later files override earlier ones. Use NCRCENV_HOME to override the home directory.

## RC File Format for UDFs

For each UDF slot (0-9), configure these keys:

NETCDF.UDF<N>.LIBRARY=<full-path-to-library>
NETCDF.UDF<N>.INIT=<initialization-function-name>
NETCDF.UDF<N>.MAGIC=<optional-magic-number>

Example .ncrc file:

# Load custom HDF format in UDF0
NETCDF.UDF0.LIBRARY=/usr/local/lib/libmyformat.so
NETCDF.UDF0.INIT=myformat_init
NETCDF.UDF0.MAGIC=MYFORMAT
# Load scientific data format in UDF3
NETCDF.UDF3.LIBRARY=/opt/scidata/lib/libscidata.so
NETCDF.UDF3.INIT=scidata_initialize
NETCDF.UDF3.MAGIC=SCIDATA
# Load analysis format in UDF7 (no magic number)
NETCDF.UDF7.LIBRARY=/home/user/analysis/libanalysis.so
NETCDF.UDF7.INIT=analysis_init

## RC Configuration Requirements

- LIBRARY: Must be a full absolute path to the shared library (.so on Unix, .dll on Windows)
- INIT: Name of the initialization function in the library
- MAGIC: Optional magic number for automatic format detection
- Both LIBRARY and INIT must be present; partial configuration is ignored with a warning

## Plugin Loading Process

Plugins are loaded during library initialization (nc_initialize()):

1. RC files are parsed
2. For each configured UDF slot:
   - Library is loaded using dlopen (Unix) or LoadLibrary (Windows)
   - Init function is located using dlsym or GetProcAddress
   - Init function is called
   - Init function must call nc_def_user_format() to register the dispatch table
3. Dispatch table ABI version is verified
4. Magic number (if provided) is optionally verified

Note: Library handles are intentionally not closed; they remain loaded for the lifetime of the process.

# Magic Numbers and Format Detection

Magic numbers enable automatic format detection when opening files.

## How Magic Numbers Work

When nc_open() is called without a specific format flag:

1. The file's first bytes are read
2. They are compared against all registered magic numbers
3. If a match is found, the corresponding UDF dispatcher is used

## Magic Number Best Practices

- Use unique, distinctive strings (4-8 bytes recommended)
- Place at the beginning of your file format
- Avoid conflicts with existing formats:
  - NetCDF-3: "CDF\001" or "CDF\002"
  - HDF5: "\211HDF\r\n\032\n"
  - NetCDF-4: Same as HDF5
- Maximum length: NC_MAX_MAGIC_NUMBER_LEN bytes

## Example with Magic Number

/* File format with magic number */
FILE *fp = fopen("mydata.dat", "wb");
fwrite("MYDATA", 1, 6, fp); /* Magic number */
/* ... write your data ... */
fclose(fp);
/* Register UDF with magic number */
nc_def_user_format(NC_UDF0 | NC_NETCDF4, &my_dispatcher, "MYDATA");
/* Open automatically detects format */
int ncid;
nc_open("mydata.dat", 0, &ncid); /* No mode flag needed! */

# Platform Considerations

## Unix/Linux/macOS

- Shared libraries: .so extension
- Dynamic loading: dlopen() and dlsym()
- Library paths: Use absolute paths or ensure libraries are in LD_LIBRARY_PATH

## Windows

- Shared libraries: .dll extension
- Dynamic loading: LoadLibrary() and GetProcAddress()
- Library paths: Use absolute paths or ensure DLLs are in system PATH

## Building Plugins

Your plugin must be compiled as a shared library:

Unix:

gcc -shared -fPIC -o libmyplugin.so myplugin.c -lnetcdf

Windows:

cl /LD myplugin.c netcdf.lib

# Security Considerations

- Full paths required: RC files must specify absolute library paths to prevent path injection attacks
- Code execution: Plugins execute arbitrary code in your process; only load trusted libraries
- Validation: The library verifies dispatch table ABI version but cannot validate plugin behavior
- Permissions: Ensure plugin libraries have appropriate file permissions

# Error Handling

Common errors and solutions:

## NC_EINVAL: Invalid dispatch table version

Cause: Plugin was compiled against a different version of netCDF-C

Solution: Recompile plugin against current netCDF-C version

## Plugin not loaded (no error)

Cause: Partial RC configuration (LIBRARY without INIT, or vice versa)

Solution: Check that both LIBRARY and INIT keys are present in RC file

## Library not found

Cause: Incorrect path in NETCDF.UDF*.LIBRARY

Solution: Use absolute path; verify file exists and has correct permissions

## Init function not found

Cause: Function name mismatch or missing export

Solution: Verify function name matches INIT key; ensure function is exported (not static)

## Init function fails

Cause: Plugin initialization error

Solution: Check plugin logs; verify nc_def_user_format() is called correctly

# Complete Example

See examples/C/udf_example.c for a complete working example of implementing a user-defined format.

# Troubleshooting

## Enable Logging

Set the NC_LOG_LEVEL environment variable to see plugin loading messages:

export NC_LOG_LEVEL=3
./myprogram

## Verify RC File is Read

Create a test RC file and check if it's being parsed:

echo "NETCDF.UDF0.LIBRARY=/tmp/test.so" > ~/.ncrc
echo "NETCDF.UDF0.INIT=test_init" >> ~/.ncrc
# Run your program and check for warnings about missing library

## Check Plugin Exports

Verify your init function is exported:

Unix:

nm -D libmyplugin.so | grep init

Windows:

dumpbin /EXPORTS myplugin.dll

## Test Plugin Loading

Use a minimal test program:

#include <netcdf.h>
#include <stdio.h>
int main() {
nc_initialize(); /* Triggers plugin loading */
printf("Initialization complete\n");
return 0;
}

# See Also

- Dispatch Table Architecture - Internal architecture
- UDF Plugin Development Guide - Creating plugins
- RC File Reference - RC file format details
- API Reference: nc_def_user_format(), nc_inq_user_format()

# References

- NetCDF-C Dispatch Layer: docs/dispatch.md
- Example Implementation: examples/C/udf_example.c
- Test Suite: nc_test4/tst_udf*.c