Skip to main content
`Ctrl`+`K`

# User-Guide

Argonne National Laboratory

Mathematics and Computer Science Division

Prepared by

S. Balay 1, S. Abhyankar 1,2, M. F. Adams 3, S. Benson 1, J. Brown 1,10, P. Brune 1, K. Buschelman 1, E. M. Constantinescu 1, L. Dalcin 4, A. Dener 1, V. Eijkhout 6, J. Faibussowitsch 1,18, W. D. Gropp 1,18, V. Hapla 8, T. Isaac 1,14, P. Jolivet 12,22, D. Karpeev 1, D. Kaushik 1, M. G. Knepley 1,9, F. Kong 1,11, S. Kruger 15, D. A. May 7,21, L. Curfman McInnes 1, R. Tran Mills 1, L. Mitchell 13,20, T. Munson 1, J. E. Roman 16, K. Rupp 1,19, P. Sanan 1,8, J. Sarich 1, B. F. Smith 1,17, H. Suh 1, S. Zampini 4, H. Zhang 1,5, H. Zhang 1, and J. Zhang 1

1 Mathematics and Computer Science Division, Argonne National Laboratory

2 Electricity Infrastructure and Buildings Division, Pacific Northwest National Laboratory

3 Computational Research, Lawrence Berkeley National Laboratory

4 Extreme Computing Research Center, King Abdullah University of Science and Technology

5 Department of Computer Science, Illinois Institute of Technology

6 Texas Advanced Computing Center, University of Texas at Austin

7 Department of Earth Sciences, University of Oxford

8 Institute of Geophysics, ETH Zurich

9 Department of Computer Science and Engineering, University at Buffalo

10 Department of Computer Science, University of Colorado, Boulder

11 Computational Frameworks, Idaho National Laboratory

12 Sorbonne Université, CNRS, LIP6

13 NVIDIA Corporation

14 College of Computing, Georgia Tech

15 Tech-X Corporation

16 DSIC, Universitat Politècnica de València

17 Flatiron Institute, Simons Foundation

18 University of Illinois, Urbana-Champaign

19 Institute for Microelectronics, TU Wien

20 Department of Computer Science, Durham University

21 Scripps Institution of Oceanography, University of California, San Diego

22 Toulouse INP, CNRS, Institute of Computer Science Research

This work was supported by the Office of Advanced Scientific Computing Research, Office of Science, U.S. Department of Energy, under Contract DE-AC02-06CH11357.

- Introduction to PETSc
  - About This Manual
  - Getting Started
    - Suggested Reading
    - Running PETSc Programs
    - Writing PETSc Programs
    - Simple PETSc Examples
  - Parallel and GPU Programming
    - MPI Parallelism
    - CPU SIMD parallelism
    - CPU OpenMP parallelism
    - GPU kernel parallelism
    - GPU stream parallelism
  - Compiling and Running Programs
  - Profiling Programs
  - Writing C/C++ or Fortran Applications
  - PETSc’s Object-Oriented Design
    - User Callbacks
  - Directory Structure
- The Solvers in PETSc/TAO
  - Vectors and Parallel Data
    - Creating Vectors
    - Common vector functions and operations
    - Assembling (putting values in) vectors
    - Basic Vector Operations
    - Local/global vectors and communicating between vectors
    - Low-level Vector Communication
    - Application Orderings
  - PetscSF - an alternative to low-level MPI calls for data communication
    - Non-contiguous storage of leafdata
    - GPU usage
    - Gathering leafdata but not reducing it
    - Optimized communication patterns
  - Matrices
    - Creating matrices
    - Low-level matrix creation routines
    - Assembling (putting values into) matrices
    - Basic Matrix Operations
    - Application Specific Custom Matrices
    - Transposes of Matrices
    - Other Matrix Operations
    - Symbolic and Numeric Stages in Sparse Matrix Operations
    - Graph Operations
    - Partitioning
  - KSP: Linear System Solvers
    - Using KSP
    - Solving Successive Linear Systems
    - Krylov Methods
    - Preconditioners
    - Solving Block Matrices with PCFIELDSPLIT
    - Solving Singular Systems
    - Using External Linear Solvers
    - Using PETSc’s MPI parallel linear solvers from a non-MPI program
  - SNES: Nonlinear Solvers
    - Basic SNES Usage
    - Function Domain Errors and infinity or NaN
    - The Nonlinear Solvers
    - General Options
    - Inexact Newton-like Methods
    - Matrix-Free Methods
    - Finite Difference Jacobian Approximations
    - Variational Inequalities
    - Nonlinear Preconditioning
  - TS: Scalable ODE and DAE Solvers
    - Basic TS Options
    - DAE Formulations
    - Using Implicit-Explicit (IMEX) Methods
    - IMEX Methods for fast-slow systems
    - GLEE methods
    - Using fully implicit methods
    - Using the Explicit Runge-Kutta timestepper with variable timesteps
    - Special Cases
    - Monitoring and visualizing solutions
    - Error control via variable time-stepping
    - Handling of discontinuities
    - Explicit integrators with finite element mass matrices
    - Performing sensitivity analysis with the TS ODE Solvers
    - Using Sundials from PETSc
    - Using TChem from PETSc
  - Solving Steady-State Problems with Pseudo-Timestepping
  - TAO: Optimization Solvers
    - Getting Started: A Simple TAO Example
    - TAO Workflow
    - TAO Algorithms
    - Advanced Options
    - Adding a Solver
  - PetscRegressor: Regression Solvers
    - Basic Regressor Usage
    - Regression Solvers
    - Linear regressor
  - PetscDA: Data Assimilation
    - Ensemble-based Data Assimilation
    - Lifecycle overview
    - Creating a PetscDA context
    - Degrees of freedom per grid point
    - Managing ensembles
    - Observation error
    - Analysis step
    - Forecast step
    - Inflation
    - Implementations
    - Command-line options
    - Viewing and monitoring
    - Further reading
- DM: Interfacing Between Solvers and Models/Discretizations
  - DM Basics
  - PetscSection: Connecting Grids to Data
    - General concept
    - Multiple Fields
    - Working with data
    - Global Sections: Constrained and Distributed Data
    - Permutation: Changing the order of array data
    - DMPlex Specific Functionality: Obtaining data from the array
  - DMPlex: Unstructured Grids
    - Representing Unstructured Grids
    - Grid Point Orientations
    - Dealing with Periodicity
    - Connecting Grids to Data Using PetscSection:
    - Data Layout using DMPLEX and PetscFE
    - Partitioning and Ordering
    - Evaluating Residuals
    - Saving and Loading DMPlex Data with HDF5
    - Metric-based mesh adaptation
  - DMSTAG: Staggered, Structured Grid
    - Terminology
    - Working with vectors and operators (matrices)
    - Coordinates
    - Numberings and internal data layout
  - Networks
    - Application flow
    - Utility functions
    - Retrieving components and number of variables
  - DM Commonalities
    - DMDA simple structured grids
    - DMSTAG simple stagger grids
    - DMPLEX unstructured meshes
    - DMNETWORK computations on graphs of nodes and connecting edges
  - Is it a programming language issue?
  - PetscDT: Discretization Technology in PETSc
    - Quadrature
    - Probability Distributions
  - PetscFE: Finite Element Infrastructure in PETSc
    - Using Pointwise Functions to Specify Finite Element Problems
    - Describing a particular finite element problem to PETSc
    - Assembling finite element residuals and Jacobians
- Additional Information
  - PETSc for Fortran Users
    - Fortran and MPI
    - Numerical Constants
    - Basic Fortran API Differences
    - Sample Fortran Programs
  - Checking the PETSc version
    - During configure/make time
    - During compile time
    - At Runtime
  - Using MATLAB with PETSc
    - Dumping Data for MATLAB
    - Sending Data to an Interactive MATLAB Session
    - Using the MATLAB Compute Engine
    - Licensing the MATLAB Compute Engine on a cluster
  - Profiling
    - Basic Profiling Information
    - Profiling Application Codes
    - Profiling Multiple Sections of Code
    - Restricting Event Logging
    - Interpreting `-info` Output: Informative Messages
    - Time
    - Saving Output to a File
    - Accurate Profiling and Paging Overheads
    - NVIDIA Nsight Systems profiling
    - ROCProfiler profiling
    - Using TAU
  - Hints for Performance Tuning
    - Maximizing Memory Bandwidth
    - Performance Pitfalls and Advice
  - STREAMS: Example Study
    - Detailed STREAMS study for large arrays
    - Detailed study with application
    - Application with the MPI linear solver server
  - The Use of BLAS and LAPACK in PETSc and external libraries
    - 32 or 64-bit BLAS/LAPACK integers
    - Shared memory BLAS/LAPACK parallelism
    - Available BLAS/LAPACK libraries
  - Other PETSc Features
    - PETSc on a process subset
    - Runtime Options
    - Viewers: Looking at PETSc Objects
    - Using SAWs with PETSc
    - Debugging
    - Error Handling
    - Numbers
    - Parallel Communication
    - Graphics
  - Developer Environments
    - Emacs Users
    - VS Code Users
    - Vi and Vim Users
    - Eclipse Users
    - Qt Creator Users
    - Visual Studio Users
    - Xcode IDE Users
  - Advanced Features of Matrices and Solvers
    - Extracting Submatrices
    - Matrix Factorization
    - Matrix-Matrix Products
    - Creating PC’s Directly
  - Running PETSc Tests
    - Quick start with the tests
    - Understanding test output and more information
    - Using the test harness for your own code