Transform Search
Encoder Algorithm

## Functions

int64_t | av1_pixel_diff_dist (const MACROBLOCK *x, int plane, int blk_row, int blk_col, const BLOCK_SIZE plane_bsize, const BLOCK_SIZE tx_bsize, unsigned int *block_mse_q8)
| Compute the pixel domain distortion.
void | av1_pick_recursive_tx_size_type_yrd (const AV1_COMP *cpi, MACROBLOCK *x, RD_STATS *rd_stats, BLOCK_SIZE bsize, int64_t ref_best_rd)
| Recursive transform size and type search.
void | av1_pick_uniform_tx_size_type_yrd (const AV1_COMP *const cpi, MACROBLOCK *x, RD_STATS *rd_stats, BLOCK_SIZE bs, int64_t ref_best_rd)
| Uniform transform size and type search.
int | av1_txfm_uvrd (const AV1_COMP *const cpi, MACROBLOCK *x, RD_STATS *rd_stats, BLOCK_SIZE bsize, int64_t ref_best_rd)
| Chroma block transform search.
void | av1_txfm_rd_in_plane (MACROBLOCK *x, const AV1_COMP *cpi, RD_STATS *rd_stats, int64_t ref_best_rd, int64_t current_rd, int plane, BLOCK_SIZE plane_bsize, TX_SIZE tx_size, FAST_TX_SEARCH_MODE ftxs_mode)
| Transform type search with fixed transform size.
int | av1_txfm_search (const AV1_COMP *cpi, MACROBLOCK *x, BLOCK_SIZE bsize, RD_STATS *rd_stats, RD_STATS *rd_stats_y, RD_STATS *rd_stats_uv, int mode_rate, int64_t ref_best_rd)
| Recursive transform size and type search.

## Detailed Description

This module describes transform search algorithm in AV1.

## Function Documentation

## av1_pixel_diff_dist()

int64_t av1_pixel_diff_dist | ( | const MACROBLOCK * | x,
|  | int | plane,
|  | int | blk_row,
|  | int | blk_col,
|  | const BLOCK_SIZE | plane_bsize,
|  | const BLOCK_SIZE | tx_bsize,
|  | unsigned int * | block_mse_q8 )

Compute the pixel domain distortion.

Compute the pixel domain distortion from diff on all visible 4x4s in the transform block.

Parameters

  [in] | x | Pointer to structure holding the data for the current encoding macroblock
  [in] | plane | Plane index
  [in] | blk_row | Block row index
  [in] | blk_col | Block col index
  [in] | plane_bsize | Current plane block size
  [in] | tx_bsize | Transform size
  [in] | block_mse_q8 | Block mse

Returns
  An int64_t value that is the block sse.

## av1_pick_recursive_tx_size_type_yrd()

void av1_pick_recursive_tx_size_type_yrd | ( | const AV1_COMP * | cpi,
|  | MACROBLOCK * | x,
|  | RD_STATS * | rd_stats,
|  | BLOCK_SIZE | bsize,
|  | int64_t | ref_best_rd )

Recursive transform size and type search.

Search for best transform size and type for luma inter blocks. The transform block partitioning can be recursive resulting in non-uniform transform sizes. The best transform size and type, if found, will be saved in the MB_MODE_INFO structure, and the corresponding RD stats will be saved in rd_stats.

Parameters

  [in] | cpi | Top-level encoder structure
  [in] | x | Pointer to structure holding the data for the current encoding macroblock
  [in] | rd_stats | Pointer to struct to keep track of the RD stats
  [in] | bsize | Current macroblock size
  [in] | ref_best_rd | Best RD cost seen for this block so far

Remarks
  Nothing is returned. The selected transform size and type will be saved in the MB_MODE_INFO structure

Referenced by av1_interpolation_filter_search().

## av1_pick_uniform_tx_size_type_yrd()

void av1_pick_uniform_tx_size_type_yrd | ( | const AV1_COMP *const | cpi,
|  | MACROBLOCK * | x,
|  | RD_STATS * | rd_stats,
|  | BLOCK_SIZE | bs,
|  | int64_t | ref_best_rd )

Uniform transform size and type search.

Search for the best transform size and type for current macroblock block, with the assumption that all the transform blocks have a uniform size (VP9 style). The selected transform size and type will be saved in the MB_MODE_INFO structure; the corresponding RD stats will be saved in rd_stats. This function may be used for both intra and inter predicted blocks.

Parameters

  [in] | cpi | Top-level encoder structure
  [in] | x | Pointer to structure holding the data for the current encoding macroblock
  [in] | rd_stats | Pointer to struct to keep track of the RD stats
  [in] | bs | Current macroblock size
  [in] | ref_best_rd | Best RD cost seen for this block so far

Remarks
  Nothing is returned. The selected transform size and type will be saved in the MB_MODE_INFO structure

Referenced by av1_handle_intra_y_mode(), av1_rd_pick_intra_sby_mode(), handle_filter_intra_mode(), intra_block_yrd(), palette_rd_y(), and rd_pick_filter_intra_sby().

## av1_txfm_uvrd()

int av1_txfm_uvrd | ( | const AV1_COMP *const | cpi,
|  | MACROBLOCK * | x,
|  | RD_STATS * | rd_stats,
|  | BLOCK_SIZE | bsize,
|  | int64_t | ref_best_rd )

Chroma block transform search.

Calculate the transform coefficient RD cost for the given chroma macroblock If the current mode is intra, then this function will compute the predictor.

Parameters

  [in] | cpi | Top-level encoder structure
  [in] | x | Pointer to structure holding the data for the current encoding macroblock
  [in] | rd_stats | Pointer to struct to keep track of the RD stats
  [in] | bsize | Current macroblock size
  [in] | ref_best_rd | Best RD cost seen for this block so far

Returns
  An integer value is returned. 0: early termination triggered, no valid rd cost available; 1: rd cost values are valid.

Referenced by av1_rd_pick_intra_sbuv_mode().

## av1_txfm_rd_in_plane()

void av1_txfm_rd_in_plane | ( | MACROBLOCK * | x,
|  | const AV1_COMP * | cpi,
|  | RD_STATS * | rd_stats,
|  | int64_t | ref_best_rd,
|  | int64_t | current_rd,
|  | int | plane,
|  | BLOCK_SIZE | plane_bsize,
|  | TX_SIZE | tx_size,
|  | FAST_TX_SEARCH_MODE | ftxs_mode )

Transform type search with fixed transform size.

Search for the best transform type and calculate the transform coefficients RD cost of the current transform block with the specified (uniform) transform size and plane. The RD results will be saved in rd_stats.

Parameters

  [in] | x | Pointer to structure holding the data for the current encoding macroblock
  [in] | cpi | Top-level encoder structure
  [in] | rd_stats | Pointer to struct to keep track of the RD stats
  [in] | ref_best_rd | Best RD cost seen for this block so far
  [in] | current_rd | Current RD cost for this block so far
  [in] | plane | Plane index
  [in] | plane_bsize | Size of the current macroblock considering sup-sampling
  [in] | tx_size | The given transform size
  [in] | ftxs_mode | Transform search mode specifying desired speed and quality tradeoff

Remarks
  Nothing is returned. The RD results will be saved in rd_stats.

## av1_txfm_search()

int av1_txfm_search | ( | const AV1_COMP * | cpi,
|  | MACROBLOCK * | x,
|  | BLOCK_SIZE | bsize,
|  | RD_STATS * | rd_stats,
|  | RD_STATS * | rd_stats_y,
|  | RD_STATS * | rd_stats_uv,
|  | int | mode_rate,
|  | int64_t | ref_best_rd )

Recursive transform size and type search.

This function combines y and uv planes' transform search processes together for inter-predicted blocks (including IntraBC), when the prediction is already generated. It first does subtraction to obtain the prediction error. Then it calls av1_pick_recursive_tx_size_type_yrd/av1_pick_uniform_tx_size_type_yrd and av1_txfm_uvrd sequentially and handles possible early terminations. The RD metrics are calculated and stored in rd_stats/_y/_uv.

Parameters

  [in] | cpi | Top-level encoder structure
  [in] | x | Pointer to structure holding the data for the current encoding macroblock
  [in] | bsize | Current macroblock size
  [in] | rd_stats | Pointer to struct to keep track of the overal RD stats
  [in] | rd_stats_y | Pointer to struct to keep track of the RD stats for the luma plane
  [in] | rd_stats_uv | Pointer to struct to keep track of the RD stats for the chroma planes
  [in] | mode_rate | Rate cost to encode the prediction mode info. of the current macroblock
  [in] | ref_best_rd | Best RD cost seen for this block so far

Returns
  An integer value is returned indicating if a valid transform candidate is found (1) or not (0).

Referenced by motion_mode_rd(), and rd_pick_intrabc_mode_sb().