Intra Mode Search
Encoder Algorithm

## Topics

| Palette Mode Search

## Data Structures

struct | IntraModeSearchState
| Variables related to intra-mode search during inter frame coding. More...

## Typedefs

typedef struct IntraModeSearchState | IntraModeSearchState
| Variables related to intra-mode search during inter frame coding.

## Functions

int | av1_handle_intra_y_mode (IntraModeSearchState *intra_search_state, const AV1_COMP *cpi, MACROBLOCK *x, BLOCK_SIZE bsize, unsigned int ref_frame_cost, const PICK_MODE_CONTEXT *ctx, RD_STATS *rd_stats_y, int64_t best_rd, int *mode_cost_y, int64_t *rd_y, int64_t *best_model_rd, int64_t top_intra_model_rd[])
| Evaluate a given luma intra-mode for inter frames.
int | av1_search_intra_uv_modes_in_interframe (IntraModeSearchState *intra_search_state, const AV1_COMP *cpi, MACROBLOCK *x, BLOCK_SIZE bsize, RD_STATS *rd_stats, const RD_STATS *rd_stats_y, RD_STATS *rd_stats_uv, int64_t best_rd)
| Search through all chroma intra-modes for inter frames.
int | av1_search_palette_mode (IntraModeSearchState *intra_search_state, const AV1_COMP *cpi, MACROBLOCK *x, BLOCK_SIZE bsize, unsigned int ref_frame_cost, PICK_MODE_CONTEXT *ctx, RD_STATS *this_rd_cost, int64_t best_rd)
| Evaluate luma palette mode for inter frames.
void | av1_search_palette_mode_luma (const AV1_COMP *cpi, MACROBLOCK *x, BLOCK_SIZE bsize, unsigned int ref_frame_cost, PICK_MODE_CONTEXT *ctx, RD_STATS *this_rd_cost, int64_t best_rd)
| Evaluate luma palette mode for inter frames.
int64_t | av1_rd_pick_intra_sby_mode (const AV1_COMP *const cpi, MACROBLOCK *x, int *rate, int *rate_tokenonly, int64_t *distortion, uint8_t *skippable, BLOCK_SIZE bsize, int64_t best_rd, PICK_MODE_CONTEXT *ctx)
| Perform intra-mode search on luma channels for intra frames.
int64_t | av1_rd_pick_intra_sbuv_mode (const AV1_COMP *const cpi, MACROBLOCK *x, int *rate, int *rate_tokenonly, int64_t *distortion, uint8_t *skippable, BLOCK_SIZE bsize, TX_SIZE max_tx_size)
| Perform intra-mode search on chroma channels.
static int | rd_pick_filter_intra_sby (const AV1_COMP *const cpi, MACROBLOCK *x, int *rate, int *rate_tokenonly, int64_t *distortion, uint8_t *skippable, BLOCK_SIZE bsize, int mode_cost, PREDICTION_MODE best_mode_so_far, int64_t *best_rd, int64_t *best_model_rd, PICK_MODE_CONTEXT *ctx)
| Search for the best filter_intra mode when coding intra frame.
static int | rd_pick_intra_angle_sbuv (const AV1_COMP *const cpi, MACROBLOCK *x, BLOCK_SIZE bsize, int rate_overhead, int64_t best_rd, int *rate, RD_STATS *rd_stats)
| Search for the best angle delta for chroma prediction.
static int | cfl_rd_pick_alpha (MACROBLOCK *const x, const AV1_COMP *const cpi, TX_SIZE tx_size, int64_t ref_best_rd, int cfl_search_range, RD_STATS *best_rd_stats, uint8_t *best_cfl_alpha_idx, int8_t *best_cfl_alpha_signs)
| Pick the optimal parameters for Chroma to Luma (CFL) component.
static int | intra_block_yrd (const AV1_COMP *const cpi, MACROBLOCK *x, BLOCK_SIZE bsize, const int *bmode_costs, int64_t *best_rd, int *rate, int *rate_tokenonly, int64_t *distortion, uint8_t *skippable, MB_MODE_INFO *best_mbmi, PICK_MODE_CONTEXT *ctx)
| Get the intra prediction by searching through tx_type and tx_size.
static void | handle_filter_intra_mode (const AV1_COMP *cpi, MACROBLOCK *x, BLOCK_SIZE bsize, const PICK_MODE_CONTEXT *ctx, RD_STATS *rd_stats_y, int mode_cost, int64_t best_rd, int64_t best_rd_so_far)
| Search for the best filter_intra mode when coding inter frame.
static int | model_intra_yrd_and_prune (const AV1_COMP *const cpi, MACROBLOCK *x, BLOCK_SIZE bsize, int64_t *best_model_rd)
| Estimate the luma rdcost of a given intra mode and try to prune it.
static void | hybrid_intra_mode_search (AV1_COMP *cpi, MACROBLOCK *const x, RD_STATS *rd_cost, BLOCK_SIZE bsize, PICK_MODE_CONTEXT *ctx)
| Hybrid intra mode search.
void | av1_rd_pick_intra_mode_sb (const struct AV1_COMP *cpi, struct macroblock *x, struct RD_STATS *rd_cost, BLOCK_SIZE bsize, PICK_MODE_CONTEXT *ctx, int64_t best_rd)
| AV1 intra mode selection for intra frames.
static int64_t | rd_pick_intrabc_mode_sb (const AV1_COMP *cpi, MACROBLOCK *x, PICK_MODE_CONTEXT *ctx, RD_STATS *rd_stats, BLOCK_SIZE bsize, int64_t best_rd)
| Search for the best intrabc predictor.
static void | search_intra_modes_in_interframe (InterModeSearchState *search_state, const AV1_COMP *cpi, MACROBLOCK *x, RD_STATS *rd_cost, BLOCK_SIZE bsize, PICK_MODE_CONTEXT *ctx, const InterModeSFArgs *sf_args, unsigned int intra_ref_frame_cost, int64_t yrd_threshold)
| Search intra modes in interframes.

## Detailed Description

This module describes intra mode search algorithm in AV1. More details will be added.

## Typedef Documentation

## IntraModeSearchState

typedef struct IntraModeSearchState IntraModeSearchState

Variables related to intra-mode search during inter frame coding.

This is a set of variables used during intra-mode search for inter frames. This includes an histogram of gradient speed features and a cache of uv prediction to avoid repeated search of chroma prediction.

## Function Documentation

## av1_handle_intra_y_mode()

int av1_handle_intra_y_mode | ( | IntraModeSearchState * | intra_search_state,
|  | const AV1_COMP * | cpi,
|  | MACROBLOCK * | x,
|  | BLOCK_SIZE | bsize,
|  | unsigned int | ref_frame_cost,
|  | const PICK_MODE_CONTEXT * | ctx,
|  | RD_STATS * | rd_stats_y,
|  | int64_t | best_rd,
|  | int * | mode_cost_y,
|  | int64_t * | rd_y,
|  | int64_t * | best_model_rd,
|  | int64_t | top_intra_model_rd[] )

Evaluate a given luma intra-mode for inter frames.

This function handles an intra-mode luma prediction when the current frame is an inter frame. This is the intra-mode counterpart of handle_inter_mode. This function performs an intra luma prediction using the mode specified by x->e_mbd.mi[0]->mode. This function does not support palette mode prediction in the luma channel.

Parameters

  [in,out] | intra_search_state | Structure to intra search state.
  [in] | cpi | Top-level encoder structure.
  [in,out] | x | Pointer to structure holding all the data for the current macroblock.
  [in] | bsize | Current partition block size.
  [in] | ref_frame_cost | The entropy cost for signaling that the current ref frame is an intra frame.
  [in] | ctx | Structure to hold the number of 4x4 blks to copy tx_type and txfm_skip arrays.
  [out] | rd_stats_y | Struct to keep track of the current intra-mode's rd_stats (luma only).
  [in] | best_rd | Best RD seen for this block so far.
  [out] | mode_cost_y | The cost needed to signal the current intra mode.
  [out] | rd_y | The rdcost of the chosen mode.
  [in] | best_model_rd | Best model RD seen for this block so far
  [in] | top_intra_model_rd | Top intra model RD seen for this block so far.

Returns
  Returns 1 if a valid intra mode is found, 0 otherwise. The corresponding values in x->e_mbd.mi[0], rd_stats_y, mode_cost_y, and rd_y are also updated. Moreover, in the first evaluation with directional mode, a prune_mask computed with histogram of gradient is also stored in intra_search_state.

References av1_pick_uniform_tx_size_type_yrd(), AV1_COMP::common, IntraModeSearchState::dir_mode_skip_mask_ready, IntraModeSearchState::directional_mode_skip_mask, macroblock::e_mbd, MB_MODE_INFO::filter_intra_mode_info, handle_filter_intra_mode(), intra_mode_info_cost_y(), SPEED_FEATURES::intra_sf, ModeCosts::mbmode_cost, macroblockd::mi, MB_MODE_INFO::mode, macroblock::mode_costs, AV1_COMP::oxcf, macroblock::rdmult, MB_MODE_INFO::ref_frame, AV1Common::seq_params, AV1_COMP::sf, IntraModeSearchState::skip_intra_modes, and ModeCosts::skip_txfm_cost.

Referenced by search_intra_modes_in_interframe().

## av1_search_intra_uv_modes_in_interframe()

int av1_search_intra_uv_modes_in_interframe | ( | IntraModeSearchState * | intra_search_state,
|  | const AV1_COMP * | cpi,
|  | MACROBLOCK * | x,
|  | BLOCK_SIZE | bsize,
|  | RD_STATS * | rd_stats,
|  | const RD_STATS * | rd_stats_y,
|  | RD_STATS * | rd_stats_uv,
|  | int64_t | best_rd )

Search through all chroma intra-modes for inter frames.

This function handles intra-mode chroma prediction when the current frame is an inter frame. This is done by calling av1_rd_pick_intra_sbuv_mode with some additional book-keeping.

Parameters

  [in,out] | intra_search_state | Structure to intra search state.
  [in] | cpi | Top-level encoder structure.
  [in,out] | x | Pointer to structure holding all the data for the current macroblock.
  [in] | bsize | Current partition block size.
  [out] | rd_stats | Struct to keep track of the current intra-mode's rd_stats (all planes).
  [out] | rd_stats_y | Struct to keep track of the current intra-mode's rd_stats (luma only).
  [out] | rd_stats_uv | Struct to keep track of the current intra-mode's rd_stats (chroma only).
  [in] | best_rd | Best RD seen for this block so far.

Returns
  Returns 1 if a valid intra mode is found, 0 otherwise. The corresponding values in x->e_mbd.mi[0], rd_stats(_y|_uv) are also updated. Moreover, in the first evocation of the function, the chroma intra mode result is cached in intra_search_state to be used in subsequent calls.

< U (Chroma) plane

References FeatureFlags::allow_screen_content_tools, MB_MODE_INFO::angle_delta, AOM_PLANE_U, av1_rd_pick_intra_sbuv_mode(), MB_MODE_INFO::bsize, AV1_COMP::common, IntraModeSearchState::dist_uvs, macroblock::e_mbd, AV1Common::features, macroblockd::mi, IntraModeSearchState::mode_uv, AV1_COMP::oxcf, MB_MODE_INFO::palette_mode_info, IntraModeSearchState::pmi_uv, IntraModeSearchState::rate_uv_intra, IntraModeSearchState::rate_uv_tokenonly, macroblock::rdmult, MB_MODE_INFO::ref_frame, IntraModeSearchState::skip_intra_modes, IntraModeSearchState::skip_uvs, IntraModeSearchState::uv_angle_delta, and MB_MODE_INFO::uv_mode.

Referenced by search_intra_modes_in_interframe().

## av1_search_palette_mode()

int av1_search_palette_mode | ( | IntraModeSearchState * | intra_search_state,
|  | const AV1_COMP * | cpi,
|  | MACROBLOCK * | x,
|  | BLOCK_SIZE | bsize,
|  | unsigned int | ref_frame_cost,
|  | PICK_MODE_CONTEXT * | ctx,
|  | RD_STATS * | this_rd_cost,
|  | int64_t | best_rd )

Evaluate luma palette mode for inter frames.

This function handles luma palette mode when the current frame is an inter frame.

Parameters

  [in] | intra_search_state | Structure to hold the best luma intra mode and cache chroma prediction for speed up.
  [in] | cpi | Top-level encoder structure.
  [in] | x | Pointer to structure holding all the data for the current macroblock.
  [in] | bsize | Current partition block size.
  [in] | ref_frame_cost | The entropy cost for signaling that the current ref frame is an intra frame.
  [in] | ctx | Structure to hold the number of 4x4 blks to copy the tx_type and txfm_skip arrays.
  [in] | this_rd_cost | Struct to keep track of palette mode's rd_stats.
  [in] | best_rd | Best RD seen for this block so far.

Returns
  Returns whether luma palette mode can skip the txfm. The corresponding mbmi, this_rd_costs, intra_search_state, and tx_type arrays in ctx are also updated.

< U (Chroma) plane

References MB_MODE_INFO::angle_delta, AOM_PLANE_U, av1_rd_pick_intra_sbuv_mode(), av1_rd_pick_palette_intra_sby(), PALETTE_BUFFER::best_palette_color_map, AV1_COMP::common, IntraModeSearchState::dist_uvs, macroblock::e_mbd, ModeCosts::mbmode_cost, macroblockd::mi, MB_MODE_INFO::mode, macroblock::mode_costs, IntraModeSearchState::mode_uv, macroblock::palette_buffer, MB_MODE_INFO::palette_mode_info, macroblockd::plane, IntraModeSearchState::pmi_uv, IntraModeSearchState::rate_uv_intra, IntraModeSearchState::rate_uv_tokenonly, macroblock::rdmult, MB_MODE_INFO::ref_frame, MB_MODE_INFO::skip_txfm, ModeCosts::skip_txfm_cost, IntraModeSearchState::skip_uvs, macroblockd::tx_type_map, IntraModeSearchState::uv_angle_delta, and MB_MODE_INFO::uv_mode.

Referenced by av1_rd_pick_inter_mode().

## av1_search_palette_mode_luma()

void av1_search_palette_mode_luma | ( | const AV1_COMP * | cpi,
|  | MACROBLOCK * | x,
|  | BLOCK_SIZE | bsize,
|  | unsigned int | ref_frame_cost,
|  | PICK_MODE_CONTEXT * | ctx,
|  | RD_STATS * | this_rd_cost,
|  | int64_t | best_rd )

Evaluate luma palette mode for inter frames.

This function handles luma palette mode when the current frame is an inter frame.

Parameters

  [in] | cpi | Top-level encoder structure.
  [in] | x | Pointer to structure holding all the data for the current macroblock.
  [in] | bsize | Current partition block size.
  [in] | ref_frame_cost | The entropy cost for signaling that the current ref frame is an intra frame.
  [in] | ctx | Structure to hold the number of 4x4 blks to copy the tx_type and txfm_skip arrays.
  [in] | this_rd_cost | Struct to keep track of palette mode's rd_stats.
  [in] | best_rd | Best RD seen for this block so far.

References av1_rd_pick_palette_intra_sby(), PALETTE_BUFFER::best_palette_color_map, macroblock::e_mbd, ModeCosts::mbmode_cost, macroblockd::mi, MB_MODE_INFO::mode, macroblock::mode_costs, macroblock::palette_buffer, MB_MODE_INFO::palette_mode_info, macroblockd::plane, macroblock::rdmult, MB_MODE_INFO::ref_frame, MB_MODE_INFO::skip_txfm, ModeCosts::skip_txfm_cost, macroblockd::tx_type_map, and MB_MODE_INFO::uv_mode.

Referenced by av1_nonrd_pick_intra_mode().

## av1_rd_pick_intra_sby_mode()

int64_t av1_rd_pick_intra_sby_mode | ( | const AV1_COMP *const | cpi,
|  | MACROBLOCK * | x,
|  | int * | rate,
|  | int * | rate_tokenonly,
|  | int64_t * | distortion,
|  | uint8_t * | skippable,
|  | BLOCK_SIZE | bsize,
|  | int64_t | best_rd,
|  | PICK_MODE_CONTEXT * | ctx )

Perform intra-mode search on luma channels for intra frames.

This function performs intra-mode search on the luma channel when the current frame is intra-only. This function does not search intrabc mode, but it does search palette and filter_intra.

Parameters

  [in] | cpi | Top-level encoder structure.
  [in] | x | Pointer to structure holding all the data for the current macroblock.
  [in] | rate | The total rate needed to predict the current chroma block.
  [in] | rate_tokenonly | The rate without the cost of sending the prediction modes. chroma block. after the reconstruction.
  [in] | distortion | The chroma distortion of the best prediction after the reconstruction.
  [in] | skippable | Whether we can skip txfm process.
  [in] | bsize | Current partition block size.
  [in] | best_rd | Best RD seen for this block so far.
  [in] | ctx | Structure to hold the number of 4x4 blks to copy the tx_type and txfm_skip arrays.

Returns
  Returns the rd_cost if this function finds a mode better than best_rd, otherwise returns INT64_MAX. This also updates the mbmi, the rate and distortion, and the tx_type arrays in ctx.

< Y (Luminance) plane

References macroblockd::above_mbmi, FeatureFlags::allow_screen_content_tools, MB_MODE_INFO::angle_delta, AOM_PLANE_Y, av1_pick_uniform_tx_size_type_yrd(), av1_rd_pick_palette_intra_sby(), PALETTE_BUFFER::best_palette_color_map, MB_MODE_INFO::bsize, WinnerModeStats::color_index_map, AV1_COMP::common, macroblock::e_mbd, IntraModeCfg::enable_angle_delta, IntraModeCfg::enable_diagonal_intra, IntraModeCfg::enable_directional_intra, IntraModeCfg::enable_paeth_intra, IntraModeCfg::enable_smooth_intra, AV1Common::features, MB_MODE_INFO::filter_intra_mode_info, intra_block_yrd(), intra_mode_info_cost_y(), SPEED_FEATURES::intra_sf, macroblockd::left_mbmi, macroblockd::lossless, macroblock::mb_mode_cache, WinnerModeStats::mbmi, macroblockd::mi, MB_MODE_INFO::mode, macroblock::mode_costs, AV1_COMP::oxcf, macroblock::palette_buffer, MB_MODE_INFO::palette_mode_info, macroblockd::plane, rd_pick_filter_intra_sby(), macroblock::rdmult, MB_MODE_INFO::segment_id, AV1Common::seq_params, set_y_mode_and_delta_angle(), AV1_COMP::sf, MB_MODE_INFO::tx_size, macroblockd::tx_type_map, macroblock::use_mb_mode_cache, macroblock::winner_mode_count, SPEED_FEATURES::winner_mode_sf, macroblock::winner_mode_stats, and ModeCosts::y_mode_costs.

Referenced by av1_rd_pick_intra_mode_sb().

## av1_rd_pick_intra_sbuv_mode()

int64_t av1_rd_pick_intra_sbuv_mode | ( | const AV1_COMP *const | cpi,
|  | MACROBLOCK * | x,
|  | int * | rate,
|  | int * | rate_tokenonly,
|  | int64_t * | distortion,
|  | uint8_t * | skippable,
|  | BLOCK_SIZE | bsize,
|  | TX_SIZE | max_tx_size )

Perform intra-mode search on chroma channels.

This function performs intra-mode search on the chroma channels. Just like av1_rd_pick_intra_sby_mode(), this function searches over palette mode (filter_intra is not available on chroma planes). Unlike av1_rd_pick_intra_sby_mode() this function is used by both inter and intra frames.

Parameters

  [in] | cpi | Top-level encoder structure.
  [in] | x | Pointer to structure holding all the data for the current macroblock.
  [in] | rate | The total rate needed to predict the current chroma block.
  [in] | rate_tokenonly | The rate without the cost of sending the prediction modes. chroma block. after the reconstruction.
  [in] | distortion | The chroma distortion of the best prediction after the reconstruction.
  [in] | skippable | Whether we can skip txfm process.
  [in] | bsize | Current partition block size.
  [in] | max_tx_size | The maximum tx_size available

Returns
  Returns the rd_cost of the best uv mode found. This also updates the mbmi, the rate and distortion, distortion.

< Y (Luminance) plane

< U (Chroma) plane

References FeatureFlags::allow_screen_content_tools, MB_MODE_INFO::angle_delta, AOM_PLANE_U, AOM_PLANE_Y, av1_rd_pick_palette_intra_sbuv(), av1_txfm_uvrd(), PALETTE_BUFFER::best_palette_color_map, MB_MODE_INFO::bsize, macroblockd::cfl, MB_MODE_INFO::cfl_alpha_idx, MB_MODE_INFO::cfl_alpha_signs, cfl_rd_pick_alpha(), AV1_COMP::common, IntraModeSearchState::dir_mode_skip_mask_ready, IntraModeSearchState::directional_mode_skip_mask, macroblock::e_mbd, IntraModeCfg::enable_angle_delta, IntraModeCfg::enable_cfl_intra, IntraModeCfg::enable_paeth_intra, IntraModeCfg::enable_smooth_intra, AV1Common::features, init_intra_mode_search_state(), intra_mode_info_cost_uv(), SPEED_FEATURES::intra_sf, ModeCosts::intra_uv_mode_cost, macroblockd::is_chroma_ref, macroblockd::mi, MB_MODE_INFO::mode, macroblock::mode_costs, AV1_COMP::optimize_seg_arr, AV1_COMP::oxcf, macroblock::palette_buffer, rd_pick_intra_angle_sbuv(), macroblock::rdmult, MB_MODE_INFO::segment_id, AV1Common::seq_params, AV1_COMP::sf, MB_MODE_INFO::skip_txfm, and MB_MODE_INFO::uv_mode.

Referenced by av1_rd_pick_intra_mode_sb(), av1_search_intra_uv_modes_in_interframe(), and av1_search_palette_mode().

## rd_pick_filter_intra_sby()

int rd_pick_filter_intra_sby | ( | const AV1_COMP *const | cpi,
|  | MACROBLOCK * | x,
|  | int * | rate,
|  | int * | rate_tokenonly,
|  | int64_t * | distortion,
|  | uint8_t * | skippable,
|  | BLOCK_SIZE | bsize,
|  | int | mode_cost,
|  | PREDICTION_MODE | best_mode_so_far,
|  | int64_t * | best_rd,
|  | int64_t * | best_model_rd,
|  | PICK_MODE_CONTEXT * | ctx )

static

Search for the best filter_intra mode when coding intra frame.

This function loops through all filter_intra modes to find the best one.

Returns
  Returns 1 if a new filter_intra mode is selected; 0 otherwise.

References av1_pick_uniform_tx_size_type_yrd(), AV1_COMP::common, macroblock::e_mbd, MB_MODE_INFO::filter_intra_mode_info, intra_mode_info_cost_y(), SPEED_FEATURES::intra_sf, macroblock::mb_mode_cache, macroblockd::mi, MB_MODE_INFO::mode, model_intra_yrd_and_prune(), AV1_COMP::oxcf, MB_MODE_INFO::palette_mode_info, macroblock::rdmult, AV1_COMP::sf, MB_MODE_INFO::tx_size, macroblockd::tx_type_map, macroblock::use_mb_mode_cache, and SPEED_FEATURES::winner_mode_sf.

Referenced by av1_rd_pick_intra_sby_mode().

## rd_pick_intra_angle_sbuv()

int rd_pick_intra_angle_sbuv | ( | const AV1_COMP *const | cpi,
|  | MACROBLOCK * | x,
|  | BLOCK_SIZE | bsize,
|  | int | rate_overhead,
|  | int64_t | best_rd,
|  | int * | rate,
|  | RD_STATS * | rd_stats )

static

Search for the best angle delta for chroma prediction.

Given a chroma directional intra prediction mode, this function will try to estimate the best delta_angle.

Returns
  Return if there is a new mode with smaller rdcost than best_rd.

References MB_MODE_INFO::angle_delta, macroblock::e_mbd, and macroblockd::mi.

Referenced by av1_rd_pick_intra_sbuv_mode().

## cfl_rd_pick_alpha()

int cfl_rd_pick_alpha | ( | MACROBLOCK *const | x,
|  | const AV1_COMP *const | cpi,
|  | TX_SIZE | tx_size,
|  | int64_t | ref_best_rd,
|  | int | cfl_search_range,
|  | RD_STATS * | best_rd_stats,
|  | uint8_t * | best_cfl_alpha_idx,
|  | int8_t * | best_cfl_alpha_signs )

static

Pick the optimal parameters for Chroma to Luma (CFL) component.

This function will use DCT_DCT followed by computing SATD (sum of absolute transformed differences) to estimate the RD score and find the best possible CFL parameter.

Then the function will apply a full RD search near the best possible CFL parameter to find the best actual CFL parameter.

Side effect: We use ths buffers in x->plane[] and xd->plane[] as throw-away buffers for RD search.

Parameters

  [in] | x | Encoder prediction block structure.
  [in] | cpi | Top-level encoder instance structure.
  [in] | tx_size | Transform size.
  [in] | ref_best_rd | Reference best RD.
  [in] | cfl_search_range | The search range of full RD search near the estimated best CFL parameter.
  [out] | best_rd_stats | RD stats of the best CFL parameter
  [out] | best_cfl_alpha_idx | Best CFL alpha index
  [out] | best_cfl_alpha_signs | Best CFL joint signs

References macroblockd::cfl, ModeCosts::cfl_cost, macroblock::e_mbd, ModeCosts::intra_uv_mode_cost, macroblockd::mi, MB_MODE_INFO::mode, macroblock::mode_costs, and macroblock::rdmult.

Referenced by av1_rd_pick_intra_sbuv_mode().

## intra_block_yrd()

int intra_block_yrd | ( | const AV1_COMP *const | cpi,
|  | MACROBLOCK * | x,
|  | BLOCK_SIZE | bsize,
|  | const int * | bmode_costs,
|  | int64_t * | best_rd,
|  | int * | rate,
|  | int * | rate_tokenonly,
|  | int64_t * | distortion,
|  | uint8_t * | skippable,
|  | MB_MODE_INFO * | best_mbmi,
|  | PICK_MODE_CONTEXT * | ctx )

inline static

Get the intra prediction by searching through tx_type and tx_size.

Currently this function is only used in the intra frame code path for winner-mode processing.

Returns
  Returns whether the current mode is an improvement over best_rd.

References av1_pick_uniform_tx_size_type_yrd(), MB_MODE_INFO::bsize, macroblock::e_mbd, intra_mode_info_cost_y(), macroblockd::lossless, macroblockd::mi, MB_MODE_INFO::mode, macroblock::rdmult, MB_MODE_INFO::segment_id, AV1_COMP::sf, MB_MODE_INFO::skip_txfm, SPEED_FEATURES::tx_sf, MB_MODE_INFO::tx_size, and macroblockd::tx_type_map.

Referenced by av1_rd_pick_intra_sby_mode().

## handle_filter_intra_mode()

void handle_filter_intra_mode | ( | const AV1_COMP * | cpi,
|  | MACROBLOCK * | x,
|  | BLOCK_SIZE | bsize,
|  | const PICK_MODE_CONTEXT * | ctx,
|  | RD_STATS * | rd_stats_y,
|  | int | mode_cost,
|  | int64_t | best_rd,
|  | int64_t | best_rd_so_far )

inline static

Search for the best filter_intra mode when coding inter frame.

This function loops through all filter_intra modes to find the best one.

Remarks
  Returns nothing, but updates the mbmi and rd_stats.

References av1_pick_uniform_tx_size_type_yrd(), AV1_COMP::common, macroblock::e_mbd, MB_MODE_INFO::filter_intra_mode_info, intra_mode_info_cost_y(), macroblockd::mi, MB_MODE_INFO::mode, macroblock::rdmult, MB_MODE_INFO::tx_size, and macroblockd::tx_type_map.

Referenced by av1_handle_intra_y_mode().

## model_intra_yrd_and_prune()

int model_intra_yrd_and_prune | ( | const AV1_COMP *const | cpi,
|  | MACROBLOCK * | x,
|  | BLOCK_SIZE | bsize,
|  | int64_t * | best_model_rd )

inline static

Estimate the luma rdcost of a given intra mode and try to prune it.

This function first makes a quick luma prediction and estimates the rdcost with a model without going through the txfm, then try to prune the current mode if the new estimate y_rd > 1.25 * best_model_rd.

Returns
  Returns 1 if the given mode is prune; 0 otherwise.

References AV1_COMP::common.

Referenced by rd_pick_filter_intra_sby().

## hybrid_intra_mode_search()

void hybrid_intra_mode_search | ( | AV1_COMP * | cpi,
|  | MACROBLOCK *const | x,
|  | RD_STATS * | rd_cost,
|  | BLOCK_SIZE | bsize,
|  | PICK_MODE_CONTEXT * | ctx )

inline static

Hybrid intra mode search.

This is top level function for mode search for intra frames in non-RD optimized case. Depending on speed feature and block size it calls either non-RD or RD optimized intra mode search.

Parameters

  [in] | cpi | Top-level encoder structure
  [in] | x | Pointer to structure holding all the data for the current macroblock
  [in] | rd_cost | Struct to keep track of the RD information
  [in] | bsize | Current block size
  [in] | ctx | Structure to hold snapshot of coding context during the mode picking process

Remarks
  Nothing is returned. Instead, the MB_MODE_INFO struct inside x is modified to store information about the best mode computed in this function. The rd_cost struct is also updated with the RD stats corresponding to the best mode found.

References av1_nonrd_pick_intra_mode(), av1_rd_pick_intra_mode_sb(), SPEED_FEATURES::rt_sf, AV1_COMP::sf, and macroblock::source_variance.

Referenced by pick_sb_modes_nonrd().

## av1_rd_pick_intra_mode_sb()

void av1_rd_pick_intra_mode_sb | ( | const struct AV1_COMP * | cpi,
|  | struct macroblock * | x,
|  | struct RD_STATS * | rd_cost,
|  | BLOCK_SIZE | bsize,
|  | PICK_MODE_CONTEXT * | ctx,
|  | int64_t | best_rd )

AV1 intra mode selection for intra frames.

Top level function for rd-based intra mode selection during intra frame encoding. This function will first search for the best luma prediction by calling av1_rd_pick_intra_sby_mode, then it searches for chroma prediction with av1_rd_pick_intra_sbuv_mode. If applicable, this function ends the search with an evaluation for intrabc.

Parameters

  [in] | cpi | Top-level encoder structure.
  [in] | x | Pointer to structure holding all the data for the current macroblock.
  [in] | rd_cost | Struct to keep track of the RD information.
  [in] | bsize | Current block size.
  [in] | ctx | Structure to hold snapshot of coding context during the mode picking process.
  [in] | best_rd | Best RD seen for this block so far.

Remarks
  Nothing is returned. Instead, the MB_MODE_INFO struct inside x is modified to store information about the best mode computed in this function. The rd_cost struct is also updated with the RD stats corresponding to the best mode found.

< U (Chroma) plane

References AOM_PLANE_U, av1_rd_pick_intra_sbuv_mode(), av1_rd_pick_intra_sby_mode(), AV1_COMP::common, macroblock::e_mbd, macroblockd::is_chroma_ref, macroblock::mbmi_ext, macroblockd::mi, macroblock::mode_costs, MB_MODE_INFO::mv, rd_pick_intrabc_mode_sb(), macroblock::rdmult, MB_MODE_INFO::ref_frame, MB_MODE_INFO::skip_mode, MB_MODE_INFO::skip_txfm, ModeCosts::skip_txfm_cost, macroblockd::tx_type_map, and MB_MODE_INFO::use_intrabc.

Referenced by hybrid_intra_mode_search(), and pick_sb_modes().

## rd_pick_intrabc_mode_sb()

int64_t rd_pick_intrabc_mode_sb | ( | const AV1_COMP * | cpi,
|  | MACROBLOCK * | x,
|  | PICK_MODE_CONTEXT * | ctx,
|  | RD_STATS * | rd_stats,
|  | BLOCK_SIZE | bsize,
|  | int64_t | best_rd )

static

Search for the best intrabc predictor.

This function performs a motion search to find the best intrabc predictor.

Returns
  Returns the best overall rdcost (including the non-intrabc modes search before this function).

References av1_txfm_search(), AV1_COMP::common, macroblockd::cur_buf, IntraBCMVCosts::dv_costs, macroblock::dv_costs, macroblock::e_mbd, KeyFrameCfg::enable_intrabc, MB_MODE_INFO::filter_intra_mode_info, MB_MODE_INFO_EXT::global_mvs, macroblockd::height, MB_MODE_INFO::interp_filters, ModeCosts::intrabc_cost, macroblock::intrabc_hash_info, IntraBCMVCosts::joint_mv, AV1EncoderConfig::kf_cfg, macroblock::mbmi_ext, macroblockd::mi, macroblockd::mi_col, macroblockd::mi_row, MB_MODE_INFO::mode, MB_MODE_INFO_EXT::mode_context, macroblock::mode_costs, MB_MODE_INFO::motion_mode, MB_MODE_INFO::mv, AV1_COMP::mv_search_params, SPEED_FEATURES::mv_sf, MotionVectorSearchParams::mv_step_param, AV1_COMP::oxcf, MB_MODE_INFO::palette_mode_info, macroblockd::plane, macroblock::rdmult, MB_MODE_INFO_EXT::ref_mv_count, macroblockd::ref_mv_stack, MB_MODE_INFO_EXT::ref_mv_stack, SPEED_FEATURES::rt_sf, MotionVectorSearchParams::search_site_cfg, AV1Common::seq_params, AV1_COMP::sf, MB_MODE_INFO::skip_txfm, macroblockd::tile, macroblockd::tx_type_map, MB_MODE_INFO::use_intrabc, MB_MODE_INFO::uv_mode, macroblockd::weight, and macroblockd::width.

Referenced by av1_rd_pick_intra_mode_sb().

## search_intra_modes_in_interframe()

void search_intra_modes_in_interframe | ( | InterModeSearchState * | search_state,
|  | const AV1_COMP * | cpi,
|  | MACROBLOCK * | x,
|  | RD_STATS * | rd_cost,
|  | BLOCK_SIZE | bsize,
|  | PICK_MODE_CONTEXT * | ctx,
|  | const InterModeSFArgs * | sf_args,
|  | unsigned int | intra_ref_frame_cost,
|  | int64_t | yrd_threshold )

inline static

Search intra modes in interframes.

This function searches for the best intra mode when the current frame is an interframe. This function however does not handle luma palette mode. Palette mode is currently handled by av1_search_palette_mode.

This function will first iterate through the luma mode candidates to find the best luma intra mode. Once the best luma mode it's found, it will then search for the best chroma mode. Because palette mode is currently not handled by here, a cache of uv mode is stored in InterModeSearchState::intra_search_state so it can be reused later by av1_search_palette_mode.

Parameters

  [in,out] | search_state | Struct keep track of the prediction mode search state in interframe.
  [in] | cpi | Top-level encoder structure.
  [in,out] | x | Pointer to struct holding all the data for the current prediction block.
  [out] | rd_cost | Stores the best rd_cost among all the prediction modes searched.
  [in] | bsize | Current block size.
  [in,out] | ctx | Structure to hold the number of 4x4 blks to copy the tx_type and txfm_skip arrays. for only the Y plane.
  [in] | sf_args | Stores the list of intra mode candidates to be searched.
  [in] | intra_ref_frame_cost | The entropy cost for signaling that the current ref frame is an intra frame.
  [in] | yrd_threshold | The rdcost threshold for luma intra mode to terminate chroma intra mode search.

Remarks
  If a new best mode is found, search_state and rd_costs are updated correspondingly. While x is also modified, it is only used as a temporary buffer, and the final decisions are stored in search_state.

References AV1EncoderConfig::algo_cfg, MB_MODE_INFO::angle_delta, av1_handle_intra_y_mode(), av1_search_intra_uv_modes_in_interframe(), IntraModeSearchState::best_intra_mode, AV1_COMP::common, macroblock::e_mbd, IntraModeCfg::enable_angle_delta, IntraModeCfg::enable_paeth_intra, IntraModeCfg::enable_smooth_intra, intra_mode_info_cost_uv(), SPEED_FEATURES::intra_sf, ModeCosts::intra_uv_mode_cost, macroblockd::is_chroma_ref, macroblockd::lossless, macroblockd::mi, MB_MODE_INFO::mode, macroblock::mode_costs, AV1_COMP::oxcf, macroblock::rdmult, SPEED_FEATURES::rt_sf, MB_MODE_INFO::segment_id, set_y_mode_and_delta_angle(), AV1_COMP::sf, AlgoCfg::sharpness, MB_MODE_INFO::skip_txfm, TxfmSearchInfo::skip_txfm, ModeCosts::skip_txfm_cost, MB_MODE_INFO::tx_size, macroblockd::tx_type_map, macroblock::txfm_search_info, MB_MODE_INFO::uv_mode, and SPEED_FEATURES::winner_mode_sf.

Referenced by av1_rd_pick_inter_mode().