SQLFLUFF(1)

NAME
sqlfluff - SQLFluff stable_version

Bored of not having a good SQL linter that works with whichever dialect you're working with? Fluff is an extensible and modular linter designed to help you write good SQL and catch errors and bad SQL before it hits your database.

Notable releases:

    •   1.0.x: First stable release, no major changes to take advantage of a point of relative stability.

    •   2.0.x: Recode of rules, whitespace fixing consolidation, sqlfluff format and removal of support for dbt versions pre 1.1. Note, that this release brings with it some breaking changes to rule coding and configuration, see Upgrading from 1.x to 2.0 <#upgrading-2-0>.

    •   3.0.x: sqlfluff fix now defaults to not asking for confirmation and the --force option was removed. Richer information returned by the sqlfluff lint command (although in a different structure to previous versions). See Upgrading to 3.x <#upgrading-3-0>.

For more detail on other releases, see our Release Notes <#releasenotes>.

Want to see where and how people are using SQLFluff in their projects? Head over to SQLFluff in the Wild <#inthewildref> for inspiration.

GETTING STARTED

To get started just install the package, make a sql file and then run SQLFluff and point it at the file. For more details or if you don't have python or pip already installed see Getting Started <#gettingstartedref>.

            $ pip install sqlfluff
            $ echo "  SELECT a  +  b FROM tbl;  " > test.sql
            $ sqlfluff lint test.sql --dialect ansi
            == [test.sql] FAIL
            L:   1 | P:   1 | LT01 | Expected only single space before 'SELECT' keyword.
                                   | Found '  '. [layout.spacing]
            L:   1 | P:   1 | LT02 | First line should not be indented.
                                   | [layout.indent]
            L:   1 | P:   1 | LT13 | Files must not begin with newlines or whitespace.
                                   | [layout.start_of_file]
            L:   1 | P:  11 | LT01 | Expected only single space before binary operator '+'.
                                   | Found '  '. [layout.spacing]
            L:   1 | P:  14 | LT01 | Expected only single space before naked identifier.
                                   | Found '  '. [layout.spacing]
            L:   1 | P:  27 | LT01 | Unnecessary trailing whitespace at end of file.
                                   | [layout.spacing]
            L:   1 | P:  27 | LT12 | Files must end with a single trailing newline.
                                   | [layout.end_of_file]
            All Finished 📜 🎉!

CONTENTS

  Getting Started

To get started with SQLFluff you'll need python and pip installed on your machine, if you're already set up, you can skip straight to Installing sqlfluff.

  Installing Python

How to install python and pip depends on what operating system you're using. In any case, the python wiki provides up to date instructions for all platforms here <https://wiki.python.org/moin/BeginnersGuide/Download>.

There's a chance that you'll be offered the choice between python versions. Support for python 2 was dropped in early 2020, so you should always opt for a version number starting with a 3. As for more specific options beyond that, SQLFluff aims to be compatible with all current python versions, and so it's best to pick the most recent.

You can confirm that python is working as expected by heading to your terminal or console of choice and typing python --version which should give you a sensible read out and not an error.

            $ python --version
            Python 3.9.1

For most people, their installation of python will come with pip (the python package manager) preinstalled. To confirm this you can type pip --version similar to python above.

            $ pip --version
            pip 21.3.1 from ...

If however, you do have python installed but not pip, then the best instructions for what to do next are on the python website <https://pip.pypa.io/en/stable/installation/>.

  Installing SQLFluff

Assuming that python and pip are already installed, then installing SQLFluff is straight forward.

            $ pip install sqlfluff

You can confirm its installation by getting SQLFluff to show its version number.

            $ sqlfluff version
            3.5.0

  Basic Usage

To get a feel for how to use SQLFluff it helps to have a small .sql file which has a simple structure and some known issues for testing. Create a file called test.sql in the same folder that you're currently in with the following content:

            SELECT a+b  AS foo,
            c AS bar from my_table

You can then run sqlfluff lint test.sql --dialect ansi to lint this file.

            $ sqlfluff lint test.sql --dialect ansi
            == [test.sql] FAIL
            L:   1 | P:   1 | LT09 | Select targets should be on a new line unless there is
                                   | only one select target.
                                   | [layout.select_targets]
            L:   1 | P:   1 | ST06 | Select wildcards then simple targets before calculations
                                   | and aggregates. [structure.column_order]
            L:   1 | P:   7 | LT02 | Expected line break and indent of 4 spaces before 'a'.
                                   | [layout.indent]
            L:   1 | P:   9 | LT01 | Expected single whitespace between naked identifier and
                                   | binary operator '+'. [layout.spacing]
            L:   1 | P:  10 | LT01 | Expected single whitespace between binary operator '+'
                                   | and naked identifier. [layout.spacing]
            L:   1 | P:  11 | LT01 | Expected only single space before 'AS' keyword. Found '
                                   | '. [layout.spacing]
            L:   2 | P:   1 | LT02 | Expected indent of 4 spaces.
                                   | [layout.indent]
            L:   2 | P:   9 | LT02 | Expected line break and no indent before 'from'.
                                   | [layout.indent]
            L:   2 | P:  10 | CP01 | Keywords must be consistently upper case.
                                   | [capitalisation.keywords]
            All Finished 📜 🎉!

You'll see that SQLFluff has failed the linting check for this file. On each of the following lines you can see each of the problems it has found, with some information about the location and what kind of problem there is. One of the errors has been found on line 1, position * (as shown by :code:`L: 1 | P: 9`) and it's a problem with rule *LT01 (for a full list of rules, see Rules Reference <#ruleref>). From this (and the following error) we can see that the problem is that there is no space either side of the + symbol in a+b. Head into the file, and correct this issue so that the file now looks like this:

            SELECT a + b  AS foo,
            c AS bar from my_table

Rerun the same command as before, and you'll see that the original error (violation of LT01) no longer shows up.

            $ sqlfluff lint test.sql --dialect ansi
            == [test.sql] FAIL
            L:   1 | P:   1 | LT09 | Select targets should be on a new line unless there is
                                   | only one select target.
                                   | [layout.select_targets]
            L:   1 | P:   1 | ST06 | Select wildcards then simple targets before calculations
                                   | and aggregates. [structure.column_order]
            L:   1 | P:   7 | LT02 | Expected line break and indent of 4 spaces before 'a'.
                                   | [layout.indent]
            L:   1 | P:  13 | LT01 | Expected only single space before 'AS' keyword. Found '
                                   | '. [layout.spacing]
            L:   2 | P:   1 | LT02 | Expected indent of 4 spaces.
                                   | [layout.indent]
            L:   2 | P:   9 | LT02 | Expected line break and no indent before 'from'.
                                   | [layout.indent]
            L:   2 | P:  10 | CP01 | Keywords must be consistently upper case.
                                   | [capitalisation.keywords]

To fix the remaining issues, we're going to use one of the more advanced features of SQLFluff, which is the fix command. This allows more automated fixing of some errors, to save you time in sorting out your sql files. Not all rules can be fixed in this way and there may be some situations where a fix may not be able to be applied because of the context of the query, but in many simple cases it's a good place to start.

For now, we only want to fix the following rules: LT02, LT12, CP01

            $ sqlfluff fix test.sql --rules LT02,LT12,CP01 --dialect ansi
            ==== finding violations ====
            == [test.sql] FAIL
            L:   1 | P:   7 | LT02 | Expected line break and indent of 4 spaces before 'a'.
                                   | [layout.indent]
            L:   2 | P:   1 | LT02 | Expected indent of 4 spaces.
                                   | [layout.indent]
            L:   2 | P:   9 | LT02 | Expected line break and no indent before 'FROM'.
                                   | [layout.indent]
            L:   2 | P:  10 | CP01 | Keywords must be consistently upper case.
                                   | [capitalisation.keywords]
            ==== fixing violations ====
            4 fixable linting violations found
            Are you sure you wish to attempt to fix these? [Y/n]

...at this point you'll have to confirm that you want to make the changes by pressing y on your keyboard...

            Are you sure you wish to attempt to fix these? [Y/n] ...
            Attempting fixes...
            Persisting Changes...
            == [test.sql] PASS
            Done. Please check your files to confirm.

If we now open up test.sql, we'll see the content is now different.

            SELECT
                a + b  AS foo,
                c AS bar
            FROM my_table

In particular:

    •   The two columns have been indented to reflect being inside the SELECT statement.

    •   The FROM keyword has been capitalised to match the other keywords.

We could also fix all of the fixable errors by not specifying --rules.

            $ sqlfluff fix test.sql --dialect ansi
            ==== finding violations ====
            == [test.sql] FAIL
            L:   1 | P:   1 | ST06 | Select wildcards then simple targets before calculations
                                   | and aggregates. [structure.column_order]
            L:   2 | P:  10 | LT01 | Expected only single space before 'AS' keyword. Found '
                                   | '. [layout.spacing]
            ==== fixing violations ====
            2 fixable linting violations found
            Are you sure you wish to attempt to fix these? [Y/n] ...
            Attempting fixes...
            Persisting Changes...
            == [test.sql] PASS
            Done. Please check your files to confirm.

If we now open up test.sql, we'll see the content has been updated again.

            SELECT
                c AS bar,
                a + b AS foo
            FROM my_table

The SQL statement is now well formatted according to all the rules defined in SQLFluff.

The --rules argument is optional, and could be useful when you or your organisation follows a slightly different convention than what we have defined.

  Custom Usage

So far we've covered the stock settings of SQLFluff, but there are many different ways that people style their sql, and if you or your organisation have different conventions, then many of these behaviours can be configured. For example, given the example above, what if we actually think that indents should only be two spaces, and rather than uppercase keywords, they should all be lowercase?

To achieve this we create a configuration file named .sqlfluff and place it in the same directory as the current file. In that file put the following content:

            [sqlfluff]
            dialect = ansi

            [sqlfluff:indentation]
            tab_space_size = 2

            [sqlfluff:rules:capitalisation.keywords]
            capitalisation_policy = lower

Then rerun the same command as before.

            $ sqlfluff fix test.sql --rules LT02,LT12,CP01,ST06,LT09,LT01

Then examine the file again, and you'll notice that the file has been fixed accordingly.

            select
              c as bar,
              a + b as foo
            from my_table

For a full list of configuration options check out Default Configuration <#defaultconfig>. Note that in our example here we've only set a few configuration values and any other configuration settings remain as per the default config. To see how these options apply to specific rules check out the "Configuration" section within each rule's documentation in Rules Reference <#ruleref>.

  Going further

From here, there are several more things to explore.

    •   To understand how SQLFluff is interpreting your file explore the parse command. You can learn more about that command and more by running sqlfluff --help or sqlfluff parse --help.

    •   To start linting more than just one file at a time, experiment with passing SQLFluff directories rather than just single files. Try running sqlfluff lint . (to lint every sql file in the current folder) or sqlfluff lint path/to/my/sqlfiles.

    •   To find out more about which rules are available, see Rules Reference <#ruleref>.

    •   To find out more about configuring SQLFluff and what other options are available, see Configuration <#config>.

    •   Once you're ready to start using SQLFluff on a project or with the rest of your team, check out Production Usage & Security <#production-use>.

One last thing to note is that SQLFluff is a relatively new project and you may find bugs or strange things while using it. If you do find anything, the most useful thing you can do is to post the issue on GitHub <https://github.com/sqlfluff/sqlfluff/issues> where the maintainers of the project can work out what to do with it. The project is in active development and so updates and fixes may come out regularly.

  Why SQLFluff?

SQL has been around for a long time, as a language for communicating with databases, like a communication protocol. More recently with the rise of data as a business function, or a domain in its own right SQL has also become an invaluable tool for defining the structure of data and analysis - not just as a one off but as a form of infrastructure as code <https://en.wikipedia.org/wiki/Infrastructure_as_code>.

As analytics transitions from a profession of people doing one-offs, and moves to building stable and reusable pieces of analytics, more and more principles from software engineering are moving in the analytics space. One of the best articulations of this is written in the viewpoint section of the docs for the open-source tool dbt <https://docs.getdbt.com/docs/viewpoint>. Two of the principles mentioned in that article are quality assurance <https://docs.getdbt.com/docs/viewpoint#quality-assurance> and modularity <https://docs.getdbt.com/docs/viewpoint#modularity>.

  Quality assurance

The primary aim of SQLFluff as a project is in service of that first aim of quality assurance <https://docs.getdbt.com/docs/viewpoint#quality-assurance>. With larger and larger teams maintaining large bodies of SQL code, it becomes more and more important that the code is not just valid but also easily comprehensible by other users of the same codebase. One way to ensure readability is to enforce a consistent style <https://www.smashingmagazine.com/2012/10/why-coding-style-matters/>, and the tools used to do this are called linters <https://en.wikipedia.org/wiki/Lint_(software)>.

Some famous linters <https://en.wikipedia.org/wiki/Lint_(software)> which are well known in the software community are flake8 <http://flake8.pycqa.org/> and jslint <https://www.jslint.com/> (the former is used to lint the SQLFluff project itself).

SQLFluff aims to fill this space for SQL.

  Modularity

SQL itself doesn't lend itself well to modularity <https://docs.getdbt.com/docs/viewpoint#modularity>, so to introduce some flexibility and reusability it is often templated <https://en.wikipedia.org/wiki/Template_processor>. Typically this is done in the wild in one of the following ways:

    1.  Using the limited inbuilt templating abilities of a programming language directly. For example in python this would be using the format string syntax <https://docs.python.org/3/library/string.html#formatstrings>:

                "SELECT {foo} FROM {tbl}".format(foo="bar", tbl="mytable")

    Which would evaluate to:

                SELECT bar FROM mytable

    2.  Using a dedicated templating library such as jinja2 <https://jinja.palletsprojects.com/>. This allows a lot more flexibility and more powerful expressions and macros. See the Templating Configuration <#templateconfig> section for more detail on how this works.

        •   Often there are tools like dbt <https://getdbt.com> or apache airflow <https://airflow.apache.org> which allow templated <https://en.wikipedia.org/wiki/Template_processor> sql to be used directly, and they will implement a library like jinja2 <https://jinja.palletsprojects.com/> under the hood themselves.

All of these templating tools are great for modularity <https://docs.getdbt.com/docs/viewpoint#modularity> but they also mean that the SQL files themselves are no longer valid SQL code, because they now contain these configured placeholder values, intended to improve modularity.

SQLFluff supports both of the templating methods outlined above, as well as dbt <https://getdbt.com> projects, to allow you to still lint these "dynamic" SQL files as part of your CI/CD pipeline (which is great 🙌), rather than waiting until you're in production (which is bad 🤦, and maybe too late).

During the CI/CD pipeline (or any time that we need to handle templated <https://en.wikipedia.org/wiki/Template_processor> code), SQLFluff needs additional info in order to interpret your templates as valid SQL code. You do so by providing dummy parameters in SQLFluff configuration files. When substituted into the template, these values should evaluate to valid SQL (so SQLFluff can check its style, formatting, and correctness), but the values don't need to match actual values used in production. This means that you can use much simpler dummy values than what you would really use. The recommendation is to use the simplest possible dummy value that still allows your code to evaluate to valid SQL so that the configuration values can be as streamlined as possible.

  Vision for SQLFluff

SQLFluff has a few components:

    1.  A generic parser for SQL which aims to be able to unify SQL written in different dialects into a comparable format. The parser.

    2.  A mechanism for measuring written SQL against a set of rules, with the added ability to fix any violations found. The linter.

    3.  An opinionated set of guidelines for how SQL should be structured and formatted. The rules.

The core vision [1] for SQLFluff is to be really good at being the linter. The reasoning for this is outlined in Why SQLFluff?.

Most of the codebase for SQLFluff is the parser, mostly because at the point of developing SQLFluff, there didn't appear to be a good option for a whitespace-aware parser that could be used instead.

With regards to the rules, SQLFluff aims to be opinionated but it also accepts that many organisations and groups have pre-existing strong conventions around how to write SQL and so ultimately SQLFluff should be flexible enough to support whichever rule set a user wishes to.

Notes

[1] Credit to this article <https://opensource.com/business/16/6/bad-practice-foss-projects-management> for highlighting the importance of a good vision.

  Guides & How-tos

This section is of short guides and articles is meant to be read alongside the rest of the documentation which is more reference-oriented.

  Setting up SQLFluff

  Rolling out SQLFluff with a new team

Rolling out SQLFluff, like rolling out any other linter or style guide, is not just about the technical rollout, but also how you introduce the tool to the team and organisation around you.
        The effect of SQLFluff should be to change your behaviours, not just your SQL.

With that in mind, it's worth reminding ourselves what we're trying to achieve with a tool like this. A set of potential success criteria might be:

    1.  Faster comprehension and collaboration by the team on a shared codebase. This includes more effective (and more enjoyable) code review on top of code which is easy to review and build upon.

    2.  Easier and faster onboarding for new team members. By adopting a style which is clean and consistent with other organisations we make it easier for new people to join the team.

    3.  Improved adoption of shared SQL from other sources. If the SQL found in open source projects is easy to read and looks familiar then you're more likely to use it. This means more reusable code across the industry.

    4.  Productive discussions around style. By defining your organisation's style guide in code, it means you can version control it, discuss changes and ultimately give a concrete output to discussions over style.

        You like leading commas? Make a PR to .sqlfluff and let's discuss with the team what the implications would be.

Consider which of these success measures is most important and most desirable for your team. Write that down.

The following steps are a guide, which you should adapt to your organisation, and in particular its level of data maturity.

  1. Assess the situation

This step is done by you, or a small group of people who already think that linting is a good idea.

    •   Run sqlfluff lint on your project with the stock configuration to find out how things work out of the box.

    •   Set up your Configuration <#config> so that things run and that you can get a readout of the errors which you would want the team to see and not the ones you don't. Great tools for this are to use .sqlfluffignore <#sqlfluffignore>, --exclude-rules or --ignore in the CLI (see CLI Reference <#cliref>).

    •   Identify which areas of your project are the worst and which are the tidiest. In particular, any areas which are particularly tidy already will be particularly useful in the next phase.

  2. Make a plan

There are three sensible rollout phases:

    1.  Pre CI/CD.

    2.  Soft CI/CD (warnings but no strict fails).

    3.  Hard CI/CD (violations mean deployments fail).

In each of these phases you have three levers to play with:

    1.  Areas of the project in which to apply rules.

    2.  Depth of rules enforced (this might also include whether to ignore parsing errors or not).

    3.  Whether to just lint changes (Using SQLFluff on changes using diff-quality <#diff-quality>), or to lint all the existing code as well.

Work out a sensible roadmap of how hard you want to go in each phase. Be clear who is responsible for changes at each phase. An example plan might look like this:

    1.  Pre CI/CD we get the tidiest area of a project to a stage that it fully passes the rules we eventually want to enforce. The core project team will do this. Liberal use of sqlfluff fix can be a lifesaver in this phase.

    2.  Soft CI/CD is applied to the whole project, team members are encouraged to write tidy SQL, but not required to.

    3.  Hard CI/CD is applied to the tidy areas of the project and also to any changes to the whole project. Anyone making changes is required to write SQL which passes check.

    4.  Hard CI/CD is applied to the whole project on not just changes, with only a few particularly problematic files explicitly ignored using .sqlfluffignore <#sqlfluffignore>.

  3. Build the need

Bring your team together to introduce both linting as a concept and also SQLFluff as a tool. At this stage it's really important that the team understand *why* this is a good thing.

Consider whether to discuss the whole plan from step 2, or whether to only talk about the first few steps. Aim to make this an empowering experience that everyone can get involved with rather than another piece of admin they need to do.

At this stage, you might also want to consider other tools in the SQLFluff ecosystem such as the SQLFluff pre-commit hook <#using-pre-commit> and the SQLFluff VSCode plugin <https://github.com/sqlfluff/vscode-sqlfluff> or SQLFluff online formatter <https://online.sqlfluff.com/>.

  4. Do, Review & Reassess

Once the plan is in motion, make sure to start putting in place norms and rituals around how you change the rules. In particular:

    •   How would someone suggest changing the style guide or enabling/disabling a rule?

    •   How do we assess whether the changes are working for the team or whether some are creating unnecessary stress?

It's normal for your usage of tools like SQLFluff to change and evolve over time. It's important to expect this change in advance, and welcome it when it happens. Always make sure you're driving toward the success measures you decided up front, rather than just resisting the change.

  5. Spread the word 😁

Did it work? If so, spread the word. Tell a friend about SQLFluff.

If you're lucky they might share your views on comma placement 🤷‍♀️.

  Developing Custom Rules

It's quite common to have organisation-, or project-specific norms and conventions you might want to enforce using SQLFluff. With a little bit of python knowledge this is very achievable with SQLFluff, and there's a plugin architecture to support that.

This guide should be read alongside the code for the SQLFluff example plugin <https://github.com/sqlfluff/sqlfluff/tree/main/plugins/sqlfluff-plugin-example> and the more technical documentation for Developing Plugins <#developingpluginsref>.

  What Plugin do I need?

When thinking about developing a rule, the following thought process will help you decide what to develop:

    1.  When do I want this rule to show a warning, when should it definitely not show one? What information do I need when evaluating whether a the rule has been followed or not? This information will tell you about the two important locations in the parse tree which will become important.

        •   The trigger location: i.e. when should the rule be called for evaluation. e.g. CP01 <#rule-CP01> triggers on keywords, because it only needs the information about that keyword to run, but LT08 <#rule-LT08> triggers on WITH statements even though it's only interested in specific pieces of whitespace, because it needs the full context of the statement to evaluate. You may with to examine the parse structure of some example queries you'd want to handle by using sqlfluff parse my_file.sql to identify the right segment. This is then specified using the crawl_behaviour attribute on the rule.

        •   The anchor location: i.e. which position will show up in the CLI readout back to the user. To continue the example of above, while LT08 <#rule-LT08> triggers on a WITH statement, it anchors on a more specific segment just after where it expected whitespace. It specifies this using the anchor argument to the LintResult <#sqlfluff.core.rules.base.LintResult> object.

    2.  How should the rule evaluate and should I implement an auto-fix? For the simplest rules, it the logic to evaluate whether there's an issue can be very simple. For example in the SQLFluff example plugin <https://github.com/sqlfluff/sqlfluff/tree/main/plugins/sqlfluff-plugin-example>, we are just checking the name of an element isn't in a configured list. Typically we recommend that for organisation-specific rules, KEEP IT SIMPLE. Some of the rules bundled with SQLFluff contain a lot of complexity for handling how to automatically fix lots of edge cases, but for your organisation it's probably not worth the overhead unless you're a very big team or come across a huge amount of poorly formatted SQL.

        •   Consider the information not just to trigger, but also whether a custom error message would be appropriate and how to get the information to construct that too. The default error message will be the first line of the rule docstring <https://en.wikipedia.org/wiki/Docstring>. Custom messages can be configured by setting the description argument of the LintResult <#sqlfluff.core.rules.base.LintResult> object.

        •   Do use the existing SQLFluff core rules as examples of what is possible and how to achieve various things - but remember that many of them implement a level of complexity and edge case handling which may not be necessary for your organisation.

    3.  How am I going to roll out my rule to the team? Thinking through this aspect of rule development is just as important as the technical aspect. Spending a lot of time on rule development for it to be rejected by the end users of it is both a waste of time and also counterproductive.

        •   Consider manually fixing any pre-existing issues in your project which would trigger the rule before rollout.

        •   Seek consensus on how strictly the rule will be enforced and what the step by step pathway is to strict enforcement.

        •   Consider beta-testing your new rule with a smaller group of users who are more engaged with SQLFluff or code quality in general.

  Plugin Discovery

One of most common questions asked with respect to custom plugins is discovery, or "how do I tell SQLFluff where my plugin is". SQLFluff uses pluggy <https://pluggy.readthedocs.io/en/latest/> as it's plugin architecture (developed by the folks at pytest <https://docs.pytest.org/en/stable/>). Pluggy uses the python packaging metadata for plugin discovery. This means that your plugin must be installed as a python package for discovery. Specifically, it must define an entry point <https://setuptools.pypa.io/en/stable/userguide/entry_point.html> for SQLFluff. When SQLFluff runs, it inspects installed python packages for this entry point and then can run any which define one. For example you'll see in the SQLFluff example plugin <https://github.com/sqlfluff/sqlfluff/tree/main/plugins/sqlfluff-plugin-example> that the pyproject.toml file has the following section:

            [project.entry-points.sqlfluff]
            # Change this name in your plugin, e.g. company name or plugin purpose.
            sqlfluff_example = "sqlfluff_plugin_example"

You can find equivalent examples for setup.cfg and setup.py in the python docs for entry point <https://setuptools.pypa.io/en/stable/userguide/entry_point.html>. This information is registered on install of your plugin, (i.e. when running pip install, or equivalent if you're using a different package manager) so if you change it later, you may need to re-install your plugin.

You can test whether your rule has been successfully discovered by running sqlfluff rules and reviewing whether your new rule has been included in the readout.

Note:
        If you're struggling with rule discovery, use the example plugin. It can be much easier to take a known working example and then modify from there:

            1.  Copy the code from the SQLFluff example plugin <https://github.com/sqlfluff/sqlfluff/tree/main/plugins/sqlfluff-plugin-example> into a local folder.

            2.  Run pip install -e /path/to/where/you/put/it.

            3.  Run sqlfluff rules, to confirm that the example plugin is present to demonstrate to yourself that discovery is functional.

            4.  Then edit the example plugin to do what you want now that discovery isn't an issue. You may have to re-run pip install ... if you change anything in the rule metadata (like the entry point, filenames or plugin location).

  Troubleshooting SQLFluff

  How to Troubleshoot SQLFluff

It can at times be tricky to troubleshoot SQLFluff as it exists within an ecosystem of other tools, and can be deployed in wide range of ways.

This step by step guide can help you narrow down what's likely going wrong and point you toward the swiftest resolution.

  1. Common Errors

There are a few error messages you may get which have relatively straightforward resolution paths.

  Parsing Errors

SQLFluff needs to be able to parse your SQL to understand it's structure. That means if it fails to parse the SQL it will give you an error message. The intent is that if SQLFluff cannot parse the SQL, then it should mean the SQL is also invalid and help you understand where and why.

For example, this is a simple query which is not valid SQL:

            select 1 2 3
            from my_table

When running sqlfluff lint or sqlfluff parse we get the following error message:

            ==== parsing violations ====
            L:   1 | P:  10 |  PRS | Line 1, Position 10: Found unparsable section: '2 3'

Furthermore if we look at the full parsing output we can see an unparsable section in the parse tree:

            [L:  1, P:  1]      |file:
            [L:  1, P:  1]      |    statement:
            [L:  1, P:  1]      |        select_statement:
            [L:  1, P:  1]      |            select_clause:
            [L:  1, P:  1]      |                keyword:                                      'select'
            [L:  1, P:  7]      |                [META] indent:
            [L:  1, P:  7]      |                whitespace:                                   ' '
            [L:  1, P:  8]      |                select_clause_element:
            [L:  1, P:  8]      |                    numeric_literal:                          '1'
            [L:  1, P:  9]      |                [META] dedent:
            [L:  1, P:  9]      |                whitespace:                                   ' '
            [L:  1, P: 10]      |                unparsable:                                   !! Expected: 'Nothing here.'
            [L:  1, P: 10]      |                    numeric_literal:                          '2'
            [L:  1, P: 11]      |                    whitespace:                               ' '
            [L:  1, P: 12]      |                    numeric_literal:                          '3'
            [L:  1, P: 13]      |            newline:                                          '\n'
            [L:  2, P:  1]      |            from_clause:
            [L:  2, P:  1]      |                keyword:                                      'from'
            [L:  2, P:  5]      |                whitespace:                                   ' '
            [L:  2, P:  6]      |                from_expression:
            [L:  2, P:  6]      |                    [META] indent:
            [L:  2, P:  6]      |                    from_expression_element:
            [L:  2, P:  6]      |                        table_expression:
            [L:  2, P:  6]      |                            table_reference:
            [L:  2, P:  6]      |                                naked_identifier:             'my_table'
            [L:  2, P: 14]      |                    [META] dedent:
            [L:  2, P: 14]      |    newline:                                                  '\n'
            [L:  3, P:  1]      |    [META] end_of_file:

SQLFluff maintains it's own version of each SQL dialect, and this may not be exhaustive for some of the dialects which are newer to SQLFluff or which are in very active development themselves. This means in some scenarios you may find a query which runs fine in your environment, but cannot be parsed by SQLFluff. This is not a "bug" per-se, but is an indicator of a gap in the SQLFluff dialect.

Many of the issues raised on GitHub relate to parsing errors like this, but it's also a great way to support the project if you feel able to contribute a dialect improvement yourself. We have a short guide on Contributing dialect changes <#contributing-dialect-changes> to walk you through the process. In the short term you can also ignore specific files from your overall project so that this specific file doesn't become a blocker for the rest. See Ignoring Errors & Files <#ignoreconfig>.

  Configuration Issues

If you're getting ether unexpected behaviour with your config, or errors because config values haven't been set correctly, it's often due to config file discovery (i.e. whether SQLFluff can find your config file, and what order it's combining config files).

For a more general guide to this topic see Setting Configuration <#setting-config>.

To help troubleshoot issues, if you run sqlfluff with a more verbose logging setting (e.g. sqlfluff lint /my/model.sql -v, or -vv, or -vvvvvv) you'll get a readout of the root config that SQLFluff is using. This can help debug which values are being used.

  2. Isolating SQLFluff

If you're still getting strange errors, then the next most useful thing you can do, both to help narrow down the cause, but also to assist with fixing a bug if you have found one, is to isolate SQLFluff from any other tools you're using in parallel:

    1.  If you're using SQLFluff with the dbt templater <#dbt-templater>, then try and recreate the error with the Jinja templater <#jinja-templater> to remove the influence of dbt and any database connection related issues.

    2.  If you're getting an error in a remote CI suite (for example on GitHub actions, or a server like Jenkins), try and recreate the issue locally on your machine using the same tools.

    3.  If you're Using pre-commit <#using-pre-commit>, Using SQLFluff on changes using diff-quality <#diff-quality> or the VSCode extension <https://github.com/sqlfluff/vscode-sqlfluff> try to recreate the issue by running the SQLFluff CLI Reference <#cliref> directly. Often this can make debugging significantly easier because some of these tools hide some of the error messages which SQLFluff gives the user to help debugging errors.

  3. Minimise the Query

Often SQL scripts can get very long, and if you're getting an error on a very long script, then it can be extremely difficult to work out what the issue is. To assist with this we recommend iteratively cutting down the file (or alternatively, iteratively building a file back up) until you have the smallest file which still exhibits the issue. Often after this step, the issue can become obvious.

    1.  If your file has multiple statements in it (i.e. statements separated by ;), then remove ones until SQLFluff no longer shows the issue. When you get to that point, add the offending one back in and remove all the others.

    2.  Simplify individual statements. For example in a SELECT statement, if you suspect the issue is coming from a particular column, then remove the others, or remove CTEs, until you've got the simplest query which still shows the issue.

  Contributing to SQLFluff

It is recommended that the following is read in conjunction with exploring the codebase. dialect_ansi.py <https://github.com/sqlfluff/sqlfluff/blob/main/src/sqlfluff/dialects/dialect_ansi.py> in particular is helpful to understand the recursive structure of segments and grammars.

You may also need to reference the Internal API <#internal-api-docs>.

  Architecture

At a high level, the behaviour of SQLFluff is divided into a few key stages. Whether calling sqlfluff lint, sqlfluff fix or sqlfluff parse, the internal flow is largely the same.

  Stage 1, the templater

This stage only applies to templated SQL. Vanilla SQL is sent straight to stage 2, the lexer.

In order to lint templated SQL, SQLFluff must first convert the 'raw' or pre-templated code into valid SQL, which can then be parsed. The templater returns both the raw and post-templated SQL so that any rule violations which occur in templated sections can be ignored and the rest mapped to their original line location for user feedback.

SQLFluff supports multiple templating engines:

            •   Jinja <https://jinja.palletsprojects.com/>

            •   SQL placeholders (e.g. SQLAlchemy parameters)

            •   Python format strings <https://docs.python.org/3/library/string.html#format-string-syntax>

            •   dbt <https://docs.getdbt.com/> (via plugin)

Under the hood dbt also uses Jinja, but in SQLFluff uses a separate mechanism which interfaces directly with the dbt python package.

For more details on how to configure the templater see Templating Configuration <#templateconfig>.

  Stage 2, the lexer

The lexer takes SQL and separates it into segments of whitespace and code. Where we can impart some high level meaning to segments, we do, but the result of this operation is still a flat sequence of typed segments (all subclasses of RawSegment).

  Stage 3, the parser

The parser is arguably the most complicated element of SQLFluff, and is relied on by all the other elements of the tool to do most of the heavy lifting.

    1.  The lexed segments are parsed using the specified dialect's grammars. In SQLFluff, grammars describe the shape of SQL statements (or their components). The parser attempts to apply each potential grammar to the lexed segments until all the segments have been matched.

    2.  In SQLFluff, segments form a tree-like structure. The top-level segment is a FileSegment, which contains zero or more StatementSegments, and so on. Before the segments have been parsed and named according to their type, they are 'raw', meaning they have no classification other than their literal value.

    3.  A segment's .match() method uses the match_grammar, on which .match() is called. SQLFluff parses in a single pass through the file, so segments will recursively match the file based on their respective grammars. In the example of a FileSegment, it first divides up the query into statements, and then the .match() method of those segments works out the structure within them.

        •

            Segments must implement a match_grammar. When .match()
                is called on a segment, this is the grammar which is used to decide whether there is a match.

        •

            Grammars combine segments or other grammars together in a
                pre-defined way. For example the OneOf grammar will match if any one of its child elements match.

        1.  During the recursion, the parser eventually reaches segments which have no children (raw segments containing a single token), and so the recursion naturally finishes.

    4.  If no match is found for a segment, the contents will be wrapped in an UnparsableSegment which is picked up as a parsing error later. This is usually facilitated by the ParseMode on some grammars which can be set to GREEDY, allowing the grammar to capture additional segments as unparsable. As an example, bracketed sections are often configured to capture anything unexpected as unparsable rather than simply failing to match if there is more than expected (which would be the default, STRICT, behaviour).

    5.  The result of the .match() method is a MatchResult which contains the instructions on how to turn the flat sequence of raw segments into a nested tree of segments. Calling .apply() on this result at the end of the matching process is what finally creates the nested structure.

When working on the parser there are a couple of design principles to keep in mind.

    •   Grammars are contained in dialects, the root dialect being the ansi dialect. The ansi dialect is used to host logic common to all dialects, and so does not necessarily adhere to the formal ansi specification. Other SQL dialects inherit from the ansi dialect, replacing or patching any segments they need to. One reason for the Ref grammar is that it allows name resolution of grammar elements at runtime and so a patched grammar with some elements overridden can still rely on lower-level elements which haven't been redeclared within the dialect

    •   All grammars and segments attempt to match as much as they can and will return partial matches where possible. It is up to the calling grammar or segment to decide whether a partial or complete match is required based on the context it is matching in.

  Stage 4, the linter

Given the complete parse tree, rule classes check for linting errors by traversing the tree, looking for segments and patterns of concern. If the rule discovers a violation, it returns a LintResult <#sqlfluff.core.rules.base.LintResult> pointing to the segment which caused the violation.

Some rules are able to fix the problems they find. If this is the case, the rule will return a list of fixes, which describe changes to be made to the tree. This can include edits, inserts, or deletions. Once the fixes have been applied, the updated tree is written to the original file.

  How to use Git

Many of the contributors may not be familiar with Git and it can be a confusing world for those new to it with perplexing terms like clone, fork, branch, merge conflicts and rebase. This guide aims to provide some information to those of you new to Git about the best way we think of working with it it is and also serve as a quick reference to some of the Git terms, or commands to use.

  Introduction to Git

This section will give some basic background to complete newbies to Git. Feel free to skip to the next section, Recommended way to use Git for SQLFluff, where we talk about how we use it on SQLFluff if you understand the basics already.

  What is Git?

Git is a distributed version control system. That mouthful basically means it's a way of keeping track of changes to our source code and other content - especially when many, many people are changing various parts of it. The distributed part of it is what makes Git so interesting (and so complicated!) - there can be many copies of our code, and that can cause fun and games when trying to keep it in sync!

The original and primary copy of a code base (called a repository or repo) is hosted on a server (e.g. GitHub), people will be working on copies in their local machine, and people may have forked a copy of the repo to another one also hosted on the server - and then that forked copy may also be copied locally to your machine. Add in different branches in any of those copies and it can quickly become quite confusing.

Git often involves working with the command line, which might be less familiar and a bit intimidating for those of you less technically minded. Graphical front end tools exist to try to replicate this command line functionality but it's helpful to have some familiarity with using Git on the command line and with a guide like this, hopefully that becomes less daunting a prospect!

  What is GitHub and how is it different than Git?

GitHub is not Git, but it is one of the most commonly used instances of Git and adds various features on top of the core versioning of code that Git handles. The main thing GitHub gives you is a Git server to store your code, and a nice web front end to manage it all through. Using the web front end you can view (and even change!) code, raise issues, open and review pull requests, use GitHub Actions to automate things (e.g. test code) and even host wiki pages like this one.

In this Wiki I've tried to differentiate between Git concepts and commands and those specific to GitHub. Other instances of Git that you might be familiar with, or use in work or other projects, include GitLab and BitBucket. They have many of the same features as GitHub.

GitHub also have a graphical front end tool called GitHub Desktop <https://desktop.github.com/> for working on on Git locally and syncing it back to GitHub. Check out the GitHub Desktop section for tips on how to use it.

SQLFluff makes extensive use of GitHub to help us manage the project and allow all the many disparate contributors to collaborate easily.

  Installing Git

While it is possible to work just using GitHub's website - especially if just comment on issues and adding your advice - managing the code really is best done locally on your own computer and then pushing changes back up to GitHub. Git is very popular and widely available (see installation instructions <https://git-scm.com/book/en/v2/Getting-Started-Installing-Git> for Windows, Mac & Linux). You may already have it installed, so to check if that's the case, open a command line and type:

            git --version

If you see a version number returned then you've passed the first step!

If not, then for Windows I recommend installing and using Git Bash <https://git-scm.com/download/win> which is a Linux-like command line. For MacOS the built in Terminal available under Launchpad is fine and running the above version check will prompt you to install XCode and Git. For Linux I presume you'll be familiar with how to install this.

  Git Repos

A Git Repository or Repo is a collection of all the code that makes up a project. Well that's not strictly true as a project may also depend on other programs and libraries, but typically they are not stored in the project repo - only the code specific to this project is stored in the repo along with config files that are used to install any necessary libraries to run the code and instead installed (e.g. using a command like npm install for node modules).

The main SQLFluff repo is available on GitHub at: <https://github.com/sqlfluff/sqlfluff>. However, we also have a few other repos for the VS Code extension and the like, available at <https://github.com/sqlfluff>.

  Git Branches

A repo will usually contain a number of branches. These are copies of the code where you can work independently on a particular item. The name branch is used because, like a tree, these can diverge from each other - though, unlike a tree, they are usually merged back when the work is complete.

There will be one main (or master) branch which everything should be merged back into when ready. Traditionally these have been called the master branch, but many projects are trying to use more inclusive language and have switched to using the name main or similar instead. SQLFluff moved to using main in 2021.

Creating a branch is very quick and is integral to how Git works. Git stores branches in an incredibly efficient way and doesn’t literally have a copy of the same code, but only differences basically. So do not feel like it's a big deal to create a branch (it's not!) and frequently creating small branches, and merging them back in to the main branch when ready is the best way to use Git. Creating large branches or reusing branches for lots of different changes is not the best way of using Git and will lead to issues.

  GitHub Pull Requests

Once your changes are ready to merge back to main you open a pull request (often shortened to PR), which creates a special type of GitHub issue which can be used to merge your changes into the main branch.

A pull request is really a GitHub concept and at the end of the day is basically a fancy way of actioning a merge in Git. Bitbucket also use the term Pull Request, while GitLab uses Merge Request. It should also not be confused with git pull, which is a Git command to pull down changes from the server (e.g. GitHub) into your local copy.

An example pull request on GitHub is shown below: [image: Screenshot of an example pull request on GitHub.] [image]

In this pull request there are the following tabs:

    •   Conversation - this allows you to give some info using GitHub markdown (including screenshots if you want). Reviewers can comment, and ask questions for you to answer, before merging the pull request into the main code.

    •   Commits - this shows a list of links to all the individual changes you made to the code. It's not that useful a tab to be honest!

    •   Checks - this shows all the automated checks run on your code so we know it's good! These are setup in the main repo using GitHub Actions (or similar) and the results are also shown at the bottom of the Conversation tab for open pull requests.

    •   Files Changed - this is one of the most useful tabs and shows each line of code changed. Reviewers should look those this tab, and can click on individual lines to make comments or code improvement suggestions <https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/commenting-on-a-pull-request> which are added to the Conversation tab and the person who opened the pull request (called the pull request author) can then answer or address the concern (including accepting any suggested code changes directly into this change with a click).

You can tag people to review your pull request, assign it to someone to deal with (not used much as kind of repeat of the author and reviewers), add labels...etc.

At the bottom of the Conversation tab you will see the following: [image: Bottom of a pull request with "Squash and Merge" and "Close" buttons.] [image]

This shows on this PR that all checks have passed and this is ready to merge. Clicking the big green "Squash and Merge" button will copy (the "Merge" part) all this code into main branch with one single commit (the "Squash" part). Usually you don't need to have all the 100s of commits you have have done while developing this code change so "Squash" is what you want but you can change it if you want.

You can also close this pull request if you change your mind with the Close button at the bottom, or add a comment with the Comment button if you make a big change to it since opening that you want people following the pull request to be aware of.

Please note you do NOT need to Close and Reopen the pull request (or even open a new pull request) when you need to make changes based on review feedback - simply pushing changes to the branch will cause any open pull request from that branch to automatically be updated and checks to automatically be rerun. It is expected (and a good thing!) to change your code based on feedback and this is very much part of the workflow of pull requests.

  GitHub Forks

As well as branches, GitHub has the concept of forks, which basically means taking a complete copy of the repo (and all its branches at that time) into your own GitHub account. You can then create a branch in that fork, and then open a pull request to to merge code from your branch on your fork, all the way back to the the original repo (called the upstream repo). It may sound like an Inception level of abstraction and confusion but it actually works quite well once you get your head around it.

Note:
        There is some confusion as to the name fork as traditionally that term was used when you wanted to take a project in a different direction than the original developers [1] - so you forked the code and never merged back again. In GitHub a fork is used to make changes outside of the original repo but usually with the intention of merging them back into the original repo once complete.

        [1] <https://drewdevault.com/2019/05/24/What-is-a-fork.html>

Why would you fork when you can just work in the original repo? Well most projects don't want people messing with the original repo so restrict permissions to only allow core contributors to create branches in the original repo. Others must therefore fork to make changes and then open pull requests to the original repo for review before they are committed.

And it's important to use the correct terminology when working with forks. Tempting as it is, the original repo should always be referred to as "original" or "upstream", and never "main" or "master" - which refer to branches within a repo. Similarly a "local" copy, or "clone" refers to the copy on your PC as we shall see and that can be of the original repo or a fork.

Another extra bit of hassle with a fork, is that you must keep it reasonably up to date with the original, upstream repo. To do that you periodically merge or rebase the fork back to the original repo which pulls down changes into your fork. We'll explain how to do that later.

  Cloning a Git Repo

To work on a project in GitHub you would normally clone a repo, which simply means taking a copy of it on your local PC. It is possible to make small edits on the GitHub.com website but it's quite limited and often doesn't allow you to run code locally to test it for example. You can clone a repo by clicking on green Code button on the repo's home page (make sure you do this on your fork and not on the main repo): [image: Screenshot of the clone button in GitHub.] [image]

This offers a number of options:

    •   "Clone with SSH" is the recommended way but is a little more complicated to set up <https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh>, but allows you to interact with GitHub without entering your GitHub password each time, and is basically mandatory if using 2FA for your GitHub account <https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh>.

    •   "Clone with HTTPS" works but requires you to enter your password each time so gets a little tiresome.

Once you copy the SSH or HTTPS URL on the command line simply go to the command line on your PC, into a directory you want to create the copy in and type the following (assuming SSH):

            git clone git@github.com:sqlfluff/sqlfluff.git

You can clone a local copy of the original repo, if you plan to (and have access to work on branches of that, or you can clone a fork of the original repo. The above example command clones the original repo location, and not the fork location - you should change the git address to the forked version when working from a fork.

After running this command you'll see the repo being downloaded locally. You can then branch, edit any of the files, or add new files, or even delete files to your hearts content. Any changes you make will only be on your machine and then you push changes back up to GitHub. We'll cover that later.

Just like with a fork, you need to keep any local up to date with both the original, upstream repo, and the GitHub version. This is done by using the git pull, git merge and git rebase commands. We'll explain how to do all that below.

  Git Merge Conflicts

When keeping all the different copies in sync you will inevitably run into the dreaded "merge conflict" - a rite of passage every developer must go through. This happens were you've changed some code, but so has someone else, and their changes has been merged into main, so when you attempt to merge (either by syncing main back to your branch to update your branch with any new changes since branching, or by attempting to open a pull request from your branch) Git will give up and say "I don't know what to do here - you deal with it!".

In actually fact, dealing with merge conflicts is actually very simple. When you open the conflicted file you'll see something like this:

            If you have questions, please
            <<<<<<< HEAD
            open an issue
            =======
            ask your question in Slack
            >>>>>>> branch-a

In this case someone changed the line to "open an issue" and merged that to main (aka HEAD) and you've also changed it to "ask your question in Slack". Git is warning you that it has been changed since branching but you also changed it. You simply need to decide what line you want and then delete all the other lines (including the ones starting <<<<, ==== and >>>>). Then git add the "resolved" file to your branch.

You can even do it directly on GitHub <https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/resolving-a-merge-conflict-on-github>.

Merge conflicts get a bad name and people think they are scary to deal with but Git actually makes it fairly easy. It will also usually only complain if the exact same line has changed — two people working on different parts of the same file usually won't see any merge conflicts.

Of course if you're both working on lots of the same code, across lots of files they can be a real pain to deal with - this is one of the main reasons to resync your branch back to the original main branch frequently, and also to work on small PRs rather than big unwieldy ones!

  Recommended way to use Git for SQLFluff

When working on SQLFluff you must fork SQLFluff to your own copy and work on that. SQLFluff contributors do not have access to create branches in the original repo.

To create your own Fork do the following:

    1.  Click the Fork button in GitHub and wait for it to complete.

    2.  Clone your fork locally - for me (tunetheweb) that means running this command on your machine git clone git@github.tunetheweb/sqlfluff.git

    3.  Add a link to upstream - git remote add upstream git@github.com:sqlfluff/sqlfluff.git (change this to the HTTPS URL if not using SSH)

It is also strongly recommended not to work on the main branch of your forked repo. When creating a new branch you will usually branch from main, so once your main has extra changes in it, it is no longer possible (or at least easy!) to create a clean branch for other work.

If you are only working on one thing at once, then using main in your fork may seem the quickest and easiest thing to do, but you'd be surprised how often you may want to pause that work for a bit and work on something else instead - and that's one of the advantages of using Git. For example if you are waiting on a pull request to be reviewed, you may want to work on another feature in the meantime on a fresh branch, which is completely independent of your other work. Or perhaps someone discovers an urgent, and easily fixed, bug in the code that you can quickly fix, before coming back to the current work. See the Switching between branches section below for more info on how to switch branches.

Working on a separate branch to main allows main to be kept in sync with upstream main, which allows new branches to be created more easily. It also allows you to merge upstream main into your branch periodically more easily. It also keeps your history of changes without a long history on each future pull request. Finally, it also also you to completely reset your main back to the same as upstream if you get it completely messed up, without losing any history in other branches.

  Resyncing your main branch to upstream

As discussed above it is recommended to use the main branch in your fork only to sync up with the main branch in the original SQLFluff repo. To do that you can do one of several things:

    1.  You can merge upstream main into your main branch - however if you have any changes, the this will add a commit message for each upstream change into your main. This means the next time you branch and open a pull request you'll see a long history of old commits, which is a bit noisy and annoying when they were nothing to do with your old pull request. Of course you shouldn't have any changes on main (because you're not using this as advised above aren't you?) so this shouldn't be an issue, but you never know...

    2.  You can rebase your main branch to be based on upstream main. This is a special Git command that basically makes it look like you just forked your main off of upstream main and then applies any changes you have made in your repo on top of that, so should lead to less risk of old commit messages, but it will still add commits for any changes you added. And again, you shouldn't have any changes on main so this shouldn't be an issue, but you never know...

    3.  You can blast away your main and basically reset it back to upstream main. This sounds extreme but is actually my preferred option to ensure main is super clean and reset back to the same as upstream main. Note this will (intentionally!) blast away any changes from your main and you will lose them completely but, again, there shouldn't be any changes here if following above advice.

So I prefer option 3 - it's the cleanest and ensures main is exactly the same as upstream, and that there will be no long history of commit messages in your next branch and pull request.

  Commands to force reset your main branch to upstream

You can use the below commands to reset your local fork to upstream and then push those changes to GitHub.

Note:
        You should ONLY do this if you are happy to lose all changes to your main branch and completely reset it as if you'd just forked it. If you have changes in flight on the main branch then you will need to either wait until they have been merged, or move them to a new branch before you can do a clean reset of main.

Check if upstream already exists:

            git remote -v

Add an upstream remote, if not already added previously:

            git remote add upstream git@github.com:sqlfluff/sqlfluff.git

Then force reset your main branch:

            git fetch upstream
            git checkout main
            git reset --hard upstream/main
            git push origin main --force

After this your should visit your forked repo on GitHub and check you get a message that "This branch is even with sqlfluff:main.": [image: A forked repo which is even with upstream.] [image]

  Creating and working on a branch

So, when you're ready to make your first changes, do the following:

    1.  Resync your local copy to upstream as per above (unless you've just forked it, in which case you can skip this step)

    2.  Make sure you are in the main branch - git checkout main

    3.  Branch main to a new branch with a meaningful branch name. For example to work on an issue to add new Posgres Create Table you'd run this command - git checkout -b postgres-create-table (note the -b which means "create a new branch please").

    4.  Make your changes using your favourite code editor (I recommend VSCode <https://code.visualstudio.com/> myself).

    5.  If making code changes to the website then test them - follow instructions in CONTRIBUTING.md <https://github.com/sqlfluff/sqlfluff/blob/main/CONTRIBUTING.md> to set up the environment and then use tox -e generate-fixture-yml,cov-init,py39,cov-report,linting to run most of the tests.

    6.  Add any new files you added in this change that you want tracked in git - e.g. git add text/fixtures/dialects/*.sql. You also need to do this to re-add merge conflict files that you have resolved. You typically do not need to add other files that are already in the repo that you have changed, as Git will track those already.

    7.  Commit your changes - git commit -a. This means “save all changes to Git (for tracked files)”. If you only want to commit some files, then git add just the files you want and then use git commit without the -a option. When you commit you need to add a message. Git will automatically open your default Git editor - usually vi which can take some getting used to but basically type your message and then type Esc : w q ! to enter command mode (Esc), start a command () save your file (w for write) and quit (q) and force override any errors on save (!). See the next section for an alternative method if you're not comfortable with vi. And btw if you want to cancel the commit at this stage, remove any message you added and type Esc q ! to quit without saving any changes and Git will see you didn't add a message and cancel the commit.

    8.  Push your committed changes to GitHub - git push. If this is your first push of this branch to GitHub, then git will tell you you have to use a slightly different command: git push --set-upstream origin postgres-create-table

    9.  If there are any changes already in GitHub (e.g. of other people are also working on this branch), then you will get and error and need to do a git pull to pull them down locally before you are able to push any more changes back up to GitHub.

    10. Repeat steps 4-9 as often as you want until you are happy to open a pull request to merge this back into SQLFluff code base.

  Committing changes - to vi or not to vi

My preferred method of committing changes is to use this:

            git commit -a

This takes all the changes for existing tracked files and adds them to the commit. New files still need to be added with git add but all files currently tracked by Git are automatically included without having to use git add. This then opens the default Git editor (usually vi) and lists the files for this commit, any files not included and allows you to add the message and complete the commit. If you close vi without adding a message then it cancels the commit.

However vi can be quite a barrier to people as it's quite a confusing editor with a "command" and an "edit" mode, and needing to remember command sequences (like ESC + w + q + !). For those not familiar with linux and vi this might be quite a stumbling block.

You can provide the message on the command line so you don't have to got near vi using a sequence of commands like this:

            git status
            git add file1 file2
            git commit -m "Committing file1 and file2"

This does require you to add the files or folders first so a bit more painful than git commit -a. Alternatively you use the -a and -m switches together:

            git status
            git commit -a -m "Committing all open files"

The downside is that, unlike the vi method, it won't show you the list of files it's going to commit, so is a bit more dangerous, hence why I prefer the vi method instead. However, if you do a git status before you commit, you should see the files that will be committed. Plus you can always revert changes if you need to.

So, in summary vi method is preferred but can be more complicated to those not familiar with it so can give commit message on command line but take care with it.

  Keeping your branch up to date

It is also recommended to merge any changes that have happened to SQLFluff code (in its main branch) into your branch periodically in case it affects your code, and particularly important to do this just before opening a PR.

To merge changes into a forked repo from upstream main do the following:

    1.  Commit any changes in the branch you are working on.

    2.  Update you local copy of upstream - git fetch upstream

    3.  Merge the changes from upstream main into your branch - git merge upstream/main

    4.  Commit the merge - git commit -a

    5.  Push all your changes up to GitHub - git push

Or to merge from a branch on the main repo do the following:

    1.  Commit any changes in the branch you are working on.

    2.  Update you local copy of main - git fetch origin main

    3.  Merge the changes from upstream main into your branch - git merge main

    4.  Commit the merge - git commit -a

    5.  Push all your changes up to GitHub - git push

  Switching between branches

Git allows working on several branches at once. This allows you to work on something else while you are stuck on one bit of work (waiting for answers, or pull request feedback, or you just fancy a break!). Use git checkout to switch between branches and use git status to ensure all your changes are committed when switching between branches.

For example, let's say you are working on feature1 branch:

            git checkout main
            git pull
            git checkout -b feature1
            # Make some changes
            git commit -m "Commit my changes for feature1"
            # Make some more changes
            git commit -m "Commit some more changes for feature1"
            # Push changes to Github.com if you want to (always good to do this in case your computer dies!)
            git push
            # Note the first time you push a new branch you will need a slightly different push comment:
            # `git push --set-upstream origin feature1`
            # Helpfully, git will tell you this if you try using just `git push`.

And then you want to take a break from feature1, in which case you should open a new branch - in most cases you want to branch from main again, and not from feature1 branch so make sure you flip back to main again if this is the case. If working on a fork, you should also check your main is up to date first - see the Resyncing your main branch to upstream section above:

            # Check your branch is clean and everything has been committed
            git status
            # Create a new branch from main (note I've not included the resyncing of main to upstream here for forks)
            git checkout main
            git pull
            git checkout -b feature2
            # Make some changes and push your new branch to GitHub
            git commit -m "Commit some changes for feature2"
            git push --set-upstream origin feature2
            # Make some more changes and push those too
            git commit -m "Commit more changes for feature2"
            git push

You are now free to switch back to feature1 if you want using git checkout (note you don't need the -b flag as that's only needed to create a new branch that doesn't exist, whereas just switching between branches that already exist don't need it):

            git checkout feature1

And then switch back to feature2 later:

            git checkout feature2

The primary concern with multiple branches like this is getting yourself confused! Using git status here is your friend to double check you're on the correct branch and all outstanding changes have been committed.

As with lots of things in Git, there are many ways to handle branches (including the git branch command), but I'd encourage you to stick with few commands and use git checkout, git commit, git push, git pull and git status as much as possible as you can do most things with those and it's already getting confusing!

The other alternative is to just create a new folder and clone the whole repo again and manage it completely separately. This can be easier, safer and less confusing for those less familiar with Git if working on a limited number of branches. However it doesn't scale very well and is not the way you're going to get the most out of Git so as soon as you go beyond a second branch I'd strongly encourage you get used to checking out between branches.

I do encourage separate folders however if you have different repos (e.g. your do some of the main work on the HTTPArchive repo, and some work on your own fork) as switching repo that a folder points to, while also possible, is adding yet more confusion to an already complex thing! 🙂

  Opening a Pull Request

Once you are finished making changes, you should take the following steps to open a pull request back to the original repo to accept your code into SQLFluff:

    1.  Merge in any changes that happened to SQLFluff code since you branches (see above).

    2.  Run all the automated tests tox -e generate-fixture-yml,cov-init,py39,cov-report,linting.

    3.  Make sure all your changes are pushed to GitHub.

    4.  Open a pull request in GitHub.

    5.  If the pull request closes an issue then you can add "Closes #123" or "Fixes #123" in the first comment and GitHub will automatically close issue #123 when the pull request is merged. If it doesn't fully close the issue, then you should instead say something like "Makes progress on #123" which will create a link on original issue but not close it.

As mentioned above, you can make more changes to your branch and push them up to GitHub and the Pull Request will automatically be updated. There is no need to close the PR and reopen a new one.

  Actioning feedback from pull requests

When your pull request is reviewed, the first bit of feedback you're likely to see if from the automated tests that run every time a pull request is opened. They take a few minutes to run and then you will then give you a satisfying green tick, or a scary red cross.

  Merge conflict checks

The first check that GitHub itself will do is check for any Git Merge Conflicts and these must be resolved before the pull request can be merged. If you merge main to your code before submitting a pull request, then it's unlikely you'll get any of these (unless someone's got in real quick while you were opening it!) which is why it's recommended to do that, but other pull requests can be action while yours is being reviewed so can still happen. Smaller pull request, touching few files reduces the chance of this so again, where at all possible, break up changes into smaller batches.

  Linter checks

Another check that we perform is linting the code in the pull request. This runs automated checks for errors or code styling and formatting issues that don't match the code conventions we use.

Python code is linted with flake8 and you can run this command on any files to see similar linting issues. You can use black to auto-fix most flake8 issues, though some need to be manually addresses.

  Code review feedback

After the automated tests have passed the code will be reviewed manually by a maintainer, or another contributor. They can ask questions, or make suggestions to change the code.

Look upon a code review as an opportunity to learn and improve your code. Feedback can be tough to hear after you've worked hard on code, but stay aware that it is meant with the best possible intention to provide feedback to you for this and future commits, and to keep our codebase to a high standard. It is not a personal sleight upon you or your code, and if you are getting annoyed with feedback I suggest you take a break and step away and read it again later, in a fresh light. Of course if if you feel that a reviewer is acting inappropriately then please raise it - we have a Code of Conduct <https://github.com/sqlfluff/sqlfluff/blob/main/CODE_OF_CONDUCT.md> and want all contributors to feel welcome. Feel free to also reach out to a maintainer if you would like to discuss something privately.

When a reviewer makes a code suggestion you can accept it right in GitHub and it will automatically update your branch. As this happens in GitHub directly, just make sure you do a git pull next time you are working locally on your code to pull down these changes. It's quite slow to accept a lot of changes this way, so if there are a lot of changes then sometimes better to change locally in your favourite editor, and then push one commit with the fixes, and then mark each of the suggestions as resolved. Any suggestions on lines which have changes since they were raised, will be marked as outdated which makes them easy to spot.

Reviewers may also make small, seemingly pedantic changes - usually they will include the word "nit", to indicate this is a "nitpick" in these. Like the linting, they can seem needless, but they help maintain our code quality.

It should also be noted that not all suggestions may be valid! Reviewers can make a mistake just as easily as the pull request author (more easily in some ways as they often will be reviewing based on reading the code rather than testing it). Feel free to push back on suggestions when you disagree with them. However, it is best to get consensus between reviewer and pull request author where at all possible so explain why you don't think you should make the change being suggested rather than just ignoring the suggestion or resolving it without a comment.

After addressing feedback, please re-request a review by clicking the little two arrow icon next to the reviewer name, or make a comment (e.g. "All feedback addresses. I think this is good to merge now."). Sometimes it's difficult to know if someone is still working on feedback and a pull request may be left unintentionally, getting out of date, because reviewers are not aware that it's good for a re-review or merge.

Only SQLFluff maintainers can merge pull requests, but every contributor can review pull requests, and merging becomes a lot easier (and so more likely!) if someone else has already had a look through the changes. Please, please, please help us reviewing_pull_requests to help spread the load!

  Draft pull requests

It is also possible to open a draft pull request, if you want early feedback on your code or approach. Please remember to convert to a full pull request when happy.

Additionally if making a large number of changes (for example if you need to update every dialect), then best to do this in only one or two files initially, open a pull request (whether draft or full) and get feedback on your approach before you spend time updating all the files! It's much easier to review code changes if they are not drowned out by lots of identical changes that needs that code in the same pull request, and much less likely to result in merge conflicts. Then the rest of the files can be added to the pull request, or a separate one opened for those (maybe a new pull request per language if you want different translators to approve any changes).

  Reviewing pull requests

As mentioned above we strongly encourage contributors to help review pull requests. This is a voluntary, collaborative effort for us all and depending on one or two people creates a bottleneck and a single point of failure for the project.

Even if a review pulls up nothing, it is important to approve the pull request - this indicates that it has been reviewed and is just as useful (if not more so) as commenting on code or making suggestions.

Do also be conscious of how feedback will be read. We have many first time contributors who may not be as familiar with language (either programming language or English language!) so do try to avoid technical terms, colloquialism...etc. Though we are aware of some very commonly used acronyms and terms (which we've included in our Glossary of terms) like LGTM ("Looks Good To Me").

Do be mindful as well that pull request authors have voluntarily spent time on this and we wish to encourage that and foster an inclusive environment. Offensive language is forbidden by our Code of Conduct <https://github.com/sqlfluff/sqlfluff/blob/main/CODE_OF_CONDUCT.md>.

Do remember that a code review is also about reviewing the authors changes, and not about showing off your own knowledge! Try not to get side-tracked but instead raise a new issue if you want to consider something else that comes up during a code review but is not directly related.

On that note, do also remember that code can be improved incrementally. Small changes are much better in Git. So, while it's OK to point out a fuller solution do remember that Perfect is the enemy of good <https://en.wikipedia.org/wiki/Perfect_is_the_enemy_of_good> and accepting a change that is an improvement and then improving further in future iterations can often be better than holding out for the perfect solution that may never come. Of course that doesn't mean we should accept code that regresses the quality, or seems like the wrong way of doing it completely!

And finally, we strongly encourage positive GitHub reactions - particular for new contributors. They give quick, encouraging, feedback and add a playful, fun tone: [image: GitHub Heart and Rocket reactions.] [image]

We discourage the negative ones though (thumbs down 👎, or confused 😕). Better to add a comment (politely!) explaining your concerns and then if others agree with you, they can "thumbs up" your comment. This keeps things on a positive tone and also means your cause for concerns are fully understood.

  GitHub Desktop

GitHub Desktop is a Windows and MacOS app that provides a visual interface to GitHub. It reduces the need to use and understand Git via the command line.

This section will provide some tips on performing some common tasks via the GitHub Desktop

  Installing GitHub Desktop

First make sure you have Git installed. See our section on Installing Git for more details.

You can then download the install file from <https://desktop.github.com/>, with further instructions from their Installing and configuring GitHub Desktop <https://docs.github.com/en/free-pro-team@latest/desktop/installing-and-configuring-github-desktop> document. Your main tasks will be to Authenticate with GitHub <https://docs.github.com/en/free-pro-team@latest/desktop/installing-and-configuring-github-desktop/authenticating-to-github> and Configuring Git for GitHub Desktop <https://docs.github.com/en/free-pro-team@latest/desktop/installing-and-configuring-github-desktop/configuring-git-for-github-desktop> so that the systems know who you are.

  Cloning the SQLFluff repo

If you have not done already, you will want to clone a copy of the <https://github.com/sqlfluff/sqlfluff> repo into your computer. The simplest way is to follow Cloning a repository from GitHub to GitHub Desktop <https://docs.github.com/en/free-pro-team@latest/desktop/contributing-and-collaborating-using-github-desktop/cloning-a-repository-from-github-to-github-desktop> where you go to the repository on the website and select "Open with GitHub Desktop". This will open a window where you can click "Clone" and the job will be done.

  Navigating GitHub Desktop

Once you have cloned repositories you will be able to select them via the "Current repository" toolbar button, just under the menu on the left. By default the sidebar will show you what edits have been made to the repository, and the main section shows actions you may want to perform.

  Updating your repository (Pull origin)

Over time the original repository will get updated and your copy will become out of date. GitHub Desktop will highlight if your repository is out of date, with an option to pull any changes from the origin so that you have the latest versions.

  Making your own edits (creating a branch)

You want to create your own branch before you start as you very likely do not have permission to edit the SQLFluff main branch. A branch is a way for you to group your own edits so you can later submit (push) them for review. Then, when they are approved, they will get merged back into the main branch.

Before creating a branch, make sure you're currently on the main branch and it is up to date (see above).

If you click on the "Current branch" tab in the toolbar you will see all the public branches in play. To create your own branch, enter a new name in the textbox at the top and click the "Create new branch" button.

  Publishing your branch

At the moment your branch is only known to you. If you want others to see it, then you need to publish it. GitHub Desktop will prompt you to do that.

Once published you and others can select your branch on the GitHub website.

  Editing your branch

You can edit the repository using your favourite editor. As you edit, GitHub Desktop will show you what changes you have made.

Note that you can change branches at any time, but I suggest you commit and push any edits (see next) before you switch as things can get confusing. If you are working with multiple branches, always keep an eye out to make sure you're on the right one when working.

  Committing and pushing your edits to the web

Every once in a while you want to store and document your changes. This can help you or others in the future. You also have to commit before you can share (push) your changes with anyone. You can quickly commit your current edits via the form to the bottom left.

Once you have commits you will be prompted to push those commits to GitHub. I typically do this straight after committing.

  Getting your changes accepted

At this point you have a branch with edits committed and everything pushed to GitHub. Once you are happy with your work, you want it to be reviewed, approved and merged back into the main repository.

For this I switch back to the website, as it is there you will be communicating with reviewers. To get this stage started you need to create a pull request. Go to the SQLFluff responsitory on GitHub <https://github.com/sqlfluff/sqlfluff>, make sure your branch is selected, then click the Pull request link and follow the instructions. This will notify the reviewers who will help you to get your changes live.

  Keeping the forked repository up to date

The main branch of your fork should be kept in sync with the original repository (rebased). Especially before you create any branches to make edits. Details on how to do this are in the Resyncing your main branch to upstream section.

  Making your own edits

This is done in the exact same way as before (i.e. in Making your own edits (creating a branch)). Create a branch from your master (make sure master is up to date using the above process), publish the branch, edit the files in the branch, commit your edits, push back to GitHub.

With a forked repository the process to get your edits accepted is about the same as before (i.e. in Getting your changes accepted). Go to the web page for your copy of the repository and create a pull request.

  Glossary of terms

This is a list of terms to those less familiar with Git/GitHub:

    •   branch - a copy of the code within a repo, where you may be working on code that is not ready to commit back to the main branch. Note Git actually only stores differences so it's not really a copy, but that's just an efficiency in Git and you can consider it a copy to all intents and purposes.

    •   fetch - a git command which downloads all changes from a remote repo (i.e. a server) to a local one.

    •   fork - a complete copy of a repo and all it's branches.

    •   LGTM - Short hand for Looks Good To Me - typically used approving a pull request.

    •   local - a complete copy of a repo on your PC

    •   main - the primary branch of SQLFluff repo. Some other repos use master for this.

    •   master - an alternative name for main branch used by some repos.

    •   merge - to copy changes from one branch into another branch.

    •   merge request - what a pull request is also known by particularly in GitLab an alternative to GitHub.

    •   origin - the server version if the repo (the opposite of local).

    •   pull - to fetch changes from a remote repo, and then merge them into this branch in one step.

    •   pull request - a way to merge changes back to the main branch. A pull request is a special issue type that allows the potential merge to be reviewed and commented on before it is merged.

    •   rebase - to bring a branch up to date, as if it had been created from now, while maintaining the existing changes on top.

    •   repo/repository - a git project which is basically a collection of files - which may exist in several branches.

    •   upstream - the original repo that a fork was created from.

  Contributing dialect changes

One of the best ways that SQLFluff users can improve SQLFluff for themselves and others is in contributing dialect changes.

Users will likely know their syntax much better than the regular maintainers and will have access to an instance of that SQL dialect to confirm changes are valid SQL in that dialect.

If you can fix your own issues then that's often the quickest way of unblocking any issues preventing you from using SQLFluff! The maintainers are all volunteers doing this in our spare time and (like you all I'm sure!), we only have so much time to work on this.

  How SQLFluff reads (or parses) SQL

SQLFluff has a lexer and parser which is built in a very modular fashion that is easy to read, understand, and expand on without any core programming skills or deep knowledge of Python or how SQLFluff operates. For more information see the Architecture Documentation <#architecture>, but will cover that briefly here to give you enough to start contributing.

We also have a robust Continuous Integration pipeline in GitHub where you can gain confidence your changes are correct and will not break other SQLFluff users, even before a regular maintainer reviews the code.

SQLFluff defines the syntax it will used in dialect files (more on this later). If you look at the dialect_ansi.py <https://github.com/sqlfluff/sqlfluff/blob/main/src/sqlfluff/dialects/dialect_ansi.py> file you will see it has syntax like this:

            class SelectClauseSegment(BaseSegment):
                """A group of elements in a select target statement."""

                type = "select_clause"
                match_grammar = StartsWith(
                    Sequence("SELECT", Ref("WildcardExpressionSegment", optional=True)),
                    terminator=OneOf(
                        "FROM",
                        "WHERE",
                        "ORDER",
                        "LIMIT",
                        "OVERLAPS",
                        Ref("SetOperatorSegment"),
                    ),
                    enforce_whitespace_preceding_terminator=True,
                )

                parse_grammar = Ref("SelectClauseSegmentGrammar")

This says the SelectClauseSegment starts with SELECT or SELECT * and ends when it encounters a FROM, WHERE, ORDER...etc. line.

The match_grammar is what is used primarily to try to match and parse the statement. It can be relatively simple (as in this case), to quickly match just the start and terminating clauses. If that is the case, then a parse_grammar is needed to actually delve into the statement itself with all the clauses and parts it is made up of. The parse_grammar can be fully defined in the class or, like above example, reference another class with the definition.

The match_grammar is used to quickly identify the start and end of this block, as parsing can be quite intensive and complicated as the parser tries various combinations of classes and segments to match the SQL (particularly optional ones like the WildcardExpressionSegment above, or when there is a choice of statements that could be used).

For some statements a quick match is not needed, and so we can delve straight into the full grammar definition. In that case the match_grammar will be sufficient and we don't need the optional parse_grammar.

Here's another statement, which only uses the match_grammar and doesn't have (or need!) an optional parse_grammar:

            class JoinOnConditionSegment(BaseSegment):
                """The `ON` condition within a `JOIN` clause."""

                type = "join_on_condition"
                match_grammar = Sequence(
                    "ON",
                    Indent,
                    OptionallyBracketed(Ref("ExpressionSegment")),
                    Dedent,
                )

You may have noticed that a segment can refer to another segment, and that is a good way of splitting up a complex SQL expression into its component parts to manage and handle them separately.

  Segment grammar options

There are a number of options when creating SQL grammar including:
Grammar | Used For | Example
"KEYWORD" | Having a raw SQL keyword | "SELECT"
Sequence() | Having a known sequence of Keywords or Segments | Sequence("SELECT", Ref("SelectClauseElementSegment"), "FROM"...)
AnyNumberOf() | Choose from a set of options which may be repeated | "SELECT", AnyNumberOf(Ref("WildcardExpressionSegment"), Ref("ColumnReferenceSegment")...)...
OneOf() | A more restrictive from a set of AnyNumberOf limited to just one option | OneOf("INNER","OUTER","FULL"), "JOIN"
Delimited() | Used for lists (e.g. comma-delimited - which is the default) | "SELECT", Delimited("SelectClauseElementSegment"), "FROM"...
Bracketed() | Used for bracketed options - like function parameters | Ref("FunctionNameSegment"), Bracketed(Ref("FunctionContentsGrammar")

Some of the keywords have extra params you can give them, the most commonly used will be optional=True. This allows you to further define the make up of a SQL statement. Here's the DeleteStatementSegment definition:

            parse_grammar = Sequence(
                "DELETE",
                Ref("FromClauseSegment"),
                Ref("WhereClauseSegment", optional=True),
            )

You can see the WHERE clause is optional (many's a head has been shaken because of deletes without WHERE clauses I'm sure, but that's what SQL syntax allows!).

Using these Grammar options, it's possible to build up complex structures to define SQL syntax.

  Segments and Grammars

A Segment is a piece of the syntax which defines a type (which can be useful to reference later in rules or parse trees). This can be through one of the functions that creates a Segment (e.g. NamedParser, SegmentGenerator...etc.) or through a class.

A Grammar is a section of syntax that can be used in a Segment. Typically these are created to avoid repeating the same code in multiple places. Think of a Grammar as an alias for a piece of syntax to avoid you having to type out the same code again and again and again.

The other good thing about Grammars is it allows other dialects to override a specific part of a Segment without having to redefine the whole thing just to tweak one small part. For example ansi defines this:

            NotOperatorGrammar=StringParser("NOT", KeywordSegment, type="keyword")

whereas mysql overrides this to:

            NotOperatorGrammar=OneOf(
                StringParser("NOT", KeywordSegment, type="keyword"),
                StringParser("!", CodeSegment, name="not_operator", type="not_operator"),
            ),

This allows MySQL to use ! in all the places that NOT was used (providing they use NotOperatorGrammar rather than hardcode the NOT keyword of course). This makes it much easier to customise syntax to a particular dialect without having to copy and paste (and maintain) nearly identical code multiple times just to add the extra ! syntax that MySQL supports to mean NOT.

  Dialects

A lot of SQL is the same no matter which particular type of SQL you are using. The basic SELECT.. FROM... WHERE statement is common to them all. However lots of different SQL dialects (Postgres, Snowflake, Oracle... etc.) have sprung up as different companies have implemented SQL, or expanded it, for their own needs.

For this reason, SQLFluff allows creating dialects, which can have different grammars from each other.

SQLFluff has all the dialects in the src/sqlfluff/dialects <https://github.com/sqlfluff/sqlfluff/tree/main/src/sqlfluff/dialects> folder. The main dialect file (that every other dialect ultimately inherits from) is the dialect_ansi.py <https://github.com/sqlfluff/sqlfluff/blob/main/src/sqlfluff/dialects/dialect_ansi.py> file.

In SQLFluff, a dialect is basically a file which inherits everything from the original ANSI dialect, and then adds or overrides parsing segments. If a dialect has the exact same SELECT, FROM and WHERE clauses as ANSI but a different :ORDER BY syntax, then only the :ORDER BY clause needs to overridden so the dialect file will be very small. For some of the other dialects where there's lots of differences (Microsoft T-SQL <#tsql-dialect-ref>!) you may be overriding a lot more.

  Lexing

I kind of skipped this part, but before a piece of SQL can be parsed, it is lexed - that is split up into symbols, and logical groupings.

An inline comment, for example, is defined as this:

            RegexLexer(
                "inline_comment",
                r"(--|#)[^\n]*",
                CommentSegment,
                segment_kwargs={"trim_start": ("--", "#")},
            ),

That is, anything after -- or # to the newline. This allows us to deal with that whole comment as one lexed block and so we don't need to define how to parse it (we even give that a parsing segment name here - CommentSegment).

For simple grammar addition, you won't need to to touch the lexing definitions as they usually cover most common ones already. But for slightly more complicated ones, you may have to add to this. So if you see lexing errors then you may have to add something here.

Lexing happens in order. So it starts reading the SQL from the start, until it has the longest lexing match, then it chomps that up, files it away as a symbol to deal with later in the parsing, and starts again with the remaining text. So if you have SELECT * FROM table WHERE col1 = 12345 it will not break that up into S, E, L...etc., but instead into SELECT, *, FROM, table...etc.

An example of where we had to override lexing, is in BigQuery we have parameterised variables which are of the form @variable_name. The ANSI lexer doesn't recognise the @ sign, so you could add a grammar or segment for that. But a better solution, since you don't need to know two parts (@ and variable_name) is to just tell the lexer to go ahead and parse the whole thing into one big symbol, that we will then use later in the parser:

            bigquery_dialect.insert_lexer_matchers(
                [
                    RegexLexer("atsign_literal", r"@[a-zA-Z_][\w]*", CodeSegment),
                ],
                before="equals",
            )

Note the before="equals" which means we tell the lexer the order of preference to try to match this symbol. For example if we'd defined an at_sign lexing rule for other, standalone @ usage, then we'd want this to be considered first, and only fall back to that if we couldn't match this.

  Keywords

Most dialects have a keywords file, listing all the keywords. Some dialects just inherit the ANSI keywords and then add or remove keywords from that. Not quite as accurate as managing the actual keywords, but a lot quicker and easier to manage usually!

Keywords are separated into RESERVED and UNRESERVED lists. RESERVED keywords have extra restrictions meaning they cannot be used as identifiers. If using a keyword in grammar (e.g. "SELECT"), then it needs to be in one of the Keywords lists so you may have to add it or you might see error's like this (showing "NAN" has not been added as a Keyword in this dialect):

            RuntimeError: Grammar refers to 'NanKeywordSegment' which was not found in the redshift dialect

Also if editing the main ANSI dialect, and adding the the ANSI keyword list, then take care to consider if it needs added to the other dialects if they will inherit this syntax - usually yes unless explicitly overridden in those dialects.

  Where to find the grammar for your database

Now that you know about some of the tools SQLFluff provides for lexing and parsing a SQL statement, what changes will you make to it? While devising ad-hoc changes to the grammar to fix particular issues can be better than nothing, the best and most robust contributions will be created by consulting the source of truth for the grammar of your dialect when mapping it to SQLFluff segments and grammars. This will help you exhaustively find all possible statements that would be accepted by the dialect.

Many computer languages are written using venerable tools like Flex <https://en.wikipedia.org/wiki/Flex_(lexical_analyser_generator)> and Bison <https://en.wikipedia.org/wiki/GNU_Bison>, or similar parser generators, and SQL database engines are no exception. You can refer to the parser specification in the source code of your database engine for the ultimate source of truth of how a SQL statement will be parsed: you might be surprised at what your SQL engine will parse due to gaps in the documentation!

You should also refer to the reference documentation for your SQL dialect to get a concise high-level overview of what the statement grammar looks like, as well as read of any further restrictions and intended use of the grammar that you find. If your SQL engine is closed-source, then you'll likely have only the reference documentation to work with. However, this will always be a less-accurate resource than the bison grammar that's actually used for code generation inside the database engine itself.

It is also extremely helpful to try parsing the queries that you put into the test fixtures to make sure that they are actually parsable by the database engine. They don't have to be valid queries per se (can refer to non-existing table names, etc), but you should confirm that they are parsable. We do not want to require that SQLFluff be able to parse a statement that the actual database engine would reject: overeager matching logic can create parsing issues elsewhere.

Here is a list of grammars and parsing techniques for some of the dialects implemented by SQLFluff:

  ANSI SQL

Unfortunately, the ANSI SQL standard is not free. If you want a licensed copy of the latest standard, it must be purchased: Part 2 <https://webstore.ansi.org/standards/iso/isoiec90752016-1646101> is the most useful section for SQLFluff since it contains the grammar. There are, however, other resources you can find on the Internet related to this standard:

    •   modern-sql.com/standard <https://modern-sql.com/standard>: has a discussion on the various parts of the standard, and links to some older/draft versions of it that are out there.

    •   jakewheat.github.io/sql-overview <https://jakewheat.github.io/sql-overview/>: has a nice browsable view of (only) the BNF grammar.

    •   web.cecs.pdx.edu/~len/sql1999.pdf <http://web.cecs.pdx.edu/~len/sql1999.pdf>: a copy of the (much older) SQL:1999 standard.

    •   developer.mimer.com/services/mimer-sql-validator/ <https://developer.mimer.com/services/mimer-sql-validator/>: the SQL-2016 validator can be used to verify if a query can be parsed using the ANSI standard.

  PostgreSQL

Simply Googling for pg <statement> will often bring up the documentation for an older PG version. Please be sure you're referring to the latest version of the documentation, as well as refer to the bison grammar.

    •   PostgreSQL Bison grammar <https://github.com/postgres/postgres/blob/master/src/backend/parser/gram.y>

    •   PostgreSQL Flex scanner <https://github.com/postgres/postgres/blob/master/src/backend/parser/scan.l>

    •   More information about the parsing stage <https://www.postgresql.org/docs/current/parser-stage.html>

    •   Reference documentation for Postgres SQL statements <https://www.postgresql.org/docs/current/sql-commands.html>

    •   To check if a statement is parseable, simply paste it into psql. If you get a ERROR: syntax error then it means that it can't be parsed. These queries do not need to be parsed by SQLFluff; please do not include them in the main test fixtures. If you get a different error, then it means the query was parsed successfully, and might have failed for a different reason (e.g. non-existing column name, etc). In that case, it's probably best if SQLFluff can also parse it.

    •   The pgsql-parser <https://www.npmjs.com/package/pgsql-parser> tool wraps the official PostgreSQL source code & bison grammar linked above into a simple CLI tool. You can use it if you want to view the exact parse tree that PG can see.

  MySQL

    •   Reference documentation for MySQL SQL statements <https://dev.mysql.com/doc/refman/8.0/en/sql-statements.html>

    •   MySQL Bison grammar <https://github.com/mysql/mysql-server/blob/8.0/sql/sql_yacc.yy>

    •   To check if a statement is parsable, simply paste it into mysql. Look for ERROR 1064 (42000): You have an error in your SQL syntax to indicate a parse error.

  Example of contributing a syntax fix

So that's a bit of theory but let's go through some actual examples of how to add to the SQLFluff code to address any issues you are seeing. In this I'm not going to explain about how to set up your Python development environment (see the Contributing to SQLFluff <#development> and the CONTRIBUTING.md <https://github.com/sqlfluff/sqlfluff/blob/main/CONTRIBUTING.md> file for that), nor how to manage Git (see our How to use Git <#using-git> guide if new to that, and we use the standard “Fork, and then open a PR” workflow common to GitHub projects).

So assuming you know (or are willing to follow above guides to find out!) how to set up Python environment, and commit via Git, how do you contribute a simple fix to a dialect for syntax you want SQLFluff to support?

  Example 1

If we look at issue #1520 <https://github.com/sqlfluff/sqlfluff/issues/1520> it was raised to say we couldn't parse this:

            CREATE OR REPLACE FUNCTION public.postgres_setof_test()
            RETURNS SETOF text

and instead returned this message:

            Found unparsable section: 'CREATE OR REPLACE FUNCTION crw_public.po...'

This was in the postgres dialect, so I had a look at dialect_postgres.py <https://github.com/sqlfluff/sqlfluff/blob/main/src/sqlfluff/dialects/dialect_postgres.py> and found the code in CreateFunctionStatementSegment which had the following:

            parse_grammar = Sequence(
                "CREATE",
                Sequence("OR", "REPLACE", optional=True),
                Ref("TemporaryGrammar", optional=True),
                "FUNCTION",
                Sequence("IF", "NOT", "EXISTS", optional=True),
                Ref("FunctionNameSegment"),
                Ref("FunctionParameterListGrammar"),
                Sequence(  # Optional function return type
                    "RETURNS",
                    OneOf(
                        Sequence(
                            "TABLE",
                            Bracketed(
                                Delimited(
                                    OneOf(
                                        Ref("DatatypeSegment"),
                                        Sequence(
                                            Ref("ParameterNameSegment"), Ref("DatatypeSegment")
                                        ),
                                    ),
                                    delimiter=Ref("CommaSegment"),
                                )
                            ),
                            optional=True,
                        ),
                        Ref("DatatypeSegment"),
                    ),
                    optional=True,
                ),
                Ref("FunctionDefinitionGrammar"),
            )

So it allowed returning a table, or a datatype.

Fixing the issue was as simple as adding the SETOF structure as another return option:

            parse_grammar = Sequence(
                "CREATE",
                Sequence("OR", "REPLACE", optional=True),
                Ref("TemporaryGrammar", optional=True),
                "FUNCTION",
                Sequence("IF", "NOT", "EXISTS", optional=True),
                Ref("FunctionNameSegment"),
                Ref("FunctionParameterListGrammar"),
                Sequence(  # Optional function return type
                    "RETURNS",
                    OneOf(
                        Sequence(
                            "TABLE",
                            Bracketed(
                                Delimited(
                                    OneOf(
                                        Ref("DatatypeSegment"),
                                        Sequence(
                                            Ref("ParameterNameSegment"), Ref("DatatypeSegment")
                                        ),
                                    ),
                                    delimiter=Ref("CommaSegment"),
                                )
                            ),
                            optional=True,
                        ),
                        Sequence(
                            "SETOF",
                            Ref("DatatypeSegment"),
                        ),
                        Ref("DatatypeSegment"),
                    ),
                    optional=True,
                ),
                Ref("FunctionDefinitionGrammar"),
            )

With that code the above item could parse.

I added a test case (covered below) and submitted pull request #1522 <https://github.com/sqlfluff/sqlfluff/pull/1522> to fix this.

  Example 2

If we look at issue #1537 <https://github.com/sqlfluff/sqlfluff/issues/1537> it was raised to say we couldn't parse this:

            select 1 from group

And threw this error:

            ==== parsing violations ====
            L:   1 | P:  10 |  PRS | Line 1, Position 10: Found unparsable section: 'from'
            L:   1 | P:  14 |  PRS | Line 1, Position 14: Found unparsable section: ' group'

The reporter had also helpfully included the parse tree (produced by sqlfluff parse):

            [L:  1, P:  1]      |file:
            [L:  1, P:  1]      |    statement:
            [L:  1, P:  1]      |        select_statement:
            [L:  1, P:  1]      |            select_clause:
            [L:  1, P:  1]      |                keyword:                                      'select'
            [L:  1, P:  7]      |                [META] indent:
            [L:  1, P:  7]      |                whitespace:                                   ' '
            [L:  1, P:  8]      |                select_clause_element:
            [L:  1, P:  8]      |                    literal:                                  '1'
            [L:  1, P:  9]      |            whitespace:                                       ' '
            [L:  1, P: 10]      |            [META] dedent:
            [L:  1, P: 10]      |            from_clause:
            [L:  1, P: 10]      |                unparsable:                                   !! Expected: 'FromClauseSegment'
            [L:  1, P: 10]      |                    keyword:                                  'from'
            [L:  1, P: 14]      |            unparsable:                                       !! Expected: 'Nothing...'
            [L:  1, P: 14]      |                whitespace:                                   ' '
            [L:  1, P: 15]      |                raw:                                          'group'
            [L:  1, P: 20]      |    newline:                                                  '\n'

So the problem was it couldn't parse the FromClauseSegment. Looking at that definition showed this:

            FromClauseTerminatorGrammar=OneOf(
                "WHERE",
                "LIMIT",
                "GROUP",
                "ORDER",
                "HAVING",
                "QUALIFY",
                "WINDOW",
                Ref("SetOperatorSegment"),
                Ref("WithNoSchemaBindingClauseSegment"),
            ),

So the parser was terminating as soon as it saw the GROUP and saying "hey we must have reached the end of the :code:`FROM` clause".

This was a little restrictive so changing that to this solved the problem:

            FromClauseTerminatorGrammar=OneOf(
                "WHERE",
                "LIMIT",
                Sequence("GROUP", "BY"),
                Sequence("ORDER", "BY"),
                "HAVING",
                "QUALIFY",
                "WINDOW",
                Ref("SetOperatorSegment"),
                Ref("WithNoSchemaBindingClauseSegment"),
            ),

You can see we simply replaced the "GROUP" by a Sequence("GROUP", "BY") so it would only match if both words were given. Rechecking the example with this changed code, showed it now parsed. We did the same for "ORDER", and also changed a few other places in the code with similar clauses and added a test case (covered below) and submitted pull request #1546 <https://github.com/sqlfluff/sqlfluff/pull/1546> to fix this.

  Example 3

As an example of using the reference grammar to fix an existing SQLFluff grammar, pull request #4744 <https://github.com/sqlfluff/sqlfluff/pull/4744> contributed the CREATE CAST / DROP CAST statements to SQLFluff from scratch for both ANSI and PostgreSQL dialects. The first step when contributing a new statement is to check whether the statement is part of the ANSI standard. If it is, then you very likely should first start by adding a generally vendor-neutral version to the SQLFluff ANSI dialect so that other dialects can inherit from it. Every database engine deviates from the ANSI standard in practice, but by adding a reasonably standard segment to the ANSI dialect, you'll probably do a reasonable thing for most other database dialects.

In this case, CREATE and DROP CAST were indeed defined in the ANSI standard <https://jakewheat.github.io/sql-overview/sql-2016-foundation-grammar.html#_11_63_user_defined_cast_definition>, as quickly revealed by a quick search of the document:

            <user-defined cast definition> ::=
                CREATE CAST <left paren>  <source data type>  AS <target data type>  <right paren>
                    WITH <cast function>
                    [ AS ASSIGNMENT ]

So the first step was to read this ANSI BNF grammar and use it to build a corresponding vendor-neutral CreateCastSegment in dialect_ansi.py <https://github.com/sqlfluff/sqlfluff/blob/main/src/sqlfluff/dialects/dialect_ansi.py>.

            class CreateCastStatementSegment(BaseSegment):
                """A `CREATE CAST` statement.
                https://jakewheat.github.io/sql-overview/sql-2016-foundation-grammar.html#_11_63_user_defined_cast_definition
                """

                type = "create_cast_statement"

                match_grammar: Matchable = Sequence(
                    "CREATE",
                    "CAST",
                    Bracketed(
                        Ref("DatatypeSegment"),
                        "AS",
                        Ref("DatatypeSegment"),
                    ),
                    "WITH",
                    Ref.keyword("SPECIFIC", optional=True),
                    OneOf(
                        "ROUTINE",
                        "FUNCTION",
                        "PROCEDURE",
                        Sequence(
                            OneOf("INSTANCE", "STATIC", "CONSTRUCTOR", optional=True),
                            "METHOD",
                        ),
                    ),
                    Ref("FunctionNameSegment"),
                    Ref("FunctionParameterListGrammar", optional=True),
                    Sequence("FOR", Ref("ObjectReferenceSegment"), optional=True),
                    Sequence("AS", "ASSIGNMENT", optional=True),
                )

            # Not shown: register the CreateCastStatementSegment in StatementSegment

As you work your way through the grammar, think about whether other parts of the SQL language might contain similar elements. For example, here we noticed that there are already segments we can reuse for data types, function names, and function parameter lists. This helped simplify our new grammar, as well as make it easy to centrally change those particular areas of the grammar in other dialects. Also consider whether there are entire new segments and grammars you should separately define in addition to the root statement segment you're writing. Introducing new and reusing existing segments adds structure to the SQLFluff parse tree that can make it easier for lint rules to analyze the tree. A strong indicator that there should be a shared segment or grammar is when the reference grammar has a symbol that is reused from multiple other symbols/statements.

After writing the ANSI segment (and corresponding tests), it was time to move on to the PostgreSQL grammar. In this case, a quick glance at the documentation <https://www.postgresql.org/docs/15/sql-createcast.html> shows us that there are some notable differences from ANSI SQL:

    •   You can only specify FUNCTION. Other keywords like ROUTINE and PROCEDURE are rejected.

    •   The SPECIFIC keyword is not supported.

    •   Most importantly: PG provides some non-standard extensions which we'd like to include, like WITHOUT FUNCTION and AS IMPLICIT.

However, we should also consult the bison grammar for CREATE CAST <https://github.com/postgres/postgres/blob/e0693faf797f997f45bee8e572e8b4288cc7eaeb/src/backend/parser/gram.y#L8938>. Bison grammars tend to be very lengthy and daunting, but the right techniques can help you quickly and easily find what you're looking for:

    •   Search for a symbol by adding a : to the end of it.

    •   Start with the highest level of the thing you are looking for. For example, start with the top-level statement symbol. With PostgreSQL, all statements end with Stmt. Putting it all together, if we search for CreateCastStmt:, that takes us right to the definition for it.

    •   Drill down into deeper parts of the parser to learn more. For example, we see function_with_argtypes in the sequence; if we want to know what that means, search for function_with_argtypes: to find it.

Examining the Bison grammar can take a few extra minutes, but it can be rewarding. You'll be surprised what you might learn. I've found entire alternate spellings of keywords in there that were not in the documentation, and which testing showed were indeed valid SQL! The grammar in PG documentation is human-maintained <https://github.com/postgres/postgres/blob/master/doc/src/sgml/ref/create_cast.sgml> and not auto-generated, so there can be and are gaps between what is parsable and what is documented.

A good approach if you're still learning might be to draft a segment from the high-level documentation, and then systematically go through it with the bison grammar and verify it's correct (and that you're not forgetting anything).

One aspect of bison grammars to be aware of is that the tend to be very recursive, because it doesn't have the high-level constructs such as AnyOf, Delimited, Bracketed, and so on that SQLFluff provides. On the other hand, SQLFluff doesn't scale well with recursion. Sometimes it's unavoidable and reasonable in many cases (e.g. parenthesized expression) to refer to another segment recursively. But many times the recursion is extremely trivial, and should always be rewritten using an existing high-level SQLFluff concept. For example, this bison grammar defines a bracketed comma-delimited list which would be better represented using Bracketed and Delimited in SQLFluff:

            func_args:  '(' func_args_list ')'              { $$ = $2; }
                        | '(' ')'                           { $$ = NIL; }
                    ;

            func_args_list:
                        func_arg                            { $$ = list_make1($1); }
                        | func_args_list ',' func_arg       { $$ = lappend($1, $3); }
                    ;

  Example 4

As an example of using the reference grammar to fix an existing SQLFluff grammar, issue #4336 <https://github.com/sqlfluff/sqlfluff/issue/4336> reported that array slices were not being parsed correctly in PostgreSQL. A simple SELECT statement was given that I further simplified to the following test case:

            SELECT a[2:2+3];

Obviously, we know that a simple query like SELECT a; would parse, so it's surely related to the array access. I started by looking up the bison grammar for PostgreSQL's SELECT statement and drilling down into it to find an array accessor symbol; searching for SelectStmt: proved to be a lucky guess to start with <https://github.com/postgres/postgres/blob/e0693faf797f997f45bee8e572e8b4288cc7eaeb/src/backend/parser/gram.y#L12497-L12504>:

            SelectStmt: select_no_parens            %prec UMINUS
                        | select_with_parens        %prec UMINUS
                    ;

Drilling down into the grammar via SelectStmt --> select_no_parens --> simple_select --> target_list --> target_el show that we are dealing with an a_expr, which is the main symbol widely used to represent an expression throughout the grammar. SQLFluff implements that as ExpressionSegment (and more specifically Expression_A_Grammar). Looking further: target_el --> a_expr --> c_expr --> columnref. Which brings us to a key rule:

            columnref:  <snip>
                        | ColId indirection
                            {
                                $$ = makeColumnRef($1, $2, @1, yyscanner);
                            }

Digging into indirection, we finally find where the array accessor is happening:

            indirection:
                        indirection_el                      { $$ = list_make1($1); }
                        | indirection indirection_el        { $$ = lappend($1, $2); }
                    ;
            indirection_el: <snip>
                        | '[' a_expr ']'
                            {
                                A_Indices *ai = makeNode(A_Indices);

                                ai->is_slice = false;
                                ai->lidx = NULL;
                                ai->uidx = $2;
                                $$ = (Node *) ai;
                            }
                        | '[' opt_slice_bound ':' opt_slice_bound ']'
                            {
                                A_Indices *ai = makeNode(A_Indices);

                                ai->is_slice = true;
                                ai->lidx = $2;
                                ai->uidx = $4;
                                $$ = (Node *) ai;
                            }
                    ;
            opt_slice_bound:
                        a_expr                              { $$ = $1; }
                        | /*EMPTY*/                         { $$ = NULL; }
                    ;

From this we observe:

    •   There is a sequence of indirection elements.

    •   There can be a simple array index provided, which is an expression.

    •   Most importantly, and most immediate to our problem, is the observation that each slice bound is optional, and if it is present, then it is an expression.

Now that we looked up the relevant PG grammar, we can dig into the corresponding SQLFluff grammar in a similar top-down way: postgres.SelectStatementSegment --> we see it's mostly a copy of the ANSI select statement, so --> ansi.SelectStatementSegment --> remember Ref always picks the dialect-specific grammar first --> postgres.SelectClauseSegment --> ansi.SelectClauseSegment.parse_grammar --> postgres.SelectClauseSegmentGrammar --> ansi.SelectClauseElementSegment --> ansi.BaseExpressionElementGrammar --> ansi.ExpressionSegment --> ansi.Expression_A_Grammar --> ansi.Expression_C_Grammar --> ansi.Expression_D_Grammar --> notice this at the end of the sequence --> postgres.Accessor_Grammar --> postgres.ArrayAccessorSegment. As you navigate, always remember to check for dialect-specific grammar before falling back to the inherited grammar (e.g. ANSI). Finally, we have found the part of the grammar that corresponds to the indirection_el in the bison grammar!

            class ArrayAccessorSegment(ansi.ArrayAccessorSegment):
                """Overwrites Array Accessor in ANSI to allow n many consecutive brackets.

                Postgres can also have array access like python [:2] or [2:] so
                numbers on either side of the slice segment are optional.
                """

                match_grammar = Sequence(
                    AnyNumberOf(
                        Bracketed(
                            Sequence(
                                OneOf(
                                    OneOf(
                                        Ref("QualifiedNumericLiteralSegment"),
                                        Ref("NumericLiteralSegment"),
                                    ),
                                    Sequence(
                                        OneOf(
                                            Ref("QualifiedNumericLiteralSegment"),
                                            Ref("NumericLiteralSegment"),
                                            optional=True,
                                        ),
                                        Ref("SliceSegment"),
                                        OneOf(
                                            Ref("QualifiedNumericLiteralSegment"),
                                            Ref("NumericLiteralSegment"),
                                        ),
                                    ),
                                    Sequence(
                                        OneOf(
                                            Ref("QualifiedNumericLiteralSegment"),
                                            Ref("NumericLiteralSegment"),
                                        ),
                                        Ref("SliceSegment"),
                                        OneOf(
                                            Ref("QualifiedNumericLiteralSegment"),
                                            Ref("NumericLiteralSegment"),
                                            optional=True,
                                        ),
                                    ),
                                ),
                            ),
                            bracket_type="square",
                        )
                    )
                )

Observing this, we can make a few observations. The most glaring are that:

    •   Only numeric literals are accepted! No expressions. Clearly, that's the source of the issue that the person reported.

    •   But while we are here, notice another problem we can fix: when a SliceSegment (a ) is present, you're forced to include a numeric literal either before or after the SliceSegment. You can't have [:], even though that's valid SQL that PG can parse.

At this point, it's a simple matter of simplifying & rewriting the grammar to fix these shortcomings and better align it with the bison grammar, which was done in pull request #4748 <https://github.com/sqlfluff/sqlfluff/pull/4748>.

  Testing your changes

So you've made your fix, you've tested it fixed the original problem so just submit that change, and all is good now?

Well, no. You want to do two further things:

    •   Test your change hasn't broken anything else. To do that you run the test suite.

    •   Add a test case, so others can check this in future.

To test your changes you'll need to have your environment set up (again see the CONTRIBUTING.md <https://github.com/sqlfluff/sqlfluff/blob/main/CONTRIBUTING.md> file for how to do that).

  Adding test cases for your changes

Adding a test case is simple. Just add a SQL file to test/fixtures/dialects/ <https://github.com/sqlfluff/sqlfluff/tree/main/test/fixtures/dialects> in the appropriate dialect directory. You can either expand an existing SQL file test case (e.g. if adding something similar to what's in there) or create a new one.

I advise adding the original SQL raised in the issue, and if you have examples from the official syntax, then they are always good test cases to add as well. For example, the Snowflake documentation has an example section <https://docs.snowflake.com/en/sql-reference/sql/select.html#examples> at the bottom of every syntax definition so just copy all them into your example file too.

You should also use the reference grammar to exhaustively test various pedantic combinations of syntax. It doesn't have to be runnable. It just needs to parse correctly into the right structure, and be a statement that can get past the parsing stage of the database engine. The documentation often includes more simple examples that might not reflect all the real-world possibilities. While referring to the reference documentation / bison grammar, try to come up with a statement that uses as much of the grammar as it can!

Be sure that you verify that the SQL statements in your test are actually parsable by the database engine! An easy way to do that is often to copy/paste the statement into the console and try running it, or use a CLI parsing tool that uses the same source code as the database engine (e.g. pgsql-parser). An error is ok (e.g. invalid column name), as long as it's not a syntax error from parsing. Check the reference section at the top of this document for dialect-specific resources.

  YML test fixture files

In addition to the SQL files, we have auto-generated YAML counterparts for them. The YAML contains the parsed version of the SQL, and having these in our source code, allows us to easily see if they change, so if someone redefines a syntax, which changes how a SQL statement is parsed, then the SQL won't change but the parse tree does, so by having that in our source code, and so checking that in with any pull request, we can spot that and make sure we're comfortable the change is expected. For most cases (except adding new test cases obviously!) you would not expect unrelated YML files to change so this is a good check.

To regenerate all the YAML files when you add or edit any test fixture SQL files run the following command:

            tox -e generate-fixture-yml

You can also do the following to only generate for a particular dialect, or only for new and changed files, which is often quicker:

            tox -e generate-fixture-yml -- --dialect postgres
            tox -e generate-fixture-yml -- --new-only

It takes a few mins to run, and regenerates all the YAML files. You can then do a git status to see any differences.

When making changes, make sure to check the post-parse structure from the test output or from the associated YAML file: check that each query element is typed correctly. Typical bugs can be that a standalone keyword (such as INTERVAL) is parsed as a function name, or that an element that should be date_part is parsed as an identifier. Typically there is no need to write assertions by hand, but it's the developer's responsibility to verify the structure from auto-generated YAML. One should not assume that everything is working just because no parsing error is raised.

  Running the test suite

For the basic setup, see the local testing section of the CONTRIBUTING.md <https://github.com/sqlfluff/sqlfluff/blob/main/CONTRIBUTING.md> file first.

There's a few ways of running the test suite. You could just run the tox command, but this will run all the test suites, for various python versions, and with and without dbt, and take a long time. Best to leave that to our CI infrastructure. You just want to run what you need to have reasonable confidence before submitting.

  Testing a single fixture

The dialects_test is parametrized to automatically pick all files under test/fixtures/dialects/.

For example if you're adding or modifying dialects/hive/select_interval.sql, you can test that with:

            tox -e py39 -- -s test/dialects/dialects_test.py -k hive-select_interval.sql

The -s flag for pytest enables printing of post-parse structure, which allows you to quickly check that each query element is typed correctly. Same can be seen in the generated fixture YAML file.

To run it a bit faster, you can invoke pytest directly (requires that you have activated the project venv):

            pytest -s test/dialects/dialects_test.py -k hive-select_interval.sql

  Running all dialect tests

The following command runs just the dialect tests, for all dialects:

            tox -e py39 -- test/dialects/dialects_test.py

The following command runs just the dialect tests, for a specific dialect:

            tox -e py39 -- test/dialects/dialects_test.py -k ansi

Or, if making a dialect change to fix a rule that is incorrectly flagging, you can just run the tests for that one rule, for example to run the LT01 <#rule-LT01> tests:

            tox -e py39 -- -k LT01 test

  Final checks before committing

For formatting and linting it's usually enough to rely on the pre-commit hook <https://github.com/sqlfluff/sqlfluff/blob/main/CONTRIBUTING.md#pre-commit-config>.

Run all tests (but only on one Python version, and without dbt):

            tox -e py311

I like to kick that off just before opening a PR but does take ~10 minutes to run.

If you want also coverage & linting, run this instead (takes even more time):

            tox -e generate-fixture-yml,cov-init,py311,cov-report,linting

Also it should be noted that the coverage tests require several versions to run (windows, and dbt) so can report missing coverage when run locally.

The rest can be left for the CI to check.

Regardless of what testing you do, GitHub will run the full regression suite when the PR is opened or updated. Note first time contributors will need a maintainer to kick off the tests until their first PR is merged.

  Black code linting

These tools are run automatically by the pre-commit hook <https://github.com/sqlfluff/sqlfluff/blob/main/CONTRIBUTING.md#pre-commit-config>, but can also be run manually for those not using that.

We use ruff <https://docs.astral.sh/ruff/> to lint our python code (being a linter ourselves we should have high quality code!). Our CI, or the tox commands above will run this and flag any errors.

In most cases running black <https://github.com/psf/black> on the python file(s) will correct any simple errors (e.g. line formatting) but for some you'll need to run ruff to see the issues and manually correct them.

  Submitting your change

We use the standard GitHub workflow so simply fork the repo, clone it locally, make the change, push it to your fork, then open a pull request back to the original SQLFluff repo. There’s lots more info in our How to use Git <#using-git> guide if you're new to Git.

Once you open the PR CI tests will run, and after 5-10mins should complete. If all green, then a maintainer will pick it up as soon as they can. Have a good, easy to understand, small PR with all the tests passing, makes it easier to review so more likely to be merged quickly.

  Questions

Feel free to open up any issues on GitHub, or join the SQLFluff Slack <#sqlfluff-slack> for any quick questions to the community/maintainers.

  Developing Rules

Rules in SQLFluff are implemented as classes inheriting from BaseRule. SQLFluff crawls through the parse tree of a SQL file, calling the rule's _eval() function for each segment in the tree. For many rules, this allows the rule code to be really streamlined and only contain the logic for the rule itself, with all the other mechanics abstracted away.

  Running Tests

The majority of the test cases for most bundled rules are "yaml test cases", i.e. test cases defined in yaml <https://yaml.org/> files. You can find those yaml fixtures on github <https://github.com/sqlfluff/sqlfluff/tree/main/test/fixtures/rules/std_rule_cases>. While this provides a very simple way to write tests, it can be occasionally tedious to run specific tests.

Within either a tox environment or virtualenv (as described in the contributing.md <https://github.com/sqlfluff/sqlfluff/blob/main/CONTRIBUTING.md> file), you can either run all of the rule yaml tests with:

            pytest test/rules/yaml_test_cases_test.py -vv

...or to just run tests for a specific rule, there are two options for a syntax to select only those tests:

            pytest -vv test/rules/ -k RF01

The -k option simply searches for the content of the argument being in the name of the test, which will match any single or combo tests for that rule. By convention, any test cases for a rule should include the code for that rule.

  Traversal Options

  recurse_into

Some rules are a poor fit for the simple traversal pattern described above. Typical reasons include:

    •   The rule only looks at a small portion of the file (e.g. the beginning or end).

    •   The rule needs to traverse the parse tree in a non-standard way.

These rules can override BaseRule's recurse_into field, setting it to False. For these rules False, _eval() is only called once, with the root segment of the tree. This can be much more efficient, especially on large files. For example, see rules LT13 and LT12 , which only look at the beginning or end of the file, respectively.

  _works_on_unparsable

By default, SQLFluff calls _eval() for all segments, even "unparsable" segments, i.e. segments that didn't match the parsing rules in the dialect. This causes issues for some rules. If so, setting _works_on_unparsable to False tells SQLFluff not to call _eval() for unparsable segments and their descendants.

  Performance-related Options

These are other fields on BaseRule. Rules can override them.

  needs_raw_stack

needs_raw_stack defaults to False. Some rules use RuleContext.raw_stack property to access earlier segments in the traversal. This can be useful, but it adds significant overhead to the linting process. For this reason, it is disabled by default.

  lint_phase

There are two phases of rule running.

1. The main phase is appropriate for most rules. These rules are assumed to interact and potentially cause a cascade of fixes requiring multiple passes. These rules run the runaway_limit number of times (default 10).

2. The post phase is for post-processing rules, not expected to trigger any downstream rules, e.g. capitalization fixes. They are run in a post-processing loop at the end. This loop is identical to the main loop, but is only run 2 times at the end (once to fix, and once again to confirm no remaining issues).

The two phases add complexity, but they also improve performance by allowing SQLFluff to run fewer rules during the main phase, which often runs several times.

NOTE: post rules also run on the first pass of the main phase so that any issues they find will be presented in the list of issues output by sqlfluff fix and sqlfluff lint.

  Developing Plugins

SQLFluff is extensible through "plugins". We use the pluggy library <https://pluggy.readthedocs.io/en/latest/> to make linting Rules pluggable, which enable users to implement rules that are just too "organization specific" to be shared, or too platform specific to be included in the core library.

Note:
        We recommend that the module in a plugin which defines all of the hook implementations (anything using the @hookimpl decorator) must be able to fully import before any rule implementations are imported. More specifically, SQLFluff must be able to both import and run any implementations of get_configs_info() before any plugin rules (i.e. any derivatives of BaseRule <#sqlfluff.core.rules.base.BaseRule>) are imported. Because of this, we recommend that rules are defined in a separate module to the root of the plugin and then only imported within the get_rules() method.

        Importing in the main body of the module was previously our recommendation and so may be the case for versions of some plugins. If one of your plugins does use imports in this way, a warning will be presented, recommending that you update your plugin.

                     # The root module will need to import `hookimpl`, but
                     # should not yet import the rule definitions for the plugin.
                     from sqlfluff.core.plugin import hookimpl

                     @hookimpl
                     def get_rules():
                         # Rules should be imported within the `get_rules` method instead
                         from my_plugin.rules import MyRule
                         return [MyRule]

  Creating a plugin

We have an example plugin in sqlfluff/plugins/sqlfluff-plugin-example <https://github.com/sqlfluff/sqlfluff/tree/main/plugins/sqlfluff-plugin-example> which you can use as a template for rules, or the sqlfluff/plugins/sqlfluff-templater-dbt <https://github.com/sqlfluff/sqlfluff/tree/main/plugins/sqlfluff-templater-dbt> which you can use as a template for templater plugins.

  Few things to note about plugins:

Currently, only Rules and Templaters can be added through plugins. Over time we expect more elements of SQLFluff will be extensible with plugins. Each plugin can implement multiple Rules or Templaters.

We recommend that the name of a plugin should start with "sqlfluff-" to be clear on the purpose of your plugin.

A plugin may need to include a default configuration if its rules are configurable: use plugin default configurations only for that reason! We advise against overwriting core configurations by using a default plugin configuration, as there is no mechanism in place to enforce precedence between the core library configs and plugin configs, and multiple plugins could clash.

A plugin Rule class name should have the structure: "Rule_PluginName_L000". The 'L' can be any letter and is meant to categorize rules; you could use the letter 'S' to denote rules that enforce security checks for example.

An important thing to note when running custom implemented rules: Run pip install -e ., inside the plugin folder so custom rules in linting are included.

A plugin Rule code includes the PluginName, so a rule "Rule_L000" in core will have code "L000", while "Rule_PluginName_L000" will have code "PluginName_L000". Codes are used to display errors, they are also used as configuration keys.

We make it easy for plugin developers to test their rules by exposing a testing library in sqlfluff.utils.testing.

  Giving feedback

Would you like to have other parts of SQLFluff be "pluggable"? Tell us about it in a GitHub issue <https://github.com/sqlfluff/sqlfluff/issues/new?assignees=&labels=enhancement&template=enhancement.md> 😄.

  Documentation Contributions

Contributing to the docs is one of the easiest and most helpful ways to help the project. Documentation changes require relatively little specialist knowledge apart from being familiar with how to use SQLFluff and the docs are read by a very wide range of people.

Documentation takes two forms:

    1.  Embedded documentation found in function and module docstrings <https://en.wikipedia.org/wiki/Docstring>.

    2.  The free-standing documentation which you're reading now, and hosted at docs.sqlfluff.com <https://docs.sqlfluff.com> (built using sphinx <https://www.sphinx-doc.org/en/master/> and ReadtheDocs <https://about.readthedocs.com/>).

The two are somewhat blurred by the use of autodoc <https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html> (and some other custom integrations), where documentation is generated directly off docstrings <https://en.wikipedia.org/wiki/Docstring> within the codebase, for example the Rules Reference <#ruleref>, CLI Reference <#cliref> and Dialects Reference <#dialectref>. To understand more about how the custom integrations we use to generate these docs, see the generate-auto-docs.py <https://github.com/sqlfluff/sqlfluff/blob/main/docs/generate-auto-docs.py> file.

  Docstrings

Embedded documentation of functions, classes and modules is most useful for developer-focussed documentation as it's most accessible in the places which those developers are working: directly in the codebase. We enforce that docstrings are present and correctly formatted using the pydocstyle rules for ruff <https://docs.astral.sh/ruff/rules/#pydocstyle-d>, which we have configured to enforce the google style of docstrings <https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html>.

  Sphinx Docs

The main documentation (which you're reading now), is build using sphinx <https://www.sphinx-doc.org/en/master/>, and written using reStructuredText <https://www.sphinx-doc.org/en/master/usage/restructuredtext/index.html> (files ending with .rst). The sphinx <https://www.sphinx-doc.org/en/master/> project offers a reStructuredText primer <https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html> for people who are new to the syntax (and the SQLFluff project uses doc8 <https://github.com/PyCQA/doc8> in the CI process to try and catch any issues early).

On top of those docs, there are a few areas worth highlighting for new (or returning) users, which are either specific to the SQLFluff project, or not particularly clear in the sphinx docs:

    •   reStructuredText <https://www.sphinx-doc.org/en/master/usage/restructuredtext/index.html> is very similar to, but differs from (the somewhat more well known) Markdown <https://www.markdownguide.org/> syntax. Importantly:

        •   *text with single asterisks* renders as italics. Use **double asterisks** for bold text.

        •   code snippets are created using the directive, rather than just lone backticks () as found in most Markdown <https://www.markdownguide.org/>.

    •   To create links to other parts of the documentation (i.e. Cross-referencing <https://www.sphinx-doc.org/en/master/usage/referencing.html>), use either the :ref: syntax.

        •   Docs for all the SQL dialects are auto generated with associated anchors to use for referencing. For example to link to the PostgreSQL <#postgres-dialect-ref> dialect docs, you can use the . Replace the postgres portion with the name of the dialect you want to link to.

        •   Docs for all the bundled rules and handled using a customer sphinx <https://www.sphinx-doc.org/en/master/> plugin, which means you can refer to them using their name or code:
             resolves to LT01 <#rule-LT01> and resolves to layout.spacing <#rule-layout.spacing>.

        •   Docs for any of the python classes and modules handled using autodoc <https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html> can be referenced as per their docs, so the sqlfluff.core.rules.base.BaseRule <#sqlfluff.core.rules.base.BaseRule> class can be referenced with . You can also use the ~ prefix (i.e. ) so that it just renders as BaseRule <#sqlfluff.core.rules.base.BaseRule>. See the docs for Cross-referencing <https://www.sphinx-doc.org/en/master/usage/referencing.html> for more details.

  Configuration

  Setting Configuration

SQLFluff accepts configuration either through the command line or through configuration files. There is rough parity between the two approaches with the exception that templating configuration must be done via a file, because it otherwise gets slightly complicated.

For details of what's available on the command line check out the CLI Reference <#cliref>.

  Configuration Files

For file based configuration SQLFluff will look for the following files in order. Later files will (if found) will be used to overwrite any values read from earlier files.

    •   setup.cfg

    •   tox.ini

    •   pep8.ini

    •   .sqlfluff

    •   pyproject.toml

Within these files, the first four will be read like a cfg file <https://docs.python.org/3/library/configparser.html>, and SQLFluff will look for sections which start with sqlfluff, and where subsections are delimited by a colon. For example the jinjacontext section will be indicated in the section started with [sqlfluff:jinjacontext].

For example, a snippet from a .sqlfluff file (as well as any of the supported cfg file types):

            [sqlfluff]
            templater = jinja
            sql_file_exts = .sql,.sql.j2,.dml,.ddl

            [sqlfluff:indentation]
            indented_joins = False
            indented_using_on = True
            template_blocks_indent = False

            [sqlfluff:templater]
            unwrap_wrapped_queries = True

            [sqlfluff:templater:jinja]
            apply_dbt_builtins = True

For the pyproject.toml file <https://www.python.org/dev/peps/pep-0518/>, all valid sections start with tool.sqlfluff and subsections are delimited by a dot. For example the jinjacontext section will be indicated in the section started with [tool.sqlfluff.jinjacontext].

For example, a snippet from a pyproject.toml file:

            [tool.sqlfluff.core]
            templater = "jinja"
            sql_file_exts = ".sql,.sql.j2,.dml,.ddl"

            [tool.sqlfluff.indentation]
            indented_joins = false
            indented_using_on = true
            template_blocks_indent = false

            [tool.sqlfluff.templater]
            unwrap_wrapped_queries = true

            [tool.sqlfluff.templater.jinja]
            apply_dbt_builtins = true

            # For rule specific configuration, use dots between the names exactly
            # as you would in .sqlfluff. In the background, SQLFluff will unpack the
            # configuration paths accordingly.
            [tool.sqlfluff.rules.capitalisation.keywords]
            capitalisation_policy = "upper"

  New Project Configuration

When setting up a new project with SQLFluff, we recommend keeping your configuration file fairly minimal. The config file should act as a form of documentation for your team i.e. a record of what decisions you've made which govern how your format your SQL. By having a more concise config file, and only defining config settings where they differ from the defaults - you are more clearly stating to your team what choices you've made.

However, there are also a few places where the default configuration is designed more for existing projects, rather than fresh projects, and so there is an opportunity to be a little stricter than you might otherwise be with an existing codebase.

Here is a simple configuration file which would be suitable for a starter project:

            [sqlfluff]

            # Supported dialects https://docs.sqlfluff.com/en/stable/perma/dialects.html
            # Or run 'sqlfluff dialects'
            dialect = snowflake

            # One of [raw|jinja|python|placeholder]
            templater = jinja

            # Comma separated list of rules to exclude, or None
            # See https://docs.sqlfluff.com/en/stable/perma/rule_disabling.html
            # AM04 (ambiguous.column_count) and ST06 (structure.column_order) are
            # two of the more controversial rules included to illustrate usage.
            exclude_rules = ambiguous.column_count, structure.column_order

            # The standard max_line_length is 80 in line with the convention of
            # other tools and several style guides. Many projects however prefer
            # something a little longer.
            # Set to zero or negative to disable checks.
            max_line_length = 120

            # CPU processes to use while linting.
            # The default is "single threaded" to allow easy debugging, but this
            # is often undesirable at scale.
            # If positive, just implies number of processes.
            # If negative or zero, implies number_of_cpus - specified_number.
            # e.g. -1 means use all processors but one. 0 means all cpus.
            processes = -1

            # If using the dbt templater, we recommend setting the project dir.
            [sqlfluff:templater:dbt]
            project_dir = ./

            [sqlfluff:indentation]
            # While implicit indents are not enabled by default. Many of the
            # SQLFluff maintainers do use them in their projects.
            allow_implicit_indents = True

            [sqlfluff:rules:aliasing.length]
            min_alias_length = 3

            # The default configuration for capitalisation rules is "consistent"
            # which will auto-detect the setting from the rest of the file. This
            # is less desirable in a new project and you may find this (slightly
            # more strict) setting more useful.
            # Typically we find users rely on syntax highlighting rather than
            # capitalisation to distinguish between keywords and identifiers.
            # Clearly, if your organisation has already settled on uppercase
            # formatting for any of these syntax elements then set them to "upper".
            # See https://stackoverflow.com/questions/608196/why-should-i-capitalize-my-sql-keywords-is-there-a-good-reason
            [sqlfluff:rules:capitalisation.keywords]
            capitalisation_policy = lower
            [sqlfluff:rules:capitalisation.identifiers]
            extended_capitalisation_policy = lower
            [sqlfluff:rules:capitalisation.functions]
            extended_capitalisation_policy = lower
            [sqlfluff:rules:capitalisation.literals]
            capitalisation_policy = lower
            [sqlfluff:rules:capitalisation.types]
            extended_capitalisation_policy = lower

            # The default configuration for the not equal convention rule is "consistent"
            # which will auto-detect the setting from the rest of the file. This
            # is less desirable in a new project and you may find this (slightly
            # more strict) setting more useful.
            [sqlfluff:rules:convention.not_equal]
            # Default to preferring the "c_style" (i.e. `!=`)
            preferred_not_equal_style = c_style

  Nesting

SQLFluff uses nesting in its configuration files, with files closer overriding (or patching, if you will) values from other files. That means you'll end up with a final config which will be a patchwork of all the values from the config files loaded up to that path. The exception to this is the value for templater, which cannot be set in config files in subdirectories of the working directory. You don't need any config files to be present to make SQLFluff work. If you do want to override any values though SQLFluff will use files in the following locations in order, with values from later steps overriding those from earlier:

    0.  [...and this one doesn't really count] There's a default config as part of the SQLFluff package. You can find this below, in the Default Configuration <#defaultconfig> section.

    1.  It will look in the user's os-specific app config directory. On macOS and Unix this is ~/.config/sqlfluff, Windows is <home>\AppData\Local\sqlfluff\sqlfluff, for any of the filenames above in the main Setting Configuration section. If multiple are present, they will patch/override each other in the order above.

    2.  It will look for the same files in the user's home directory (~).

    3.  [if the current working directory is a subdirectory of the user's home directory (~)] It will look for the same files in all directories between the user's home directory (~), and the current working directory.

    4.  It will look for the same files in the current working directory.

    5.  [if parsing a file in a subdirectory of the current working directory] It will look for the same files in every subdirectory between the current working dir and the file directory.

    6.  It will look for the same files in the directory containing the file being linted.

This whole structure leads to efficient configuration, in particular in projects which utilise a lot of complicated templating.

  In-File Configuration Directives

In addition to configuration files mentioned above, SQLFluff also supports comment based configuration switching in files. This allows specific SQL file to modify a default configuration if they have specific needs.

When used, these apply to the whole file, and are parsed from the file in an initial step before the rest of the file is properly parsed. This means they can be used for both rule configuration and also for parsing configuration.

To use these, the syntax must start as an inline sql comment beginning with sqlfluff (i.e. -- sqlfluff). The line is then interpreted as a colon-separated address of the configuration value you wish to set. A few common examples are shown below:

            -- Set Indented Joins
            -- sqlfluff:indentation:indented_joins:True

            -- Set a smaller indent for this file
            -- sqlfluff:indentation:tab_space_size:2

            -- Set keywords to be capitalised
            -- sqlfluff:rules:capitalisation.keywords:capitalisation_policy:upper

            SELECT *
            FROM a
              JOIN b USING(c)

We recommend only using this configuration approach for configuration that applies to one file in isolation. For configuration changes for areas of a project or for whole projects we recommend Nesting of configuration files.

This syntax is very similar to the method for Ignoring individual lines <#inline-ignoring-errors>.

  Rule Configuration

Rules can be configured with the .sqlfluff config files.

Common rule configurations can be set in the [sqlfluff:rules] section.

For example:

            [sqlfluff:rules]
            allow_scalar = True
            single_table_references = consistent
            unquoted_identifiers_policy = all

Rule specific configurations are set in rule specific subsections.

For example, enforce that keywords are upper case by configuring the rule CP01 <#rule-CP01>:

            [sqlfluff:rules:capitalisation.keywords]
            # Keywords
            capitalisation_policy = upper

All possible options for rule sections are documented in Rules Reference <#ruleref>.

For an overview of the most common rule configurations that you may want to tweak, see Default Configuration <#defaultconfig> (and use Rules Reference <#ruleref> to find the available alternatives).

  Enabling and Disabling Rules

The decision as to which rules are applied to a given file is applied on a file by file basis, by the effective configuration for that file. There are two configuration values which you can use to set this:

    •   rules, which explicitly enables the specified rules. If this parameter is unset or empty for a file, this implies "no selection" and so "all rules" is taken to be the meaning.

    •   exclude_rules, which explicitly disables the specified rules. This parameter is applied after the rules parameter so can be used to subtract from the otherwise enabled set.

Each of these two configuration values accept a comma separated list of references. Each of those references can be:

    •   a rule code e.g. LN01

    •   a rule name e.g. layout.indent

    •   a rule alias, which is often a deprecated code e.g. L003

    •   a rule group e.g. layout or capitalisation

These different references can be mixed within a given expression, which results in a very powerful syntax for selecting exactly which rules are active for a given file.

Note:
        It's worth mentioning here that the application of rules and exclude_rules, with groups, aliases and names, in projects with potentially multiple nested configuration files defining different rules for different areas of a project can get very confusing very fast. While this flexibility is intended for users to take advantage of, we do have some recommendations about how to do this is a way that remains manageable.

        When considering configuration inheritance, each of rules and exclude_rules will totally overwrite any values in parent config files if they are set in a child file. While the subtraction operation between both of them is calculated "per file", there is no combination operation between two definitions of rules (just one overwrites the other).

        The effect of this is that we recommend one of two approaches:

            1.  Simply only use rules. This has the upshot of each area of your project being very explicit in which rules are enabled. When that changes for part of your project you just reset the whole list of applicable rules for that part of the project.

            2.  Set a single rules value in your master project config file and then only use exclude_rules in sub-configuration files to turn off specific rules for parts of the project where those rules are inappropriate. This keeps the simplicity of only having one value which is inherited, but allows slightly easier and simpler rollout of new rules because we manage by exception.

For example, to disable the rules LT08 <#rule-LT08> and RF02 <#rule-RF02>:

            [sqlfluff]
            exclude_rules = LT08, RF02

To enable individual rules, configure rules, respectively.

For example, to enable RF02 <#rule-RF02>:

            [sqlfluff]
            rules = RF02

Rules can also be enabled/disabled by their grouping. Right now, the only rule grouping is core. This will enable (or disable) a select group of rules that have been deemed 'core rules'.

            [sqlfluff]
            rules = core

More information about 'core rules' can be found in the Rules Reference <#ruleref>.

Additionally, some rules have a special force_enable configuration option, which allows to enable the given rule even for dialects where it is disabled by default. The rules that support this can be found in the Rules Reference <#ruleref>.

The default values can be seen in Default Configuration <#defaultconfig>.

See Ignoring Errors & Files <#ignoreconfig> for more information on how to turn ignore particular rules for specific lines, sections or files.

  Downgrading rules to warnings

To keep displaying violations for specific rules, but not have those issues lead to a failed run, rules can be downgraded to warnings. Rules set as warnings won't cause a file to fail, but will still be shown in the CLI to warn users of their presence.

The configuration of this behaves very like exclude_rules above:

            [sqlfluff]
            warnings = LT01, LT04

With this configuration, files with no other issues (other than those set to warn) will pass. If there are still other issues, then the file will still fail, but will show both warnings and failures.

            == [test.sql] PASS
            L:   2 | P:   9 | LT01 | WARNING: Missing whitespace before +
            == [test2.sql] FAIL
            L:   2 | P:   8 | CP02 | Unquoted identifiers must be consistently upper case.
            L:   2 | P:  11 | LT01 | WARNING: Missing whitespace before +

This is particularly useful as a transitional tool when considering the introduction on new rules on a project where you might want to make users aware of issues without blocking their workflow (yet).

You can use either rule code or rule name for this setting.

  Layout & Spacing Configuration

The [sqlfluff:layout] section of the config controls the treatment of spacing and line breaks across all rules. To understand more about this section, see the section of the docs dedicated to layout: Configuring Layout <#layoutconfig>.

  Layout & Whitespace Configuration

If there is one part of building a linter that is going to be controversial it's going to be whitespace (closely followed by cApiTaLiSaTiOn 😁).

More specifically, whitespace divides into three key themes:

    1.  Spacing: The amount of whitespace between elements on the same line.

    2.  Line Breaks: The choice of where within the code it is inappropriate, appropriate or even compulsory to have a line break.

    3.  Indentation: Given a line break, how much whitespace should precede the first code element on that line.

SQLFluff aims to be opinionated on this theme, but also configurable (see Configuring Layout). The tool will have a default viewpoint and will aim to have views on all of the important aspects of SQL layout, but if you (or your organisation) don't like those views then we aim to allow enough configuration that you can lint in line with your views, and still use SQLFluff. For more information on how to configure rules to your own viewpoint see Configuration <#config>.

Note:
        This section of the docs handles the intent and reasoning behind how layout is handled by SQLFluff. For a deeper look at how this is achieved internally see sqlfluff.utils.reflow: Whitespace Reflow API <#reflowinternals>.

  Spacing

Of the different elements of whitespace, spacing is likely the least controversial. By default, all elements are separated by a single space character. Except for very specific circumstances (see section on Aligned elements), any additional space between elements is usually unwanted and a distraction for the reader. There are however several common cases where no whitespace is more appropriate, which fall into two cases (for more details on where to configure these see Configuring layout and spacing).

    1.  No whitespace but a newline is allowed. This option is configured using the touch option in the spacing_* configuration settings. The most common example of this is the spacing around commas. For example SELECT a , b would be unusual and more normally be written SELECT a, b. Inserting a newline between the a and comma would not cause issues and may even be desired, for example:

                SELECT
                   col_a
                   , col_b
                   -- Newline present before column
                   , col_c
                   -- When inline, comma should still touch element before.
                   , GREATEST(col_d, col_e) as col_f
                FROM tbl_a

    2.  No whitespace and a newline is not allowed. This option is configured using the inline option in the spacing_* configuration settings. The most common example of this is spacing within the parts of qualified identifier e.g. my_schema.my_table. If a newline were present between the . and either my_schema or my_table, then the expression would not parse and so no newlines should be allowed.

  Aligned elements

A special case of spacing is where elements are set to be aligned within some limits. This is not enabled by default, but can be be configured to achieve layouts like:

            SELECT
               a           AS first_column,
               b           AS second_column,
               (a + b) / 2 AS third_column
            FROM foo AS bar

In this example, the alias expressions are all aligned with each other. To configure this, SQLFluff needs to know what elements to align and how far to search to find elements which should be aligned with each other. The configuration to achieve this layout is:

            [sqlfluff:layout:type:alias_expression]
            # We want non-default spacing _before_ the alias expressions.
            spacing_before = align
            # We want to align them within the next outer select clause.
            # This means for example that alias expressions within the FROM
            # or JOIN clause would _not_ be aligned with them.
            align_within = select_clause
            # The point at which to stop searching outward for siblings, which
            # in this example would likely be the boundary of a CTE. Stopping
            # when we hit brackets is usually a good rule of thumb for this
            # configuration.
            align_scope = bracketed

Of these configuration values, the align_scope is potentially the least obvious. The following example illustrates the impact it has.

            -- With
            --    align_scope = bracketed
            --    align_within = select_clause

            WITH foo as (
               SELECT
                  a,
                  b,
                  c     AS first_column
                  d + e AS second_column
            )

            SELECT
               a           AS first_column,
               (a + b) / 2 AS third_column
            FROM foo AS bar;

            -- With
            --    align_scope = bracketed
            --    align_within = statement

            WITH foo as (
               SELECT
                  a,
                  b,
                  c     AS first_column
                  d + e AS second_column
            )

            SELECT
               a           AS first_column,
               (a + b) / 2 AS third_column
            FROM foo       AS bar            -- Now the FROM alias is also aligned.

            -- With
            --    align_scope = file
            --    align_within = select_clause

            WITH foo as (
               SELECT
                  a,
                  b,
                  c        AS first_column   -- Now the aliases here are aligned
                  d + e    AS second_column  -- with the outer query.
            )

            SELECT
               a           AS first_column,
               (a + b) / 2 AS third_column
            FROM foo AS bar

            -- With
            --    align_scope = file
            --    align_within = statement

  Templating and alignment coordinate space

When using templating (e.g. Jinja), alignment is for human readability and stable diffs. SQLFluff aligns based on the source (visible) positions whenever templated (non-literal) segments are involved in the alignment scope. This prevents excessive padding caused by the rendered output being longer than the source template. If all segments are literal (non-templated), alignment uses the regular templated working positions.

Example (Jinja templating, align alias expressions within a select clause):

            select
                {{ "longtemplated" }} as test_key,
                b                     as b_col

The alignment above is computed against the source text so that both lines line up visually in the editor, regardless of the rendered length of {{ "longtemplated" }}.

  Advanced: coordinate space override

You can optionally force the coordinate space either:

    1.  via the alignment constraint suffix (available for spacing_before and spacing_after), or

    2.  via the alignment_coordinate_space key in layout config for the target type.

            [sqlfluff:layout:type:alias_expression]
            spacing_before = align:alias_expression:select_clause:bracketed:source

Alternatively, the equivalent can be configured more declaratively:

            [sqlfluff:layout:type:alias_expression]
            spacing_before = align
            align_within = select_clause
            align_scope = bracketed
            alignment_coordinate_space = source

Supported values are source and templated. In most cases, source is the recommended choice for readability.

            WITH foo as (
               SELECT
                  a,
                  b,
                  c        AS first_column,
                  d + e    AS second_column
            )

            SELECT
               a           AS first_column,
               (a + b) / 2 AS third_column
            FROM foo       AS bar

  Line Breaks

When controlling line breaks, we are trying to achieve a few different things:

    1.  Do we have enough line breaks that line length doesn't become excessive. Long lines are hard to read, especially given that readers may be on varying screen sizes or have multiple windows open. This is (of course) configurable, but the default is 80 characters (in line with the dbt Labs SQL style guide <https://github.com/dbt-labs/corp/blob/main/dbt_style_guide.md>.)

    2.  Is the positioning of blank lines (i.e. lines with nothing other than whitespace on them) appropriate. There are some circumstances where a blank line is desired (e.g. between CTEs). There are others where they are not, in particular multiple blank lines, for example at the beginning of a file.

    3.  Where we do have line breaks, are they positioned appropriately and consistently with regards to other elements around them. This is most common when it comes to commas, and whether they should be leading (e.g. , my_column) or trailing (e.g. my_column,). In less common cases, it may also be desirable for some elements to have both a line break before and after (e.g. a set operator such as UNION).

  Indentation

Lastly, given we have multiple lines of SQL, to what extent should we indent some lines to provide visual cues to the structure of that SQL. It's important to note that SQL is not whitespace sensitive in its interpretation and that means that any principles we apply here are entirely for the benefit of humans. Your database doesn't care.

The indentation therefore should be treated as a hint to the reader of the structure of the code. This explains the common practice within most languages that nested elements (for example the contents of a set of brackets in a function call) should be indented one step from the outer elements. It's also convention that elements with the same level in a nested structure should have the same indentation, at least with regards to their local surroundings. As an example:

            SELECT
               nested_within_select AS first_column,
               some_function(
                  nested_within_function,
                  also_nested_within_function
               ) AS indented_the_same_as_opening_bracket
            FROM indented_the_same_as_select

  Comment Indents

Note:
        The notes here about block comments are not implemented prior to 2.0.x. They should be coming in that release or soon after.

Comments are dealt with differently, depending on whether they're block comments (/* like this */), which might optionally include newlines, or inline comments (-- like this) which are necessarily only on one line.

    •   Block comments cannot share a line with any code elements (so in effect they must start on their own new line), they cannot be followed by any code elements on the same line (and so in effect must be followed by a newline, if we are to avoid trailing whitespace). None of the lines within the block comment may have an indent less than the first line of the block comment (although additional indentation within a comment is allowed), and that first line should be aligned with the first code element following the block comment.

                SELECT
                   /* This is a block comment starting on a new line
                   which contains a newline (continuing with at least
                   the same indent.
                      - potentially containing greater indents
                      - having no other code following it in the same line
                      - and aligned with the line of code following it */
                   this_column as what_we_align_the_column_to
                FROM my_table

    •   Inline comments can be on the same line as other code, but are subject to the same line-length restrictions. If they don't fit on the same line (or if it just looks nicer) they can also be the only element on a line. In this latter case, they should be aligned with the first code element following the comment.

                SELECT
                   -- This is fine
                   this_column as what_we_align_to,
                   another_column as something_short,  -- Is ok
                   case
                      -- This is aligned correctly with below
                      when indented then take_care
                      else try_harder
                   end as the_general_guidance
                -- Even here we align with the line below
                FROM my_table

    Note:
            When fixing issues with comment indentation, SQLFluff will attempt to keep comments in their original position but if line length concerns make this difficult, it will either abandon the fix, or move same line comments up and before the line they are currently on. This is in line with the assumption that comments on their own line refer to the elements of code which they come before, not after.

  Hanging Indents

One approach to indenting nested elements is a layout called a hanging indent. In this layout, there is no line break before the first nested element, but subsequent elements are indented to match the line position of that first element. Two examples might be:

            -- A select statement with two hanging indents:
            SELECT no_line_break_before_me,
                   indented_to_match_the_first,
                   1 + (a
                        + b) AS another_more_complex_example
            FROM my_table;

            -- This TSQL example is also in essence a hanging indent:
            DECLARE @prv_qtr_1st_dt DATETIME,
                    @last_qtr INT,
                    @last_qtr_first_mn INT,
                    @last_qtr_yr INT;

In some circumstances this layout can be quite neat (the DECLARE statement is a good example of this), however once indents are nested or indentation styles are mixed it can rapidly become confusing (as partially shown in the first example). Additionally, unless the leading element of the first line is very short, hanging indents use much larger indents than a traditional simple indent where a line break is used before the first element.

Hanging indents have been supported in SQLFluff up to the 1.x versions, however they will no longer by supported from 2.0.0 onwards. This is due to the ambiguity which they bring to fixing poorly formatted SQL. Take the following code:

            SELECT   this_is,
            badly_formatted, code_and,
               not_obvious,
                  what_was,
            intended FROM my_table

Given the lack of line break between SELECT and this_is, it would appear that the user is intending a hanging indent, however it is also plausible that they did not and they just forgot to add a line break between them. This ambiguity is unhelpful, both for SQLFluff as a tool, but also for people who write SQL that there two ways of indenting their SQL. Given SQLFluff aims to provide consistency in SQL layout and remove some of the burden of needing to make choices like this - and that it would be very unusual to keep only hanging indents and disable traditional ones - the only route left to consistency is to not allow hanging indents. Starting in 2.0.0, any hanging indents detected will be converted to traditional indents.

  Implicit Indents

A close cousin of the hanging indent is the implicit indent. While it does look a little like a hanging indent, it's much more consistent in its behaviour and is supported from SQLFluff 2.0.0 onwards.

An implicit indent is exactly like a normal indent, but doesn't have to be actually taken to influence the indentation of lines after it - it just needs to be left un-closed before the end of the line. These are normally available in clauses which take the form of KEYWORD <expression>, like WHERE clauses or CASE expressions.

            -- This WHERE clause here takes advantage of an implicit indent.
            SELECT *
            FROM my_table
            WHERE condition_a
               AND condition_b;

            -- With implicit indents disabled (which is currently the
            -- default), the above formulation is not allowed, and instead
            -- there should be a newline immediately after `WHERE` (which
            -- is the location of the _implicit_ indent).
            SELECT *
            FROM my_table
            WHERE
               condition_a
               AND condition_b;

When addressing both indentation and line-length, implicit indents allow a slightly more compact layout, without significant drawbacks in legibility. They also enable a style much closer to some established style guides.

They are however not recommended by many of the major style guides at time of writing (including the dbt Labs SQL style guide <https://github.com/dbt-labs/corp/blob/main/dbt_style_guide.md> and the Mozilla SQL style guide <https://docs.telemetry.mozilla.org/concepts/sql_style.html#joins>), and so are disabled by default. To enable them, set the allow_implicit_indents flag in sqluff.indentation to True.

  Templated Indents

SQLFluff supports templated elements in code, such as those offered by jinja2 (or dbt which relies on it). For simple cases, templated elements are handled as you would expect by introducing additional indents into the layout.

            SELECT
               a,
               {% for n in ['b', 'c', 'd'] %}
                  -- This section is indented relative to 'a' because
                  -- it is inside a jinja for loop.
                  {{ n }},
               {% endfor %}
               e
            FROM my_table

This functionality can be turned off if you wish using the template_blocks_indent option in your Configuration <#config>.

It's important to note here, that SQLFluff lints the code after it has been rendered, and so only has access to code which is still present after that process.

            SELECT
               a,
               {% if False %}
               -- This section of the code cannot be linted because
               -- it is never rendered due to the `if False` condition.
               my    + poorly
                  +   spaced - and/indented AS    section_of_code
               {% endif %}
               e
            FROM my_table

More complex templated cases are usually characterised by templated tags cutting across the parse tree. This more formally is where the opening and closing tags of a templated section exist at different levels in the parsed structure. Starting in version 2.x, these will be treated differently (Prior to version 2.x, situations like this were sometimes handled inconsistently or incorrectly).

Indentation should act as a visual cue to the structure of the written SQL, and as such, the most important thing is that template tags belonging to the same block structure use the same indentation. In the example below, this is the opening and closing elements of the second if statement. If treated as a simple case, these tags would have different indents, because they are at different levels of the parse tree and so clearly there is a conflict to be resolved.

The view SQLFluff takes on how to resolve this conflict is to pull all of the tags in this section down to the indent of the least indented (in the example below that would be the closing endif tag). This is similar to the treatment of C Preprocessor Directives <https://www.cprogramming.com/reference/preprocessor/>, which are treated somewhat as being outside the structure of the rest of the file. In these cases, the content is also not further indented as in the simple case because it makes it harder to line up elements within the affected section and outside (in the example below the SELECT and FROM are a good illustration).

            SELECT
               a,
               {% if True %}
                  -- This is a simple case. The opening and closing tag are
                  -- both at the same level within the SELECT clause.
                  simple_case AS example,
               {% endif %}
               b,
            {% if True %}
               -- This is a complex case. The opening tag is within the SELECT
               -- clause, but the closing tag is outside the statement
               -- entirely.
               complex_case AS example
            FROM table_option_one
            {% else %}
               complex_case_two AS example
            FROM table_option_two
            {% endif %}

  Configuring Layout

Configuration for layout is spread across three places:

    1.  Indent behavior for particular dialect elements is controlled by the parser. This is because in the background SQLFluff inserts Indent and Dedent tokens into the parse tree where those things are expected. For more detail see Configuring indent locations.

    2.  Configuration for the spacing and line position of particular types of element (such as commas or operators) is set in the layout section of the config file. For more detail see Configuring layout and spacing.

    3.  Some elements of layout are still controlled by rules directly. These are usually very specific cases, see Rules Reference <#ruleref> for more details.

  Configuring indent locations

One of the key areas for this is the indentation of the JOIN expression, which we'll use as an example.

Semantically, a JOIN expression is part of the FROM expression and therefore would be expected to be indented. However according to many of the most common SQL style guides (including the dbt Labs SQL style guide <https://github.com/dbt-labs/corp/blob/main/dbt_style_guide.md> and the Mozilla SQL style guide <https://docs.telemetry.mozilla.org/concepts/sql_style.html#joins>) the JOIN keyword is expected to at the same indent as the FROM keyword. By default, SQLFluff sides with the current consensus, which is to not indent the JOIN keyword, however this is one element which is configurable.

By setting values in the sqlfluff:indentation section of your config file you can control how this is parsed.

For example, the default indentation would be as follows:

            SELECT
               a,
               b
            FROM my_table
            JOIN another_table
               ON
                  condition1
                  AND condition2

By setting your config file to:

            [sqlfluff:indentation]
            indented_joins = True

Then the expected indentation will be:

            SELECT
               a,
               b
            FROM my_table
               JOIN another_table
                  ON
                     condition1
                     AND condition2

There is a similar indented_using_on config (defaulted to True) which can be set to False to prevent the USING or ON clause from being indented, in which case the original SQL would become:

            SELECT
               a,
               b
            FROM my_table
            JOIN another_table
            ON
               condition1
               AND condition2

It's worth noting at this point, that for some users, the additional line break after ON is unexpected, and this is a good example of an implicit indent. By setting your config to:

            [sqlfluff:indentation]
            indented_using_on = False
            allow_implicit_indents = True

Then the expected indentation will be:

            SELECT
               a,
               b
            FROM my_table
            JOIN another_table
            ON condition1
               AND condition2

There is also a similar indented_on_contents config (defaulted to True) which can be set to False to align any AND subsections of an ON block with each other. If set to False (assuming implicit indents are still enabled) the original SQL would become:

            SELECT
               a,
               b
            FROM my_table
            JOIN another_table
               ON condition1
               AND condition2

These can also be combined, so if indented_using_on config is set to False, indented_on_contents is also set to False, and allow_implicit_indents is set to True then the SQL would become:

            SELECT
               a,
               b
            FROM my_table
            JOIN another_table
            ON condition1
            AND condition2

There is also a similar indented_ctes config (defaulted to False) which can be set to True to enforce CTEs to be indented within the WITH clause:

            WITH
               some_cte AS (
                  SELECT 1 FROM table1
               ),

               some_other_cte AS (
                  SELECT 1 FROM table1
               )

            SELECT 1 FROM some_cte

There is also a similar indented_then config (defaulted to True) which can be set to False to allow THEN without an indent after WHEN:

            SELECT
               a,
               CASE
                  WHEN b >= 42 THEN
                     1
                  ELSE 0
               END AS c
            FROM some_table

By default, SQLFluff aims to follow the most common approach to indentation. However, if you have other versions of indentation which are supported by published style guides, then please submit an issue on GitHub to have that variation supported by SQLFluff.

  Configuring layout and spacing

The [sqlfluff:layout] section of the config controls the treatment of spacing and line breaks across all rules. The syntax of this section is very expressive; however in normal use, only very small alterations should be necessary from the Default Configuration <#defaultconfig>.

The syntax of the section headings here select by type, which corresponds to the type defined in the dialect. For example the following section applies to elements of the type comma, i.e. ,.

            [sqlfluff:layout:type:comma]
            spacing_before = touch
            line_position = trailing

Within these configurable sections there are a few key elements which are available:

    •   Spacing Elements: spacing_before, spacing_after and spacing_within. For each of these options, there are a few possible settings:

        •   The default spacing for all elements is single unless otherwise specified. In this state, elements will be spaced with a single space character unless there is a line break between them.

        •   The value of touch allows line breaks, but if no line break is present, then no space should be present. A great example of this is the spacing before commas (as shown in the config above), where line breaks may be allowed, but if not they should touch the element before.

        •   Both of the above can be qualified with the :inline modifier - which prevents newlines within the segment. This is best illustrated by the spacing found in a qualified identifier like my_schema.my_table which uses touch:inline or other clauses where we want to force some elements to be on the same line.

    •   Line Position: set using the line_position option. By default this is unset, which implies no particular line position requirements. The available options are:

        •   trailing and leading, which are most common in the placement of commas. Both of these settings also allow the option of a comma on its own on a line, or in the middle of a line, but if there is a line break on either side then they make sure it's on the correct side. By default we assume trailing commas, but if you (or your organisation) have settled on leading commas then you should add the following section to your config:

                    [sqlfluff:layout:type:comma]
                    line_position = leading

        •   alone, which means if there is a line break on either side, then there must be a line break on both sides (i.e. that it should be the only thing on that line.

        •   All of the above options can be qualified with the :strict modifier - which prevents the inline case. For example:

                    -- Setting line_position to just `alone`
                    -- within [sqlfluff:layout:type:set_operator]
                    -- would not allow:
                    SELECT a
                    UNION SELECT b;
                    -- ...or...
                    SELECT a UNION
                    SELECT b;
                    -- but *would* allow both of the following:
                    SELECT a UNION SELECT b;
                    SELECT a
                    UNION
                    SELECT b;

                    -- However the default is set to `alone:strict`
                    -- then the *only* acceptable configuration is:
                    SELECT a
                    UNION
                    SELECT b;

    •

        Keyword Line Position: set using the keyword_line_position option.
            By default for most clauses this is unset, which implies no particular keyword line position requirements. The available options are:

            •   leading and alone, which are most common in the placement of keywords. Both of these settings also allow the option of a keyword to end on a line. By default we assume leading WHERE keywords, but if you (or your organisation) have settled on alone WHERE keywords then you should add the following section to your config:

                        [sqlfluff:layout:type:where_clause]
                        keyword_line_position = alone

            •   trailing, which means there should be a line break after the keyword. This is fairly uncommon but may apply to the ON keyword after a join. If you (or your organisation) have settled on trailing ON keywords then you should add the following section to your config:

                        [sqlfluff:layout:type:join_on_condition]
                        keyword_line_position = trailing

            •   The keyword positioning is valid across a number of different clauses. For example, to apply the leading directive to the PARTITION BY clause you would add the following configuration:

                        [sqlfluff:layout:type:partitionby_clause]
                        keyword_line_position = leading

            •   If you (or your organisation) would prefer to unset the Default Configuration <#defaultconfig> of some options, you may clear it by setting the configuration to none.

                        [sqlfluff:layout:type:where_clause]
                        keyword_line_position = none

                        [sqlfluff:layout:type:orderby_clause]
                        keyword_line_position = none

                        [sqlfluff:layout:type:groupby_clause]
                        keyword_line_position = none

                        [sqlfluff:layout:type:having_clause]
                        keyword_line_position = none

    •

        Exclusions: The keyword_line_position_exclusions option allows you to
            exclude specific types of segments from the keyword_line_position rule. This is useful when certain segments, such as window specifications or aggregate functions, should not follow the same keyword line position rules as other segments in the same clause.

            For example, to exclude window specifications from the ORDER BY clause's keyword line position rule, you can configure it as follows:

                    [sqlfluff:layout:type:orderby_clause]
                    keyword_line_position = leading
                    keyword_line_position_exclusions = window_specification

        This configuration ensures that the ORDER BY clause follows the leading rule, except for window specifications, which are allowed to remain inline.

        You can also specify multiple exclusions by separating them with commas:

                    [sqlfluff:layout:type:orderby_clause]
                    keyword_line_position = leading
                    keyword_line_position_exclusions = window_specification, aggregate_order_by

        In this case, both window specifications and aggregate functions with ORDER BY clauses are excluded from the leading rule.

        Example Usage:

        With the above configuration, the following SQL would pass:

                    SELECT
                    a,
                    b,
                    ROW_NUMBER() OVER (PARTITION BY c ORDER BY d) AS e,
                    STRING_AGG(a ORDER BY b, c)
                    FROM f
                    JOIN g
                    ON g.h = f.h

        However, the following SQL would fail because the outer ORDER BY clause does not follow the leading rule:

                    SELECT
                    a,
                    b,
                    ROW_NUMBER() OVER (PARTITION BY c ORDER BY d) AS e,
                    STRING_AGG(a ORDER BY b, c)
                    FROM f
                    JOIN g
                    ON g.h = f.h ORDER BY a

        The corrected version would be:

                    SELECT
                    a,
                    b,
                    ROW_NUMBER() OVER (PARTITION BY c ORDER BY d) AS e,
                    STRING_AGG(a ORDER BY b, c)
                    FROM f
                    JOIN g
                    ON g.h = f.h
                    ORDER BY a

  Templating Configuration

This section explains how to configure templating for SQL files.

When writing SQL files, users might utilise some kind of templating. The SQL file itself is written with placeholders which get rendered to proper SQL at run time. This can range from very simple placeholder templating to complex Jinja templating.

SQLFluff supports templated sections in SQL, see Stage 1, the templater <#templater>. This is achieved by the following set of operations:

    1.  SQLFluff pre-renders the templated SQL

    2.  SQLFluff applies the lint and fix operations to the rendered file

    3.  SQLFluff backports the rule violations to the templated section of the SQL.

SQLFluff does not automatically have access to the same environment used in production template setup. This means it is necessary to either provide that environment or provide dummy values to effectively render the template and generate valid SQL. Refer to the templater sections below for details.

SQLFluff natively supports the following templating engines

    •   Jinja templater <#jinja-templater>

    •   Placeholder templater <#placeholder-templater>

    •   Python templater <#python-templater>

Also, SQLFluff has an integration to use dbt as a templater.

    •   dbt templater <#dbt-templater> (via plugin which is covered in a different section).

Note:
        Templaters may not be able to generate a rendered SQL that cover the entire raw file.

        For example, if the raw SQL uses a {% if condition %} block, the rendered version of the template will only include either the {% then %} or the {% else %} block (depending on the provided configuration for the templater), but not both.

        In this case, because SQLFluff linting can only operate on the output of the templater, some areas of the raw SQL will never be seen by the linter and will not be covered by lint rules.

        This is functionality we hope to support in future.

  Jinja templater

The Jinja templater uses Jinja2 <https://jinja.palletsprojects.com/> to render templates.

There are multiple, complementary ways of configuring the Jinja templater.

    •   Reading variables and Jinja macros directly from the SQLFLuff config file

    •   Loading macros from a path

    •   Using a library

  Overview of Jinja templater's configuration options
Configuration | Variables | Macros | Filters | Documentation
Config file | ✅ | ✅ | ❌ | Complex Jinja Variable Templating and Jinja Macro Templating (from config)
Macro Path | ❌ | ✅ | ❌ | Jinja Macro Templating (from file)
Library | ✅ | ✅ | ✅ | Library Templating

For example, a snippet from a .sqlfluff file that uses all config options:

            [sqlfluff]
            templater = jinja

            [sqlfluff:templater:jinja]
            apply_dbt_builtins = True
            load_macros_from_path = my_macros
            loader_search_path = included_templates
            library_path = sqlfluff_libs
            exclude_macros_from_path = my_macros_exclude

            [sqlfluff:templater:jinja:context]
            my_list = ['a', 'b', 'c']
            MY_LIST = ("d", "e", "f")
            my_where_dict = {"field_1": 1, "field_2": 2}

            [sqlfluff:templater:jinja:macros]
            a_macro_def = {% macro my_macro(n) %}{{ n }} + {{ n * 2 }}{% endmacro %}

  Complex Jinja Variable Templating

Apart from the Generic variable templating that is supported for all templaters, two more advanced features of variable templating are available for Jinja.

case sensitivity and native python types. Both are illustrated in the following example:

            [sqlfluff:templater:jinja:context]
            my_list = ['a', 'b', 'c']
            MY_LIST = ("d", "e", "f")
            my_where_dict = {"field_1": 1, "field_2": 2}

            SELECT
                {% for elem in MY_LIST %}
                    '{{elem}}' {% if not loop.last %}||{% endif %}
                {% endfor %} as concatenated_list
            FROM tbl
            WHERE
                {% for field, value in my_where_dict.items() %}
                    {{field}} = {{value}} {% if not loop.last %}and{% endif %}
                {% endfor %}

...will render as...

            SELECT
                'd' || 'e' || 'f' as concatenated_list
            FROM tbl
            WHERE
                field_1 = 1 and field_2 = 2

Note that the variable was replaced in a case sensitive way and that the settings in the config file were interpreted as native python types.

  Jinja Macro Templating (from config)

Macros (which also look and feel like functions are available only in the jinja templater. Similar to Generic Variable Templating <#generic-variable-templating>, these are specified in config files, what's different in this case is how they are named. Similar to the context section above, macros are configured separately in the macros section of the config. Consider the following example.

If passed the following .sql file:

            SELECT {{ my_macro(6) }} FROM some_table

...and the following configuration in .sqlfluff in the same directory (note the tight control of whitespace):

            [sqlfluff:templater:jinja:macros]
            a_macro_def = {% macro my_macro(n) %}{{ n }} + {{ n * 2 }}{% endmacro %}

...then before parsing, the sql will be transformed to:

            SELECT 6 + 12 FROM some_table

Note that in the code block above, the variable name in the config is a_macro_def, and this isn't apparently otherwise used anywhere else. Broadly this is accurate, however within the configuration loader this will still be used to overwrite previous values in other config files. As such this introduces the idea of config blocks which could be selectively overwritten by other configuration files downstream as required.

  Jinja Macro Templating (from file)

In addition to macros specified in the config file, macros can also be loaded from files or folders. This is specified in the config file:

            [sqlfluff:templater:jinja]
            load_macros_from_path = my_macros,other_macros

load_macros_from_path is a comma-separated list of .sql files or folders. Locations are relative to the config file. For example, if the config file above was found at /home/my_project/.sqlfluff, then SQLFluff will look for macros in the folders /home/my_project/my_macros/ and /home/my_project/other_macros/, including any of their subfolders. Any macros defined in the config will always take precedence over a macro defined in the path.

exclude_macros_from_path works in the same manner as load_macros_from_path but allows you to have sqlfluff ignore certain macros. This can be useful if you have custom jinja tags.

Macros loaded from these files are available in every .sql file without requiring a Jinja include or import. They are loaded into the Jinja Global Namespace <https://jinja.palletsprojects.com/en/3.1.x/api/#global-namespace>.

Note: The load_macros_from_path setting also defines the search path for Jinja include <https://jinja.palletsprojects.com/en/3.1.x/templates/#include> or import <https://jinja.palletsprojects.com/en/3.1.x/templates/#import>. As with loaded macros, subdirectories are also supported. For example, if load_macros_from_path is set to my_macros, and there is a file my_macros/subdir/my_file.sql, you can do:

            {% include 'subdir/my_file.sql' %}

If you would like to define the Jinja search path without also loading the macros into the global namespace, use the loader_search_path setting instead.

Note:
        Throughout the templating process whitespace will still be treated rigorously, and this includes newlines. In particular you may choose to provide dummy macros in your configuration different from the actual macros used in production.

        REMEMBER: The reason SQLFluff supports macros is to enable it to parse templated sql without it being a blocker. It shouldn't be a requirement that the templating is accurate - it only needs to work well enough that parsing and linting are helpful.

  Builtin Jinja Macro Blocks

One of the main use cases which inspired SQLFluff as a project was dbt <https://www.getdbt.com/>. It uses jinja templating extensively and leads to some users maintaining large repositories of sql files which could potentially benefit from some linting.

Note:
        SQLFluff has now a tighter integration with dbt through the "dbt" templater. It is the recommended templater for dbt projects. If used, it eliminates the need for the overrides described in this section.

        To use the dbt templater, go to dbt templater <#dbt-templater>.

SQLFluff anticipates this use case and provides some built in macro blocks in the Default Configuration <#defaultconfig> which assist in getting started with dbt <https://www.getdbt.com/> projects. In particular it provides mock objects for:

    •   ref: The mock version of this provided simply returns the model reference as the name of the table. In most cases this is sufficient.

    •   config: A regularly used macro in dbt <https://www.getdbt.com/> to set configuration values. For linting purposes, this makes no difference and so the provided macro simply returns nothing.

Note:
        If there are other builtin macros which would make your life easier, consider submitting the idea (or even better a pull request) on github <https://www.github.com/sqlfluff/sqlfluff>.

  Library Templating

If using SQLFluff with jinja as your templater, you may have library function calls within your sql files that can not be templated via the normal macro templating mechanisms:

            SELECT foo, bar FROM baz {{ dbt_utils.group_by(2) }}

To template these libraries, you can use the sqlfluff:jinja:library_path config option:

            [sqlfluff:templater:jinja]
            library_path = sqlfluff_libs

This will pull in any python modules from that directory and allow sqlfluff to use them in templates. In the above example, you might define a file at sqlfluff_libs/dbt_utils.py as:

            def group_by(n):
                return "GROUP BY 1,2"

If an __init__.py is detected, it will be loaded alongside any modules and submodules found within the library path.

            SELECT
               {{ custom_sum('foo', 'bar') }},
               {{ foo.bar.another_sum('foo', 'bar') }}
            FROM
               baz

sqlfluff_libs/__init__.py:

            def custom_sum(a: str, b: str) -> str:
                return a + b

sqlfluff_libs/foo/__init__.py:

            # empty file

sqlfluff_libs/foo/bar.py:

            def another_sum(a: str, b: str) -> str:
               return a + b

Additionally, the library can be used to expose Jinja Filters <https://jinja.palletsprojects.com/en/3.1.x/templates/#filters> to the Jinja environment used by SQLFluff.

This is achieve by setting a global variable named SQLFLUFF_JINJA_FILTERS. SQLFLUFF_JINJA_FILTERS is a dictionary where

    •   dictionary keys map to the Jinja filter name

    •   dictionary values map to the Python callable

For example, to make the Airflow filter ds available to SQLFLuff, add the following to the __init__.py of the library:

            # https://github.com/apache/airflow/blob/main/airflow/templates.py#L53
            def ds_filter(value: datetime.date | datetime.time | None) -> str | None:
               """Date filter."""
               if value is None:
                   return None
               return value.strftime("%Y-%m-%d")

            SQLFLUFF_JINJA_FILTERS = {"ds": ds_filter}

Now, ds can be used in SQL

            SELECT "{{ "2000-01-01" | ds }}";

  Jinja loader search path

The Jinja environment can be configured to search for files included with include <https://jinja.palletsprojects.com/en/3.1.x/templates/#include> or import <https://jinja.palletsprojects.com/en/3.1.x/templates/#import> in a list of folders. This is specified in the config file:

            [sqlfluff:templater:jinja]
            loader_search_path = included_templates,other_templates

loader_search_path is a comma-separated list of folders. Locations are relative to the config file. For example, if the config file above was found at /home/my_project/.sqlfluff, then SQLFluff will look for included files in the folders /home/my_project/included_templates/ and /home/my_project/other_templates/, including any of their subfolders. For example, this will read from /home/my_project/included_templates/my_template.sql:

            {% include 'included_templates/my_template.sql' %}

Any folders specified in the load_macros_from_path setting are automatically appended to the loader_search_path. It is not necessary to specify a given directory in both settings.

Unlike the load_macros_from_path setting, any macros within these folders are not automatically loaded into the global namespace. They must be explicitly imported using the import <https://jinja.palletsprojects.com/en/3.1.x/templates/#import> Jinja directive. If you would like macros to be automatically included in the global Jinja namespace, use the load_macros_from_path setting instead.

  Interaction with --ignore=templating

Ignoring Jinja templating errors provides a way for users to use SQLFluff while reducing or avoiding the need to spend a lot of time adding variables to [sqlfluff:templater:jinja:context].

When --ignore=templating is enabled, the Jinja templater behaves a bit differently. This additional behavior is usually but not always helpful for making the file at least partially parsable and fixable. It definitely doesn’t guarantee that every file can be fixed, but it’s proven useful for some users.

Here's how it works:

    •   Within the expanded SQL, undefined variables are automatically replaced with the corresponding string value.

    •   If you do: {% include query %}, and the variable query is not defined, it returns a “file” containing the string query.

    •   If you do: {% include "query_file.sql" %}, and that file does not exist or you haven’t configured a setting for load_macros_from_path or loader_search_path, it returns a “file” containing the text query_file.

For example:

            select {{ my_variable }}
            from {% include "my_table.sql" %}

is interpreted as:

            select my_variable
            from my_table

The values provided by the Jinja templater act a bit (not exactly) like a mixture of several types:

    •   str

    •   int

    •   list

    •   Jinja's Undefined class <https://jinja.palletsprojects.com/en/3.1.x/api/#jinja2.Undefined>

Because the values behave like Undefined, it's possible to replace them using Jinja's default() filter <https://jinja.palletsprojects.com/en/3.1.x/templates/#jinja-filters.default>. For example:

            select {{ my_variable | default("col_a") }}
            from my_table

is interpreted as:

            select col_a
            from my_table

  Placeholder templater

Libraries such as SQLAlchemy or Psycopg use different parameter placeholder styles to mark where a parameter has to be inserted in the query.

For example a query in SQLAlchemy can look like this:

            SELECT * FROM table WHERE id = :myid

At runtime :myid will be replace by a value provided by the application and escaped as needed, but this is not standard SQL and cannot be parsed as is.

In order to parse these queries is then necessary to replace these placeholders with sample values, and this is done with the placeholder templater.

Placeholder templating can be enabled in the config using:

            [sqlfluff]
            templater = placeholder

A few common styles are supported:

             -- colon
             WHERE bla = :my_name

             -- colon_nospaces
             -- (use with caution as more prone to false positives)
             WHERE bla = table:my_name

             -- colon_optional_quotes
             SELECT :"column" FROM :table WHERE bla = :'my_name'

             -- numeric_colon
             WHERE bla = :2

             -- pyformat
             WHERE bla = %(my_name)s

             -- dollar
             WHERE bla = $my_name or WHERE bla = ${my_name}

             -- dollar_surround (DbUp compatible variable)
             WHERE bla = $my_name$

             -- question_mark
             WHERE bla = ?

             -- numeric_dollar
             WHERE bla = $3 or WHERE bla = ${3}

             -- percent
             WHERE bla = %s

             -- ampersand
             WHERE bla = &s or WHERE bla = &{s} or USE DATABASE MARK_{ENV}

These can be configured by setting param_style to the names above:

            [sqlfluff:templater:placeholder]
            param_style = colon
            my_name = 'john'

then you can set sample values for each parameter, like my_name above. Notice that the value needs to be escaped as it will be replaced as a string during parsing. When the sample values aren't provided, the templater will use parameter names themselves by default.

When parameters are positional, like question_mark, then their name is simply the order in which they appear, starting with 1.

            [sqlfluff:templater:placeholder]
            param_style = question_mark
            1 = 'john'

In case you need a parameter style different from the ones above, you can pass a custom regex.

            [sqlfluff:templater:placeholder]
            param_regex = __(?P<param_name>[\w_]+)__
            my_name = 'john'

N.B. quotes around param_regex in the config are interpreted literally by the templater. e.g. param_regex='__(?P<param_name>[w_]+)__' matches '__some_param__' not __some_param__

the named parameter param_name will be used as the key to replace, if missing, the parameter is assumed to be positional and numbers are used instead.

Also consider making a pull request to the project to have your style added, it may be useful to other people and simplify your configuration.

  Python templater

Uses native Python f-strings. As described in Generic Variable Templating <#generic-variable-templating>, an example usage would look be configured as follows:

If passed the following .sql file:

            SELECT * FROM {tbl_name}

...and the following configuration in .sqlfluff in the same directory:

            [sqlfluff]
            templater = python

            [sqlfluff:templater:python:context]
            tbl_name = my_table

...then before parsing, the sql will be transformed to:

            SELECT * FROM my_table

  Complex Python Variable Templating

Python string formatting <https://docs.python.org/3/library/string.html#format-string-syntax> supports accessing object attributes via dot notation (e.g. {foo.bar}). However, since we cannot create Python objects within configuration files, we need a workaround in order to provide dummy values to render templates containing these values. The SQLFluff python templater will interpret any variable containing a "." as a dictionary lookup on the magic fixed context key sqlfluff.

            -- this SQL
            SELECT * FROM {foo.bar}

            -- becomes this
            SELECT * FROM {sqlfluff["foo.bar"]}

..which can be populated using the following configuration:

            [sqlfluff:templater:python:context]
            sqlfluff = {"foo.bar": "abc"}

  dbt templater

Note:
        From sqlfluff version 0.7.0 onwards, the dbt templater has been moved to a separate plugin and python package. Projects that were already using the dbt templater may initially fail after an upgrade to 0.7.0+. See the installation instructions below to install the dbt templater.

        dbt templating is still a relatively new feature added in 0.4.0 and is still in very active development! If you encounter an issue, please let us know in a GitHub issue or on the SQLFluff slack workspace.

dbt is not the default templater for SQLFluff (it is jinja). dbt is a complex tool, so using the default jinja templater will be simpler. You should be aware when using the dbt templater that you will be exposed to some of the complexity of dbt. Users may wish to try both templaters and choose according to how they intend to use SQLFluff.

A simple rule of thumb might be:

    •   If you are using SQLFluff in a CI/CD context, where speed is not critical but accuracy in rendering sql is, then the dbt templater may be more appropriate.

    •   If you are using SQLFluff in an IDE or on a git hook, where speed of response may be more important, then the jinja templater may be more appropriate.

Pros:

    •   Most (potentially all) macros will work

Cons:

    •   More complex, e.g. using it successfully may require deeper understanding of your models and/or macros (including third-party macros)

        •   More configuration decisions to make

        •   Best practices are not yet established or documented

    •   If your dbt model files access a database at compile time, using SQLFluff with the dbt templater will also require access to a database.

        •   Note that you can often point SQLFluff and the dbt templater at a test database (i.e. it doesn't have to be the production database).

    •   Runs slower

  Installation & Configuration

In order to get started using SQLFluff with a dbt project you will first need to install the relevant dbt adapter <https://docs.getdbt.com/docs/available-adapters> for your dialect and the sqlfluff-templater-dbt package using your package manager of choice (e.g. pip install dbt-postgres sqlfluff-templater-dbt) and then will need the following configuration:

In .sqlfluff:

            [sqlfluff]
            templater = dbt

In .sqlfluffignore:

            target/
            # dbt <1.0.0
            dbt_modules/
            # dbt >=1.0.0
            dbt_packages/
            macros/

You can set the dbt project directory, profiles directory and profile with:

            [sqlfluff:templater:dbt]
            project_dir = <relative or absolute path to dbt_project directory, can be overridden by env var DBT_PROJECT_DIR>
            profiles_dir = <relative or absolute path to the directory that contains the profiles.yml file, can be overridden by env var DBT_PROFILES_DIR>
            profile = <dbt profile>
            target = <dbt target>
            dbt_skip_compilation_error = <True or False, default is True>

Note:
        If the profiles_dir setting is omitted, SQLFluff will look for the profile in the default location, which varies by operating system. On Unix-like operating systems (e.g. Linux or macOS), the default profile directory is ~/.dbt/. On Windows, you can determine your default profile directory by running dbt debug --config-dir.

Note:
        A fatal error can be raised at compile time. That can sometimes happen for SQLFluff related reasons (it used to happen if we tried to compile ephemeral models in the wrong order), but more often because a macro tries to query a table at compile time which doesn't exist. By default, dbt_skip_compilation_error parameter is set to True, that's why such errors will be ignored. However if you want to see them, you can set it to False and SQLFluff will raise a fatal error.

To use builtin dbt Jinja functions SQLFluff provides a configuration option that enables usage within templates.

            [sqlfluff:templater:jinja]
            apply_dbt_builtins = True

This will provide dbt macros like ref, var, is_incremental(). If the need arises builtin dbt macros can be customised via Jinja macros in .sqlfluff configuration file.

            [sqlfluff:templater:jinja:macros]
            # Macros provided as builtins for dbt projects
            dbt_ref = {% macro ref(model_ref) %}{{model_ref}}{% endmacro %}
            dbt_source = {% macro source(source_name, table) %}{{source_name}}_{{table}}{% endmacro %}
            dbt_config = {% macro config() %}{% for k in kwargs %}{% endfor %}{% endmacro %}
            dbt_var = {% macro var(variable, default='') %}item{% endmacro %}
            dbt_is_incremental = {% macro is_incremental() %}True{% endmacro %}

If your project requires that you pass variables to dbt through command line, you can specify them in template:dbt:context section of .sqlfluff. See below configuration and its equivalent dbt command:

            [sqlfluff:templater:dbt:context]
            my_variable = 1

            dbt run --vars '{"my_variable": 1}'

  Known Caveats

    •   To use the dbt templater, you must set templater = dbt in the .sqlfluff config file in the directory where sqlfluff is run. The templater cannot be changed in .sqlfluff files in subdirectories.

    •   In SQLFluff 0.4.0 using the dbt templater requires that all files within the root and child directories of the dbt project must be part of the project. If there are deployment scripts which refer to SQL files not part of the project for instance, this will result in an error. You can overcome this by adding any non-dbt project SQL files to .sqlfluffignore.

  Generic Variable Templating

Variables are available in all the templaters. By default the templating engine will expect variables for templating to be available in the config, and the templater will be look in the section corresponding to the context for that templater. By convention, the config for the jinja templater is found in the sqlfluff:templater:jinja:context section, the config for the python templater is found in the sqlfluff:templater:python:context section, the one for the placeholder templater is found in the sqlfluff:templater:placeholder:context section.

For example, if passed the following .sql file:

            SELECT {{ num_things }} FROM {{ tbl_name }} WHERE id > 10 LIMIT 5

...and the following configuration in .sqlfluff in the same directory:

            [sqlfluff:templater:jinja:context]
            num_things=456
            tbl_name=my_table

...then before parsing, the sql will be transformed to:

            SELECT 456 FROM my_table WHERE id > 10 LIMIT 5

Note:
        If there are variables in the template which cannot be found in the current configuration context, then this will raise a SQLTemplatingError and this will appear as a violation without a line number, quoting the name of the variable that couldn't be found.

  Ignoring Errors & Files

  Ignoring individual lines

Similar to flake8's ignore <https://flake8.pycqa.org/en/3.1.1/user/ignoring-errors.html#in-line-ignoring-errors>, individual lines can be ignored by adding -- noqa to the end of the line. Additionally, specific rules can be ignored by quoting their code or the category.

            -- Ignore all errors
            SeLeCt  1 from tBl ;    -- noqa

            -- Ignore rule CP02 & rule CP03
            SeLeCt  1 from tBl ;    -- noqa: CP02,CP03

            -- Ignore all parsing errors
            SeLeCt from tBl ;       -- noqa: PRS

Note:
        It should be noted that ignoring TMP and PRS errors can lead to incorrect sqlfluff lint and sqfluff fix results as SQLFluff can misinterpret the SQL being analysed.

  Ignoring line ranges

Similar to pylint's "pylint" directive" <http://pylint.pycqa.org/en/latest/user_guide/message-control.html>, ranges of lines can be ignored by adding -- noqa:disable=<rule>[,...] | all to the line. Following this directive, specified rules (or all rules, if "all" was specified) will be ignored until a corresponding -- noqa:enable=<rule>[,...] | all directive.

            -- Ignore rule AL02 from this line forward
            SELECT col_a a FROM foo -- noqa: disable=AL02

            -- Ignore all rules from this line forward
            SELECT col_a a FROM foo -- noqa: disable=all

            -- Enforce all rules from this line forward
            SELECT col_a a FROM foo -- noqa: enable=all

  .sqlfluffignore

Similar to Git's <https://git-scm.com/docs/gitignore#_pattern_format> .gitignore and Docker's <https://docs.docker.com/engine/reference/builder/#dockerignore-file> .dockerignore, SQLFluff supports a .sqlfluffignore file to control which files are and aren't linted. Under the hood we use the python pathspec library <https://python-path-specification.readthedocs.io/> which also has a brief tutorial in their documentation.

An example of a potential .sqlfluffignore placed in the root of your project would be:

            # Comments start with a hash.

            # Ignore anything in the "temp" path
            /temp/

            # Ignore anything called "testing.sql"
            testing.sql

            # Ignore any ".tsql" files
            *.tsql

Ignore files can also be placed in subdirectories of a path which is being linted and the sub files will also be applied within that subdirectory.

  Ignoring types of errors

General categories of errors can be ignored using the --ignore command line option or the ignore setting in .sqlfluffignore. Types of errors that can be ignored include:

    •   lexing

    •   linting

    •   parsing

    •   templating

  Default Configuration

The default configuration is as follows, note the Builtin Jinja Macro Blocks <#builtin-jinja-blocks> in section [sqlfluff:templater:jinja:macros] as referred to above.

Note:
        This shows the entire default config. We do not recommend that users copy this whole config as the starter config file for their project.

        This is for two reasons:

            1.  The config file should act as a form of documentation for your team. A record of what decisions you've made which govern how your format your sql. By having a more concise config file, and only defining config settings where they differ from the defaults - you are more clearly stating to your team what choices you've made.

            2.  As the project evolves, the structure of the config file may change and we will attempt to make changes as backward compatible as possible. If you have not overridden a config setting in your project, we can easily update the default config to match your expected behaviour over time. We may also find issues with the default config which we can also fix in the background. However, the longer your local config file, the more work it will be to update and migrate your config file between major versions.

        If you are starting a fresh project and are looking for a good starter config, check out the New Project Configuration <#starter-config> section above.

            [sqlfluff]
            # verbose is an integer (0-2) indicating the level of log output
            verbose = 0
            # Turn off color formatting of output
            nocolor = None
            # Supported dialects https://docs.sqlfluff.com/en/stable/perma/dialects.html
            # Or run 'sqlfluff dialects'
            dialect = None
            # One of [raw|jinja|python|placeholder]
            templater = jinja
            # Comma separated list of rules to check, default to all
            rules = all
            # Comma separated list of rules to exclude, or None
            exclude_rules = None
            # Below controls SQLFluff output, see max_line_length for SQL output
            output_line_length = 80
            # Number of passes to run before admitting defeat
            runaway_limit = 10
            # Ignore errors by category (one or more of the following, separated by commas: lexing,linting,parsing,templating)
            ignore = None
            # Warn only for rule codes (one of more rule codes, separated by commas: e.g. LT01,LT02)
            # Also works for templating and parsing errors by using TMP or PRS
            warnings = None
            # Whether to warn about unneeded '-- noqa:' comments.
            warn_unused_ignores = False
            # Ignore linting errors found within sections of code coming directly from
            # templated code (e.g. from within Jinja curly braces. Note that it does not
            # ignore errors from literal code found within template loops.
            ignore_templated_areas = True
            # can either be autodetect or a valid encoding e.g. utf-8, utf-8-sig
            encoding = autodetect
            # Ignore inline overrides (e.g. to test if still required)
            disable_noqa = False
            # Ignore inline overrides except those listed.
            # This take priority over `disabled_noqa`.
            disable_noqa_except = None
            # Comma separated list of file extensions to lint
            # NB: This config will only apply in the root folder
            sql_file_exts = .sql,.sql.j2,.dml,.ddl
            # Allow fix to run on files, even if they contain parsing errors
            # Note altering this is NOT RECOMMENDED as can corrupt SQL
            fix_even_unparsable = False
            # Very large files can make the parser effectively hang.
            # The more efficient check is the _byte_ limit check which
            # is enabled by default. The previous _character_ limit check
            # is still present for backward compatibility. This will be
            # removed in a future version.
            # Set either to 0 to disable.
            large_file_skip_char_limit = 0
            large_file_skip_byte_limit = 20000
            # CPU processes to use while linting.
            # If positive, just implies number of processes.
            # If negative or zero, implies number_of_cpus - specified_number.
            # e.g. -1 means use all processors but one. 0  means all cpus.
            processes = 1
            # Max line length is set by default to be in line with the dbt style guide.
            # https://github.com/dbt-labs/corp/blob/main/dbt_style_guide.md
            # Set to zero or negative to disable checks.
            max_line_length = 80
            # NOTE: Templater Variant rendering should currently be considered EXPERIMENTAL.
            # Only set `render_variant_limit` to more than 1 if you know what you're doing!
            # Implementation of this will also depend on your templater.
            render_variant_limit = 1

            [sqlfluff:indentation]
            # See https://docs.sqlfluff.com/en/stable/perma/indent_locations.html
            indent_unit = space
            tab_space_size = 4
            indented_joins = False
            indented_ctes = False
            indented_using_on = True
            indented_on_contents = True
            indented_then = True
            indented_then_contents = True
            allow_implicit_indents = False
            template_blocks_indent = True
            # This is a comma separated list of elements to skip
            # indentation edits to.
            skip_indentation_in = script_content
            # If comments are found at the end of long lines, we default to moving
            # them to the line _before_ their current location as the convention is
            # that a comment precedes the line it describes. However if you prefer
            # comments moved _after_, this configuration setting can be set to "after".
            trailing_comments = before
            # To exclude comment lines from indentation entirely set this to "True".
            ignore_comment_lines = False

            # Layout configuration
            # See https://docs.sqlfluff.com/en/stable/perma/layout_spacing.html
            [sqlfluff:layout:type:comma]
            spacing_before = touch
            line_position = trailing

            [sqlfluff:layout:type:binary_operator]
            spacing_within = touch
            line_position = leading

            [sqlfluff:layout:type:column_path_operator]
            spacing_within = touch
            line_position = leading

            [sqlfluff:layout:type:statement_terminator]
            spacing_before = touch
            line_position = trailing

            [sqlfluff:layout:type:end_of_file]
            spacing_before = touch

            [sqlfluff:layout:type:set_operator]
            line_position = alone:strict

            [sqlfluff:layout:type:start_bracket]
            spacing_after = touch

            [sqlfluff:layout:type:end_bracket]
            spacing_before = touch

            [sqlfluff:layout:type:start_square_bracket]
            spacing_after = touch

            [sqlfluff:layout:type:end_square_bracket]
            spacing_before = touch

            [sqlfluff:layout:type:start_angle_bracket]
            spacing_after = touch

            [sqlfluff:layout:type:end_angle_bracket]
            spacing_before = touch

            [sqlfluff:layout:type:casting_operator]
            spacing_before = touch
            spacing_after = touch:inline

            [sqlfluff:layout:type:slice]
            spacing_before = touch
            spacing_after = touch

            [sqlfluff:layout:type:dot]
            spacing_before = touch
            spacing_after = touch

            [sqlfluff:layout:type:comparison_operator]
            spacing_within = touch
            line_position = leading

            [sqlfluff:layout:type:assignment_operator]
            spacing_within = touch
            line_position = leading

            [sqlfluff:layout:type:object_reference]
            spacing_within = touch:inline

            [sqlfluff:layout:type:numeric_literal]
            spacing_within = touch:inline

            [sqlfluff:layout:type:sign_indicator]
            spacing_after = touch:inline

            [sqlfluff:layout:type:tilde]
            spacing_after = touch:inline

            [sqlfluff:layout:type:function_name]
            spacing_within = touch:inline

            [sqlfluff:layout:type:function_contents]
            spacing_before = touch:inline

            [sqlfluff:layout:type:function_parameter_list]
            spacing_before = touch:inline

            [sqlfluff:layout:type:array_type]
            spacing_within = touch:inline

            [sqlfluff:layout:type:typed_array_literal]
            spacing_within = touch

            [sqlfluff:layout:type:sized_array_type]
            spacing_within = touch

            [sqlfluff:layout:type:struct_type]
            spacing_within = touch:inline

            [sqlfluff:layout:type:bracketed_arguments]
            spacing_before = touch:inline

            [sqlfluff:layout:type:match_condition]
            spacing_within = touch:inline

            [sqlfluff:layout:type:typed_struct_literal]
            spacing_within = touch

            [sqlfluff:layout:type:semi_structured_expression]
            spacing_within = touch:inline
            spacing_before = touch:inline

            [sqlfluff:layout:type:array_accessor]
            spacing_before = touch:inline

            [sqlfluff:layout:type:colon]
            spacing_before = touch

            [sqlfluff:layout:type:colon_delimiter]
            spacing_before = touch
            spacing_after = touch

            [sqlfluff:layout:type:path_segment]
            spacing_within = touch

            [sqlfluff:layout:type:sql_conf_option]
            spacing_within = touch

            [sqlfluff:layout:type:sqlcmd_operator]
            # NOTE: This is the spacing between the operator and the colon
            spacing_before = touch

            [sqlfluff:layout:type:slash]
            spacing_before = any
            spacing_after = any

            [sqlfluff:layout:type:comment]
            spacing_before = any
            spacing_after = any

            [sqlfluff:layout:type:pattern_expression]
            # Snowflake pattern expressions shouldn't have their spacing changed.
            spacing_within = any

            [sqlfluff:layout:type:bind_variable]
            spacing_within = touch

            [sqlfluff:layout:type:placeholder]
            # Placeholders exist "outside" the rendered SQL syntax
            # so we shouldn't enforce any particular spacing around
            # them.
            spacing_before = any
            spacing_after = any

            [sqlfluff:layout:type:common_table_expression]
            # The definition part of a CTE should fit on one line where possible.
            # For users which regularly define column names in their CTEs they
            # may which to relax this config to just `single`.
            spacing_within = single:inline

            # By setting a selection of clauses to "alone", we hint to the reflow
            # algorithm that in the case of a long single line statement, the
            # first place to add newlines would be around these clauses.
            # Setting this to "alone:strict" would always _force_ line breaks
            # around them even if the line isn't too long.
            [sqlfluff:layout:type:select_clause]
            line_position = alone

            [sqlfluff:layout:type:where_clause]
            line_position = alone
            keyword_line_position = leading
            keyword_line_position_exclusions = pipe_operator_clause

            [sqlfluff:layout:type:from_clause]
            line_position = alone

            [sqlfluff:layout:type:join_clause]
            line_position = alone

            [sqlfluff:layout:type:groupby_clause]
            line_position = alone
            keyword_line_position = leading

            [sqlfluff:layout:type:orderby_clause]
            # NOTE: Order by clauses appear in many places other than in a select
            # clause. To avoid unexpected behaviour we use `leading` in this
            # case rather than `alone`.
            line_position = leading
            keyword_line_position = leading
            keyword_line_position_exclusions =
                window_specification,
                aggregate_order_by,
                withingroup_clause,
                pipe_operator_clause

            [sqlfluff:layout:type:having_clause]
            line_position = alone
            keyword_line_position = leading

            [sqlfluff:layout:type:limit_clause]
            line_position = alone

            # Template loop tokens shouldn't dictate spacing around them.
            [sqlfluff:layout:type:template_loop]
            spacing_before = any
            spacing_after = any

            [sqlfluff:templater]
            unwrap_wrapped_queries = True

            [sqlfluff:templater:jinja]
            apply_dbt_builtins = True

            # Some rules can be configured directly from the config common to other rules
            [sqlfluff:rules]
            allow_scalar = True
            single_table_references = consistent
            unquoted_identifiers_policy = all

            [sqlfluff:rules:capitalisation.keywords]
            # Keywords
            capitalisation_policy = consistent
            # Comma separated list of words to ignore for this rule
            ignore_words = None
            ignore_words_regex = None

            [sqlfluff:rules:capitalisation.identifiers]
            # Unquoted identifiers
            extended_capitalisation_policy = consistent
            # Comma separated list of words to ignore for this rule
            ignore_words = None
            ignore_words_regex = None

            [sqlfluff:rules:capitalisation.functions]
            # Function names
            extended_capitalisation_policy = consistent
            # Comma separated list of words to ignore for this rule
            ignore_words = None
            ignore_words_regex = None

            [sqlfluff:rules:capitalisation.literals]
            # Null & Boolean Literals
            capitalisation_policy = consistent
            # Comma separated list of words to ignore for this rule
            ignore_words = None
            ignore_words_regex = None

            [sqlfluff:rules:capitalisation.types]
            # Data Types
            extended_capitalisation_policy = consistent
            # Comma separated list of words to ignore for this rule
            ignore_words = None
            ignore_words_regex = None

            [sqlfluff:rules:ambiguous.join]
            # Fully qualify JOIN clause
            fully_qualify_join_types = inner

            [sqlfluff:rules:ambiguous.column_references]
            # GROUP BY/ORDER BY column references
            group_by_and_order_by_style = consistent

            [sqlfluff:rules:aliasing.table]
            # Aliasing preference for tables
            aliasing = explicit

            [sqlfluff:rules:aliasing.column]
            # Aliasing preference for columns
            aliasing = explicit

            [sqlfluff:rules:aliasing.unused]
            alias_case_check = dialect

            [sqlfluff:rules:aliasing.length]
            min_alias_length = None
            max_alias_length = None

            [sqlfluff:rules:aliasing.forbid]
            # Avoid table aliases in from clauses and join conditions.
            # Disabled by default for all dialects unless explicitly enabled.
            # We suggest instead using aliasing.length (AL06) in most cases.
            force_enable = False

            [sqlfluff:rules:convention.not_equal]
            # Consistent usage of preferred "not equal to" comparison
            preferred_not_equal_style = consistent

            [sqlfluff:rules:convention.select_trailing_comma]
            # Trailing commas
            select_clause_trailing_comma = forbid

            [sqlfluff:rules:convention.count_rows]
            # Consistent syntax to count all rows
            prefer_count_1 = False
            prefer_count_0 = False

            [sqlfluff:rules:convention.terminator]
            # Semi-colon formatting approach
            multiline_newline = False
            require_final_semicolon = False

            [sqlfluff:rules:convention.blocked_words]
            # Comma separated list of blocked words that should not be used
            blocked_words = None
            blocked_regex = None
            match_source = False

            [sqlfluff:rules:convention.quoted_literals]
            # Consistent usage of preferred quotes for quoted literals
            preferred_quoted_literal_style = consistent
            # Disabled for dialects that do not support single and double quotes for quoted literals (e.g. Postgres)
            force_enable = False

            [sqlfluff:rules:convention.casting_style]
            # SQL type casting
            preferred_type_casting_style = consistent

            [sqlfluff:rules:references.from]
            # References must be in FROM clause
            # Disabled for some dialects (e.g. bigquery)
            force_enable = False

            [sqlfluff:rules:references.qualification]
            # Comma separated list of words to ignore for this rule
            ignore_words = None
            ignore_words_regex = None
            subqueries_ignore_external_references = False

            [sqlfluff:rules:references.consistent]
            # References must be consistently used
            # Disabled for some dialects (e.g. bigquery)
            force_enable = False

            [sqlfluff:rules:references.keywords]
            # Keywords should not be used as identifiers.
            unquoted_identifiers_policy = aliases
            quoted_identifiers_policy = none
            # Comma separated list of words to ignore for this rule
            ignore_words = None
            ignore_words_regex = None

            [sqlfluff:rules:references.special_chars]
            # Special characters in identifiers
            unquoted_identifiers_policy = all
            quoted_identifiers_policy = all
            allow_space_in_identifier = False
            additional_allowed_characters = None
            ignore_words = None
            ignore_words_regex = None

            [sqlfluff:rules:references.quoting]
            # Policy on quoted and unquoted identifiers
            prefer_quoted_identifiers = False
            prefer_quoted_keywords = False
            ignore_words = None
            ignore_words_regex = None
            case_sensitive = True

            [sqlfluff:rules:layout.long_lines]
            # Line length
            ignore_comment_lines = False
            ignore_comment_clauses = False

            [sqlfluff:rules:layout.newlines]
            maximum_empty_lines_between_statements = 2
            maximum_empty_lines_inside_statements = 1

            [sqlfluff:rules:layout.select_targets]
            wildcard_policy = single

            [sqlfluff:rules:structure.subquery]
            # By default, allow subqueries in from clauses, but not join clauses
            forbid_subquery_in = join

            [sqlfluff:rules:structure.join_condition_order]
            preferred_first_table_in_join_clause = earlier

  Production Usage & Security

SQLFluff is designed to be used both as a utility for developers but also to be part of CI/CD <https://en.wikipedia.org/wiki/Continuous_integration> pipelines.

  Security Considerations

A full list of Security Advisories is available on GitHub <https://github.com/sqlfluff/sqlfluff/security/advisories>.

Given the context of how SQLFluff is designed to be used, there are three different tiers of access which users may have access to manipulate how the tool functions in a secure environment.

    1.  Users may have edit access to the SQL code which is being linted. While SQLFluff does not execute the SQL itself, in the process of the templating step <#templater> (in particular via jinja or dbt), certain macros may have the ability to execute arbitrary SQL code (e.g. the dbt run_query macro <https://docs.getdbt.com/reference/dbt-jinja-functions/run_query>). For the Jinja templater, SQLFluff uses the Jinja2 SandboxedEnvironment <https://jinja.palletsprojects.com/en/3.0.x/sandbox/#jinja2.sandbox.SandboxedEnvironment> to limit the execution on unsafe code. When looking to further secure this situation, see below for ways to limit the ability of users to import other libraries.

    2.  Users may have edit access to the SQLFluff :ref:`config-files`. In some (perhaps, many) environments, the users who can edit SQL files may also be able to access and edit the Configuration Files <#config-files>. It's important to note that because of In-File Configuration Directives <#in-file-config>, that users who can edit SQL files which are designed to be linted, will also have access to the vast majority of any configuration options available in Configuration Files <#config-files>. This means that there is minimal additional protection from restricting access to Configuration Files <#config-files> for users who already have access to edit the linting target files (as described above).

    3.  Users may have access to change how SQLFluff is invoked. SQLFluff can be invoked either as a command line too or via the python API. Typically the method is fixed for a given application. When thinking about how to restrict the ability of users to call insecure code, SQLFluff aims to provide options at the point of invocation. In particular, as described above, the primary risk vector for SQLFluff is the macro environment as described in Templating Configuration <#templateconfig>. To restrict users being able to bring arbitrary python methods into sqlfluff via the library_path configuration value (see Library Templating <#jinja-library-templating>), we recommend that for secure environments you override this config value either by providing an override option to the FluffConfig object if using the Python API or via the --library-path CLI option:

        To disable this option entirely via the CLI:

                $ sqlfluff lint my_path --library-path none

    To disable this option entirely via the python API:

                """This is an example of providing config overrides."""

                from sqlfluff.core import FluffConfig, Linter

                sql = "SELECT 1\n"

                config = FluffConfig(
                    overrides={
                        "dialect": "snowflake",
                        # NOTE: We explicitly set the string "none" here rather
                        # than a None literal so that it overrides any config
                        # set by any config files in the path.
                        "library_path": "none",
                    }
                )

                linted_file = Linter(config=config).lint_string(sql)

                assert linted_file.get_violations() == []

  Using SQLFluff directly as a CLI application

The SQLFluff CLI application <#cliref> is a python application which means if depends on your host python environment (see Installing SQLFluff <#installingsqlfluff>).

The exit code <https://shapeshed.com/unix-exit-codes/> provided by SQLFluff when run as a command line utility is designed to assist usefulness in deployment pipelines. If no violations are found then the exit code <https://shapeshed.com/unix-exit-codes/> will be 0. If violations are found then a non-zero code will be returned which can be interrogated to find out more.

    •   An error code of 0 means operation success, no issues found.

    •   An error code of 1 means operation success, issues found. For example this might mean that a linting issue was found, or that one file could not be parsed.

    •   An error code of 2 means an error occurred and the operation could not be completed. For example a configuration issue or an internal error within SQLFluff.

For details of what commands and options are available in the CLI see the CLI Reference <#cliref>.

  Using SQLFluff on changes using diff-quality

For projects with large amounts of (potentially imperfect) SQL code, the full SQLFluff output could be very large, which can be distracting -- perhaps the CI build for a one-line SQL change shouldn't encourage the developer to fix lots of unrelated quality issues.

To support this use case, SQLFluff integrates with a quality checking tool called diff-quality. By running SQLFluff using diff-quality (rather than running it directly), you can limit the the output to the new or modified SQL in the branch (aka pull request or PR) containing the proposed changes.

Currently, diff-quality requires that you are using git for version control.

NOTE: Installing SQLFluff automatically installs the diff_cover package that provides the diff-quality tool.

  Adding diff-quality to your builds

In your CI build script:

    1.  Set the current working directory to the git repository containing the SQL code to be checked.

    2.  Run diff-quality, specifying SQLFluff as the underlying tool:

            $ diff-quality --violations sqlfluff

The output will look something like:

            -------------
            Diff Quality
            Quality Report: sqlfluff
            Diff: origin/master...HEAD, staged and unstaged changes
            -------------
            sql/audience_size_queries/constraints/_postcondition_check_gdpr_compliance.sql (0.0%):
            sql/audience_size_queries/constraints/_postcondition_check_gdpr_compliance.sql:5: Unquoted Identifiers must be consistently upper case.
            -------------
            Total:   1 line
            Violations: 1 line
            % Quality: 0%
            -------------

These messages are basically the same as those provided directly by SQLFluff, although the format is a little different. Note that diff-quality only lists the line _numbers_, not the character position. If you need the character position, you will need to run SQLFluff directly.

Note:
        When using diff-quality with .sqlfluff Configuration Files <#config-files>, and especially together with the dbt templater <#dbt-templater>, it can be really easy to run into issues with file discovery. There are a few steps you can take to make it much less likely that this will happen:

            1.  diff-quality needs to be run from the root of your git repository (so that it can find the git metadata).

            2.  SQLFluff works best if the bulk of the configuration is done from a single .sqlfluff file, which should be in the root of your git repository.

            3.  If using dbt templater <#dbt-templater>, then either place your dbt_project.yml file in the same root folder, or if you put it in a subfolder, then only invoke diff-quality and sqlfluff from the root and define the subfolder that the dbt project lives in using the .sqlfluff config file.

        By aligning the paths of all three, you should be able to achieve a robust setup. If each is rooted in different paths if can be very difficult to achieve the same result, and the resulting behaviour can be difficult to debug.

        To debug any issues relating to this setup, we recommend occasionally running sqlfluff directly using the main cli (i.e. calling sqlfluff lint my/project/path) and check whether that route gives you the results you expect. diff-quality should behave as though it's calling the SQLFluff CLI from the same path that you invoke diff-quality.

For more information on diff-quality and the diff_cover package, see the documentation <https://github.com/Bachmann1234/diff_cover> on their github repository. It covers topics such as:

    •   Generating HTML reports

    •   Controlling which branch to compare against (i.e. to determine new/changed lines). The default is origin/main.

    •   Configuring diff-quality to return an error code if the quality is too low.

    •   Troubleshooting

  Using pre-commit

pre-commit <https://pre-commit.com/> is a framework to manage git "hooks" triggered right before a commit is made.

A git hook <https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks> is a git feature to "fire off custom scripts" when specific actions occur.

Using pre-commit <https://pre-commit.com/> with SQLFluff is a good way to provide automated linting to SQL developers.

With pre-commit <https://pre-commit.com/>, you also get the benefit of only linting/fixing the files that changed.

SQLFluff comes with two pre-commit <https://pre-commit.com/> hooks:

    •   sqlfluff-lint: returns linting errors.

    •   sqlfluff-fix: attempts to fix rule violations.

Warning:
        For safety reasons, sqlfluff-fix by default will not make any fixes in files that had templating or parse errors, even if those errors were ignored using noqa or --ignore`.

        Although it is not advised, you can tell SQLFluff to try and fix these files by overriding the fix_even_unparsable setting in .sqlfluff config file or using the sqlfluff fix --FIX-EVEN-UNPARSABLE command line option.

        Overriding this behavior may break your SQL. If you use this override, always be sure to review any fixes applied to files with templating or parse errors to verify they are okay.

You should create a file named .pre-commit-config.yaml at the root of your git project, which should look like this:

            repos:
            - repo: https://github.com/sqlfluff/sqlfluff
              rev: stable_version
              hooks:
                - id: sqlfluff-lint
                  # For dbt projects, this installs the dbt "extras".
                  # You will need to select the relevant dbt adapter for your dialect
                  # (https://docs.getdbt.com/docs/available-adapters):
                  # additional_dependencies: ['<dbt-adapter>', 'sqlfluff-templater-dbt']
                - id: sqlfluff-fix
                  # Arbitrary arguments to show an example
                  # args: [--rules, "LT02,CP02"]
                  # additional_dependencies: ['<dbt-adapter>', 'sqlfluff-templater-dbt']

When trying to use the dbt templater <`dbt-project-configuration`>, uncomment the additional_dependencies to install the extras. This is equivalent to running pip install <dbt-adapter> sqlfluff-templater-dbt.

You can specify the version of dbt-adapter used in pre-commit <https://pre-commit.com/>, for example:

            additional_dependencies : ['dbt-bigquery==1.0.0', 'sqlfluff-templater-dbt']

See the list of available dbt-adapters <https://docs.getdbt.com/docs/available-adapters>.

Note that you can pass the same arguments available through the CLI using args:.

  Ignoring files while using pre-commit

Under the hood, pre-commit <https://pre-commit.com/> works by passing specific files to SQLFluff. For example, if the only two files that are modified in your commit are file_a.sql and file_b.sql, then the command which is called in the background is sqlfluff lint file_a.sql file_b.sql. While this is efficient, it does produce some unwanted noise when also using .sqlfluffignore <#sqlfluffignore>. This is because SQLFluff is designed to allow users to override an ignore configuration by passing the name of the file directly. This makes a lot of sense in a CLI context, but less so in the context of being invoked by pre-commit <https://pre-commit.com/>.

To avoid noisy logs when using both pre-commit <https://pre-commit.com/> and .sqlfluffignore <#sqlfluffignore>, we recommend also setting the exclude argument in your .pre-commit-config.yaml file (either the top level config <https://pre-commit.com/#top_level-exclude> or the hook specific config <https://pre-commit.com/#config-exclude>). This will prevent files matching the given pattern being passed to SQLFluff and so silence any warnings about the .sqlfluffignore <#sqlfluffignore> being overridden.

  Using GitHub Actions <https://github.com/features/actions> to Annotate PRs

There are two way to utilize SQLFluff to annotate Github PRs.

    1.  When sqlfluff lint is run with the --format github-annotation-native option, it produces output formatted as Github workflow commands <https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#setting-a-notice-message> which are converted into pull request annotations by Github.

    2.  When sqlfluff lint is run with the --format github-annotation option, it produces output compatible with this action from yuzutech <https://github.com/yuzutech/annotations-action>. Which uses Github API to annotate the SQL in GitHub pull requests.

Warning:
        At present (December 2023), limitations put in place by Github mean that only the first 10 annotations will be displayed if the first option (using github-annotation-native) is used. This is a not something that SQLFluff can control itself and so we currently recommend using the the second option above and the action from yuzutech <https://github.com/yuzutech/annotations-action>.

        There is an open feature request <https://github.com/orgs/community/discussions/68471> for GitHub Actions which you can track to follow this issue.

For more information and examples on using SQLFluff in GitHub Actions, see the sqlfluff-github-actions repository <https://github.com/sqlfluff/sqlfluff-github-actions>.

  Reference

  Dialects Reference

SQLFluff is designed to be flexible in supporting a variety of dialects. Not all potential dialects are supported so far, but several have been implemented by the community. Below are a list of the currently available dialects. Each inherits from another, up to the root ansi dialect.

For a canonical list of supported dialects, run the sqlfluff dialects command, which will output a list of the current dialects available on your installation of SQLFluff.

Note:
        For technical users looking to add new dialects or add new features to existing ones, the dependent nature of how dialects have been implemented is to try and reduce the amount of repetition in how different elements are defined. As an example, when we say that the AWS Redshift dialect inherits from the PostgreSQL dialect this is not because there is an agreement between those projects which means that features in one must end up in the other, but that the design of the AWS Redshift dialect was heavily inspired by the postgres dialect and therefore when defining the dialect within sqlfuff it makes sense to use PostgreSQL as a starting point rather than starting from scratch.

        Consider when adding new features to a dialect:

            •   Should I be adding it just to this dialect, or adding it to a parent dialect?

            •   If I'm creating a new dialect, which dialect would be best to inherit from?

            •   Will the feature I'm adding break any downstream dependencies within dialects which inherit from this one?

  ANSI

Label: ansi

Default Casing: UPPERCASE

Quotes: String Literals: '', Identifiers: ""

This is the base dialect which holds most of the definitions of common SQL commands and structures. If the dialect which you're actually using isn't specifically implemented by SQLFluff, using this dialect is a good place to start.

This dialect doesn't intend to be brutal in adhering to (and only to) the ANSI SQL spec (mostly because ANSI charges for access to that spec). It aims to be a representation of vanilla SQL before any other project adds their spin to it, and so may contain a slightly wider set of functions than actually available in true ANSI SQL.

  AWS Athena

Label: athena

Inherits from: ANSI

Default Casing: lowercase

Quotes: String Literals: '', "" or , Identifiers: "" or

The dialect for Athena <https://aws.amazon.com/athena/> on Amazon Web Services (AWS).

  Google BigQuery

Label: bigquery

Inherits from: ANSI

Default Casing: BigQuery resolves unquoted column identifiers case insensitively, and table/dataset identifiers case sensitively (by default, unless is_case_insensitive is set for the latter). Unless specified, columns are returned in the case which they were defined in, which means columns can be re-cased in the result set without aliasing e.g. if a table is defined with CREATE TEMPORARY TABLE foo (col1 int, COL2 int) then SELECT * FROM foo returns col1 and COL2 in the result, but SELECT COL1, col2 FROM foo returns COL1 and col2 in the result.

Quotes: String Literals: '', "", @ or @@ (with the quoted options, also supporting variants prefixes with r/R (for raw/regex expressions) or b/B (for byte strings)), Identifiers: "" or .

The dialect for BigQuery <https://cloud.google.com/bigquery/> on Google Cloud Platform (GCP).

  ClickHouse

Label: clickhouse

Inherits from: ANSI

Default Casing: Clickhouse is case sensitive throughout, regardless of quoting. An unquoted reference to an object using the wrong case will raise an UNKNOWN_IDENTIFIER error.

Quotes: String Literals: '', Identifiers: "" or . Note as above, that because identifiers are always resolved case sensitively, the only reason for quoting identifiers is when they contain invalid characters or reserved keywords.

The dialect for ClickHouse <https://clickhouse.com/>.

  Databricks

Label: databricks

Inherits from: Apache Spark SQL

The dialect for Databricks <https://databricks.com/>.

  IBM Db2

Label: db2

Inherits from: ANSI

The dialect for IBM Db2 <https://www.ibm.com/analytics/db2>.

  Apache Doris

Label: doris

Inherits from: MySQL

Default Casing: lowercase
        Quotes: String Literals: '', ", Identifiers: .

        The dialect for Apache Doris <https://doris.apache.org/>.

  DuckDB

Label: duckdb

Inherits from: PostgreSQL

Default Casing: DuckDB stores all identifiers in the case they were defined, however all identifier resolution is case-insensitive (when unquoted, and more unusually, also when quoted). See the DuckDB Identifiers Documentation <https://duckdb.org/docs/sql/dialect/keywords_and_identifiers> for more details.

Quotes: String Literals: '', Identifiers: "" or ''

The dialect for DuckDB <https://duckdb.org/>.

  Exasol

Label: exasol

Inherits from: ANSI

The dialect for Exasol <https://www.exasol.com/>.

  Apache Flink SQL

Label: flink

Inherits from: ANSI

Default Casing: FlinkSQL is case insensitive with both quoted and unquoted identifiers.

Quotes: String Literals: ' (single quotes), Identifiers: .

The dialect for Apache Flink SQL <https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/#noqa:E501>. This dialect includes FlinkSQL-specific syntax for stream processing, table operations, and connector configurations.

FlinkSQL supports advanced features like: - ROW data types for complex nested structures - Table connectors (Kafka, BigQuery, etc.) - Watermark definitions for event time processing - Computed columns and metadata columns - Temporal table functions

  Greenplum

Label: greenplum

Inherits from: PostgreSQL

The dialect for Greenplum <https://www.greenplum.org/>.

  Apache Hive

Label: hive

Inherits from: ANSI

The dialect for Apache Hive <https://hive.apache.org/>.

  Apache Impala

Label: impala

Inherits from: Apache Hive

The dialect for Apache Impala <https://impala.apache.org/>.

  MariaDB

Label: mariadb

Inherits from: MySQL

Default Casing: lowercase

Quotes: String Literals: '', "" or @, Identifiers: .

The dialect for MariaDB <https://www.mariadb.org/>.

  Materialize

Label: materialize

Inherits from: PostgreSQL

The dialect for Materialize <https://materialize.com/>.

  MySQL

Label: mysql

Inherits from: ANSI

Default Casing: lowercase

Quotes: String Literals: '', "" or @, Identifiers: .

The dialect for MySQL <https://www.mysql.com/>.

  Oracle

Label: oracle

Inherits from: ANSI

The dialect for Oracle <https://www.oracle.com/database/technologies/appdev/sql.html> SQL. Note: this does include PL/SQL.

  PostgreSQL

Label: postgres

Inherits from: ANSI

Default Casing: lowercase

Quotes: String Literals: '', Identifiers: "".

This is based around the PostgreSQL spec <https://www.postgresql.org/docs/current/reference.html>. Many other SQL dialects are often based on the PostreSQL syntax. If you're running an unsupported dialect, then this is often the dialect to use (until someone makes a specific dialect).

  AWS Redshift

Label: redshift

Inherits from: PostgreSQL

Default Casing: lowercase (unless configured to be case sensitive with all identifiers using the enable_case_sensitive_identifier configuration value, see the Redshift Names & Identifiers Docs <https://docs.aws.amazon.com/redshift/latest/dg/r_names.html>).

Quotes: String Literals: '', Identifiers: "".

The dialect for Redshift <https://aws.amazon.com/redshift/> on Amazon Web Services (AWS).

  Snowflake

Label: snowflake

Inherits from: ANSI

Default Casing: UPPERCASE

Quotes: String Literals: '', Identifiers: ""

The dialect for Snowflake <https://docs.snowflake.com/en/sql-reference.html>, which has much of its syntax inherited from PostgreSQL.

  Salesforce Object Query Language (SOQL)

Label: soql

Inherits from: ANSI

The dialect for SOQL <https://developer.salesforce.com/docs/atlas.en-us.soql_sosl.meta/soql_sosl/sforce_api_calls_soql.htm> (Salesforce Object Query Language).

  Apache Spark SQL

Label: sparksql

Inherits from: ANSI

Default Casing: SparkSQL is case insensitive with both quoted and unquoted identifiers (_"delimited"_ identifiers in Spark terminology). See the Spark Identifiers <https://spark.apache.org/docs/latest/sql-ref-identifier.html> docs.

Quotes: String Literals: '' or "", Identifiers: .

The dialect for Apache Spark SQL <https://spark.apache.org/docs/latest/sql-ref.html>. This includes relevant syntax from Apache Hive for commands that permit Hive Format. Spark SQL extensions provided by the Delta Lake <https://docs.delta.io/latest/quick-start.html#set-up-apache-spark-with-delta-lake> project are also implemented in this dialect.

This implementation focuses on the Ansi Compliant Mode <https://spark.apache.org/docs/latest/sql-ref-ansi-compliance.html> introduced in Spark3, instead of being Hive Compliant. The introduction of ANSI Compliance provides better data quality and easier migration from traditional DBMS.

Versions of Spark prior to 3.x will only support the Hive dialect.

  SQLite

Label: sqlite

Inherits from: ANSI

Default Casing: Not specified in the docs, but through testing it appears that SQLite stores column names in whatever case they were defined, but is always case-insensitive when resolving those names.

Quotes: String Literals: '' (or "" if not otherwise resolved to an identifier), Identifiers: "", [] or . See the SQLite Keywords Docs <https://sqlite.org/lang_keywords.html> for more details.

The dialect for SQLite <https://www.sqlite.org/>.

  StarRocks

Label: starrocks

Inherits from: MySQL

Default Casing: lowercase
        Quotes: String Literals: '', ", Identifiers: .

        The dialect for StarRocks <https://www.starrocks.io/>.

  Teradata

Label: teradata

Inherits from: ANSI

The dialect for Teradata <https://www.teradata.co.uk/>.

  Trino

Label: trino

Inherits from: ANSI

Default Casing: lowercase, although the case of a reference is used in the result set column label. If a column is defined using CREATE TEMPORARY TABLE foo (COL1 int), then SELECT * FROM foo returns a column labelled col1, however SELECT COL1 FROM foo returns a column labelled COL1.

Quotes: String Literals: '', Identifiers: ""

The dialect for Trino <https://trino.io/docs/current/>.

  Microsoft T-SQL

Label: tsql

Inherits from: ANSI

The dialect for T-SQL <https://docs.microsoft.com/en-us/sql/t-sql/language-reference> (aka Transact-SQL).

  Vertica

Label: vertica

Inherits from: ANSI

The dialect for Vertica <https://www.vertica.com/documentation/vertica/all/>.

  Rules Reference

This page is an index of available rules which are bundled with SQLFluff.

    •   For information on how to configure which rules are enabled for your project see Enabling and Disabling Rules <#ruleselection>.

    •   If you just want to turn rules on or off for specific files, or specific sections of files, see Ignoring Errors & Files <#ignoreconfig>.

    •   For more information on how to configure the rules which you do enable see Rule Configuration <#ruleconfig>.

  Core Rules

Certain rules belong to the core rule group. In order for a rule to be designated as core, it must meet the following criteria:

    •   Stable

    •   Applies to most dialects

    •   Could detect a syntax issue

    •   Isn’t too opinionated toward one style (e.g. the dbt style guide)

Core rules can also make it easier to roll out SQLFluff to a team by only needing to follow a 'common sense' subset of rules initially, rather than spending time understanding and configuring all the rules, some of which your team may not necessarily agree with.

We believe teams will eventually want to enforce more than just the core rules, and we encourage everyone to explore all the rules and customize a rule set that best suites their organization.

See the Configuration <#config> section for more information on how to enable only core rules by default.

  Rule Index
Bundle | Rule Name | Code | Aliases
Aliasing bundle | aliasing.table | AL01 | L011
aliasing.column | AL02 | L012
aliasing.expression | AL03 | L013
aliasing.unique.table | AL04 | L020
aliasing.unused | AL05 | L025
aliasing.length | AL06 | L066
aliasing.forbid | AL07 | L031
aliasing.unique.column | AL08 |
aliasing.self_alias.column | AL09 |
Ambiguous bundle | ambiguous.distinct | AM01 | L021
ambiguous.union | AM02 | L033
ambiguous.order_by | AM03 | L037
ambiguous.column_count | AM04 | L044
ambiguous.join | AM05 | L051
ambiguous.column_references | AM06 | L054
ambiguous.set_columns | AM07 | L068
ambiguous.join_condition | AM08 |
ambiguous.order_by_limit | AM09 |
Capitalisation bundle | capitalisation.keywords | CP01 | L010
capitalisation.identifiers | CP02 | L014
capitalisation.functions | CP03 | L030
capitalisation.literals | CP04 | L040
capitalisation.types | CP05 | L063
Convention bundle | convention.not_equal | CV01 | L061
convention.coalesce | CV02 | L060
convention.select_trailing_comma | CV03 | L038
convention.count_rows | CV04 | L047
convention.is_null | CV05 | L049
convention.terminator | CV06 | L052
convention.statement_brackets | CV07 | L053
convention.left_join | CV08 | L055
convention.blocked_words | CV09 | L062
convention.quoted_literals | CV10 | L064
convention.casting_style | CV11 | L067
convention.join_condition | CV12 |
Jinja bundle | jinja.padding | JJ01 | L046
Layout bundle | layout.spacing | LT01 | L001, L005, L006, L008, L023, L024, L039, L048, L071
layout.indent | LT02 | L002, L003, L004
layout.operators | LT03 | L007
layout.commas | LT04 | L019
layout.long_lines | LT05 | L016
layout.functions | LT06 | L017
layout.cte_bracket | LT07 | L018
layout.cte_newline | LT08 | L022
layout.select_targets | LT09 | L036
layout.select_modifiers | LT10 | L041
layout.set_operators | LT11 | L065
layout.end_of_file | LT12 | L009, layout.end-of-file
layout.start_of_file | LT13 | L050
layout.keyword_newline | LT14 |
layout.newlines | LT15 |
References bundle | references.from | RF01 | L026
references.qualification | RF02 | L027
references.consistent | RF03 | L028
references.keywords | RF04 | L029
references.special_chars | RF05 | L057
references.quoting | RF06 | L059
Structure bundle | structure.else_null | ST01 | L035
structure.simple_case | ST02 | L043
structure.unused_cte | ST03 | L045
structure.nested_case | ST04 | L058
structure.subquery | ST05 | L042
structure.column_order | ST06 | L034
structure.using | ST07 | L032
structure.distinct | ST08 | L015
structure.join_condition_order | ST09 |
structure.constant_expression | ST10 |
structure.unused_join | ST11 |
structure.consecutive_semicolons | ST12 |
TSQL bundle | tsql.sp_prefix | TQ01 | L056

  Aliasing bundle

    Code AL01

    Rule aliasing.table

  Implicit/explicit aliasing of table.

Aliasing of table to follow preference (requiring an explicit AS is the default).

Anti-pattern

In this example, the alias voo is implicit.

            SELECT
                voo.a
            FROM foo voo

Best practice

Add AS to make it explicit.

            SELECT
                voo.a
            FROM foo AS voo

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``aliasing.table``

            **Aliases**: ``L011``

            **Groups**: ``all``, ``aliasing``

            **Configuration**

            * ``aliasing``: Should alias have an explicit AS or is implicit aliasing required? Must be one of ``['implicit', 'explicit']``.

    Code AL02

    Rule aliasing.column

  Implicit/explicit aliasing of columns.

Aliasing of columns to follow preference (explicit using an AS clause is default).

Anti-pattern

In this example, the alias for column a is implicit.

            SELECT
                a alias_col
            FROM foo

Best practice

Add AS to make it explicit.

            SELECT
                a AS alias_col
            FROM foo

            **Name**: ``aliasing.column``

            **Aliases**: ``L012``

            **Groups**: ``all``, ``core``, ``aliasing``

            **Configuration**

            * ``aliasing``: Should alias have an explicit AS or is implicit aliasing required? Must be one of ``['implicit', 'explicit']``.

    Code AL03

    Rule aliasing.expression

  Column expression without alias. Use explicit AS clause.

Anti-pattern

In this example, there is no alias for both sums.

            SELECT
                sum(a),
                sum(b)
            FROM foo

Best practice

Add aliases.

            SELECT
                sum(a) AS a_sum,
                sum(b) AS b_sum
            FROM foo

            **Name**: ``aliasing.expression``

            **Aliases**: ``L013``

            **Groups**: ``all``, ``core``, ``aliasing``

            **Configuration**

            * ``allow_scalar``: Whether or not to allow a single element in the  select clause to be without an alias. Must be one of ``[True, False]``.

    Code AL04

    Rule aliasing.unique.table

  Table aliases should be unique within each clause.

Reusing table aliases is very likely a coding error.

Anti-pattern

In this example, the alias t is reused for two different tables:

            SELECT
                t.a,
                t.b
            FROM foo AS t, bar AS t

            -- This can also happen when using schemas where the
            -- implicit alias is the table name:

            SELECT
                a,
                b
            FROM
                2020.foo,
                2021.foo

Best practice

Make all tables have a unique alias.

            SELECT
                f.a,
                b.b
            FROM foo AS f, bar AS b

            -- Also use explicit aliases when referencing two tables
            -- with the same name from two different schemas.

            SELECT
                f1.a,
                f2.b
            FROM
                2020.foo AS f1,
                2021.foo AS f2

            **Name**: ``aliasing.unique.table``

            **Aliases**: ``L020``

            **Groups**: ``all``, ``core``, ``aliasing``, ``aliasing.unique``

    Code AL05

    Rule aliasing.unused

  Tables should not be aliased if that alias is not used.

Anti-pattern

            SELECT
                a
            FROM foo AS zoo

Best practice

Use the alias or remove it. An unused alias makes code harder to read without changing any functionality.

            SELECT
                zoo.a
            FROM foo AS zoo

            -- Alternatively...

            SELECT
                a
            FROM foo

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``aliasing.unused``

            **Aliases**: ``L025``

            **Groups**: ``all``, ``core``, ``aliasing``

            **Configuration**

            * ``alias_case_check``: How to handle comparison casefolding in an alias. Must be one of ``['dialect', 'case_insensitive', 'quoted_cs_naked_upper', 'quoted_cs_naked_lower', 'case_sensitive']``.

    Code AL06

    Rule aliasing.length

  Enforce table alias lengths in from clauses and join conditions.

Anti-pattern

In this example, alias o is used for the orders table.

            SELECT
                SUM(o.amount) as order_amount,
            FROM orders as o

Best practice

Avoid aliases. Avoid short aliases when aliases are necessary.

See also: AL07.

            SELECT
                SUM(orders.amount) as order_amount,
            FROM orders

            SELECT
                replacement_orders.amount,
                previous_orders.amount
            FROM
                orders AS replacement_orders
            JOIN
                orders AS previous_orders
                ON replacement_orders.id = previous_orders.replacement_id

            **Name**: ``aliasing.length``

            **Aliases**: ``L066``

            **Groups**: ``all``, ``core``, ``aliasing``

            **Configuration**

            * ``max_alias_length``: The maximum length of an alias to allow without raising a violation. Must be one of ``range(0, 1000)``.
            * ``min_alias_length``: The minimum length of an alias to allow without raising a violation. Must be one of ``range(0, 1000)``.

    Code AL07

    Rule aliasing.forbid

  Avoid table aliases in from clauses and join conditions.

Note:
        This rule was taken from the dbt Style Guide <https://github.com/dbt-labs/corp/blob/main/dbt_style_guide.md> which notes that:
                Avoid table aliases in join conditions (especially initialisms) - it's harder to understand what the table called "c" is compared to "customers".

        This rule is controversial and for many larger databases avoiding alias is neither realistic nor desirable. In particular for BigQuery due to the complexity of backtick requirements and determining whether a name refers to a project or dataset so automated fixes can potentially break working SQL code. For most users AL06 is likely a more appropriate linting rule to drive a sensible behaviour around aliasing.

        The stricter treatment of aliases in this rule may be useful for more focused projects, or temporarily as a refactoring tool because the fix routine of the rule can remove aliases.

        This rule is disabled by default for all dialects it can be enabled with the force_enable = True flag.

Anti-pattern

In this example, alias o is used for the orders table, and c is used for customers table.

            SELECT
                COUNT(o.customer_id) as order_amount,
                c.name
            FROM orders as o
            JOIN customers as c on o.id = c.user_id

Best practice

Avoid aliases.

            SELECT
                COUNT(orders.customer_id) as order_amount,
                customers.name
            FROM orders
            JOIN customers on orders.id = customers.user_id

            -- Self-join will not raise issue

            SELECT
                table1.a,
                table_alias.b,
            FROM
                table1
                LEFT JOIN table1 AS table_alias ON
                    table1.foreign_key = table_alias.foreign_key

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``aliasing.forbid``

            **Aliases**: ``L031``

            **Groups**: ``all``, ``aliasing``

            **Configuration**

            * ``force_enable``: Run this rule even for dialects where this rule is disabled by default. Must be one of ``[True, False]``.

    Code AL08

    Rule aliasing.unique.column

  Column aliases should be unique within each clause.

Reusing column aliases is very likely a coding error. Note that while in many dialects, quoting an identifier makes it case-sensitive this rule always compares in a case-insensitive way. This is because columns with the same name, but different case, are still confusing and potentially ambiguous to other readers.

In situations where it is necessary to have columns with the same name (whether they differ in case or not) we recommend disabling this rule for either just the line, or the whole file.

Anti-pattern

In this example, the alias foo is reused for two different columns:

            SELECT
                a as foo,
                b as foo
            FROM tbl;

            -- This can also happen when referencing the same column
            -- column twice, or aliasing an expression to the same
            -- name as a column:

            SELECT
                foo,
                foo,
                a as foo
            FROM tbl;

Best practice

Make all columns have a unique alias.

            SELECT
                a as foo,
                b as bar
            FROM tbl;

            -- Avoid also using the same column twice unless aliased:

            SELECT
                foo as foo1,
                foo as foo2,
                a as foo3
            FROM tbl;

            **Name**: ``aliasing.unique.column``

            **Groups**: ``all``, ``core``, ``aliasing``, ``aliasing.unique``

    Code AL09

    Rule aliasing.self_alias.column

  Column aliases should not alias to itself, i.e. self-alias.

Renaming the column to itself is a redundant piece of SQL, which doesn't affect its functionality. This rule only applies when aliasing to an exact copy of the column reference (e.g. foo as foo or "BAR" as "BAR", see note below on more complex examples). Aliases which effectively change the casing of an identifier are still allowed.

Note:
        This rule works in conjunction with references.quoting (RF06) and capitalisation.identifiers (CP02) to handle self aliases with mixed quoting and casing. In the situation that these two rules are not enabled then this rule will only fix the strict case where the quoting and casing of the alias and reference are the same.

        If those two rules are enabled, the fixes applied may result in a situation where this rule can kick in as a secondary effect. For example this Snowflake <#snowflake-dialect-ref> query:

                    -- Original Query. AL09 will not trigger because casing and
                    -- quoting are different. RF06 will however fix the unnecessary
                    -- quoting of "COL".
                    SELECT "COL" AS col FROM table;
                    -- After RF06, the query will look like this, at which point
                    -- CP02 will see the inconsistent capitalisation. Depending
                    -- on the configuration it will change one of the identifiers.
                    -- Let's assume the default configuration of "consistent".
                    SELECT COL AS col FROM table;
                    -- After CP02, the alias and the reference will be the same
                    -- and at this point AL09 can take over and remove the alias.
                    SELECT COL AS COL FROM table;
                    -- ..resulting in:
                    SELECT COL FROM table;

        This interdependence between the rules, and the configuration options offered by each one means a variety of outcomes can be achieved by enabling and disabling each one. See Enabling and Disabling Rules <#ruleselection> and Rule Configuration <#ruleconfig> for more details.

Anti-pattern

Aliasing the column to itself, where not necessary for changing the case of an identifier.

            SELECT
                col AS col,
                "Col" AS "Col",
                COL AS col
            FROM table;

Best practice

Not to use alias to rename the column to its original name. Self-aliasing leads to redundant code without changing any functionality, unless used to effectively change the case of the identifier.

            SELECT
                col,
                "Col"
                COL,
            FROM table;

            -- Re-casing aliasing is still allowed where necessary, i.e.
            SELECT
                col as "Col",
                "col" as "COL"
            FROM table;

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``aliasing.self_alias.column``

            **Groups**: ``all``, ``core``, ``aliasing``

  Ambiguous bundle

    Code AM01

    Rule ambiguous.distinct

  Ambiguous use of DISTINCT in a SELECT statement with GROUP BY.

When using GROUP BY a DISTINCT` clause should not be necessary as every non-distinct SELECT clause must be included in the GROUP BY clause.

Anti-pattern

DISTINCT and GROUP BY are conflicting.

            SELECT DISTINCT
                a
            FROM foo
            GROUP BY a

Best practice

Remove DISTINCT or GROUP BY. In our case, removing GROUP BY is better.

            SELECT DISTINCT
                a
            FROM foo

            **Name**: ``ambiguous.distinct``

            **Aliases**: ``L021``

            **Groups**: ``all``, ``core``, ``ambiguous``

    Code AM02

    Rule ambiguous.union

  UNION [DISTINCT|ALL] is preferred over just UNION.

Note:
        This rule is only enabled for dialects that support UNION and UNION DISTINCT (ansi, bigquery, clickhouse, databricks, db2, hive, mysql, redshift, snowflake, and trino).

Anti-pattern

In this example, UNION DISTINCT should be preferred over UNION, because explicit is better than implicit.

            SELECT a, b FROM table_1
            UNION
            SELECT a, b FROM table_2

Best practice

Specify DISTINCT or ALL after UNION (note that DISTINCT is the default behavior).

            SELECT a, b FROM table_1
            UNION DISTINCT
            SELECT a, b FROM table_2

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``ambiguous.union``

            **Aliases**: ``L033``

            **Groups**: ``all``, ``core``, ``ambiguous``

    Code AM03

    Rule ambiguous.order_by

  Ambiguous ordering directions for columns in order by clause.

Anti-pattern

            SELECT
                a, b
            FROM foo
            ORDER BY a, b DESC

Best practice

If any columns in the ORDER BY clause specify ASC or DESC, they should all do so.

            SELECT
                a, b
            FROM foo
            ORDER BY a ASC, b DESC

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``ambiguous.order_by``

            **Aliases**: ``L037``

            **Groups**: ``all``, ``ambiguous``

    Code AM04

    Rule ambiguous.column_count

  Query produces an unknown number of result columns.

Anti-pattern

Querying all columns using * produces a query result where the number or ordering of columns changes if the upstream table's schema changes. This should generally be avoided because it can cause slow performance, cause important schema changes to go undetected, or break production code. For example:

    •   If a query does SELECT t.* and is expected to return columns a, b, and c, the actual columns returned will be wrong/different if columns are added to or deleted from the input table.

    •   UNION and DIFFERENCE clauses require the inputs have the same number of columns (and compatible types).

    •   JOIN queries may break due to new column name conflicts, e.g. the query references a column c which initially existed in only one input table but a column of the same name is added to another table.

    •   CREATE TABLE (<<column schema>>) AS SELECT *

            WITH cte AS (
                SELECT * FROM foo
            )

            SELECT * FROM cte
            UNION
            SELECT a, b FROM t

Best practice

Somewhere along the "path" to the source data, specify columns explicitly.

            WITH cte AS (
                SELECT * FROM foo
            )

            SELECT a, b FROM cte
            UNION
            SELECT a, b FROM t

            **Name**: ``ambiguous.column_count``

            **Aliases**: ``L044``

            **Groups**: ``all``, ``ambiguous``

    Code AM05

    Rule ambiguous.join

  Join clauses should be fully qualified.

By default this rule is configured to enforce fully qualified INNER JOIN clauses, but not [LEFT/RIGHT/FULL] OUTER JOIN. If you prefer a stricter lint then this is configurable.

Anti-pattern

A join is used without specifying the kind of join.

             SELECT
                 foo
             FROM bar
             JOIN baz;

Best practice

Use INNER JOIN rather than JOIN.

             SELECT
                 foo
             FROM bar
             INNER JOIN baz;

             This rule is ``sqlfluff fix`` compatible.

             **Name**: ``ambiguous.join``

             **Aliases**: ``L051``

             **Groups**: ``all``, ``ambiguous``

             **Configuration**

             * ``fully_qualify_join_types``: Which types of JOIN clauses should be fully qualified? Must be one of ``['inner', 'outer', 'both']``.

    Code AM06

    Rule ambiguous.column_references

  Inconsistent column references in GROUP BY/ORDER BY clauses.

Note:
        ORDER BY clauses from WINDOW clauses are ignored by this rule.

Anti-pattern

A mix of implicit and explicit column references are used in a GROUP BY clause.

             SELECT
                 foo,
                 bar,
                 sum(baz) AS sum_value
             FROM fake_table
             GROUP BY
                 foo, 2;

             -- The same also applies to column
             -- references in ORDER BY clauses.

             SELECT
                 foo,
                 bar
             FROM fake_table
             ORDER BY
                 1, bar;

Best practice

Reference all GROUP BY/ORDER BY columns either by name or by position.

             -- GROUP BY: Explicit
             SELECT
                 foo,
                 bar,
                 sum(baz) AS sum_value
             FROM fake_table
             GROUP BY
                 foo, bar;

             -- ORDER BY: Explicit
             SELECT
                 foo,
                 bar
             FROM fake_table
             ORDER BY
                 foo, bar;

             -- GROUP BY: Implicit
             SELECT
                 foo,
                 bar,
                 sum(baz) AS sum_value
             FROM fake_table
             GROUP BY
                 1, 2;

             -- ORDER BY: Implicit
             SELECT
                 foo,
                 bar
             FROM fake_table
             ORDER BY
                 1, 2;

             **Name**: ``ambiguous.column_references``

             **Aliases**: ``L054``

             **Groups**: ``all``, ``core``, ``ambiguous``

             **Configuration**

             * ``group_by_and_order_by_style``: The expectation for using explicit column name references or implicit positional references. Must be one of ``['consistent', 'implicit', 'explicit']``.

    Code AM07

    Rule ambiguous.set_columns

  Queries within set query produce different numbers of columns.

Anti-pattern

When writing set expressions, all queries must return the same number of columns.

            WITH cte AS (
                SELECT
                    a,
                    b
                FROM foo
            )
            SELECT * FROM cte
            UNION
            SELECT
                c,
                d,
                e
             FROM t

Best practice

Always specify columns when writing set queries and ensure that they all seleect same number of columns

            WITH cte AS (
                SELECT a, b FROM foo
            )
            SELECT
                a,
                b
            FROM cte
            UNION
            SELECT
                c,
                d
            FROM t

            **Name**: ``ambiguous.set_columns``

            **Aliases**: ``L068``

            **Groups**: ``all``, ``ambiguous``

    Code AM08

    Rule ambiguous.join_condition

  Implicit cross join detected.

Anti-pattern

Cross joins are valid, but rare in the wild - and more often created by mistake than on purpose. This rule catches situations where a cross join has been specified, but not explicitly and so the risk of a mistaken cross join is highly likely.

             SELECT
                 foo
             FROM bar
             JOIN baz;

Best practice

Use CROSS JOIN.

             SELECT
                 foo
             FROM bar
             CROSS JOIN baz;

             This rule is ``sqlfluff fix`` compatible.

             **Name**: ``ambiguous.join_condition``

             **Groups**: ``all``, ``ambiguous``

    Code AM09

    Rule ambiguous.order_by_limit

  Use of LIMIT and OFFSET without ORDER BY may lead to non-deterministic results.

When using LIMIT or OFFSET, it's generally recommended to include an ORDER BY clause to ensure deterministic results.

Anti-pattern

The following query has LIMIT and OFFSET without ORDER BY, which may return different results in successive executions.

            SELECT *
            FROM foo
            LIMIT 10 OFFSET 5;

Best practice

Include an ORDER BY clause:

            SELECT *
            FROM foo
            ORDER BY id
            LIMIT 10 OFFSET 5;

            **Name**: ``ambiguous.order_by_limit``

            **Groups**: ``all``, ``ambiguous``

  Capitalisation bundle

    Code CP01

    Rule capitalisation.keywords

  Inconsistent capitalisation of keywords.

Anti-pattern

In this example, select is in lower-case whereas FROM is in upper-case.

            select
                a
            FROM foo

Best practice

Make all keywords either in upper-case or in lower-case.

            SELECT
                a
            FROM foo

            -- Also good

            select
                a
            from foo

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``capitalisation.keywords``

            **Aliases**: ``L010``

            **Groups**: ``all``, ``core``, ``capitalisation``

            **Configuration**

            * ``capitalisation_policy``: The capitalisation policy to enforce. Must be one of ``['consistent', 'upper', 'lower', 'capitalise']``.
            * ``ignore_words``: Comma separated list of words to ignore from rule.
            * ``ignore_words_regex``: Words to ignore from rule if they are a partial match for the regular expression. To ignore only full matches you can use ``^`` (beginning of text) and ``$`` (end of text). Due to regular expression operator precedence, it is good practice to use parentheses around everything between ``^`` and ``$``.

    Code CP02

    Rule capitalisation.identifiers

  Inconsistent capitalisation of unquoted identifiers.

This rule applies to all unquoted identifiers, whether references or aliases, and whether they refer to columns or other objects (such as tables or schemas).

Note:
        In most dialects, unquoted identifiers are treated as case-insensitive and so the fixes proposed by this rule do not change the interpretation of the query. HOWEVER, some databases (notably Google BigQuery <#bigquery-dialect-ref>, Trino <#trino-dialect-ref> and ClickHouse <#clickhouse-dialect-ref>) do take the casing of unquoted identifiers into account when determining the casing of the column heading in the result.

        As this feature is only present in a few dialects, and not widely understood by users, we regard it as an antipattern. It is more widely understood that if the case of an identifier matters, then it should be quoted. If you, or your organisation, do wish to rely on this feature, we recommend that you disabled this rule (see Enabling and Disabling Rules <#ruleselection>).

Anti-pattern

In this example, unquoted identifier a is in lower-case but B is in upper-case.

            select
                a,
                B
            from foo

In this more complicated example, there are a mix of capitalisations in both reference and aliases of columns and tables. That inconsistency is acceptable when those identifiers are quoted, but not when unquoted.

            select
                col_1 + Col_2 as COL_3,
                "COL_4" as Col_5
            from Foo as BAR

Best practice

Ensure all unquoted identifiers are either in upper-case or in lower-case.

            select
                a,
                b
            from foo;

            -- ...also good...

            select
                A,
                B
            from foo;

            --- ...or for comparison with our more complex example, this too:

            select
                col_1 + col_2 as col_3,
                "COL_4" as col_5
            from foo as bar

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``capitalisation.identifiers``

            **Aliases**: ``L014``

            **Groups**: ``all``, ``core``, ``capitalisation``

            **Configuration**

            * ``extended_capitalisation_policy``: The capitalisation policy to enforce, extended with PascalCase, snake_case, and camelCase. This is separate from ``capitalisation_policy`` as it should not be applied to keywords.Camel, Pascal, and Snake will never be inferred when the policy is set to consistent. This is because snake can cause destructive changes to the identifier, and unlinted code is too easily mistaken for camel and pascal. If, when set to consistent, no consistent case is found, it will default to upper. Must be one of ``['consistent', 'upper', 'lower', 'pascal', 'capitalise', 'snake', 'camel']``.
            * ``ignore_words``: Comma separated list of words to ignore from rule.
            * ``ignore_words_regex``: Words to ignore from rule if they are a partial match for the regular expression. To ignore only full matches you can use ``^`` (beginning of text) and ``$`` (end of text). Due to regular expression operator precedence, it is good practice to use parentheses around everything between ``^`` and ``$``.
            * ``unquoted_identifiers_policy``: Types of unquoted identifiers to flag violations for. Must be one of ``['all', 'aliases', 'column_aliases', 'table_aliases']``.

    Code CP03

    Rule capitalisation.functions

  Inconsistent capitalisation of function names.

Anti-pattern

In this example, the two SUM functions don't have the same capitalisation.

            SELECT
                sum(a) AS aa,
                SUM(b) AS bb
            FROM foo

Best practice

Make the case consistent.

            SELECT
                sum(a) AS aa,
                sum(b) AS bb
            FROM foo

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``capitalisation.functions``

            **Aliases**: ``L030``

            **Groups**: ``all``, ``core``, ``capitalisation``

            **Configuration**

            * ``extended_capitalisation_policy``: The capitalisation policy to enforce, extended with PascalCase, snake_case, and camelCase. This is separate from ``capitalisation_policy`` as it should not be applied to keywords.Camel, Pascal, and Snake will never be inferred when the policy is set to consistent. This is because snake can cause destructive changes to the identifier, and unlinted code is too easily mistaken for camel and pascal. If, when set to consistent, no consistent case is found, it will default to upper. Must be one of ``['consistent', 'upper', 'lower', 'pascal', 'capitalise', 'snake', 'camel']``.
            * ``ignore_words``: Comma separated list of words to ignore from rule.
            * ``ignore_words_regex``: Words to ignore from rule if they are a partial match for the regular expression. To ignore only full matches you can use ``^`` (beginning of text) and ``$`` (end of text). Due to regular expression operator precedence, it is good practice to use parentheses around everything between ``^`` and ``$``.

    Code CP04

    Rule capitalisation.literals

  Inconsistent capitalisation of boolean/null literal.

Anti-pattern

In this example, null and false are in lower-case whereas TRUE is in upper-case.

            select
                a,
                null,
                TRUE,
                false
            from foo

Best practice

Ensure all literal null/true/false literals are consistently upper or lower case

            select
                a,
                NULL,
                TRUE,
                FALSE
            from foo

            -- Also good

            select
                a,
                null,
                true,
                false
            from foo

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``capitalisation.literals``

            **Aliases**: ``L040``

            **Groups**: ``all``, ``core``, ``capitalisation``

            **Configuration**

            * ``capitalisation_policy``: The capitalisation policy to enforce. Must be one of ``['consistent', 'upper', 'lower', 'capitalise']``.
            * ``ignore_words``: Comma separated list of words to ignore from rule.
            * ``ignore_words_regex``: Words to ignore from rule if they are a partial match for the regular expression. To ignore only full matches you can use ``^`` (beginning of text) and ``$`` (end of text). Due to regular expression operator precedence, it is good practice to use parentheses around everything between ``^`` and ``$``.

    Code CP05

    Rule capitalisation.types

  Inconsistent capitalisation of datatypes.

Anti-pattern

In this example, int and unsigned are in lower-case whereas VARCHAR is in upper-case.

            CREATE TABLE t (
                a int unsigned,
                b VARCHAR(15)
            );

Best practice

Ensure all datatypes are consistently upper or lower case

            CREATE TABLE t (
                a INT UNSIGNED,
                b VARCHAR(15)
            );

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``capitalisation.types``

            **Aliases**: ``L063``

            **Groups**: ``all``, ``core``, ``capitalisation``

            **Configuration**

            * ``extended_capitalisation_policy``: The capitalisation policy to enforce, extended with PascalCase, snake_case, and camelCase. This is separate from ``capitalisation_policy`` as it should not be applied to keywords.Camel, Pascal, and Snake will never be inferred when the policy is set to consistent. This is because snake can cause destructive changes to the identifier, and unlinted code is too easily mistaken for camel and pascal. If, when set to consistent, no consistent case is found, it will default to upper. Must be one of ``['consistent', 'upper', 'lower', 'pascal', 'capitalise', 'snake', 'camel']``.
            * ``ignore_words``: Comma separated list of words to ignore from rule.
            * ``ignore_words_regex``: Words to ignore from rule if they are a partial match for the regular expression. To ignore only full matches you can use ``^`` (beginning of text) and ``$`` (end of text). Due to regular expression operator precedence, it is good practice to use parentheses around everything between ``^`` and ``$``.

  Convention bundle

    Code CV01

    Rule convention.not_equal

  Consistent usage of != or <> for "not equal to" operator.

Anti-pattern

            SELECT * FROM X WHERE 1 <> 2 AND 3 != 4;

Best practice

Ensure all "not equal to" comparisons are consistent, not mixing != and <>.

            SELECT * FROM X WHERE 1 != 2 AND 3 != 4;

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``convention.not_equal``

            **Aliases**: ``L061``

            **Groups**: ``all``, ``convention``

            **Configuration**

            * ``preferred_not_equal_style``: The style for using not equal to operator. Defaults to ``consistent``. Must be one of ``['consistent', 'c_style', 'ansi']``.

    Code CV02

    Rule convention.coalesce

  Use COALESCE instead of IFNULL or NVL.

Anti-pattern

IFNULL or NVL are used to fill NULL values.

            SELECT ifnull(foo, 0) AS bar,
            FROM baz;

            SELECT nvl(foo, 0) AS bar,
            FROM baz;

Best practice

Use COALESCE instead. COALESCE is universally supported, whereas Redshift doesn't support IFNULL and BigQuery doesn't support NVL. Additionally, COALESCE is more flexible and accepts an arbitrary number of arguments.

            SELECT coalesce(foo, 0) AS bar,
            FROM baz;

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``convention.coalesce``

            **Aliases**: ``L060``

            **Groups**: ``all``, ``convention``

    Code CV03

    Rule convention.select_trailing_comma

  Trailing commas within select clause.

Note:
        For many database backends this is allowed. For some users this may be something they wish to enforce (in line with Python best practice). Many database backends regard this as a syntax error, and as such the SQLFluff default is to forbid trailing commas in the select clause.

Anti-pattern

            SELECT
                a,
                b,
            FROM foo

Best practice

            SELECT
                a,
                b
            FROM foo

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``convention.select_trailing_comma``

            **Aliases**: ``L038``

            **Groups**: ``all``, ``core``, ``convention``

            **Configuration**

            * ``select_clause_trailing_comma``: Should trailing commas within select clauses be required or forbidden? Must be one of ``['forbid', 'require']``.

    Code CV04

    Rule convention.count_rows

  Use consistent syntax to express "count number of rows".

    Note:
        If both prefer_count_1 and prefer_count_0 are set to true then prefer_count_1 has precedence.

COUNT(*), COUNT(1), and even COUNT(0) are equivalent syntaxes in many SQL engines due to optimizers interpreting these instructions as "count number of rows in result".

The ANSI-92 <http://msdn.microsoft.com/en-us/library/ms175997.aspx> spec mentions the COUNT(*) syntax specifically as having a special meaning:
        If COUNT(*) is specified, then the result is the cardinality of T.

So by default, SQLFluff enforces the consistent use of COUNT(*).

If the SQL engine you work with, or your team, prefers COUNT(1) or COUNT(0) over COUNT(*), you can configure this rule to consistently enforce your preference.

Anti-pattern

            select
                count(1)
            from table_a

Best practice

Use count(*) unless specified otherwise by config prefer_count_1, or prefer_count_0 as preferred.

            select
                count(*)
            from table_a

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``convention.count_rows``

            **Aliases**: ``L047``

            **Groups**: ``all``, ``core``, ``convention``

            **Configuration**

            * ``prefer_count_0``: Should count(0) be preferred over count(*) and count(1)? Must be one of ``[True, False]``.
            * ``prefer_count_1``: Should count(1) be preferred over count(*) and count(0)? Must be one of ``[True, False]``.

    Code CV05

    Rule convention.is_null

  Comparisons with NULL should use "IS" or "IS NOT".

Anti-pattern

In this example, the = operator is used to check for NULL values.

            SELECT
                a
            FROM foo
            WHERE a = NULL

Best practice

Use IS or IS NOT to check for NULL values.

            SELECT
                a
            FROM foo
            WHERE a IS NULL

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``convention.is_null``

            **Aliases**: ``L049``

            **Groups**: ``all``, ``core``, ``convention``

    Code CV06

    Rule convention.terminator

  Statements must end with a semi-colon.

Anti-pattern

A statement is not immediately terminated with a semi-colon. The • represents space.

             SELECT
                 a
             FROM foo

             ;

             SELECT
                 b
             FROM bar••;

Best practice

Immediately terminate the statement with a semi-colon.

             SELECT
                 a
             FROM foo;

             This rule is ``sqlfluff fix`` compatible.

             **Name**: ``convention.terminator``

             **Aliases**: ``L052``

             **Groups**: ``all``, ``convention``

             **Configuration**

             * ``multiline_newline``: Should semi-colons be placed on a new line after multi-line statements? Must be one of ``[True, False]``.
             * ``require_final_semicolon``: Should final semi-colons be required? (N.B. forcing trailing semi-colons is not recommended for dbt users as it can cause issues when wrapping the query within other SQL queries). Must be one of ``[True, False]``.

    Code CV07

    Rule convention.statement_brackets

  Top-level statements should not be wrapped in brackets.

Anti-pattern

A top-level statement is wrapped in brackets.

             (SELECT
                 foo
             FROM bar)

             -- This also applies to statements containing a sub-query.

             (SELECT
                 foo
             FROM (SELECT * FROM bar))

Best practice

Don't wrap top-level statements in brackets.

             SELECT
                 foo
             FROM bar

             -- Likewise for statements containing a sub-query.

             SELECT
                 foo
             FROM (SELECT * FROM bar)

             This rule is ``sqlfluff fix`` compatible.

             **Name**: ``convention.statement_brackets``

             **Aliases**: ``L053``

             **Groups**: ``all``, ``convention``

    Code CV08

    Rule convention.left_join

  Use LEFT JOIN instead of RIGHT JOIN.

Anti-pattern

RIGHT JOIN is used.

             SELECT
                 foo.col1,
                 bar.col2
             FROM foo
             RIGHT JOIN bar
                 ON foo.bar_id = bar.id;

Best practice

Refactor and use LEFT JOIN instead.

             SELECT
                 foo.col1,
                 bar.col2
             FROM bar
             LEFT JOIN foo
                 ON foo.bar_id = bar.id;

             **Name**: ``convention.left_join``

             **Aliases**: ``L055``

             **Groups**: ``all``, ``convention``

    Code CV09

    Rule convention.blocked_words

  Block a list of configurable words from being used.

This generic rule can be useful to prevent certain keywords, functions, or objects from being used. Only whole words can be blocked, not phrases, nor parts of words.

This block list is case insensitive.

Example use cases

    •   We prefer BOOL over BOOLEAN and there is no existing rule to enforce this. Until such a rule is written, we can add BOOLEAN to the deny list to cause a linting error to flag this.

    •   We have deprecated a schema/table/function and want to prevent it being used in future. We can add that to the denylist and then add a -- noqa: CV09 for the few exceptions that still need to be in the code base for now.

Anti-pattern

If the blocked_words config is set to deprecated_table,bool then the following will flag:

            SELECT * FROM deprecated_table WHERE 1 = 1;
            CREATE TABLE myschema.t1 (a BOOL);

Best practice

Do not used any blocked words:

            SELECT * FROM another_table WHERE 1 = 1;
            CREATE TABLE myschema.t1 (a BOOLEAN);

            **Name**: ``convention.blocked_words``

            **Aliases**: ``L062``

            **Groups**: ``all``, ``convention``

            **Configuration**

            * ``blocked_regex``: Optional, regex of blocked pattern which should not be used in statements.
            * ``blocked_words``: Optional, comma-separated list of blocked words which should not be used in statements.
            * ``match_source``: Optional, also match regex of blocked pattern before applying templating.

    Code CV10

    Rule convention.quoted_literals

  Consistent usage of preferred quotes for quoted literals.

Some databases allow quoted literals to use either single or double quotes. Prefer one type of quotes as specified in rule setting, falling back to alternate quotes to reduce the need for escapes.

Dollar-quoted raw strings are excluded from this rule, as they are mostly used for literal UDF Body definitions.

Note:
        This rule only checks quoted literals and not quoted identifiers as they often cannot interchange single and double quotes

        This rule is only enabled for dialects that allow single and double quotes for quoted literals (currently bigquery, databricks, hive, mysql, sparksql). It can be enabled for other dialects with the force_enable = True flag.

Anti-pattern

             select
                 "abc",
                 'abc',
                 "\"",
                 "abc" = 'abc'
             from foo

Best practice

Ensure all quoted literals use preferred quotes, unless escaping can be reduced by using alternate quotes.

             select
                 "abc",
                 "abc",
                 '"',
                 "abc" = "abc"
             from foo

             This rule is ``sqlfluff fix`` compatible.

             **Name**: ``convention.quoted_literals``

             **Aliases**: ``L064``

             **Groups**: ``all``, ``convention``

             **Configuration**

             * ``force_enable``: Run this rule even for dialects where this rule is disabled by default. Must be one of ``[True, False]``.
             * ``preferred_quoted_literal_style``: Preferred quoting style to use for the quoted literals. If set to ``consistent`` quoting style is derived from the first quoted literal in the file. Must be one of ``['consistent', 'single_quotes', 'double_quotes']``.

    Code CV11

    Rule convention.casting_style

  Enforce consistent type casting style.

Note:
        This is only compatible with 2-arguments CONVERT as some dialects allow an optional 3rd argument e.g TSQL, which cannot be rewritten into CAST. This rule is disabled by default for Teradata because it supports different type casting apart from CONVERT and :: e.g DATE '2007-01-01', '9999-12-31' (DATE).

Anti-pattern

Using mixture of CONVERT, :: and CAST when preferred_type_casting_style config is set to consistent (default).

            SELECT
                CONVERT(int, 1) AS bar,
                100::int::text,
                CAST(10 AS text) AS coo
            FROM foo;

Best practice

Use consistent type casting style.

            SELECT
                CAST(1 AS int) AS bar,
                CAST(CAST(100 AS int) AS text),
                CAST(10 AS text) AS coo
            FROM foo;

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``convention.casting_style``

            **Aliases**: ``L067``

            **Groups**: ``all``, ``convention``

            **Configuration**

            * ``preferred_type_casting_style``: The expectation for using sql type casting. Must be one of ``['consistent', 'shorthand', 'convert', 'cast']``.

    Code CV12

    Rule convention.join_condition

  Use JOIN ... ON ... instead of WHERE ... for join conditions.

Anti-pattern

Using WHERE clause for join conditions.

            SELECT
                foo.a
                , bar.b
            FROM foo
            JOIN bar
            WHERE foo.x = bar.y;

Best practice

Use JOIN ON clause for join condition.

            SELECT
                foo.a
                , bar.b
            FROM foo
            JOIN bar
            ON foo.x = bar.y;

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``convention.join_condition``

            **Groups**: ``all``, ``convention``

  Jinja bundle

    Code JJ01

    Rule jinja.padding

  Jinja tags should have a single whitespace on either side.

This rule is only active if the jinja templater (or one of it's subclasses, like the dbt templater) are used for the current file.

Anti-pattern

Jinja tags with either no whitespace or very long whitespace are hard to read.

             SELECT {{    a     }} from {{ref('foo')}}

Best practice

A single whitespace surrounding Jinja tags, alternatively longer gaps containing newlines are acceptable.

             SELECT {{ a }} from {{ ref('foo') }};
             SELECT {{ a }} from {{
                 ref('foo')
             }};

             This rule is ``sqlfluff fix`` compatible.

             **Name**: ``jinja.padding``

             **Aliases**: ``L046``

             **Groups**: ``all``, ``core``, ``jinja``

  Layout bundle

    Code LT01

    Rule layout.spacing

  Inappropriate Spacing.

This rule checks for an enforces the spacing as configured in Configuring Layout <#layoutconfig>. This includes excessive whitespace, trailing whitespace at the end of a line and also the wrong spacing between elements on the line. Because of this wide reach you may find that you wish to add specific configuration in your project to tweak how specific elements are treated. Rather than configuration on this specific rule, use the sqlfluff.layout section of your configuration file to customise how this rule operates.

The • character represents a space in the examples below.

Anti-pattern

            SELECT
                a,        b(c) as d••
            FROM foo••••
            JOIN bar USING(a)

Best practice

    •   Unless an indent or preceding a comment, whitespace should be a single space.

    •   There should also be no trailing whitespace at the ends of lines.

    •   There should be a space after USING so that it's not confused for a function.

            SELECT
                a, b(c) as d
            FROM foo
            JOIN bar USING (a)

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``layout.spacing``

            **Aliases**: ``L001``, ``L005``, ``L006``, ``L008``, ``L023``, ``L024``, ``L039``, ``L048``, ``L071``

            **Groups**: ``all``, ``core``, ``layout``

    Code LT02

    Rule layout.indent

  Incorrect Indentation.

Anti-pattern

The • character represents a space and the → character represents a tab. In this example, the third line contains five spaces instead of four and the second line contains two spaces and one tab.

             SELECT
             ••→a,
             •••••b
             FROM foo

Best practice

Change the indentation to use a multiple of four spaces. This example also assumes that the indent_unit config value is set to space. If it had instead been set to tab, then the indents would be tabs instead.

             SELECT
             ••••a,
             ••••b
             FROM foo

             This rule is ``sqlfluff fix`` compatible.

             **Name**: ``layout.indent``

             **Aliases**: ``L002``, ``L003``, ``L004``

             **Groups**: ``all``, ``core``, ``layout``

    Code LT03

    Rule layout.operators

  Operators should follow a standard for being before/after newlines.

The configuration for whether operators should be trailing or leading is part of Configuring Layout <#layoutconfig>. The default configuration is:

            [sqlfluff:layout:type:binary_operator]
            line_position = leading

            [sqlfluff:layout:type:comparison_operator]
            line_position = leading

Anti-pattern

In this example, if line_position = leading (or unspecified, as is the default), then the operator + should not be at the end of the second line.

            SELECT
                a +
                b
            FROM foo

Best practice

If line_position = leading (or unspecified, as this is the default), place the operator after the newline.

            SELECT
                a
                + b
            FROM foo

If line_position = trailing, place the operator before the newline.

            SELECT
                a +
                b
            FROM foo

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``layout.operators``

            **Aliases**: ``L007``

            **Groups**: ``all``, ``layout``

    Code LT04

    Rule layout.commas

  Leading/Trailing comma enforcement.

The configuration for whether operators should be trailing or leading is part of Configuring Layout <#layoutconfig>. The default configuration is:

            [sqlfluff:layout:type:comma]
            line_position = trailing

Anti-pattern

There is a mixture of leading and trailing commas.

            SELECT
                a
                , b,
                c
            FROM foo

Best practice

By default, SQLFluff prefers trailing commas. However it is configurable for leading commas. The chosen style must be used consistently throughout your SQL.

            SELECT
                a,
                b,
                c
            FROM foo

            -- Alternatively, set the configuration file to 'leading'
            -- and then the following would be acceptable:

            SELECT
                a
                , b
                , c
            FROM foo

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``layout.commas``

            **Aliases**: ``L019``

            **Groups**: ``all``, ``layout``

    Code LT05

    Rule layout.long_lines

  Line is too long.

This rule is sqlfluff fix compatible.

Name: layout.long_lines

Aliases: L016

Groups: all, core, layout

Configuration

    •   ignore_comment_clauses: Should comment clauses (e.g. column comments) be ignored when linting line lengths? Must be one of [True, False].

    •   ignore_comment_lines: Should lines that contain only whitespace and comments be ignored when linting line lengths? Must be one of [True, False].

    Code LT06

    Rule layout.functions

  Function name not immediately followed by parenthesis.

Anti-pattern

In this example, there is a space between the function and the parenthesis.

            SELECT
                sum (a)
            FROM foo

Best practice

Remove the space between the function and the parenthesis.

            SELECT
                sum(a)
            FROM foo

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``layout.functions``

            **Aliases**: ``L017``

            **Groups**: ``all``, ``core``, ``layout``

    Code LT07

    Rule layout.cte_bracket

  WITH clause closing bracket should be on a new line.

Anti-pattern

In this example, the closing bracket is on the same line as CTE.

             WITH zoo AS (
                 SELECT a FROM foo)

             SELECT * FROM zoo

Best practice

Move the closing bracket on a new line.

            WITH zoo AS (
                SELECT a FROM foo
            )

            SELECT * FROM zoo

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``layout.cte_bracket``

            **Aliases**: ``L018``

            **Groups**: ``all``, ``core``, ``layout``

    Code LT08

    Rule layout.cte_newline

  Blank line expected but not found after CTE closing bracket.

Anti-pattern

There is no blank line after the CTE closing bracket. In queries with many CTEs, this hinders readability.

            WITH plop AS (
                SELECT * FROM foo
            )
            SELECT a FROM plop

Best practice

Add a blank line.

            WITH plop AS (
                SELECT * FROM foo
            )

            SELECT a FROM plop

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``layout.cte_newline``

            **Aliases**: ``L022``

            **Groups**: ``all``, ``core``, ``layout``

    Code LT09

    Rule layout.select_targets

  Select targets should be on a new line unless there is only one select target.

Note:
        By default, a wildcard (e.g. SELECT *) is considered a single select target. If you want it to be treated as multiple select targets, configure wildcard_policy = multiple.

Anti-pattern

Multiple select targets on the same line.

            select a, b
            from foo;

            -- Single select target on its own line.

            SELECT
                a
            FROM foo;

Best practice

Multiple select targets each on their own line.

            select
                a,
                b
            from foo;

            -- Single select target on the same line as the ``SELECT``
            -- keyword.

            SELECT a
            FROM foo;

            -- When select targets span multiple lines, however they
            -- can still be on a new line.

            SELECT
                SUM(
                    1 + SUM(
                        2 + 3
                    )
                ) AS col
            FROM test_table;

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``layout.select_targets``

            **Aliases**: ``L036``

            **Groups**: ``all``, ``layout``

            **Configuration**

            * ``wildcard_policy``: Treatment of wildcards. Defaults to ``single``. Must be one of ``['single', 'multiple']``.

    Code LT10

    Rule layout.select_modifiers

  SELECT modifiers (e.g. DISTINCT) must be on the same line as SELECT.

Anti-pattern

            select
                distinct a,
                b
            from x

Best practice

            select distinct
                a,
                b
            from x

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``layout.select_modifiers``

            **Aliases**: ``L041``

            **Groups**: ``all``, ``core``, ``layout``

    Code LT11

    Rule layout.set_operators

  Set operators should be surrounded by newlines.

Anti-pattern

In this example, UNION ALL is not on a line itself.

            SELECT 'a' AS col UNION ALL
            SELECT 'b' AS col

Best practice

            SELECT 'a' AS col
            UNION ALL
            SELECT 'b' AS col

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``layout.set_operators``

            **Aliases**: ``L065``

            **Groups**: ``all``, ``core``, ``layout``

    Code LT12

    Rule layout.end_of_file

  Files must end with a single trailing newline.

Anti-pattern

The content in file does not end with a single trailing newline. The $ represents end of file.

             SELECT
                 a
             FROM foo$

             -- Ending on an indented line means there is no newline
             -- at the end of the file, the • represents space.

             SELECT
             ••••a
             FROM
             ••••foo
             ••••$

             -- Ending on a semi-colon means the last line is not a
             -- newline.

             SELECT
                 a
             FROM foo
             ;$

             -- Ending with multiple newlines.

             SELECT
                 a
             FROM foo

             $

Best practice

Add trailing newline to the end. The $ character represents end of file.

             SELECT
                 a
             FROM foo
             $

             -- Ensuring the last line is not indented so is just a
             -- newline.

             SELECT
             ••••a
             FROM
             ••••foo
             $

             -- Even when ending on a semi-colon, ensure there is a
             -- newline after.

             SELECT
                 a
             FROM foo
             ;
             $

             This rule is ``sqlfluff fix`` compatible.

             **Name**: ``layout.end_of_file``

             **Aliases**: ``L009``, ``layout.end-of-file``

             **Groups**: ``all``, ``core``, ``layout``

    Code LT13

    Rule layout.start_of_file

  Files must not begin with newlines or whitespace.

Anti-pattern

The file begins with newlines or whitespace. The ^ represents the beginning of the file.

             ^

             SELECT
                 a
             FROM foo

             -- Beginning on an indented line is also forbidden,
             -- (the • represents space).

             ••••SELECT
             ••••a
             FROM
             ••••foo

Best practice

Start file on either code or comment. (The ^ represents the beginning of the file.)

             ^SELECT
                 a
             FROM foo

             -- Including an initial block comment.

             ^/*
             This is a description of my SQL code.
             */
             SELECT
                 a
             FROM
                 foo

             -- Including an initial inline comment.

             ^--This is a description of my SQL code.
             SELECT
                 a
             FROM
                 foo

             This rule is ``sqlfluff fix`` compatible.

             **Name**: ``layout.start_of_file``

             **Aliases**: ``L050``

             **Groups**: ``all``, ``layout``

    Code LT14

    Rule layout.keyword_newline

  Keyword clauses should follow a standard for being before/after newlines.

Anti-pattern

In this example, the keyword are not at the beginning of or alone on the line.

            SELECT 'a' AS col FROM tab WHERE x = 4 ORDER BY y LIMIT 5

Best practice

            SELECT 'a' AS col
            FROM tab
            WHERE x = 4
            ORDER BY y
            LIMIT 5

            SELECT 'a' AS col
            FROM
                tab
            WHERE
                x = 4
            ORDER BY
                y
            LIMIT
                5

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``layout.keyword_newline``

            **Groups**: ``all``, ``layout``

    Code LT15

    Rule layout.newlines

  Too many consecutive blank lines.

Anti-pattern

In this example, the maximum number of empty lines inside a statement is set to 0.

            SELECT 'a' AS col
            FROM tab

            WHERE x = 4
            ORDER BY y

            LIMIT 5
            ;

Best practice

            SELECT 'a' AS col
            FROM tab
            WHERE x = 4
            ORDER BY y
            LIMIT 5
            ;

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``layout.newlines``

            **Groups**: ``all``, ``layout``

            **Configuration**

            * ``maximum_empty_lines_between_statements``: The maximum number of empty lines allowed between statements. Note that currently, the gap _before_ and _after_ the semicolon is considered 'between' statements. Must be one of ``range(0, 1000)``.
            * ``maximum_empty_lines_inside_statements``: The maximum number of empty lines allowed inside statements. Must be one of ``range(0, 1000)``.

  References bundle

    Code RF01

    Rule references.from

  References cannot reference objects not present in FROM clause.

Note:
        This rule is disabled by default for Athena, BigQuery, Databricks, DuckDB, Hive, Redshift, SOQL and SparkSQL due to the support of things like structs and lateral views which trigger false positives. It can be enabled with the force_enable = True flag.

Anti-pattern

In this example, the reference vee has not been declared.

            SELECT
                vee.a
            FROM foo

Best practice

Remove the reference.

            SELECT
                a
            FROM foo

            **Name**: ``references.from``

            **Aliases**: ``L026``

            **Groups**: ``all``, ``core``, ``references``

            **Configuration**

            * ``force_enable``: Run this rule even for dialects where this rule is disabled by default. Must be one of ``[True, False]``.

    Code RF02

    Rule references.qualification

  References should be qualified if select has more than one referenced table/view.

Note:
        Except if they're present in a USING clause.

Anti-pattern

In this example, the reference vee has not been declared, and the variables a and b are potentially ambiguous.

            SELECT a, b
            FROM foo
            LEFT JOIN vee ON vee.a = foo.a

Best practice

Add the references.

            SELECT foo.a, vee.b
            FROM foo
            LEFT JOIN vee ON vee.a = foo.a

            **Name**: ``references.qualification``

            **Aliases**: ``L027``

            **Groups**: ``all``, ``references``

            **Configuration**

            * ``subqueries_ignore_external_references``: If ``True``, parent query references are not included as potentially ambiguous in subqueries. Defaults to ``False``. Must be one of ``[True, False]``.

    Code RF03

    Rule references.consistent

  Column references should be qualified consistently in single table statements.

Note:
        For BigQuery, Hive and Redshift this rule is disabled by default. This is due to historical false positives associated with STRUCT data types. This default behaviour may be changed in the future. The rule can be enabled with the force_enable = True flag.

"consistent" will be fixed to "qualified" if inconsistency is found.

Anti-pattern

In this example, only the reference to b is qualified.

            SELECT
                a,
                foo.b
            FROM foo

Best practice

Either all column references should be qualified, or all unqualified.

            SELECT
                a,
                b
            FROM foo

            -- Also good

            SELECT
                foo.a,
                foo.b
            FROM foo

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``references.consistent``

            **Aliases**: ``L028``

            **Groups**: ``all``, ``references``

            **Configuration**

            * ``force_enable``: Run this rule even for dialects where this rule is disabled by default. Must be one of ``[True, False]``.
            * ``single_table_references``: The expectation for references in single-table select. Must be one of ``['consistent', 'qualified', 'unqualified']``.

    Code RF04

    Rule references.keywords

  Keywords should not be used as identifiers.

Although unreserved keywords can be used as identifiers, and reserved words can be used as quoted identifiers, best practice is to avoid where possible, to avoid any misunderstandings as to what the alias represents.

Note:
        Note that reserved keywords cannot be used as unquoted identifiers and will cause parsing errors and so are not covered by this rule.

Anti-pattern

In this example, SUM (built-in function) is used as an alias.

            SELECT
                sum.a
            FROM foo AS sum

Best practice

Avoid keywords as the name of an alias.

            SELECT
                vee.a
            FROM foo AS vee

            **Name**: ``references.keywords``

            **Aliases**: ``L029``

            **Groups**: ``all``, ``references``

            **Configuration**

            * ``ignore_words``: Comma separated list of words to ignore from rule.
            * ``ignore_words_regex``: Words to ignore from rule if they are a partial match for the regular expression. To ignore only full matches you can use ``^`` (beginning of text) and ``$`` (end of text). Due to regular expression operator precedence, it is good practice to use parentheses around everything between ``^`` and ``$``.
            * ``quoted_identifiers_policy``: Types of quoted identifiers to flag violations for. Must be one of ``['all', 'aliases', 'column_aliases', 'table_aliases', 'none']``.
            * ``unquoted_identifiers_policy``: Types of unquoted identifiers to flag violations for. Must be one of ``['all', 'aliases', 'column_aliases', 'table_aliases']``.

    Code RF05

    Rule references.special_chars

  Do not use special characters in identifiers.

Anti-pattern

Using special characters within identifiers when creating or aliasing objects.

            CREATE TABLE DBO.ColumnNames
            (
                [Internal Space] INT,
                [Greater>Than] INT,
                [Less<Than] INT,
                Number# INT
            )

Best practice

Identifiers should include only alphanumerics and underscores.

            CREATE TABLE DBO.ColumnNames
            (
                [Internal_Space] INT,
                [GreaterThan] INT,
                [LessThan] INT,
                NumberVal INT
            )

            **Name**: ``references.special_chars``

            **Aliases**: ``L057``

            **Groups**: ``all``, ``references``

            **Configuration**

            * ``additional_allowed_characters``: Optional list of extra allowed characters, in addition to alphanumerics (A-Z, a-z, 0-9) and underscores.
            * ``allow_space_in_identifier``: Should spaces in identifiers be allowed? Must be one of ``[True, False]``.
            * ``ignore_words``: Comma separated list of words to ignore from rule.
            * ``ignore_words_regex``: Words to ignore from rule if they are a partial match for the regular expression. To ignore only full matches you can use ``^`` (beginning of text) and ``$`` (end of text). Due to regular expression operator precedence, it is good practice to use parentheses around everything between ``^`` and ``$``.
            * ``quoted_identifiers_policy``: Types of quoted identifiers to flag violations for. Must be one of ``['all', 'aliases', 'column_aliases', 'table_aliases', 'none']``.
            * ``unquoted_identifiers_policy``: Types of unquoted identifiers to flag violations for. Must be one of ``['all', 'aliases', 'column_aliases', 'table_aliases']``.

    Code RF06

    Rule references.quoting

  Unnecessary quoted identifier.

This rule will fail if the quotes used to quote an identifier are (un)necessary depending on the force_quote_identifier configuration. This rule applies to both column references and their aliases. The default (safe) behaviour is designed not to unexpectedly corrupt SQL. That means the circumstances in which quotes can be safely removed depends on the current dialect would resolve the unquoted variant of the identifier (see below for examples).

Additionally this rule may be configured to a more aggressive setting by setting case_sensitive to False, in which case quotes will be removed regardless of the casing of the contained identifier. Any identifiers which contain special characters, spaces or keywords will still be left quoted. This setting is more appropriate for projects or teams where there is more control over the inputs and outputs of queries, and where it's more viable to institute rules such as enforcing that all identifiers are the default casing (and therefore meaning that using quotes to change the case of identifiers is unnecessary).
Dialect group | ✅ Example where quotes are safe to remove. | ⚠️ Examples where quotes are not safe to remove.
Natively UPPERCASE dialects e.g. Snowflake, BigQuery, TSQL & Oracle. | Identifiers which, without quotes, would resolve to the default casing of FOO i.e. "FOO". | Identifiers where the quotes are necessary to preserve case (e.g. "Foo" or "foo"), or where the identifier contains something invalid without the quotes such as keywords or special characters e.g. "SELECT", "With Space" or "Special&Characters".
Natively lowercase dialects e.g. Athena, Hive & Postgres | Identifiers which, without quotes, would resolve to the default casing of foo i.e. "foo". | Identifiers where the quotes are necessary to preserve case (e.g. "Foo" or "foo"), or where the identifier contains something invalid without the quotes such as keywords or special characters e.g. "SELECT", "With Space" or "Special&Characters".
Case insensitive dialects e.g. DuckDB <#duckdb-dialect-ref> or Apache Spark SQL <#sparksql-dialect-ref> | Any identifiers which are valid without quotes: e.g. "FOO", "foo", "Foo", "fOo", FOO and foo would all resolve to the same object. | Identifiers which contain something invalid without the quotes such as keywords or special characters e.g. "SELECT", "With Space" or "Special&Characters".

This rule is closely associated with (and constrained by the same above factors) as aliasing.self_alias.column (AL09).

When prefer_quoted_identifiers = False (default behaviour), the quotes are unnecessary, except for reserved keywords and special characters in identifiers.

Anti-pattern

In this example, valid unquoted identifiers, that are not also reserved keywords, are needlessly quoted.

            SELECT "foo" as "bar";  -- For lowercase dialects like Postgres
            SELECT "FOO" as "BAR";  -- For uppercase dialects like Snowflake

Best practice

Use unquoted identifiers where possible.

            SELECT foo as bar;  -- For lowercase dialects like Postgres
            SELECT FOO as BAR;  -- For uppercase dialects like Snowflake

            -- Note that where the case of the quoted identifier requires
            -- the quotes to remain, or where the identifier cannot be
            -- unquoted because it would be invalid to do so, the quotes
            -- may remain. For example:
            SELECT
                "Case_Sensitive_Identifier" as is_allowed,
                "Identifier with spaces or speci@l characters" as this_too,
                "SELECT" as also_reserved_words
            FROM "My Table With Spaces"

When prefer_quoted_identifiers = True, the quotes are always necessary, no matter if the identifier is valid, a reserved keyword, or contains special characters.

Note:
        Note due to different quotes being used by different dialects supported by SQLFluff, and those quotes meaning different things in different contexts, this mode is not sqlfluff fix compatible.

Anti-pattern

In this example, a valid unquoted identifier, that is also not a reserved keyword, is required to be quoted.

            SELECT 123 as foo

Best practice Use quoted identifiers.

            SELECT 123 as "foo" -- For ANSI, ...
            -- or
            SELECT 123 as `foo` -- For BigQuery, MySql, ...

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``references.quoting``

            **Aliases**: ``L059``

            **Groups**: ``all``, ``references``

            **Configuration**

            * ``case_sensitive``: If ``False``, comparison is done case in-sensitively. Defaults to ``True``. Must be one of ``[True, False]``.
            * ``ignore_words``: Comma separated list of words to ignore from rule.
            * ``ignore_words_regex``: Words to ignore from rule if they are a partial match for the regular expression. To ignore only full matches you can use ``^`` (beginning of text) and ``$`` (end of text). Due to regular expression operator precedence, it is good practice to use parentheses around everything between ``^`` and ``$``.
            * ``prefer_quoted_identifiers``: If ``True``, requires every identifier to be quoted. Defaults to ``False``. Must be one of ``[True, False]``.
            * ``prefer_quoted_keywords``: If ``True``, requires every keyword used as an identifier to be quoted. Defaults to ``False``. Must be one of ``[True, False]``.

  Structure bundle

    Code ST01

    Rule structure.else_null

  Do not specify else null in a case when statement (redundant).

Anti-pattern

            select
                case
                    when name like '%cat%' then 'meow'
                    when name like '%dog%' then 'woof'
                    else null
                end
            from x

Best practice

Omit else null

            select
                case
                    when name like '%cat%' then 'meow'
                    when name like '%dog%' then 'woof'
                end
            from x

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``structure.else_null``

            **Aliases**: ``L035``

            **Groups**: ``all``, ``structure``

    Code ST02

    Rule structure.simple_case

  Unnecessary CASE statement.

Anti-pattern

CASE statement returns booleans.

            select
                case
                    when fab > 0 then true
                    else false
                end as is_fab
            from fancy_table

            -- This rule can also simplify CASE statements
            -- that aim to fill NULL values.

            select
                case
                    when fab is null then 0
                    else fab
                end as fab_clean
            from fancy_table

            -- This also covers where the case statement
            -- replaces NULL values with NULL values.

            select
                case
                    when fab is null then null
                    else fab
                end as fab_clean
            from fancy_table

Best practice

Reduce to WHEN condition within COALESCE function.

            select
                coalesce(fab > 0, false) as is_fab
            from fancy_table

            -- To fill NULL values.

            select
                coalesce(fab, 0) as fab_clean
            from fancy_table

            -- NULL filling NULL.

            select fab as fab_clean
            from fancy_table

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``structure.simple_case``

            **Aliases**: ``L043``

            **Groups**: ``all``, ``structure``

    Code ST03

    Rule structure.unused_cte

  Query defines a CTE (common-table expression) but does not use it.

Anti-pattern

Defining a CTE that is not used by the query is harmless, but it means the code is unnecessary and could be removed.

            WITH cte1 AS (
              SELECT a
              FROM t
            ),
            cte2 AS (
              SELECT b
              FROM u
            )

            SELECT *
            FROM cte1

Best practice

Remove unused CTEs.

            WITH cte1 AS (
              SELECT a
              FROM t
            )

            SELECT *
            FROM cte1

            **Name**: ``structure.unused_cte``

            **Aliases**: ``L045``

            **Groups**: ``all``, ``core``, ``structure``

    Code ST04

    Rule structure.nested_case

  Nested CASE statement in ELSE clause could be flattened.

Anti-pattern

In this example, the outer CASE's ELSE is an unnecessary, nested CASE.

            SELECT
              CASE
                WHEN species = 'Cat' THEN 'Meow'
                ELSE
                CASE
                   WHEN species = 'Dog' THEN 'Woof'
                END
              END as sound
            FROM mytable

Best practice

Move the body of the inner CASE to the end of the outer one.

            SELECT
              CASE
                WHEN species = 'Cat' THEN 'Meow'
                WHEN species = 'Dog' THEN 'Woof'
              END AS sound
            FROM mytable

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``structure.nested_case``

            **Aliases**: ``L058``

            **Groups**: ``all``, ``structure``

    Code ST05

    Rule structure.subquery

  Join/From clauses should not contain subqueries. Use CTEs instead.

By default this rule is configured to allow subqueries within FROM clauses but not within JOIN clauses. If you prefer a stricter lint then this is configurable.

Note:
        Some dialects don't allow CTEs, and for those dialects this rule makes no sense and should be disabled.

Anti-pattern

            select
                a.x, a.y, b.z
            from a
            join (
                select x, z from b
            ) using(x)

Best practice

            with c as (
                select x, z from b
            )
            select
                a.x, a.y, c.z
            from a
            join c using(x)

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``structure.subquery``

            **Aliases**: ``L042``

            **Groups**: ``all``, ``structure``

            **Configuration**

            * ``forbid_subquery_in``: Which clauses should be linted for subqueries? Must be one of ``['join', 'from', 'both']``.

    Code ST06

    Rule structure.column_order

  Select wildcards then simple targets before calculations and aggregates.

Anti-pattern

            select
                a,
                *,
                row_number() over (partition by id order by date) as y,
                b
            from x

Best practice

Order select targets in ascending complexity

            select
                *,
                a,
                b,
                row_number() over (partition by id order by date) as y
            from x

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``structure.column_order``

            **Aliases**: ``L034``

            **Groups**: ``all``, ``structure``

    Code ST07

    Rule structure.using

  Prefer specifying join keys instead of using USING.

Note:
        This rule was originally taken from the dbt Style Guide <https://github.com/dbt-labs/corp/blob/main/dbt_style_guide.md> which notes that:
                Certain warehouses have inconsistencies in USING results (specifically Snowflake).

        In fact dbt removed it from their style guide in February 2022 <https://github.com/dbt-labs/corp/pull/58>. However, some like the rule, so for now we will keep it in SQLFluff, but encourage those that do not find value in the rule, to turn it off.

Note:
        This rule is disabled for ClickHouse as it supports USING without brackets which this rule does not support.

Anti-pattern

            SELECT
                table_a.field_1,
                table_b.field_2
            FROM
                table_a
            INNER JOIN table_b USING (id)

Best practice

Specify the keys directly

            SELECT
                table_a.field_1,
                table_b.field_2
            FROM
                table_a
            INNER JOIN table_b
                ON table_a.id = table_b.id

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``structure.using``

            **Aliases**: ``L032``

            **Groups**: ``all``, ``structure``

    Code ST08

    Rule structure.distinct

  DISTINCT used with parentheses.

Anti-pattern

In this example, parentheses are not needed and confuse DISTINCT with a function. The parentheses can also be misleading about which columns are affected by the DISTINCT (all the columns!).

            SELECT DISTINCT(a), b FROM foo

Best practice

Remove parentheses to be clear that the DISTINCT applies to both columns.

            SELECT DISTINCT a, b FROM foo

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``structure.distinct``

            **Aliases**: ``L015``

            **Groups**: ``all``, ``structure``, ``core``

    Code ST09

    Rule structure.join_condition_order

  Joins should list the table referenced earlier/later first.

This rule will break conditions from join clauses down into subconditions using the "and" and "or" binary operators.

Subconditions that are made up of a qualified column reference, a comparison operator and another qualified column reference are then evaluated to check whether they list the table that was referenced earlier - or later, depending on the preferred_first_table_in_join_clause configuration.

Subconditions that do not follow that pattern are ignored by this rule.

Note:
        Joins in WHERE clauses are currently not supported by this rule.

Anti-pattern

In this example, the tables that were referenced later are listed first and the preferred_first_table_in_join_clause configuration is set to earlier.

            select
                foo.a,
                foo.b,
                bar.c
            from foo
            left join bar
                -- This subcondition does not list
                -- the table referenced earlier first:
                on bar.a = foo.a
                -- Neither does this subcondition:
                and bar.b = foo.b

Best practice

List the tables that were referenced earlier first.

            select
                foo.a,
                foo.b,
                bar.c
            from foo
            left join bar
                on foo.a = bar.a
                and foo.b = bar.b

            This rule is ``sqlfluff fix`` compatible.

            **Name**: ``structure.join_condition_order``

            **Groups**: ``all``, ``structure``

            **Configuration**

            * ``preferred_first_table_in_join_clause``: Which table to list first when joining two tables. Defaults to ``earlier``. Must be one of ``['earlier', 'later']``.

    Code ST10

    Rule structure.constant_expression

  Redundant constant expression.

Including an expression that always evaluates to either TRUE or FALSE regardless of the input columns is unnecessary and makes statements harder to read and understand.

Constant conditions are sometimes mistakes (by mistyping the column name intended), and sometimes the result of incorrect information that they are necessary in some circumstances. In the former case, they can sometimes result in a cartesian join if it was supposed to be a join condition. Given the ambiguity of intent, this rule does not suggest an automatic fix, and instead invites the user to resolve the problem manually.

Anti-pattern

            SELECT *
            FROM my_table
            -- This following WHERE clause is redundant.
            WHERE my_table.col = my_table.col

Best practice

            SELECT *
            FROM my_table
            -- Replace with a condition that includes meaningful logic,
            -- or remove the condition entirely.
            WHERE my_table.col > 3

            **Name**: ``structure.constant_expression``

            **Groups**: ``all``, ``structure``

    Code ST11

    Rule structure.unused_join

  Joined table not referenced in query.

This rule will check if there are any tables that are referenced in the FROM or JOIN clause of a SELECT statement, but where no columns from that table are referenced in the any of the other clauses. Because some types of join are often used as filters, or to otherwise control granularity without being referenced (e.g. INNER and CROSS), this rule only applies to explicit OUTER joins (i.e. LEFT, RIGHT and FULL joins).

This rule relies on all of the column references in the SELECT statement being qualified with at least the table name, and so is designed to work alongside references.qualification (RF02). This is because without the knowledge of what columns exist in each upstream table, the rule is unable to resolve which table an unqualified column reference is pulled from.

This rule does not propose a fix, because it assumes that it an unused table is a mistake, but doesn't know whether the mistake was the join, or the mistake was not using it.

Anti-pattern

In this example, the table bar is included in the JOIN clause but not columns from it are referenced in

            SELECT
                foo.a,
                foo.b
            FROM foo
            LEFT JOIN bar ON foo.a = bar.a

Best practice

Remove the join, or use the table.

            SELECT foo.a, vee.b
            FROM foo;

            SELECT
                foo.a,
                foo.b,
                bar.c
            FROM foo
            LEFT JOIN bar ON foo.a = bar.a

In the (very rare) situations that it is logically necessary to include a table in a join clause, but not otherwise refer to it (likely for granularity reasons, or as a stepping stone to another table), we recommend ignoring this rule for that specific line by using -- noqa: ST11 at the end of the line.

    Code ST12

    Rule structure.consecutive_semicolons

  Consecutive semicolons detected.

This rule flags runs of two or more semicolons (optionally with intervening whitespace/newlines) which usually indicate an empty statement or an accidental duplicate terminator.

Anti-pattern

             SELECT 1;;
             ;;SELECT 2;

Best practice

Collapse duplicate semicolons unless intentionally separating batches.

             SELECT 1;
             SELECT 2;

             This rule is ``sqlfluff fix`` compatible.

             **Name**: ``structure.consecutive_semicolons``

             **Groups**: ``all``, ``structure``

  TSQL bundle

    Code TQ01

    Rule tsql.sp_prefix

  SP_ prefix should not be used for user-defined stored procedures in T-SQL.

Anti-pattern

The SP_ prefix is used to identify system procedures and can adversely affect performance of the user-defined stored procedure. It can also break system procedures if there is a naming conflict.

             CREATE PROCEDURE dbo.sp_pull_data
             AS
             SELECT
                 ID,
                 DataDate,
                 CaseOutput
             FROM table1

Best practice

Use a different name for the stored procedure.

             CREATE PROCEDURE dbo.pull_data
             AS
             SELECT
                 ID,
                 DataDate,
                 CaseOutput
             FROM table1

             -- Alternatively prefix with USP_ to
             -- indicate a user-defined stored procedure.

             CREATE PROCEDURE dbo.usp_pull_data
             AS
             SELECT
                 ID,
                 DataDate,
                 CaseOutput
             FROM table1

             **Name**: ``tsql.sp_prefix``

             **Aliases**: ``L056``

             **Groups**: ``all``, ``tsql``

  CLI Reference

  sqlfluff

SQLFluff is a modular SQL linter for humans.

            sqlfluff [OPTIONS] COMMAND [ARGS]...

Options

    --version
        Show the version and exit.

Examples:

            sqlfluff lint --dialect postgres .

            sqlfluff lint --dialect mysql --rules ST05 my_query.sql

            sqlfluff fix --dialect sqlite --rules LT10,ST05 src/queries

            sqlfluff parse --dialect duckdb --templater jinja path/my_query.sql

  dialects

Show the current dialects available.

            sqlfluff dialects [OPTIONS]

Options

    -n, --nocolor, --color
        No color - output will be without ANSI color codes.

    -v, --verbose
        Verbosity, how detailed should the output be. This is stackable, so -vv is more verbose than -v. For the most verbose option try -vvvv or -vvvvv.

    --version
        Show the version and exit.

  fix

Fix SQL files.

PATH is the path to a sql file or directory to lint. This can be either a file ('path/to/file.sql'), a path ('directory/of/sql/files'), a single ('-') character to indicate reading from stdin or a dot/blank ('.'/' ') which will be interpreted like passing the current working directory as a path argument.

            sqlfluff fix [OPTIONS] [PATHS]...

Options

    -n, --nocolor, --color
        No color - output will be without ANSI color codes.

    -v, --verbose
        Verbosity, how detailed should the output be. This is stackable, so -vv is more verbose than -v. For the most verbose option try -vvvv or -vvvvv.

    --version
        Show the version and exit.

    --stdin-filename <stdin_filename>
        When using stdin as an input, load the configuration as if the contents of stdin was in a file in the listed location. This is useful for some editors that pass file contents from the editor that might not match the content on disk.

    --library-path <library_path>
        Override the library_path value from the [sqlfluff:templater:jinja] configuration value. Set this to 'none' to disable entirely. This overrides any values set by users in configuration files or inline directives.

    --disable-noqa-except <disable_noqa_except>
        Ignore all but the listed rules inline noqa comments.

    --disable-noqa
        Set this flag to ignore inline noqa comments.

    --logger <logger>
        Choose to limit the logging to one of the loggers.

        Options
            templater | lexer | parser | linter | rules | config

    --bench
        Set this flag to engage the benchmarking tool output.

    -i, --ignore <ignore>
        Ignore particular families of errors so that they don't cause a failed run. For example --ignore parsing would mean that any parsing errors are ignored and don't influence the success or fail of a run. --ignore behaves somewhat like noqa comments, except it applies globally. Multiple options are possible if comma separated: e.g. --ignore parsing,templating.

    --encoding <encoding>
        Specify encoding to use when reading and writing files. Defaults to autodetect.

    --ignore-local-config
        Ignore config files in default search path locations. This option allows the user to lint with the default config or can be used in conjunction with --config to only reference the custom config file.

    --config <extra_config_path>
        Include additional config file. By default the config is generated from the standard configuration files described in the documentation. This argument allows you to specify an additional configuration file that overrides the standard configuration files. N.B. cfg format is required.

    -e, --exclude-rules <exclude_rules>
        Exclude specific rules. For example specifying --exclude-rules LT01 will remove rule LT01 (Unnecessary trailing whitespace) from the set of considered rules. This could either be the allowlist, or the general set if there is no specific allowlist. Multiple rules can be specified with commas e.g. --exclude-rules LT01,LT02 will exclude violations of rule LT01 and rule LT02.

    -r, --rules <rules>
        Narrow the search to only specific rules. For example specifying --rules LT01 will only search for rule LT01 (Unnecessary trailing whitespace). Multiple rules can be specified with commas e.g. --rules LT01,LT02 will specify only looking for violations of rule LT01 and rule LT02.

    -t, --templater <templater>
        The templater to use (default=jinja)

        Options
            raw | jinja | python | placeholder

    -d, --dialect <dialect>
        The dialect of SQL to lint

    --disregard-sqlfluffignores
        Perform the operation regardless of .sqlfluffignore configurations

    --warn-unused-ignores
        Warn about unneeded '-- noqa:' comments.

    --persist-timing <persist_timing>
        A filename to persist the timing information for a linting run to in csv format for external analysis. NOTE: This feature should be treated as beta, and the format of the csv file may change in future releases without warning.

    --disable-progress-bar
        Disables progress bars.

    -p, --processes <processes>
        The number of parallel processes to run. Positive numbers work as expected. Zero and negative numbers will work as number_of_cpus - number. e.g -1 means all cpus except one. 0 means all cpus.

    -f, --force
        [DEPRECATED - From 3.0 onward this is the default behaviour] Apply fixes will also be applied file by file, during the linting process, rather than waiting until all files are linted before fixing.

    --check
        Analyse all files and ask for confirmation before applying any fixes. Fixes will be applied all together at the end of the operation.

    -q, --quiet
        Reduces the amount of output to stdout to a minimal level. This is effectively the opposite of -v. NOTE: It will only take effect if -f/--force is also set.

    -x, --fixed-suffix <fixed_suffix>
        An optional suffix to add to fixed files.

    --FIX-EVEN-UNPARSABLE
        Enables fixing of files that have templating or parse errors. Note that the similar-sounding '--ignore' or 'noqa' features merely prevent errors from being displayed. For safety reasons, the 'fix'command will not make any fixes in files that have templating or parse errors unless '--FIX-EVEN-UNPARSABLE' is enabled on the command lineor in the .sqlfluff config file.

    --show-lint-violations
        Show lint violations

Arguments

    PATHS
        Optional argument(s)

  format

Autoformat SQL files.

This effectively force applies sqlfluff fix with a known subset of fairly stable rules. Enabled rules are ignored, but rule exclusions (via CLI) or config are still respected.

PATH is the path to a sql file or directory to lint. This can be either a file ('path/to/file.sql'), a path ('directory/of/sql/files'), a single ('-') character to indicate reading from stdin or a dot/blank ('.'/' ') which will be interpreted like passing the current working directory as a path argument.

            sqlfluff format [OPTIONS] [PATHS]...

Options

    -n, --nocolor, --color
        No color - output will be without ANSI color codes.

    -v, --verbose
        Verbosity, how detailed should the output be. This is stackable, so -vv is more verbose than -v. For the most verbose option try -vvvv or -vvvvv.

    --version
        Show the version and exit.

    --stdin-filename <stdin_filename>
        When using stdin as an input, load the configuration as if the contents of stdin was in a file in the listed location. This is useful for some editors that pass file contents from the editor that might not match the content on disk.

    --library-path <library_path>
        Override the library_path value from the [sqlfluff:templater:jinja] configuration value. Set this to 'none' to disable entirely. This overrides any values set by users in configuration files or inline directives.

    --disable-noqa-except <disable_noqa_except>
        Ignore all but the listed rules inline noqa comments.

    --disable-noqa
        Set this flag to ignore inline noqa comments.

    --logger <logger>
        Choose to limit the logging to one of the loggers.

        Options
            templater | lexer | parser | linter | rules | config

    --bench
        Set this flag to engage the benchmarking tool output.

    -i, --ignore <ignore>
        Ignore particular families of errors so that they don't cause a failed run. For example --ignore parsing would mean that any parsing errors are ignored and don't influence the success or fail of a run. --ignore behaves somewhat like noqa comments, except it applies globally. Multiple options are possible if comma separated: e.g. --ignore parsing,templating.

    --encoding <encoding>
        Specify encoding to use when reading and writing files. Defaults to autodetect.

    --ignore-local-config
        Ignore config files in default search path locations. This option allows the user to lint with the default config or can be used in conjunction with --config to only reference the custom config file.

    --config <extra_config_path>
        Include additional config file. By default the config is generated from the standard configuration files described in the documentation. This argument allows you to specify an additional configuration file that overrides the standard configuration files. N.B. cfg format is required.

    -e, --exclude-rules <exclude_rules>
        Exclude specific rules. For example specifying --exclude-rules LT01 will remove rule LT01 (Unnecessary trailing whitespace) from the set of considered rules. This could either be the allowlist, or the general set if there is no specific allowlist. Multiple rules can be specified with commas e.g. --exclude-rules LT01,LT02 will exclude violations of rule LT01 and rule LT02.

    -r, --rules <rules>
        Narrow the search to only specific rules. For example specifying --rules LT01 will only search for rule LT01 (Unnecessary trailing whitespace). Multiple rules can be specified with commas e.g. --rules LT01,LT02 will specify only looking for violations of rule LT01 and rule LT02.

    -t, --templater <templater>
        The templater to use (default=jinja)

        Options
            raw | jinja | python | placeholder

    -d, --dialect <dialect>
        The dialect of SQL to lint

    --disregard-sqlfluffignores
        Perform the operation regardless of .sqlfluffignore configurations

    --warn-unused-ignores
        Warn about unneeded '-- noqa:' comments.

    --persist-timing <persist_timing>
        A filename to persist the timing information for a linting run to in csv format for external analysis. NOTE: This feature should be treated as beta, and the format of the csv file may change in future releases without warning.

    --disable-progress-bar
        Disables progress bars.

    -p, --processes <processes>
        The number of parallel processes to run. Positive numbers work as expected. Zero and negative numbers will work as number_of_cpus - number. e.g -1 means all cpus except one. 0 means all cpus.

    -x, --fixed-suffix <fixed_suffix>
        An optional suffix to add to fixed files.

Arguments

    PATHS
        Optional argument(s)

  lint

Lint SQL files via passing a list of files or using stdin.

PATH is the path to a sql file or directory to lint. This can be either a file ('path/to/file.sql'), a path ('directory/of/sql/files'), a single ('-') character to indicate reading from stdin or a dot/blank ('.'/' ') which will be interpreted like passing the current working directory as a path argument.

Linting SQL files:
        sqlfluff lint path/to/file.sql sqlfluff lint directory/of/sql/files

Linting a file via stdin (note the lone '-' character):
        cat path/to/file.sql | sqlfluff lint - echo 'select col from tbl' | sqlfluff lint -

            sqlfluff lint [OPTIONS] [PATHS]...

Options

    -n, --nocolor, --color
        No color - output will be without ANSI color codes.

    -v, --verbose
        Verbosity, how detailed should the output be. This is stackable, so -vv is more verbose than -v. For the most verbose option try -vvvv or -vvvvv.

    --version
        Show the version and exit.

    --stdin-filename <stdin_filename>
        When using stdin as an input, load the configuration as if the contents of stdin was in a file in the listed location. This is useful for some editors that pass file contents from the editor that might not match the content on disk.

    --library-path <library_path>
        Override the library_path value from the [sqlfluff:templater:jinja] configuration value. Set this to 'none' to disable entirely. This overrides any values set by users in configuration files or inline directives.

    --disable-noqa-except <disable_noqa_except>
        Ignore all but the listed rules inline noqa comments.

    --disable-noqa
        Set this flag to ignore inline noqa comments.

    --logger <logger>
        Choose to limit the logging to one of the loggers.

        Options
            templater | lexer | parser | linter | rules | config

    --bench
        Set this flag to engage the benchmarking tool output.

    -i, --ignore <ignore>
        Ignore particular families of errors so that they don't cause a failed run. For example --ignore parsing would mean that any parsing errors are ignored and don't influence the success or fail of a run. --ignore behaves somewhat like noqa comments, except it applies globally. Multiple options are possible if comma separated: e.g. --ignore parsing,templating.

    --encoding <encoding>
        Specify encoding to use when reading and writing files. Defaults to autodetect.

    --ignore-local-config
        Ignore config files in default search path locations. This option allows the user to lint with the default config or can be used in conjunction with --config to only reference the custom config file.

    --config <extra_config_path>
        Include additional config file. By default the config is generated from the standard configuration files described in the documentation. This argument allows you to specify an additional configuration file that overrides the standard configuration files. N.B. cfg format is required.

    -e, --exclude-rules <exclude_rules>
        Exclude specific rules. For example specifying --exclude-rules LT01 will remove rule LT01 (Unnecessary trailing whitespace) from the set of considered rules. This could either be the allowlist, or the general set if there is no specific allowlist. Multiple rules can be specified with commas e.g. --exclude-rules LT01,LT02 will exclude violations of rule LT01 and rule LT02.

    -r, --rules <rules>
        Narrow the search to only specific rules. For example specifying --rules LT01 will only search for rule LT01 (Unnecessary trailing whitespace). Multiple rules can be specified with commas e.g. --rules LT01,LT02 will specify only looking for violations of rule LT01 and rule LT02.

    -t, --templater <templater>
        The templater to use (default=jinja)

        Options
            raw | jinja | python | placeholder

    -d, --dialect <dialect>
        The dialect of SQL to lint

    --disregard-sqlfluffignores
        Perform the operation regardless of .sqlfluffignore configurations

    --warn-unused-ignores
        Warn about unneeded '-- noqa:' comments.

    --persist-timing <persist_timing>
        A filename to persist the timing information for a linting run to in csv format for external analysis. NOTE: This feature should be treated as beta, and the format of the csv file may change in future releases without warning.

    --disable-progress-bar
        Disables progress bars.

    -p, --processes <processes>
        The number of parallel processes to run. Positive numbers work as expected. Zero and negative numbers will work as number_of_cpus - number. e.g -1 means all cpus except one. 0 means all cpus.

    -f, --format <format>
        What format to return the lint result in (default=human).

        Options
            human | json | yaml | github-annotation | github-annotation-native | none

    --write-output <write_output>
        Optionally provide a filename to write the results to, mostly used in tandem with --format. NB: Setting an output file re-enables normal stdout logging.

    --annotation-level <annotation_level>
        When format is set to "github-annotation" or "github-annotation-native", default annotation level (default="warning"). "failure" and "error" are equivalent. Any rules configured only as warnings will always come through with type "notice" regardless of this option.

        Options
            notice | warning | failure | error

    --nofail
        If set, the exit code will always be zero, regardless of violations found. This is potentially useful during rollout.

Arguments

    PATHS
        Optional argument(s)

  parse

Parse SQL files and just spit out the result.

PATH is the path to a sql file or directory to lint. This can be either a file ('path/to/file.sql'), a path ('directory/of/sql/files'), a single ('-') character to indicate reading from stdin or a dot/blank ('.'/' ') which will be interpreted like passing the current working directory as a path argument.

            sqlfluff parse [OPTIONS] PATH

Options

    -n, --nocolor, --color
        No color - output will be without ANSI color codes.

    -v, --verbose
        Verbosity, how detailed should the output be. This is stackable, so -vv is more verbose than -v. For the most verbose option try -vvvv or -vvvvv.

    --version
        Show the version and exit.

    --stdin-filename <stdin_filename>
        When using stdin as an input, load the configuration as if the contents of stdin was in a file in the listed location. This is useful for some editors that pass file contents from the editor that might not match the content on disk.

    --library-path <library_path>
        Override the library_path value from the [sqlfluff:templater:jinja] configuration value. Set this to 'none' to disable entirely. This overrides any values set by users in configuration files or inline directives.

    --disable-noqa-except <disable_noqa_except>
        Ignore all but the listed rules inline noqa comments.

    --disable-noqa
        Set this flag to ignore inline noqa comments.

    --logger <logger>
        Choose to limit the logging to one of the loggers.

        Options
            templater | lexer | parser | linter | rules | config

    --bench
        Set this flag to engage the benchmarking tool output.

    -i, --ignore <ignore>
        Ignore particular families of errors so that they don't cause a failed run. For example --ignore parsing would mean that any parsing errors are ignored and don't influence the success or fail of a run. --ignore behaves somewhat like noqa comments, except it applies globally. Multiple options are possible if comma separated: e.g. --ignore parsing,templating.

    --encoding <encoding>
        Specify encoding to use when reading and writing files. Defaults to autodetect.

    --ignore-local-config
        Ignore config files in default search path locations. This option allows the user to lint with the default config or can be used in conjunction with --config to only reference the custom config file.

    --config <extra_config_path>
        Include additional config file. By default the config is generated from the standard configuration files described in the documentation. This argument allows you to specify an additional configuration file that overrides the standard configuration files. N.B. cfg format is required.

    -e, --exclude-rules <exclude_rules>
        Exclude specific rules. For example specifying --exclude-rules LT01 will remove rule LT01 (Unnecessary trailing whitespace) from the set of considered rules. This could either be the allowlist, or the general set if there is no specific allowlist. Multiple rules can be specified with commas e.g. --exclude-rules LT01,LT02 will exclude violations of rule LT01 and rule LT02.

    -r, --rules <rules>
        Narrow the search to only specific rules. For example specifying --rules LT01 will only search for rule LT01 (Unnecessary trailing whitespace). Multiple rules can be specified with commas e.g. --rules LT01,LT02 will specify only looking for violations of rule LT01 and rule LT02.

    -t, --templater <templater>
        The templater to use (default=jinja)

        Options
            raw | jinja | python | placeholder

    -d, --dialect <dialect>
        The dialect of SQL to lint

    -c, --code-only
        Output only the code elements of the parse tree.

    -m, --include-meta
        Include meta segments (indents, dedents and placeholders) in the output. This only applies when outputting json or yaml.

    -f, --format <format>
        What format to return the parse result in.

        Options
            human | json | yaml | none

    --write-output <write_output>
        Optionally provide a filename to write the results to, mostly used in tandem with --format. NB: Setting an output file re-enables normal stdout logging.

    --parse-statistics
        Set this flag to enabled detailed debugging readout on the use of terminators in the parser.

    --nofail
        If set, the exit code will always be zero, regardless of violations found. This is potentially useful during rollout.

Arguments

    PATH
        Required argument

  render

Render SQL files and just spit out the result.

PATH is the path to a sql file. This should be either a single file file ('path/to/file.sql') or a single ('-') character to indicate reading from stdin.

            sqlfluff render [OPTIONS] PATH

Options

    -n, --nocolor, --color
        No color - output will be without ANSI color codes.

    -v, --verbose
        Verbosity, how detailed should the output be. This is stackable, so -vv is more verbose than -v. For the most verbose option try -vvvv or -vvvvv.

    --version
        Show the version and exit.

    --stdin-filename <stdin_filename>
        When using stdin as an input, load the configuration as if the contents of stdin was in a file in the listed location. This is useful for some editors that pass file contents from the editor that might not match the content on disk.

    --library-path <library_path>
        Override the library_path value from the [sqlfluff:templater:jinja] configuration value. Set this to 'none' to disable entirely. This overrides any values set by users in configuration files or inline directives.

    --disable-noqa-except <disable_noqa_except>
        Ignore all but the listed rules inline noqa comments.

    --disable-noqa
        Set this flag to ignore inline noqa comments.

    --logger <logger>
        Choose to limit the logging to one of the loggers.

        Options
            templater | lexer | parser | linter | rules | config

    --bench
        Set this flag to engage the benchmarking tool output.

    -i, --ignore <ignore>
        Ignore particular families of errors so that they don't cause a failed run. For example --ignore parsing would mean that any parsing errors are ignored and don't influence the success or fail of a run. --ignore behaves somewhat like noqa comments, except it applies globally. Multiple options are possible if comma separated: e.g. --ignore parsing,templating.

    --encoding <encoding>
        Specify encoding to use when reading and writing files. Defaults to autodetect.

    --ignore-local-config
        Ignore config files in default search path locations. This option allows the user to lint with the default config or can be used in conjunction with --config to only reference the custom config file.

    --config <extra_config_path>
        Include additional config file. By default the config is generated from the standard configuration files described in the documentation. This argument allows you to specify an additional configuration file that overrides the standard configuration files. N.B. cfg format is required.

    -e, --exclude-rules <exclude_rules>
        Exclude specific rules. For example specifying --exclude-rules LT01 will remove rule LT01 (Unnecessary trailing whitespace) from the set of considered rules. This could either be the allowlist, or the general set if there is no specific allowlist. Multiple rules can be specified with commas e.g. --exclude-rules LT01,LT02 will exclude violations of rule LT01 and rule LT02.

    -r, --rules <rules>
        Narrow the search to only specific rules. For example specifying --rules LT01 will only search for rule LT01 (Unnecessary trailing whitespace). Multiple rules can be specified with commas e.g. --rules LT01,LT02 will specify only looking for violations of rule LT01 and rule LT02.

    -t, --templater <templater>
        The templater to use (default=jinja)

        Options
            raw | jinja | python | placeholder

    -d, --dialect <dialect>
        The dialect of SQL to lint

Arguments

    PATH
        Required argument

  rules

Show the current rules in use.

            sqlfluff rules [OPTIONS]

Options

    -n, --nocolor, --color
        No color - output will be without ANSI color codes.

    -v, --verbose
        Verbosity, how detailed should the output be. This is stackable, so -vv is more verbose than -v. For the most verbose option try -vvvv or -vvvvv.

    --version
        Show the version and exit.

  version

Show the version of sqlfluff.

            sqlfluff version [OPTIONS]

Options

    -n, --nocolor, --color
        No color - output will be without ANSI color codes.

    -v, --verbose
        Verbosity, how detailed should the output be. This is stackable, so -vv is more verbose than -v. For the most verbose option try -vvvv or -vvvvv.

    --version
        Show the version and exit.

  Python API

SQLFluff exposes a public api for other python applications to use. A basic example of this usage is given here, with the documentation for each of the methods below.

            """This is an example of how to use the simple sqlfluff api."""

            from typing import Any, Iterator, Union

            import sqlfluff

            #  -------- LINTING ----------

            my_bad_query = "SeLEct  *, 1, blah as  fOO  from mySchema.myTable"

            # Lint the given string and return an array of violations in JSON representation.
            lint_result = sqlfluff.lint(my_bad_query, dialect="bigquery")
            # lint_result =
            # [
            #     {
            #         "code": "CP01",
            #         "line_no": 1,
            #         "line_pos": 1,
            #         "description": "Keywords must be consistently upper case.",
            #     }
            #     ...
            # ]

            #  -------- FIXING ----------

            # Fix the given string and get a string back which has been fixed.
            fix_result_1 = sqlfluff.fix(my_bad_query, dialect="bigquery")
            # fix_result_1 = 'SELECT  *, 1, blah AS  foo  FROM myschema.mytable\n'

            # We can also fix just specific rules.
            fix_result_2 = sqlfluff.fix(my_bad_query, rules=["CP01"])
            # fix_result_2 = 'SELECT  *, 1, blah AS  fOO  FROM mySchema.myTable'

            # Or a subset of rules...
            fix_result_3 = sqlfluff.fix(my_bad_query, rules=["CP01", "CP02"])
            # fix_result_3 = 'SELECT  *, 1, blah AS  fOO  FROM myschema.mytable'

            #  -------- PARSING ----------

            # Parse the given string and return a JSON representation of the parsed tree.
            parse_result = sqlfluff.parse(my_bad_query)
            # parse_result = {'file': {'statement': {...}, 'newline': '\n'}}

            # This JSON structure can then be parsed as required.
            # An example usage is shown below:

            def get_json_segment(
                parse_result: dict[str, Any], segment_type: str
            ) -> Iterator[Union[str, dict[str, Any], list[dict[str, Any]]]]:
                """Recursively search JSON parse result for specified segment type.

                Args:
                    parse_result (Dict[str, Any]): JSON parse result from `sqlfluff.fix`.
                    segment_type (str): The segment type to search for.

                Yields:
                    Iterator[Union[str, dict[str, Any], list[dict[str, Any]]]]:
                    Retrieves children of specified segment type as either a string for a raw
                    segment or as JSON or an array of JSON for non-raw segments.
                """
                for k, v in parse_result.items():
                    if k == segment_type:
                        yield v
                    elif isinstance(v, dict):
                        yield from get_json_segment(v, segment_type)
                    elif isinstance(v, list):
                        for s in v:
                            yield from get_json_segment(s, segment_type)

            # e.g. Retrieve array of JSON for table references.
            table_references = list(get_json_segment(parse_result, "table_reference"))
            print(table_references)
            # [[{'identifier': 'mySchema'}, {'dot': '.'}, {'identifier': 'myTable'}]]

            # Retrieve raw table name from last identifier in the table reference.
            for table_reference in table_references:
                table_name = list(get_json_segment(parse_result, "naked_identifier"))[-1]
                print(f"table_name: {table_name}")
            # table_name: myTable

  Simple API commands

Sqlfluff is a SQL linter for humans.

    fix(sql: str, dialect: str | None = None, rules: list[str] | None = None, exclude_rules: list[str] | None = None, config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig> | None = None, config_path: str | None = None, fix_even_unparsable: bool | None = None) -> str
        Fix a SQL string.

        Parameters

            •   sql (str) -- The SQL to be fixed.

            •   dialect (Optional[str], optional) -- A reference to the dialect of the SQL to be fixed. Defaults to ansi.

            •   rules (Optional[list[str], optional) -- A subset of rule references to fix for. Defaults to None.

            •   exclude_rules (Optional[list[str], optional) -- A subset of rule references to avoid fixing for. Defaults to None.

            •   config (Optional[FluffConfig], optional) -- A configuration object to use for the operation. Defaults to None.

            •   config_path (Optional[str], optional) -- A path to a .sqlfluff config, which is only used if a config is not already provided. Defaults to None.

            •   fix_even_unparsable (bool, optional) -- Optional override for the corresponding SQLFluff configuration value.

        Returns
            str for the fixed SQL if possible.

    lint(sql: str, dialect: str | None = None, rules: list[str] | None = None, exclude_rules: list[str] | None = None, config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig> | None = None, config_path: str | None = None) -> list[dict[str, Any]]
        Lint a SQL string.

        Parameters

            •   sql (str) -- The SQL to be linted.

            •   dialect (Optional[str], optional) -- A reference to the dialect of the SQL to be linted. Defaults to ansi.

            •   rules (Optional[list[str], optional) -- A list of rule references to lint for. Defaults to None.

            •   exclude_rules (Optional[list[str], optional) -- A list of rule references to avoid linting for. Defaults to None.

            •   config (Optional[FluffConfig], optional) -- A configuration object to use for the operation. Defaults to None.

            •   config_path (Optional[str], optional) -- A path to a .sqlfluff config, which is only used if a config is not already provided. Defaults to None.

        Returns
            list[dict[str, Any]] for each violation found.

    parse(sql: str, dialect: str | None = None, config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig> | None = None, config_path: str | None = None) -> dict[str, Any]
        Parse a SQL string.

        Parameters

            •   sql (str) -- The SQL to be parsed.

            •   dialect (Optional[str], optional) -- A reference to the dialect of the SQL to be parsed. Defaults to ansi.

            •   config (Optional[FluffConfig], optional) -- A configuration object to use for the operation. Defaults to None.

            •   config_path (Optional[str], optional) -- A path to a .sqlfluff config, which is only used if a config is not already provided. Defaults to None.

        Returns
            Dict[str, Any] JSON containing the parsed structure.

    Note:
            In the case of multiple potential variants from the raw source file only the first variant is returned by the simple API. For access to the other variants, use the underlying main API directly.

  Advanced API usage

The simple API presents only a fraction of the functionality present within the core SQLFluff library. For more advanced use cases, users can import the Linter() and FluffConfig() classes from sqlfluff.core. As of version 0.4.0 this is considered as experimental only as the internals may change without warning in any future release. If you come to rely on the internals of SQLFluff, please post an issue on GitHub to share what you're up to. This will help shape a more reliable, tidy and well documented public API for use.

  Configuring SQLFluff

You can use FluffConfig() class to configure SQLFluff behaviour.

            """This is an example of providing config overrides."""

            from sqlfluff.core import FluffConfig, Linter

            sql = "SELECT 1\n"

            config = FluffConfig(
                overrides={
                    "dialect": "snowflake",
                    # NOTE: We explicitly set the string "none" here rather
                    # than a None literal so that it overrides any config
                    # set by any config files in the path.
                    "library_path": "none",
                }
            )

            linted_file = Linter(config=config).lint_string(sql)

            assert linted_file.get_violations() == []

Instances of FluffConfig() can be created manually, or parsed.

            """An example to show a few ways of configuring the API."""

            import sqlfluff
            from sqlfluff.core import FluffConfig, Linter

            # #######################################
            # The simple API can be configured in three ways.

            # 1. Limited keyword arguments
            sqlfluff.fix("SELECT  1", dialect="bigquery")

            # 2. Providing the path to a config file
            sqlfluff.fix("SELECT  1", config_path="test/fixtures/.sqlfluff")

            # 3. Providing a preconfigured FluffConfig object.
            # NOTE: This is the way of configuring SQLFluff which will give the most control.

            # 3a. FluffConfig objects can be created directly from a dictionary of values.
            config = FluffConfig(configs={"core": {"dialect": "bigquery"}})
            # 3b. FluffConfig objects can be created from a config file in a string.
            config = FluffConfig.from_string("[sqlfluff]\ndialect=bigquery\n")
            # 3c. FluffConfig objects can be created from a config file in multiple strings
            #     to simulate the effect of multiple nested config strings.
            config = FluffConfig.from_strings(
                # NOTE: Given these two strings, the resulting dialect would be "mysql"
                # as the later files take precedence.
                "[sqlfluff]\ndialect=bigquery\n",
                "[sqlfluff]\ndialect=mysql\n",
            )
            # 3d. FluffConfig objects can be created from a path containing a config file.
            config = FluffConfig.from_path("test/fixtures/")
            # 3e. FluffConfig objects can be from keyword arguments
            config = FluffConfig.from_kwargs(dialect="bigquery", rules=["LT01"])

            # The FluffConfig is then provided via a config argument.
            sqlfluff.fix("SELECT  1", config=config)

            # #######################################
            # The core API is always configured using a FluffConfig object.

            # When instantiating a Linter (or Parser), a FluffConfig must be provided
            # on instantiation. See above for details on how to create a FluffConfig.
            linter = Linter(config=config)

            # The provided config will then be used in any operations.

            lint_result = linter.lint_string("SELECT  1", fix=True)
            fixed_string = lint_result.fix_string()
            # NOTE: The "True" element shows that fixing was a success.
            assert fixed_string == ("SELECT 1", True)

Supported dialects and rules are available through list_dialects() and list_rules().

            """This is an example of how get basic options from sqlfluff."""

            import sqlfluff

            #  -------- DIALECTS ----------

            dialects = sqlfluff.list_dialects()
            # dialects = [DialectTuple(label='ansi', name='ansi', inherits_from='nothing'), ...]
            dialect_names = [dialect.label for dialect in dialects]
            # dialect_names = ["ansi", "snowflake", ...]

            #  -------- RULES ----------

            rules = sqlfluff.list_rules()
            # rules = [
            #     RuleTuple(
            #         code='Example_LT01',
            #         description='ORDER BY on these columns is forbidden!'
            #     ),
            #     ...
            # ]
            rule_codes = [rule.code for rule in rules]
            # rule_codes = ["LT01", "LT02", ...]

  Advanced API reference

The core elements of sqlfluff.

    class Lexer(config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig> | None = None, last_resort_lexer: StringLexer | None = None, dialect: str | None = None)
        The Lexer class actually does the lexing step.

        elements_to_segments(elements: list[TemplateElement], templated_file: TemplatedFile) -> tuple[RawSegment, ...]
            Convert a tuple of lexed elements into a tuple of segments.

        lex(raw: str | TemplatedFile) -> tuple[tuple[BaseSegment, ...], list[SQLLexError]]
            Take a string or TemplatedFile and return segments.

            If we fail to match the whole string, then we must have found something that we cannot lex. If that happens we should package it up as unlexable and keep track of the exceptions.

        static lex_match(forward_string: str, lexer_matchers: list[StringLexer]) -> LexMatch
            Iteratively match strings using the selection of submatchers.

        static map_template_slices(elements: list[LexedElement], template: TemplatedFile) -> list[TemplateElement]
            Create a tuple of TemplateElement from a tuple of LexedElement.

            This adds slices in the templated file to the original lexed elements. We'll need this to work out the position in the source file.

        static violations_from_segments(segments: tuple[RawSegment, ...]) -> list[SQLLexError]
            Generate any lexing errors for any unlexables.

    class Linter(config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig> | None = None, formatter: FormatterInterface | None = None, dialect: str | None = None, rules: list[str] | None = None, user_rules: list[type[BaseRule <#sqlfluff.core.rules.base.BaseRule>]] | None = None, exclude_rules: list[str] | None = None)
        The interface class to interact with the linter.

        classmethod allowed_rule_ref_map(reference_map: dict[str, set[str]], disable_noqa_except: str | None) -> dict[str, set[str]]
            Generate a noqa rule reference map.

        fix(tree: BaseSegment, config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig> | None = None, fname: str | None = None, templated_file: TemplatedFile | None = None) -> tuple[BaseSegment, list[SQLBaseError]]
            Return the fixed tree and violations from lintfix when we're fixing.

        get_rulepack(config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig> | None = None) -> RulePack <#sqlfluff.core.rules.base.RulePack>
            Get hold of a set of rules.

        lint(tree: BaseSegment, config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig> | None = None, fname: str | None = None, templated_file: TemplatedFile | None = None) -> list[SQLBaseError]
            Return just the violations from lintfix when we're only linting.

        classmethod lint_fix_parsed(tree: BaseSegment, config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>, rule_pack: RulePack <#sqlfluff.core.rules.base.RulePack>, fix: bool = False, fname: str | None = None, templated_file: TemplatedFile | None = None, formatter: FormatterInterface | None = None) -> tuple[BaseSegment, list[SQLBaseError], IgnoreMask | None, list[tuple[str, str, float]]]
            Lint and optionally fix a tree object.

        classmethod lint_parsed(parsed: ParsedString, rule_pack: RulePack <#sqlfluff.core.rules.base.RulePack>, fix: bool = False, formatter: FormatterInterface | None = None, encoding: str = 'utf8') -> LintedFile
            Lint a ParsedString and return a LintedFile.

        lint_path(path: str, fix: bool = False, ignore_non_existent_files: bool = False, ignore_files: bool = True, processes: int | None = None) -> LintedDir
            Lint a path.

        lint_paths(paths: tuple[str, ...], fix: bool = False, ignore_non_existent_files: bool = False, ignore_files: bool = True, processes: int | None = None, apply_fixes: bool = False, fixed_file_suffix: str = '', fix_even_unparsable: bool = False, retain_files: bool = True) -> LintingResult
            Lint an iterable of paths.

        classmethod lint_rendered(rendered: RenderedFile, rule_pack: RulePack <#sqlfluff.core.rules.base.RulePack>, fix: bool = False, formatter: FormatterInterface | None = None) -> LintedFile
            Take a RenderedFile and return a LintedFile.

        lint_string(in_str: str = '', fname: str = '<string input>', fix: bool = False, config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig> | None = None, encoding: str = 'utf8') -> LintedFile
            Lint a string.

            Returns
                an object representing that linted file.

            Return type
                LintedFile

        lint_string_wrapped(string: str, fname: str = '<string input>', fix: bool = False, stdin_filename: str | None = None) -> LintingResult
            Lint strings directly.

        static load_raw_file_and_config(fname: str, root_config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>) -> tuple[str, FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>, str]
            Load a raw file and the associated config.

        parse_path(path: str, parse_statistics: bool = False) -> Iterator[ParsedString]
            Parse a path of sql files.

            NB: This a generator which will yield the result of each file within the path iteratively.

        classmethod parse_rendered(rendered: RenderedFile, parse_statistics: bool = False) -> ParsedString
            Parse a rendered file.

        parse_string(in_str: str, fname: str = '<string>', config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig> | None = None, encoding: str = 'utf-8', parse_statistics: bool = False) -> ParsedString
            Parse a string.

        static remove_templated_errors(linting_errors: list[SQLBaseError]) -> list[SQLBaseError]
            Filter a list of lint errors, removing those from the templated slices.

        render_file(fname: str, root_config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>) -> RenderedFile
            Load and render a file with relevant config.

        render_string(in_str: str, fname: str, config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>, encoding: str) -> RenderedFile
            Template the file.

        rule_tuples() -> list[RuleTuple]
            A simple pass through to access the rule tuples of the rule set.

    class Parser(config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig> | None = None, dialect: str | None = None)
        Instantiates parsed queries from a sequence of lexed raw segments.

        parse(segments: Sequence[BaseSegment], fname: str | None = None, parse_statistics: bool = False) -> BaseSegment | None
            Parse a series of lexed tokens using the current dialect.

  Internal API

Anything within this section should only be necessary for people who are developing plugins or rules to interact with SQLFluff on a deeper level or people who've decided to help the project by contributing to SQLFluff <#development>.

As these docs are some of the least commonly used, you may find that not all modules are documented directly here, and so you may find it instructive to read this together with docstrings and comments directly within the SQLFluff codebase on GitHub <https://github.com/sqlfluff/sqlfluff>.

  sqlfluff.core.config: Configuration & FluffConfig

When using the Python API, there are additional options for configuration beyond those specified in the Setting Configuration <#setting-config> section of the main docs. Internally, SQLFluff uses a consistent FluffConfig class which is then made accessible to different parts of the tool during linting and fixing.

As described in the Nesting <#nesting> section of the configuration docs, multiple nested documentation files can be used in a project and the result is a combined config object which contains the resulting union of those files. Under the hood, this is stored in a dict object, and it's possible get and set individual values, using get() & set_value(), but also get entire portions of that config dict using get_section().

  Methods for creating config mappings

When instantiating a FluffConfig object, there are a few options to set specific config values (such as dialect or rules), but to access the full available set of features it's best to pass in a dict of the values you want to set.

This config dict is a nested object, where the colon (:) characters from the .sqlfluff config files, delimit the keys. For example, take the following config file:

            [sqlfluff:rules:capitalisation.keywords]
            capitalisation_policy = lower

This would be represented in the config dict as below. See that the nested structure has been created by splitting the keys on the colon (:) characters:

            configs = {
                "rules":{
                    "capitalisation.keywords": {
                        "capitalisation_policy": "lower"
                    }
                }
            }

The following methods are provided to allow conversion of a selection of file formats into a consistent mapping object for instantiating a FluffConfig object.

    load_config_string(config_string: str, configs: ConfigMappingType | None = None, working_path: str | None = None) -> ConfigMappingType
        Load a config from a string in ini format.

        Parameters

            •   config_string (str) -- The raw config file as a string. The content is assumed to be in the the .ini format of a .sqlfluff file (i.e. not in .toml format).

            •   configs (ConfigMappingType, optional) -- A base set of configs to merge the loaded configs onto. If not provided, the result will contain only the values loaded from the string.

            •   working_path (str, optional) -- The working path to use for the resolution of any paths specified in the config. If not provided then os.getcwd() is used as a default.

        Returns
            A nested dictionary of config values.

        Return type
            ConfigMappingType

    load_config_file(file_dir: str, file_name: str, configs: ConfigMappingType | None = None) -> ConfigMappingType
        Load a config file from the filesystem.

        Parameters

            •   file_dir (str) -- The path to the location of file to be loaded. This should be a reference to the directory only and not include the filename itself. Any paths in the loaded file are resolved relative to this location.

            •   file_name (str) -- The filename of the file to be loaded. If the filename is pyproject.toml then the file is loaded in toml format, but otherwise is assumed to be in ini format (as per .sqlfluff).

            •   configs (ConfigMappingType, optional) -- A base set of configs to merge the loaded configs onto. If not provided, the result will contain only the values loaded from the string.

        Returns
            A nested dictionary of config values.

        Return type
            ConfigMappingType

    load_config_resource(package: str, file_name: str) -> ConfigMappingType
        Load a config resource from a python package.

        Parameters

            •   package (str) -- The name of the python package to load the resource from.

            •   file_name (str) -- The filename of the file to be loaded. If the filename is pyproject.toml then the file is loaded in toml format, but otherwise is assumed to be in ini format (as per .sqlfluff).

        Returns
            A nested dictionary of config values.

        Return type
            ConfigMappingType

    This is primarily used when loading configuration bundled with a SQLFluff plugin, or to load the default config for SQLFluff itself. By loading config from the package directly we avoid some of the path resolution which otherwise occurs. This is also more compatible with mypyc because it avoids the use of the __file__ attribute to find the default config.

    Any paths found in the loaded config are resolved relative to os.getcwd().

    For more information about resource loading, see the docs for importlib: <https://docs.python.org/3/library/importlib.resources.html>

    load_config_at_path(path: str) -> ConfigMappingType
        Load config files at a given path.

        Parameters
            path (str) -- The directory to search for config files.

        Returns
            A nested dictionary of config values.

        Return type
            ConfigMappingType

    This function will search for all valid config files at the given path, load any found and combine them into a config mapping. If multiple valid files are found, they are resolved in priority order, where pyproject.toml is given the highest precedence, followed by .sqlfluff, pep8.ini, tox.ini and finally setup.cfg.

    By accepting only a path string, we enable efficient caching of results, such that configuration can be reused between files without reloading the information from disk.

    load_config_up_to_path(path: str, extra_config_path: str | None = None, ignore_local_config: bool = False) -> ConfigMappingType
        Loads a selection of config files from both the path and its parent paths.

        Parameters

            •   path (str) -- The directory which is the target of the search. Config files in subdirectories will not be loaded by this method, but valid config files between this path and the current working path will.

            •   extra_config_path (str, optional) -- An additional path to load config from. This path is not used in iterating through intermediate paths, and is loaded last (taking the highest precedence in combining the loaded configs).

            •   ignore_local_config (bool, optional, defaults to False) -- If set to True, this skips loading configuration from the user home directory (~) or appdir path.

        Returns
            A nested dictionary of config values.

        Return type
            ConfigMappingType

    We layer each of the configs on top of each other, starting with any home or user configs (e.g. in appdir or home (~)), then any local project configuration and then any explicitly specified config paths.

  The FluffConfig object

    class FluffConfig(configs: ConfigMappingType | None = None, extra_config_path: str | None = None, ignore_local_config: bool = False, overrides: ConfigMappingType | None = None, plugin_manager: pluggy.PluginManager | None = None, require_dialect: bool = True)
        The persistent object for internal methods to access configuration.

        This class is designed to be instantiated once for each file and then be reused by each part of the process. For multiple files in the same path, a parent object will be created for the each path and then variants of it are created for each file. The object itself contains the references to any long lived objects which might be used by multiple parts of the codebase such as the dialect and the templater (both of which can be resource intensive to load & instantiate), which allows (for example), multiple files to reuse the same instance of the relevant dialect.

        It is also designed to pickle well for use in parallel operations.

        Parameters

            •   configs (ConfigMappingType, optional) -- A nested dict of config values from which to construct the config.

            •   extra_config_path (str, optional) -- An optional additional path to load config files from. These are loaded last if found and take precedence over any pre-existing config values. Note that when provided directly to the class, this path is not loaded for the class in question (it's assumed that has already been done, and the results are incorporated in the configs argument), but it is passed onward to child config instances, which will use it.

            •   ignore_local_config (bool, optional, defaults to False) -- If set to True, this skips loading configuration from the user home directory (~) or appdir path.

            •   overrides (ConfigMappingType, optional) -- A additional set of configs to merge into the core section of the config object at the end. These values take precedence over all other provided values and are inherited by child configs. For example, override values provided in the CLI use this method to apply to all files in a linting operation. Note that this mapping dict only applies to the core section and so cannot be used for all values.

            •   plugin_manager (PluginManager, optional) -- Optional pre-loaded config manager. Generally users should not need to provide this, as the class will fetch it's own if not provided. This argument is used when creating new class instances to avoid reloading the manager.

    Note:
            Methods for accessing internal properties on the config are not particularly standardised as the project currently assumes that few other tools are using this interface directly. If you or your project would like more formally supported methods for access to the config object, raise an issue on GitHub with the kind of things you'd like to achieve.

        copy() -> FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>
            Create a copy of this FluffConfig.

            Copies created using this method can safely be modified without those changes propagating back up to the object which was originally copied.

            Returns
                A shallow copy of this config object but with a deep copy of the internal _configs dict.

            Return type
                FluffConfig

        diff_to(other: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>) -> ConfigMappingType
            Compare this config to another.

            This is primarily used in the CLI logs to indicate to the user what values have been changed for each file compared to the root config for the project.

            Parameters
                other (FluffConfig) -- Another config object to compare against. We will return keys from this object that are not in other or are different to those in other.

            Returns
                A filtered dict of items in this config that are not in the other or are different to the other.

            Return type
                dict

        classmethod from_kwargs(dialect: str | None = None, rules: list[str] | None = None, exclude_rules: list[str] | None = None, require_dialect: bool = True) -> FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>
            Instantiate a config from a subset of common options.

            Parameters

                •   dialect (str, optional) -- The name of the dialect to use.

                •   rules (list of str, optional) -- A list of rules to include. Rule specifiers can be codes, names, groups or aliases. If not set, defaults to all rules.

                •   exclude_rules (list of str, optional) -- A list of rules to exclude. Rule specifiers can be codes, names, groups or aliases. If not set, does not exclude any rules.

                •   require_dialect (bool, optional, default is True) -- When True an error will be raise if the dialect config value is unset.

            Returns
                The loaded config object.

            Return type
                FluffConfig

        This is a convenience method for the ways that the public classes like Linter(), Parser() and Lexer() allow a subset of attributes to be set directly rather than requiring a pre-made FluffConfig.

        classmethod from_path(path: str, extra_config_path: str | None = None, ignore_local_config: bool = False, overrides: ConfigMappingType | None = None, plugin_manager: pluggy.PluginManager | None = None, require_dialect: bool = True) -> FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>
            Loads a config object given a particular path.

            Parameters

                •   path (str) -- The target path to load config files from. Files found between the working path and this path are also loaded and nested with files closest to this target path taking precedence.

                •   extra_config_path (str, optional) -- An optional additional path to load config files from. These are loaded last if found and take precedence over any pre-existing config values.

                •   ignore_local_config (bool, optional, defaults to False) -- If set to True, this skips loading configuration from the user home directory (~) or appdir path.

                •   overrides (ConfigMappingType, optional) -- A additional set of configs to merge into the core section of the config object at the end. These values take precedence over all other provided values and are inherited by child configs. Note that this mapping dict only applies to the core section and so cannot be used for all values.

                •   plugin_manager (PluginManager, optional) -- Optional pre-loaded config manager. Generally users should not need to provide this, as the class will fetch it's own if not provided. This argument is used when creating new class instances to avoid reloading the manager.

                •   require_dialect (bool, optional, default is True) -- When True an error will be raise if the dialect config value is unset.

            Returns
                The loaded config object.

            Return type
                FluffConfig

        classmethod from_root(extra_config_path: str | None = None, ignore_local_config: bool = False, overrides: ConfigMappingType | None = None, require_dialect: bool = True) -> FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>
            Loads a config object based on the root directory.

            Parameters

                •   extra_config_path (str, optional) -- An optional additional path to load config files from. These are loaded last if found and take precedence over any pre-existing config values.

                •   ignore_local_config (bool, optional, defaults to False) -- If set to True, this skips loading configuration from the user home directory (~) or appdir path.

                •   overrides (ConfigMappingType, optional) -- A additional set of configs to merge into the config object at the end. These values take precedence over all other provided values and are inherited by child configs. For example, override values provided in the CLI use this method to apply to all files in a linting operation.

                •   require_dialect (bool, optional, default is True) -- When True an error will be raise if the dialect config value is unset.

            Returns
                The loaded config object.

            Return type
                FluffConfig

        classmethod from_string(config_string: str, overrides: ConfigMappingType | None = None) -> FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>
            Loads a config object from a single config string.

            Parameters

                •   config_string (str) -- The config string, assumed to be in ini format (like a .sqlfluff file).

                •   overrides (ConfigMappingType, optional) -- A additional set of configs to merge into the config object at the end. These values take precedence over all other provided values and are inherited by child configs. For example, override values provided in the CLI use this method to apply to all files in a linting operation.

            Returns
                The loaded config object.

            Return type
                FluffConfig

        classmethod from_strings(*config_strings: str, overrides: ConfigMappingType | None = None) -> FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>
            Loads a config object given a series of nested config strings.

            Parameters

                •   *config_strings (str) -- An iterable of config strings, assumed to be in ini format (like a .sqlfluff file).

                •   overrides (ConfigMappingType, optional) -- A additional set of configs to merge into the config object at the end. These values take precedence over all other provided values and are inherited by child configs. For example, override values provided in the CLI use this method to apply to all files in a linting operation.

            Returns
                The loaded config object.

            Return type
                FluffConfig

        Config strings are incorporated from first to last, treating the first element as the "root" config, and then later config strings will take precedence over any earlier values.

        get(val: str, section: str | Iterable[str] = 'core', default: Any = None) -> Any
            Get a particular value from the config.

            Parameters

                •   val (str) -- The name of the config value to get.

                •   section (str or iterable of str, optional) -- The "path" to the config value. For values in the main [sqlfluff] section of the config, which are stored in the core section of the config this can be omitted.

                •   default -- The value to return if the config value was not found. If no default is provided, then a KeyError will be raised if no value was found.

        The following examples show how to fetch various default values:

            >>> FluffConfig(overrides={"dialect": "ansi"}).get("dialect")
            'ansi'

            >>> config = FluffConfig(overrides={"dialect": "ansi"})
            >>> config.get("tab_space_size", section="indentation")
            4

            >>> FluffConfig(overrides={"dialect": "ansi"}).get(
            ...     "capitalisation_policy",
            ...     section=["rules", "capitalisation.keywords"]
            ... )
            'consistent'

        get_section(section: str | Iterable[str]) -> Any
            Return a whole section of config as a dict.

            If the element found at the address is a value and not a section, it is still returned and so this can be used as a more advanced from of the basic get method.

            Parameters
                section -- An iterable or string. If it's a string we load that root section. If it's an iterable of strings, then we treat it as a path within the dictionary structure.

        get_templater(**kwargs: Any) -> RawTemplater
            Instantiate the configured templater.

        get_templater_class() -> type[RawTemplater]
            Get the configured templater class.

            Note:
                This is mostly useful to call directly when rules want to determine the type of a templater without (in particular to work out if it's a derivative of the jinja templater), without needing to instantiate a full templater. Instantiated templaters don't pickle well, so aren't automatically passed around between threads/processes.

        iter_vals(cfg: ConfigMappingType | None = None) -> Iterable[tuple[int, str, ConfigValueOrListType]]
            Return an iterable of tuples representing keys.

            Parameters
                cfg (optional) -- An optional config mapping to format instead. If not provided, we use the internal config object of the FluffConfig.

        This is primarily to enable formatting of config objects in the CLI.

        We show values before dicts, the tuple contains an indent value to know what level of the dict we're in. Dict labels will be returned as a blank value before their content.

        make_child_from_path(path: str, require_dialect: bool = True) -> FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>
            Make a child config at a path but pass on overrides and extra_config_path.

            Parameters

                •   path (str) -- The path to load the new config object from, inheriting the content of the calling FluffConfig as base values.

                •   require_dialect (bool, optional, default is True) -- When True an error will be raise if the dialect config value is unset.

            Returns
                A new config object which copies the current config object, but overriding any values set by config values loaded from the given path.

            Return type
                FluffConfig

        process_inline_config(config_line: str, fname: str) -> None
            Process an inline config command and update self.

            Parameters

                •   config_line (str) -- The inline config section to be processed. This should usually begin with -- sqlfluff:.

                •   fname (str) -- The name of the current file being processed. This is used purely for logging purposes in the case that an invalid config string is provided so that any error messages can reference the file with the issue.

            >>> cfg = FluffConfig(overrides={"dialect": "ansi"})
            >>> cfg.process_inline_config(
            ...     "-- sqlfluff:dialect:postgres",
            ...     "test.sql"
            ... )
            >>> cfg.get("dialect")
            'postgres'

        process_raw_file_for_config(raw_str: str, fname: str) -> None
            Process a full raw file for inline config and update self.

            Parameters

                •   raw_str (str) -- The full SQL script to evaluate for inline configs.

                •   fname (str) -- The name of the current file being processed. This is used purely for logging purposes in the case that an invalid config string is provided so that any error messages can reference the file with the issue.

            >>> cfg = FluffConfig(overrides={"dialect": "ansi"})
            >>> cfg.process_raw_file_for_config(
            ...     "-- sqlfluff:dialect:postgres",
            ...     "test.sql"
            ... )
            >>> cfg.get("dialect")
            'postgres'

        set_value(config_path: Iterable[str], val: Any) -> None
            Set a value at a given path.

            Parameters

                •   config_path -- An iterable of strings. Each should be a one of the elements which is colon delimited in a standard config file.

                •   val -- The value to set at the given path.

            >>> cfg = FluffConfig(overrides={"dialect": "ansi"})
            >>> cfg.set_value(["dialect"], "postgres")
            >>> cfg.get("dialect")
            'postgres'

            >>> cfg = FluffConfig(overrides={"dialect": "ansi"})
            >>> cfg.set_value(["indentation", "tab_space_size"], 2)
            >>> cfg.get("tab_space_size", section="indentation")
            2

        verify_dialect_specified() -> None
            Check if the config specifies a dialect, raising an error if not.

            Raises
                SQLFluffUserError -- If dialect config value is unset. The content
                    of the error contains user-facing instructions on what dialects
                    are available and how to set the dialect.

  sqlfluff.core.rules.base: Base Rule Classes

Implements the base rule class.

Rules crawl through the trees returned by the parser and evaluate particular rules.

The intent is that it should be possible for the rules to be expressed as simply as possible, with as much of the complexity abstracted away.

The evaluation function should take enough arguments that it can evaluate the position of the given segment in relation to its neighbors, and that the segment which finally "triggers" the error, should be the one that would be corrected OR if the rule relates to something that is missing, then it should flag on the segment FOLLOWING, the place that the desired element is missing.

    class BaseRule(code: str, description: str, **kwargs: Any)
        The base class for a rule.

        Parameters

            •   code (str) -- The identifier for this rule, used in inclusion or exclusion.

            •   description (str) -- A human readable description of what this rule does. It will be displayed when any violations are found.

        crawl(tree: BaseSegment, dialect: Dialect, fix: bool, templated_file: TemplatedFile | None, ignore_mask: IgnoreMask | None, fname: str | None, config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>) -> tuple[list[SQLLintError], tuple[RawSegment, ...], list[LintFix], dict[str, Any] | None]
            Run the rule on a given tree.

            Returns
                A tuple of (vs, raw_stack, fixes, memory)

        static discard_unsafe_fixes(lint_result: LintResult <#sqlfluff.core.rules.base.LintResult>, templated_file: TemplatedFile | None) -> None
            Remove (discard) LintResult fixes if they are "unsafe".

            By removing its fixes, a LintResult will still be reported, but it will be treated as _unfixable_.

        static filter_meta(segments: Sequence[BaseSegment], keep_meta: bool = False) -> tuple[BaseSegment, ...]
            Filter the segments to non-meta.

            Or optionally the opposite if keep_meta is True.

        classmethod get_config_ref() -> str
            Return the config lookup ref for this rule.

            If a name is defined, it's the name - otherwise the code.

            The name is a much more understandable reference and so makes config files more readable. For backward compatibility however we also support the rule code for those without names.

        classmethod get_parent_of(segment: BaseSegment, root_segment: BaseSegment) -> BaseSegment | None
            Return the segment immediately containing segment.

            NB: This is recursive.

            Parameters

                •   segment -- The segment to look for.

                •   root_segment -- Some known parent of the segment we're looking for (although likely not the direct parent in question).

        static split_comma_separated_string(raw: str | list[str]) -> list[str]
            Converts comma separated string to List, stripping whitespace.

    class LintResult(anchor: BaseSegment | None = None, fixes: list[LintFix] | None = None, memory: Any | None = None, description: str | None = None, source: str | None = None)
        A class to hold the results of a rule evaluation.

        Parameters

            •   anchor (BaseSegment, optional) -- A segment which represents the position of the problem. NB: Each fix will also hold its own reference to position, so this position is mostly for alerting the user to where the problem is.

            •   fixes (list of LintFix, optional) -- An array of any fixes which would correct this issue. If not present then it's assumed that this issue will have to manually fixed.

            •   memory (dict, optional) -- An object which stores any working memory for the rule. The memory returned in any LintResult will be passed as an input to the next segment to be crawled.

            •   description (str, optional) -- A description of the problem identified as part of this result. This will override the description of the rule as what gets reported to the user with the problem if provided.

            •   source (str, optional) -- A string identifier for what generated the result. Within larger libraries like reflow this can be useful for tracking where a result came from.

        to_linting_error(rule: BaseRule <#sqlfluff.core.rules.base.BaseRule>) -> SQLLintError | None
            Convert a linting result to a SQLLintError if appropriate.

    class RuleGhost(code, name, description)

        code
            Alias for field number 0

        description
            Alias for field number 2

        name
            Alias for field number 1

    class RuleLoggingAdapter(logger, extra=None, merge_extra=False)
        A LoggingAdapter for rules which adds the code of the rule to it.

        process(msg: str, kwargs: Any) -> tuple[str, Any]
            Add the code element to the logging message before emit.

    class RuleManifest(code: str, name: str, description: str, groups: tuple[str, ...], aliases: tuple[str, ...], rule_class: type[BaseRule <#sqlfluff.core.rules.base.BaseRule>])
        Element in the rule register.

    class RuleMetaclass(name: str, bases: list[BaseRule <#sqlfluff.core.rules.base.BaseRule>], class_dict: dict[str, Any])
        The metaclass for rules.

        This metaclass provides provides auto-enrichment of the rule docstring so that examples, groups, aliases and names are added.

        The reason we enrich the docstring is so that it can be picked up by autodoc and all be displayed in the sqlfluff docs.

    class RulePack(rules: list[BaseRule <#sqlfluff.core.rules.base.BaseRule>], reference_map: dict[str, set[str]])
        A bundle of rules to be applied.

        This contains a set of rules, post filtering but also contains the mapping required to interpret any noqa messages found in files.

        The reason for this object is that rules are filtered and instantiated into this pack in the main process when running in multi-processing mode so that user defined rules can be used without reference issues.

        rules
            A filtered list of instantiated rules to be applied to a given file.

            Type
                list of BaseRule

        reference_map
            A mapping of rule references to the codes they refer to, e.g. {"my_ref": {"LT01", "LT02"}}. The references (i.e. the keys) may be codes, groups, aliases or names. The values of the mapping are sets of rule codes only. This object acts as a lookup to be able to translate selectors (which may contain diverse references) into a consolidated list of rule codes. This mapping contains the full set of rules, rather than just the filtered set present in the rules attribute.

            Type
                dict

        codes() -> Iterator[str]
            Returns an iterator through the codes contained in the pack.

    class RuleSet(name: str, config_info: dict[str, ConfigInfo])
        Class to define a ruleset.

        A rule set is instantiated on module load, but the references to each of its classes are instantiated at runtime. This means that configuration values can be passed to those rules live and be responsive to any changes in configuration from the path that the file is in.

        Rules should be fetched using the get_rulelist() command which also handles any filtering (i.e. allowlisting and denylisting).

        New rules should be added to the instance of this class using the register() decorator. That decorator registers the class, but also performs basic type and name-convention checks.

        The code for the rule will be parsed from the name, the description from the docstring. The eval function is assumed that it will be overridden by the subclass, and the parent class raises an error on this function if not overridden.

        copy() -> RuleSet <#sqlfluff.core.rules.base.RuleSet>
            Return a copy of self with a separate register.

        get_rulepack(config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>) -> RulePack <#sqlfluff.core.rules.base.RulePack>
            Use the config to return the appropriate rules.

            We use the config both for allowlisting and denylisting, but also for configuring the rules given the given config.

        register(cls: type[BaseRule <#sqlfluff.core.rules.base.BaseRule>], plugin: PluginSpec | None = None) -> type[BaseRule <#sqlfluff.core.rules.base.BaseRule>]
            Decorate a class with this to add it to the ruleset.

                    @myruleset.register
                    class Rule_LT01(BaseRule):
                        "Description of rule."

                        def eval(self, **kwargs):
                            return LintResult()

        We expect that rules are defined as classes with the name Rule_XXXX where XXXX is of the form LNNN, where L is a letter (literally L for linting by default) and N is a three digit number.

        If this receives classes by any other name, then it will raise a ValueError.

        rule_reference_map() -> dict[str, set[str]]
            Generate a rule reference map for looking up rules.

            Generate the master reference map. The priority order is: codes > names > groups > aliases (i.e. if there's a collision between a name and an alias - we assume the alias is wrong)

  sqlfluff.utils.functional: Functional Traversal API

These newer modules provide a higher-level API for rules working with segments and slices. Rules that need to navigate or search the parse tree may benefit from using these. Eventually, the plan is for all rules to use these modules. As of December 30, 2021, 17+ rules use these modules.

The modules listed below are submodules of sqlfluff.utils.functional.

  segments Module

Surrogate class for working with Segment collections.

    class Segments(*segments: BaseSegment, templated_file: TemplatedFile | None = None)
        Encapsulates a sequence of one or more BaseSegments.

        The segments may or may not be contiguous in a parse tree. Provides useful operations on a sequence of segments to simplify rule creation.

        all(predicate: Callable[[BaseSegment], bool] | None = None) -> bool
            Do all the segments match?

        any(predicate: Callable[[BaseSegment], bool] | None = None) -> bool
            Do any of the segments match?

        apply(fn: Callable[[BaseSegment], Any]) -> list[Any]
            Apply function to every item.

        children(predicate: Callable[[BaseSegment], bool] | None = None) -> Segments <#sqlfluff.utils.functional.segments.Segments>
            Returns an object with children of the segments in this object.

        find(segment: BaseSegment | None) -> int
            Returns index if found, -1 if not found.

        first(predicate: Callable[[BaseSegment], bool] | None = None) -> Segments <#sqlfluff.utils.functional.segments.Segments>
            Returns the first segment (if any) that satisfies the predicates.

        get(index: int = 0, *, default: BaseSegment | None = None) -> BaseSegment | None
            Return specified item. Returns default if index out of range.

        iterate_segments(predicate: Callable[[BaseSegment], bool] | None = None) -> Iterable[Segments <#sqlfluff.utils.functional.segments.Segments>]
            Loop over each element as a fresh Segments.

        last(predicate: Callable[[BaseSegment], bool] | None = None) -> Segments <#sqlfluff.utils.functional.segments.Segments>
            Returns the last segment (if any) that satisfies the predicates.

        property raw_segments: Segments <#sqlfluff.utils.functional.segments.Segments>
            Get raw segments underlying the segments.

        property raw_slices: RawFileSlices <#sqlfluff.utils.functional.raw_file_slices.RawFileSlices>
            Raw slices of the segments, sorted in source file order.

        recursive_crawl(*seg_type: str, recurse_into: bool = True) -> Segments <#sqlfluff.utils.functional.segments.Segments>
            Recursively crawl for segments of a given type.

        recursive_crawl_all() -> Segments <#sqlfluff.utils.functional.segments.Segments>
            Recursively crawl all descendant segments.

        reversed() -> Segments <#sqlfluff.utils.functional.segments.Segments>
            Return the same segments in reverse order.

        select(select_if: Callable[[BaseSegment], bool] | None = None, loop_while: Callable[[BaseSegment], bool] | None = None, start_seg: BaseSegment | None = None, stop_seg: BaseSegment | None = None) -> Segments <#sqlfluff.utils.functional.segments.Segments>
            Retrieve range/subset.

            NOTE: Iterates the segments BETWEEN start_seg and stop_seg, i.e. those segments are not included in the loop.

  segment_predicates Module

Defines commonly used segment predicates for rule writers.

For consistency, all the predicates in this module are implemented as functions returning functions. This avoids rule writers having to remember the distinction between normal functions and functions returning functions.

This is not necessarily a complete set of predicates covering all possible requirements. Rule authors can define their own predicates as needed, either as regular functions, lambda, etc.

    and_(*functions: Callable[[BaseSegment], bool]) -> Callable[[BaseSegment], bool]
        Returns a function that computes the functions and-ed together.

    get_type() -> Callable[[BaseSegment], str]
        Returns a function that gets segment type.

    is_code() -> Callable[[BaseSegment], bool]
        Returns a function that checks if segment is code.

    is_comment() -> Callable[[BaseSegment], bool]
        Returns a function that checks if segment is comment.

    is_keyword(*keyword_name: str) -> Callable[[BaseSegment], bool]
        Returns a function that determines if it's a matching keyword.

    is_meta() -> Callable[[BaseSegment], bool]
        Returns a function that checks if segment is meta.

    is_raw() -> Callable[[BaseSegment], bool]
        Returns a function that checks if segment is raw.

    is_templated() -> Callable[[BaseSegment], bool]
        Returns a function that checks if segment is templated.

    is_type(*seg_type: str) -> Callable[[BaseSegment], bool]
        Returns a function that determines if segment is one of the types.

    is_whitespace() -> Callable[[BaseSegment], bool]
        Returns a function that checks if segment is whitespace.

    not_(fn: Callable[[BaseSegment], bool]) -> Callable[[BaseSegment], bool]
        Returns a function that computes: not fn().

    or_(*functions: Callable[[BaseSegment], bool]) -> Callable[[BaseSegment], bool]
        Returns a function that computes the functions or-ed together.

    raw_is(*raws: str) -> Callable[[BaseSegment], bool]
        Returns a function that determines if segment matches one of the raw inputs.

    raw_slices(segment: BaseSegment, templated_file: TemplatedFile | None) -> RawFileSlices <#sqlfluff.utils.functional.raw_file_slices.RawFileSlices>
        Returns raw slices for a segment.

    raw_upper_is(*raws: str) -> Callable[[BaseSegment], bool]
        Returns a function that determines if segment matches one of the raw inputs.

    templated_slices(segment: BaseSegment, templated_file: TemplatedFile | None) -> TemplatedFileSlices
        Returns raw slices for a segment.

  raw_file_slices Module

Surrogate class for working with RawFileSlice collections.

    class RawFileSlices(*raw_slices: RawFileSlice, templated_file: TemplatedFile | None = None)
        Encapsulates a sequence of one or more RawFileSlice.

        The slices may or may not be contiguous in a file. Provides useful operations on a sequence of slices to simplify rule creation.

        all(predicate: Callable[[RawFileSlice], bool] | None = None) -> bool
            Do all the raw slices match?

        any(predicate: Callable[[RawFileSlice], bool] | None = None) -> bool
            Do any of the raw slices match?

        select(select_if: Callable[[RawFileSlice], bool] | None = None, loop_while: Callable[[RawFileSlice], bool] | None = None, start_slice: RawFileSlice | None = None, stop_slice: RawFileSlice | None = None) -> RawFileSlices <#sqlfluff.utils.functional.raw_file_slices.RawFileSlices>
            Retrieve range/subset.

            NOTE: Iterates the slices BETWEEN start_slice and stop_slice, i.e. those slices are not included in the loop.

  raw_file_slice_predicates Module

Defines commonly used raw file slice predicates for rule writers.

For consistency, all the predicates in this module are implemented as functions returning functions. This avoids rule writers having to remember the distinction between normal functions and functions returning functions.

This is not necessarily a complete set of predicates covering all possible requirements. Rule authors can define their own predicates as needed, either as regular functions, lambda, etc.

    is_slice_type(*slice_types: str) -> Callable[[RawFileSlice], bool]
        Returns a function that determines if segment is one of the types.

  sqlfluff.utils.reflow: Whitespace Reflow API

Many rules supported by SQLFluff involve the spacing and layout of different elements, either to enforce a particular layout or just to add or remove code elements in a way sensitive to the existing layout configuration. The way this is achieved is through some centralised utilities in the sqlfluff.utils.reflow module.

This module aims to achieve several things:

    •   Less code duplication by implementing reflow logic in only one place.

    •   Provide a streamlined interface for rules to easily utilise reflow logic.

        •   Given this requirement, it's important that reflow utilities work within the existing framework for applying fixes to potentially templated code. We achieve this by returning LintFix objects which can then be returned by each rule wanting to use this logic.

    •   Provide a consistent way of configuring layout requirements. For more details on configuration see Configuring Layout <#layoutconfig>.

To support this, the module provides a ReflowSequence class which allows access to all of the relevant operations which can be used to reformat sections of code, or even a whole file. Unless there is a very good reason, all rules should use this same approach to ensure consistent treatment of layout.

    class ReflowSequence(elements: list[ReflowBlock <#sqlfluff.utils.reflow.elements.ReflowBlock> | ReflowPoint <#sqlfluff.utils.reflow.elements.ReflowPoint>], root_segment: BaseSegment, reflow_config: ReflowConfig, depth_map: DepthMap, lint_results: list[LintResult <#sqlfluff.core.rules.base.LintResult>] | None = None)
        Class for keeping track of elements in a reflow operation.

        This acts as the primary route into using the reflow routines. It acts in a way that plays nicely within a rule context in that it accepts segments and configuration, while allowing access to modified segments and a series of LintFix objects, which can be returned by the calling rule.

        Sequences are made up of alternating ReflowBlock and ReflowPoint objects (even if some points have no segments). This is validated on construction.

        Most operations also return ReflowSequence objects such that operations can be chained, and then the resultant fixes accessed at the last stage, for example:

                fixes = (
                    ReflowSequence.from_around_target(
                        context.segment,
                        root_segment=context.parent_stack[0],
                        config=context.config,
                    )
                    .rebreak()
                    .get_fixes()
                )

        break_long_lines() -> ReflowSequence <#sqlfluff.utils.reflow.ReflowSequence>
            Rebreak any remaining long lines in a sequence.

            This assumes that reindent() has already been applied.

        classmethod from_around_target(target_segment: BaseSegment, root_segment: BaseSegment, config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>, sides: str = 'both') -> ReflowSequence <#sqlfluff.utils.reflow.ReflowSequence>
            Generate a sequence around a target.

            Parameters

                •   target_segment (RawSegment) -- The segment to center around when considering the sequence to construct.

                •   root_segment (BaseSegment) -- The relevant root segment (usually the base FileSegment).

                •   config (FluffConfig) -- A config object from which to load the spacing behaviours of different segments.

                •   sides (str) -- Limit the reflow sequence to just one side of the target. Default is two sided ("both"), but set to "before" or "after" to limit to either side.

        NOTE: We don't just expand to the first block around the target but to the first code element, which means we may swallow several comment blocks in the process.

        To evaluate reflow around a specific target, we need need to generate a sequence which goes for the preceding raw to the following raw. i.e. at least: block - point - block - point - block (where the central block is the target).

        classmethod from_raw_segments(segments: Sequence[RawSegment], root_segment: BaseSegment, config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>, depth_map: DepthMap | None = None) -> ReflowSequence <#sqlfluff.utils.reflow.ReflowSequence>
            Construct a ReflowSequence from a sequence of raw segments.

            This is intended as a base constructor, which others can use. In particular, if no depth_map argument is provided, this method will generate one in a potentially inefficient way. If the calling method has access to a better way of inferring a depth map (for example because it has access to a common root segment for all the content), it should do that instead and pass it in.

        classmethod from_root(root_segment: BaseSegment, config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>) -> ReflowSequence <#sqlfluff.utils.reflow.ReflowSequence>
            Generate a sequence from a root segment.

            Parameters

                •   root_segment (BaseSegment) -- The relevant root segment (usually the base FileSegment).

                •   config (FluffConfig) -- A config object from which to load the spacing behaviours of different segments.

        get_fixes() -> list[LintFix]
            Get the current fix buffer.

            We're hydrating them here directly from the LintResult objects, so for more accurate results, consider using .get_results(). This method is particularly useful when consolidating multiple results into one.

        get_raw() -> str
            Get the current raw representation.

        get_results() -> list[LintResult <#sqlfluff.core.rules.base.LintResult>]
            Return the current result buffer.

        insert(insertion: RawSegment, target: RawSegment, pos: str = 'before') -> ReflowSequence <#sqlfluff.utils.reflow.ReflowSequence>
            Returns a new ReflowSequence with the new element inserted.

            Insertion is always relative to an existing element. Either before or after it as specified by pos. This generates appropriate creation LintFix objects to direct the linter to insert those elements.

            Note:
                This method is currently excluded from test coverage because it is not actively used by any rule. It was previously utilized by rule AL01, but that rule now uses an alternative implementation. The method is retained for potential future reuse or reference.

        rebreak(rebreak_type: Literal['lines', 'keywords'] = 'lines') -> ReflowSequence <#sqlfluff.utils.reflow.ReflowSequence>
            Returns a new ReflowSequence corrected line breaks.

            This intentionally does not handle indentation, as the existing indents are assumed to be correct.

            Note:
                Currently this only moves existing segments around line breaks (e.g. for operators and commas), but eventually this method will also handle line length considerations too.

        reindent() -> ReflowSequence <#sqlfluff.utils.reflow.ReflowSequence>
            Reindent lines within a sequence.

        replace(target: BaseSegment, edit: Sequence[BaseSegment]) -> ReflowSequence <#sqlfluff.utils.reflow.ReflowSequence>
            Returns a new ReflowSequence with edit elements replaced.

            This generates appropriate replacement LintFix objects to direct the linter to modify those elements.

        respace(strip_newlines: bool = False, filter: str = 'all') -> ReflowSequence <#sqlfluff.utils.reflow.ReflowSequence>
            Returns a new ReflowSequence with points respaced.

            Parameters

                •   strip_newlines (bool) -- Optionally strip newlines before respacing. This is primarily used on focused sequences to coerce objects onto a single line. This does not apply any prioritisation to which line breaks to remove and so is not a substitute for the full reindent or reflow methods.

                •   filter (str) -- Optionally filter which reflow points to respace. Default configuration is all. Other options are line_break which only respaces points containing a newline or followed by an end_of_file marker, or inline which is the inverse of line_break. This is most useful for filtering between trailing whitespace and fixes between content on a line.

        NOTE this method relies on the embodied results being correct so that we can build on them.

        without(target: RawSegment) -> ReflowSequence <#sqlfluff.utils.reflow.ReflowSequence>
            Returns a new ReflowSequence without the specified segment.

            This generates appropriate deletion LintFix objects to direct the linter to remove those elements.

            Note:
                This method is currently excluded from test coverage because it is not actively used by any rule. It was previously utilized by rule AL01, but that rule now uses an alternative implementation. The method is retained for potential future reuse or reference.

    class ReflowPoint(segments: tuple[RawSegment, ...])
        Class for keeping track of editable elements in reflow.

        This class, and its sibling ReflowBlock, should not normally be manipulated directly by rules, but instead should be manipulated using ReflowSequence.

        It holds segments which can be changed during a reflow operation such as whitespace and newlines.It may also contain Indent and Dedent elements.

        It holds no configuration and is influenced by the blocks on either side, so that any operations on it usually have that configuration passed in as required.

        property class_types: set[str]
            Get the set of contained class types.

            Parallel to BaseSegment.class_types

        get_indent() -> str | None
            Get the current indent (if there).

        get_indent_impulse() -> IndentStats
            Get the change in intended indent balance from this point.

        get_indent_segment_vals(exclude_block_indents=False) -> list[int]
            Iterate through any indent segments and extract their values.

        indent_to(desired_indent: str, after: BaseSegment | None = None, before: BaseSegment | None = None, description: str | None = None, source: str | None = None) -> tuple[list[LintResult <#sqlfluff.core.rules.base.LintResult>], ReflowPoint <#sqlfluff.utils.reflow.elements.ReflowPoint>]
            Coerce a point to have a particular indent.

            If the point currently contains no newlines, one will be introduced and any trailing whitespace will be effectively removed.

            More specifically, the newline is inserted before the existing whitespace, with the new indent being a replacement for that same whitespace.

            For placeholder newlines or indents we generate appropriate source fixes.

        is_all_unrendered() -> bool
            Return whether this element is all unrendered.

            Returns True if contains only whitespace, indents, template loops or placeholders.

            Note: * ReflowBlocks will contain the placeholders and loops * ReflowPoints will contain whitespace, indents and newlines.

        num_newlines() -> int
            Return the number of newlines in this element.

            These newlines are either newline segments or contained within consumed sections of whitespace. This counts both.

        property pos_marker: PositionMarker | None
            Get the first position marker of the element.

        property raw: str
            Get the current raw representation.

        respace_point(prev_block: ReflowBlock <#sqlfluff.utils.reflow.elements.ReflowBlock> | None, next_block: ReflowBlock <#sqlfluff.utils.reflow.elements.ReflowBlock> | None, root_segment: BaseSegment, lint_results: list[LintResult <#sqlfluff.core.rules.base.LintResult>], strip_newlines: bool = False, anchor_on: str = 'before') -> tuple[list[LintResult <#sqlfluff.core.rules.base.LintResult>], ReflowPoint <#sqlfluff.utils.reflow.elements.ReflowPoint>]
            Respace a point based on given constraints.

            NB: This effectively includes trailing whitespace fixes.

            Deletion and edit fixes are generated immediately, but creations are paused to the end and done in bulk so as not to generate conflicts.

            Note that the strip_newlines functionality exists here as a slight exception to pure respacing, but as a very simple case of positioning line breaks. The default operation of respace does not enable it, however it exists as a convenience for rules which wish to use it.

    class ReflowBlock(segments: tuple[RawSegment, ...], spacing_before: str, spacing_after: str, line_position: str | None, depth_info: DepthInfo, stack_spacing_configs: dict[int, str], line_position_configs: dict[int, str], keyword_line_position: str | None, keyword_line_position_configs: dict[int, str], keyword_line_position_exclusions: str | list[str] | None, keyword_line_position_exclusions_configs: dict[int, str | list[str]])
        Class for keeping track of elements to reflow.

        This class, and its sibling ReflowPoint, should not normally be manipulated directly by rules, but instead should be manipulated using ReflowSequence.

        It holds segments to reflow and also exposes configuration regarding how they are expected to reflow around others. Typically it holds only a single element, which is usually code or a templated element. Because reflow operations control spacing, it would be very unusual for this object to be modified; as such it exposes relatively few methods.

        The attributes exposed are designed to be "post configuration" i.e. they should reflect configuration appropriately.

        property class_types: set[str]
            Get the set of contained class types.

            Parallel to BaseSegment.class_types

        depth_info: DepthInfo
            Metadata on the depth of this segment within the parse tree which is used in inferring how and where line breaks should exist.

        classmethod from_config(segments: tuple[RawSegment, ...], config: ReflowConfig, depth_info: DepthInfo) -> ReflowBlock <#sqlfluff.utils.reflow.elements.ReflowBlock>
            Construct a ReflowBlock while extracting relevant configuration.

            This is the primary route to construct a ReflowBlock, as is allows all of the inference of the spacing and position configuration from the segments it contains and the appropriate config objects.

        is_all_unrendered() -> bool
            Return whether this element is all unrendered.

            Returns True if contains only whitespace, indents, template loops or placeholders.

            Note: * ReflowBlocks will contain the placeholders and loops * ReflowPoints will contain whitespace, indents and newlines.

        keyword_line_position: str | None
            Desired line position for this block's keywords. See Configuring layout and spacing <#layoutspacingconfig>

        keyword_line_position_configs: dict[int, str]
            Desired keyword line position configurations for parent segments of the segment in this block. See Configuring layout and spacing <#layoutspacingconfig>

        keyword_line_position_exclusions: str | list[str] | None
            Parent segments which this block's keyword line positioning should not apply. See Configuring layout and spacing <#layoutspacingconfig>

        keyword_line_position_exclusions_configs: dict[int, str | list[str]]
            Configurations for parent segments which this block's keyword line positioning should not apply. See Configuring layout and spacing <#layoutspacingconfig>

        line_position: str | None
            Desired line position for this block. See Configuring layout and spacing <#layoutspacingconfig>

        line_position_configs: dict[int, str]
            Desired line position configurations for parent segments of the segment in this block. See Configuring layout and spacing <#layoutspacingconfig>

        num_newlines() -> int
            Return the number of newlines in this element.

            These newlines are either newline segments or contained within consumed sections of whitespace. This counts both.

        property pos_marker: PositionMarker | None
            Get the first position marker of the element.

        property raw: str
            Get the current raw representation.

        spacing_after: str
            Desired spacing after this block. See Configuring layout and spacing <#layoutspacingconfig>

        spacing_before: str
            Desired spacing before this block. See Configuring layout and spacing <#layoutspacingconfig>

        stack_spacing_configs: dict[int, str]
            Desired spacing configurations for parent segments of the segment in this block. See Configuring layout and spacing <#layoutspacingconfig>

  Release Notes

This page aims to act as a guide for migrating between major SQLFluff releases. Necessarily this means that bugfix releases, or releases requiring no change for the user are not mentioned. For full details of each individual release, see the detailed changelog <https://github.com/sqlfluff/sqlfluff/blob/main/CHANGELOG.md>.

  Upgrading to 3.x

This release makes a couple of potentially breaking changes:

    •   It drops support for python 3.7, which reached end of life in June 2023.

    •   It migrates to pyproject.toml rather than setup.cfg as the python packaging configuration file (although keeping setuptools as the default backend).

    •   The serialised output for sqlfluff lint now contains more information about the span of linting issues and initial proposed fixes. Beside the new fields, the original fields of line_pos and line_no have been renamed to start_line_pos and start_line_no, to distinguish them from the new fields starting end_*.

    •   When linting from stdin, if there are no violations found - before this version, the serialised response would be simply an empty list ([]). From 3.0 onwards, there will now be a record for the file with some statistics, but the violations section of the response for that file will still be an empty list.

    •   The default annotation_level set by the --annotation-level option on the sqlfluff lint command has been changed from notice to warning, to better distinguish linting errors from warnings, which always now have the level of notice. This is only relevant when using the github-annotation or github-annotation-native formats.

    •   The previously deprecated --disable_progress_bar on :code:lint, fix and format has now been removed entirely. Please migrate to --disable-progress-bar to continue using this option.

    •   The --force option on sqlfluff fix is now the default behaviour and so the option has been deprecated. A new --check option has been introduced which mimics the old default behaviour. This has been changed as it enables significantly lower memory overheads when linting and fixing large projects.

  Upgrading to 2.3

This release include two minor breaking changes which will only affect users engaged in performance optimisation of SQLFluff itself.

    •   The --profiler option on sqlfluff parse has been removed. It was only present on the parse command and not lint or fix, and it is just as simple to invoke the python cProfiler directly.

    •   The --recurse cli option and sqlfluff.recurse configuration option have both been removed. They both existed purely for debugging the parser, and were never used in a production setting. The improvement in other debugging messages when unparsable sections are found means that this option is no longer necessary.

  Upgrading to 2.2

This release changes some of the interfaces between SQLFluff core and our plugin ecosystem. The only breaking change is in the interface between SQLFluff and templater plugins (which are not common in the ecosystem, hence why this is only a minor and not a major release).

For all plugins, we also recommend a different structure for their imports (especially for rule plugins which are more common in the ecosystem) - for performance and stability reasons. Some users had been experiencing very long import times with previous releases as a result of the layout of plugin imports. Users with affected plugins will begin to see a warning from this release onward, which can be resolved for their plugin by updating to a new version of that plugin which follows the guidelines.

  Templater plugins

Templaters before this version would pass a make_template() callable to the slicing methods as part of being able to map the source file. This method would accept a str and return a jinja2.environment.Template object to allow the templater to render multiple variants of the template to do the slicing operation (which allows linting issues found in templated files to be mapped accurately back to their position in the unrendered source file). This approach is not very generalisable, and did not support templating operations with libraries other than jinja2.

As a result, we have amended the interface to instead pass a render_func() callable, which accepts a str and returns a str. This works fine for the jinja templater (and by extension the dbt templater) as they can simply wrap the original callable with a method that calls render() on the original Template object. It also however opens up the door to other templating engines, and in particular to remote templaters which might pass unrendered code over a HTTP connection for rendering.

Specifically:

    •   The slice_file() method of the base templater classes no longer accepts an optional make_template argument or a templated_str argument.

    •   Instead a render_func callable should be passed which can be called to generate the templated_str on demand.

    •   Unlike the optional make_template - render_func is not optional and should always be present.

  Rule plugins

We recommend that the module in a plugin which defines all of the hook implementations (anything using the @hookimpl decorator) must be able to fully import before any rule implementations are imported. More specifically, SQLFluff must be able to both import and run any implementations of get_configs_info() before any plugin rules (i.e. any derivatives of BaseRule <#sqlfluff.core.rules.base.BaseRule>) are imported. Because of this, we recommend that rules are defined in a separate module to the root of the plugin and then only imported within the get_rules() method.

Importing in the main body of the module was previously our recommendation and so may be the case for versions of some plugins. If one of your plugins does use imports in this way, a warning will be presented from this version onward, recommending that you update your plugin.

See the Developing Plugins <#developingpluginsref> section of the docs for an example.

  Upgrading from 1.x to 2.0

Upgrading to 2.0 brings several important breaking changes:

    •   All bundled rules have been recoded, both from generic L00X formats into groups within similar codes (e.g. an aliasing group with codes of the format AL0X), but also given names to allow much clearer referencing (e.g. aliasing.column).

    •   Rule Configuration <#ruleconfig> now uses the rule name rather than the rule code to specify the section. Any unrecognised references in config files (whether they are references which do match existing rules by code or alias, or whether the match no rules at all) will raise warnings at runtime.

    •   A complete re-write of layout and whitespace handling rules (see Layout & Whitespace Configuration <#layoutref>), and with that a change in how layout is configured (see Configuring Layout <#layoutconfig>) and the combination of some rules that were previously separate. One example of this is that the legacy rules L001, L005, L006, L008, L023, L024, L039, L048 & L071 have been combined simply into LT01 <#rule-LT01>.

  Recommended upgrade steps

To upgrade smoothly between versions, we recommend the following sequence:

    1.  The upgrade path will be simpler if you have a slimmer configuration file. Before upgrading, consider removing any sections from your configuration file (often .sqlfluff, see Configuration <#config>) which match the current Default Configuration <#defaultconfig>. There is no need to respecify defaults in your local config if they are not different to the stock config.

    2.  In a local (or other non-production) environment, upgrade to SQLFluff 2.0.x. We recommend using a compatible release <https://peps.python.org/pep-0440/#compatible-release> specifier such as ~=2.0.0, to ensure any minor bugfix releases are automatically included.

    3.  Examine your configuration file (as mentioned above), and evaluate how rules are currently specified. We recommend primarily using either rules or exclude_rules rather than both, as detailed in Enabling and Disabling Rules <#ruleselection>. Using either the sqlfluff rules CLI command or the online Rules Reference <#ruleref>, replace all references to legacy rule codes (i.e. codes of the form L0XX). Specifically:

        •   In the rules and exclude_rules config values. Here, consider using group specifiers or names to make your config simpler to read and understand (e.g. capitalisation, is much more understandable than CP01,CP02,CP03,CP04,CP05, but the two specifiers will have the same effect). Note that while legacy codes will still be understood here (because they remain valid as aliases for those rules) - you may find that some rules no longer exist in isolation and so these references may be misleading. e.g. L005 is now an alias for layout.spacing <#rule-layout.spacing> but that rule is much more broad ranging than the original scope of L005, which was only spacing around commas.

        •   In Rule Configuration <#ruleconfig>. In particular here, legacy references to rule codes are no longer valid, will raise warnings, and until resolved, the configuration in those sections will be ignored. The new section references should include the rule name (e.g. [sqlfluff:rules:capitalisation.keywords] rather than [sqlfluff:rules:L010]). This switch is designed to make configuration files more readable, but we cannot support backward compatibility here without also having to resolve the potential ambiguity of the scenario where both code-based and name-based are both used.

        •   Review the Configuring Layout <#layoutconfig> documentation, and check whether any indentation or layout configuration should be revised.

    4.  Check your project for In-File Configuration Directives <#in-file-config> which refer to rule codes. Alter these in the same manner as described above for configuration files.

    5.  Test linting your project for unexpected linting issues. Where found, consider whether to use sqlfluff fix to repair them in bulk, or (if you disagree with the changes) consider changing which rules you enable or their configuration accordingly. In particular you may notice:

        •   The indentation rule (L003 as was, now LT02 <#rule-LT02>) has had a significant rewrite, and while much more flexible and accurate, it is also more specific. Note that Hanging Indents <#hangingindents> are no longer supported, and that while not enabled by default, many users may find the enabling Implicit Indents <#implicitindents> fits their organisation's style better.

        •   The spacing rule (LT01 <#rule-LT01>: layout.spacing <#rule-layout.spacing>) has a much wider scope, and so may pick up spacing issues that were not previously enforced. If you disagree with any of these, you can override the sqlfluff:layout sections of the config with different (or just more liberal settings, like any).

  Example 2.0 config

To illustrate the points above, this is an illustrative example config for a 2.0 compatible project. Note that the config is fairly brief and sets only the values which differ from the default config.

            [sqlfluff]
            dialect = snowflake
            templater = dbt
            max_line_length = 120

            # Exclude some specific rules based on a mixture of codes and names
            exclude_rules = RF02, RF03, RF04, ST06, ST07, AM05, AM06, convention.left_join, layout.select_targets

            [sqlfluff:indentation]
            # Enabling implicit indents for this project.
            # See https://docs.sqlfluff.com/en/stable/perma/indent_locations.html
            allow_implicit_indents = True

            # Add a few specific rule configurations, referenced by the rule names
            # and not by the rule codes.
            [sqlfluff:rules:capitalisation.keywords]
            capitalisation_policy = lower

            [sqlfluff:rules:capitalisation.identifiers]
            capitalisation_policy = lower

            [sqlfluff:rules:capitalisation.functions]
            extended_capitalisation_policy = lower

            # An example of setting a custom layout specification which
            # is more lenient than default config.
            [sqlfluff:layout:type:set_operator]
            line_position = alone

  Upgrading to 1.4

This release brings several internal changes, and acts as a prelude to 2.0.0. In particular, the following config values have changed:

    •   sqlfluff:rules:L007:operator_new_lines has been changed to sqlfluff:layout:type:binary_operator:line_position.

    •   sqlfluff:rules:comma_style and sqlfluff:rules:L019:comma_style have both been consolidated into sqlfluff:layout:type:comma:line_position.

If any of these values have been set in your config, they will be automatically translated to the new values at runtime, and a warning will be shown. To silence the warning, update your config file to the new values. For more details on configuring layout see Configuring Layout <#layoutconfig>.

  Upgrading to 1.3

This release brings several potentially breaking changes to the underlying parse tree. For users of the cli tool in a linting context you should notice no change. If however your application relies on the structure of the SQLFluff parse tree or the naming of certain elements within the yaml format, then this may not be a drop-in replacement. Specifically:

    •   The addition of a new end_of_file` meta segment at the end of the parse structure.

    •   The addition of a template_loop` meta segment to signify a jump backward in the source file within a loop structure (e.g. a jinja for` loop).

    •   Much more specific types on some raw segments, in particular identifier and literal type segments will now appear in the parse tree with their more specific type (which used to be called name) e.g. naked_identifier, quoted_identifier, numeric_literal etc...

If using the python api, the parent type (such as identifier) will still register if you call .is_type("identifier"), as this function checks all inherited types. However the eventual type returned by .get_type()` will now be (in most cases) what used to be accessible at .name. The name attribute will be deprecated in a future release.

  Upgrading to 1.2

This release introduces the capability to automatically skip large files, and sets default limits on the maximum file size before a file is skipped. Users should see a performance gain, but may experience warnings associated with these skipped files.

  Upgrades pre 1.0

    •   0.13.x new rule for quoted literals, option to remove hanging indents in rule L003, and introduction of ignore_words_regex.

    •   0.12.x dialect is now mandatory, the spark3 dialect was renamed to sparksql and datatype capitalisation was extracted from L010 to it's own rule L063.

    •   0.11.x rule L030 changed to use extended_capitalisation_policy.

    •   0.10.x removed support for older dbt versions < 0.20 and stopped fix attempting to fix unparsable SQL.

    •   0.9.x refinement of the Simple API, dbt 1.0.0 compatibility, and the official SQLFluff Docker image.

    •   0.8.x an improvement to the performance of the parser, a rebuild of the Jinja Templater, and a progress bar for the CLI.

    •   0.7.x extracted the dbt templater to a separate plugin and removed the exasol_fs dialect (now merged in with the main exasol).

    •   0.6.x introduced parallel processing, which necessitated a big re-write of several innards.

    •   0.5.x introduced some breaking changes to the API.

    •   0.4.x dropped python 3.5, added the dbt templater, source mapping and also introduced the python API.

    •   0.3.x drops support for python 2.7 and 3.4, and also reworks the handling of indentation linting in a potentially not backward compatible way.

    •   0.2.x added templating support and a big restructure of rules and changed how users might interact with SQLFluff on templated code.

    •   0.1.x involved a major re-write of the parser, completely changing the behaviour of the tool with respect to complex parsing.

  SQLFluff in the Wild

Want to find other people who are using SQLFluff in production use cases? Want to brag about how you're using it? Just want to show solidarity with the project and provide a testimonial for it?

Just add a section below by raising a PR on GitHub by editing this file ✏️ <https://github.com/sqlfluff/sqlfluff/edit/main/docs/source/inthewild.rst>.

    •   SQLFluff in production dbt <http://www.getdbt.com/> projects at tails.com <https://tails.com>. We use the SQLFluff cli as part of our CI pipeline in codeship <https://codeship.com> to enforce certain styles in our SQL codebase (with over 650 models) and keep code quality high. Contact @alanmcruickshank <https://github.com/alanmcruickshank>.

    •   Netlify <https://www.netlify.com>'s data team uses SQLFluff with dbt <http://www.getdbt.com/> to keep code quality in more than 350 models (and growing). Previously, we had our SQL Guidelines defined in a site hosted with Netlify, and now we're enforcing these rules in our CI workflow thanks to SQLFluff.

    •   Drizly's <https://www.drizly.com> analytics team uses SQLFluff with dbt <http://www.getdbt.com/> for over 700 models as part of our CI checks in GitHub. Before SQLFluff, we had SQL best practices outlined in a google doc and had to manually enforce through PR comments. We're now able to enforce much of our style guide automatically through SQLFluff.

    •   Petal's <https://www.petalcard.com> data-eng team runs SQLFluff on our 100+ model dbt <http://www.getdbt.com/> project. As a pre-commit hook and as a CI check, SQLFluff helps keep our SQL readable and consistent.

    •   Surfline <https://www.surfline.com/>'s Analytics Engineering team implemented SQLFluff as part of our continuous integration (CI) suite across our entire dbt <http://www.getdbt.com/> project (700+ models). We implement the CI suite using GitHub Actions and Workflows <https://docs.github.com/en/actions>. The benefits of using SQLFluff at Surfline are:

        •   The SQL in our dbt models is consistent and easily readable.

        •   Our style guide is maintained as code, not a README that is rarely updated.

        •   Reduced burden on Analytics Engineers to remember every single style rule.

        •   New Analytics Engineers can quickly see and learn what "good SQL" looks like at Surfline and start writing it from day 1.

    •   The HTTP Archive <https://httparchive.org> uses SQLFluff to automatically check for quality and consistency of code submitted by the many contributors to this project. In particular our annual Web Almanac <https://almanac.httparchive.org> attracts hundreds of volunteers to help analyse our BigQuery dataset and being able automatically lint Pull Requests through GitHub Actions is a fantastic way to help us maintain our growing repository of over a thousand queries <https://github.com/HTTPArchive/almanac.httparchive.org/tree/main/sql>.

    •   Brooklyn Data Co <https://www.brooklyndata.co> has a dbt_artifacts <https://github.com/brooklyn-data/dbt_artifacts> dbt package from which runs SQLFluff in CI to lint pull requests automatically. It uses the GitHub Actions workflow <https://github.com/sqlfluff/sqlfluff-github-actions/tree/main/menu_of_workflows/surfline> contributed by Greg Clunies, with annotations on pull requests to make it easy for contributors to see where their SQL has failed any rules. See an example pull request with SQLFluff annotations <https://github.com/brooklyn-data/dbt_artifacts/pull/74/files>.

    •   Markerr <https://www.markerr.com> has tightly integrated SQLFluff into our CI/CD process for data model changes and process improvements. Since adopting SQLFluff across the organization, the clarity of our SQL code has risen dramatically, freeing up review time to focus on deeper data and process-specific questions.

    •   Symend <https://www.symend.com> has a microservices platform supporting our SaaS product. We use SQLFLuff in the CI/CD process of several of our data-oriented microservices. Among other things, it validates our database migration scripts, deployed using schemachange <https://github.com/Snowflake-Labs/schemachange> and we have near-term plans to implement it for our dbt <http://www.getdbt.com/> projects.

    •   At CarePay <https://www.carepay.com> we use SQLFLuff to lint and fix all our dbt models as well as several other SQL heavy projects. Locally we use SQLFluff with pre-commit and have also integrated it into our CI/CD pipelines.

    •   Core Analytics Team from Typeform <https://www.typeform.com/> and videoask <https://www.videoask.com/> uses SQLFluff in the production dbt <http://www.getdbt.com/> project for building our datawarehouse layer for both products:

        •   We use it locally in our day to day work, helping us to write cleaner code.

        •   We added SQLFluff to our CI processes, so during a PR we can check that any new or modified sql file has a consistent and easy-to-read format.

  SQLFluff Slack

We have a fast-growing community on Slack <https://join.slack.com/t/sqlfluff/shared_invite/zt-2qtu36kdt-OS4iONPbQ3aCz2DIbYJdWg>, come and join us!

  SQLFluff on Twitter

Follow us On Twitter @SQLFluff <https://twitter.com/SQLFluff> for announcements and other related posts.

    •   Index <>

    •   Module Index <>

    •   Search Page <>

Author
Alan Cruickshank

Copyright
2024, Alan Cruickshank