--- sidebar_position: 5 sidebar_label: Custom Rules --- Custom Rules | Regal # Custom Rules Regal is built to be easily extended. Using custom rules is a great way to enforce naming conventions, best practices or more opinionated rules across teams and organizations. There are two types of custom rules to be aware of — those that are included with Regal in the `custom` category, and those that you write yourself. The rules in the `custom` category provide a way to enforce common organizational requirements, like naming conventions, by means of _configuration_ rather than code. If your requirements aren't fulfilled by the rules in this category, your other option is to write your own custom rules using Rego. ## Your Own Custom Rules If you'd like to provide your own linter rules for a project, you may do so by placing them in a `rules` directory inside the `.regal` directory preferably placed in the root of your project (which is also where custom configuration resides). The directory structure of a policy repository with custom linter rules might then look something like this: ```text . ├── .regal │ ├── config.yaml │ └── rules │ ├── naming.rego │ └── naming_test.rego └── policy ├── authz.rego └── authz_test.rego ``` If you so prefer, custom rules may also be provided using the `--rules` option for `regal lint`, which may point either to a Rego file, or a directory containing Rego files and potentially data (JSON or YAML). ## Creating a New Rule The simplest way to create a new rule is to use the `regal new rule` command. This command provides scaffolding for quickly creating a new rule, including a file for testing. The command has two required arguments: `--category` and `--name`, which should be self-explanatory. To create a new custom rule: ```shell regal new rule --category naming --name foo-bar-baz ``` This will create a `.regal/rules` directory in the current working directory, if one does not already exist, and place a starter policy and a test in a directory structure based on `--category` and `--name` in it. Following the above example would create the following directory structure under `.regal/rules`: ```text custom/regal/rules/naming/foo-bar-baz/foo_bar_baz.rego custom/regal/rules/naming/foo-bar-baz/foo_bar_baz_test.rego ``` If you'd rather create this directory structure in some other place than the current working directory, you may use the `--output` flag to specify a different location. The generated rule includes a simple example, which can be verified by running `regal test .regal/rules`. Modify the rule and the test to suit your needs! If you'd like to create a new built-in rule for submitting a PR in Regal, you may add the `--type builtin` flag to the command (the default is `custom`). This will create a similar scaffolding under `bundle/regal/rules` in the Regal repository. ## Developing Rules Regal rules works primarily with the [abstract syntax tree](https://en.wikipedia.org/wiki/Abstract_syntax_tree) (AST) provided by OPA's parser as input. The top level item in the AST is the [module](https://pkg.go.dev/github.com/open-policy-agent/opa/ast#Module), which contains nodes for everything found in a policy, like the package declaration, imports and rules. Since OPA expects JSON as input, the module and all its child nodes are serialized, and then made available as `input` in Regal linter rule policies. The `opa parse --format json` command can be used to get an idea of what the structure of the serialized AST looks like. However, recent versions of Regal leverage an optimized AST JSON representation called [roast](https://www.openpolicyagent.org/projects/regal/custom-rules/roast), which is both more compact and performant to traverse as part of linting. See the roast docs for more information on how the format differs from the "normal" OPA AST. In order to view the JSON AST representation of a policy, use the `regal parse` command. This works similarly to `opa parse`, but only outputs the roast JSON format, including additions made by Regal. If we were to write the simplest policy possible, and parse it using `regal parse`, it would contain nothing but a package declaration: ```rego title="policy.rego" package policy ``` Using `regal parse policy.rego`, we're provided with the AST of the above policy: ```json { "package": { "location": "1:1:1:8", "path": [ { "type": "var", "value": "data" }, { "location": "1:9:1:15", "type": "string", "value": "policy" } ] }, "regal": { "file": { "name": "policy.rego", "lines": [ "package policy", "" ], "abs": "/Users/anderseknert/tmp/custom/policy.rego" }, "environment": { "path_separator": "/" } } } ``` As trivial as may be, it's enough to build our first linter rule! Let's say we'd like to enforce a uniform naming convention on any policy in a repository. Packages may be named anything, but must start with the name of the organization (Acme Corp). So `package acme.corp.policy` should be allowed, but not `package policy` or `package policy.acme.corp`. One exception: policy authors should be allowed to write policy for the `system.log` package provided by OPA to allow [masking](https://www.openpolicyagent.org/docs/management-decision-logs/#masking-sensitive-data) sensitive data from decision logs. An example policy to implement this requirement might look something like this: ```rego # METADATA # description: All packages must use "acme.corp" base name # related_resources: # - description: documentation # ref: https://www.acmecorp.example.org/docs/regal/package # schemas: # - input: schema.regal.ast package custom.regal.rules.naming["acme-corp-package"] import data.regal.result report contains violation if { not acme_corp_package not system_log_package violation := result.fail(rego.metadata.chain(), result.location(input.package.path[1])) } acme_corp_package if { input.package.path[1].value == "acme" input.package.path[2].value == "corp" } system_log_package if { input.package.path[1].value == "system" input.package.path[2].value == "log" } ``` Starting from top to bottom, these are the components comprising our custom rule: 1. The package of custom rules **must** start with `custom.regal.rules`, followed by the category of the rule, and the title (which is commonly quoted as rule names use `-` for spaces). 1. The `data.regal.result` provides some helpers for formatting the result of a violation for inclusion in a report. 1. Regal rules make heavy use of [metadata annotations](https://www.openpolicyagent.org/docs/policy-language/#annotations) in order to document the purpose of the rule, along with any other information that could potentially be useful. All rule packages **must** have a `description`. Providing links to additional documentation under `related_resources` is recommended, but not required. 1. Note the `schema` attribute present in the metadata annotation. Adding this is optional, but highly recommended, as it will make the compiler aware of the structure of the input, i.e. the AST. This allows the compiler to fail when unknown attributes are referenced, due to typos or other mistakes. The compiler will also fail when an attribute is referenced using a type it does not have, like referring to a string as if it was a number. Set to `schema.regal.ast` to use the AST schema provided by Regal. 1. Regal will evaluate any rule named `report` in each linter policy, so at least one `report` rule **must** be present. 1. In our example `report` rule, we evaluate another rule (`acme_corp_package`) in order to know if the package name starts with `acme.corp`, and another rule (`system_log_package`) to know if it starts with `system.log`. If neither of the conditions are true, the rule fails and violation is created. 1. The violation is created by calling `result.fail`, which takes the metadata from the package (using `rego.metadata.chain` which conveniently also includes the path of the package) and returns a result, which will later be included in the final report provided by Regal. 1. The `result.location` helps extract the location from the element failing the test. Make sure to use it! ### Rule Development Workflow In addition to making use of the `regal parse` command to inspect the AST of a policy, using Regal's [language server](https://www.openpolicyagent.org/projects/regal/language-server) for rule development provides the absolute best rule development experience. #### Code Lens for Evaluation If you're using VS Code and the [OPA VS Code extension](https://github.com/open-policy-agent/vscode-opa), you may use the [Code Lens for Evaluation](https://www.openpolicyagent.org/projects/regal/language-server#code-lenses-evaluation) to directly evaluate packages and rules using the `input.json` file as input, and see the result directly in your editor on the line you clicked to evaluate. To start evaluating a policy against your custom rule. First turn the parse result of the policy into an input file: ```shell regal parse path/to/policy.rego > input.json ``` You should now be able evaluate your custom rule against the `input.json` AST: ![Code Lens for Evaluation of custom rule](../assets/evalcustom.png) **Tips:** - You can hover the inlined result to see the full output - Calls to `print` inside rule bodies will have the print output displayed on the same line As another convenience, any `.rego` file where the first comment in the policy is `# regal eval:use-as-input` will have the evaluation feature automatically use the AST of the file as input. This allows building queries against the AST of the policy you're working on, providing an extremely fast feedback loop for developing new rules! ![Use AST of file as input](../assets/lsp/eval_use_as_input.png) #### Test-Driven Development Using a test-driven approach to custom rule development is a great way to both understand how your rule works, and to assert that it works as expected even as you make changes to the code. Use the `regal test` command the same way as you would use `opa test`: ```shell regal test .regal/rules ``` To debug failures in your test, the `--var-values` flag can help by providing more information about which values failed to match the expected output. You can also use the `print` function anywhere in your policy, which will have it's output printed by the test runner. Tests commonly first parse a policy, then provide that as input to the rule being tested. You can either use the built-in `regal.parse_module(name, policy)` function to parse a policy, or one of the provided helpers in the `regal.ast` package: Using `ast.with_rego_v1(policy)` will have a package declararation and `import rego.v1` added to the policy, allowing you to get straight to what you actually want to test for: ```rego test_fail_constant_condition if { module := ast.with_rego_v1(`allow if true`) report := rule.report with input as module count(report) == 1 some violation in report violation.title == "constant-conditoon" } ``` The `ast.policy(policy)` adds only a package declaration and not `import rego.v1`. ## Aggregate Rules Aggregate rules are a special type of rule that allows you to collect data from multiple files before making a decision. This is needed in some cases where a single policy file won't be enough for a linter rule to make a decision. For example, you may want to enforce that in a policy repository, there must be at least one package annotated with an `authors` attribute. This requires first collecting annotation data from all the provided policies, and then have the rule use this data to make a decision. The structure of an aggregate rule is similar to a regular rule, but with a few notable differences. The most significant one is that linting happens in two separate phases — one that aggregates data from files, and one that actually lints and reports violations using that data. ```rego # METADATA # description: | # There must be at least one boolean rule named `allow`, and it must # have a default value of `false` # related_resources: # - description: documentation # ref: https://www.acmecorp.example.org/docs/regal/aggregate-allow # schemas: # - input: schema.regal.ast package custom.regal.rules.organizational["at-least-one-allow"] import data.regal.ast import data.regal.result aggregate contains entry if { # ast.rules is input.rules with functions filtered out some rule in ast.rules # search for rule named allow ast.ref_to_string(rule.head.ref) == "allow" # make sure it's a default assignment # ideally we'll want more than that, but the *requirement* is only # that such a rule exists... rule["default"] == true # ...and that it defaults to false rule.head.value.type == "boolean" rule.head.value.value == false # if found, collect the result into our aggregate collection # we don't really need the location here, but showing for demonstration entry := { # optional metadata here "package": input.package, } } # METADATA # description: | # This is called once all aggregates have been collected. Note the use of a # different schema here for type checking, as the input is no longer the AST # of a Rego policy, but our collected data. # schemas: # - input: schema.regal.aggregate aggregate_report contains violation if { # input.aggregate contains only the entries collected by *this* aggregate rule, # so you don't need to worry about counting entries from other sources here! count(input.aggregate) == 0 # no aggregated data found, so we'll report a violation # another rule may of course want to make use of the data collected in the aggregation violation := result.fail(rego.metadata.chain(), { "message": "At least one rule named `allow` must exist, and it must have a default value of `false`", }) } ``` As you can see, the aggregate rule is split into two parts — one that collects data (`aggregate`), and one that reports violations (`aggregate_report`). Use of helper functions like `result.aggregate` is optional, but **highly** recommended, as it will have any aggregate entry contain information like file, location and package, which is useful both for reporting, but also for debugging. Use a `print` or two in the `aggregate_report` rule to see exactly what's included! ## Parsing and Testing Regal provides a few tools mirrored from OPA in order to help test and debug custom rules. These are necessary since OPA is not aware of the custom [built-in functions](#built-in-functions) included in Regal, and will fail when encountering e.g. `regal.parse_module` in a custom linter policy. The following commands are included with Regal to help you author custom rules: - `regal parse` works similarly to `opa parse`, but will always output JSON and include location information, and any additional data added to the AST by Regal. Use this if you want to know exactly what the `input` will look like for any given policy, when provided to Regal for linting. - `regal test` works like `opa test`, but aware of any custom Regal additions, and the schema used for the AST. Use this to test custom linter rules, e.g. `regal test .regal/rules`. Note that the `print` built-in function is enabled for `regal test`. Good to use for quick debugging! ## Built-in Functions Regal provides a few custom built-in functions tailor-made for linter policies. ### `regal.parse_module(filename, policy)` Works just like `rego.parse_module`, but provides an AST including location information, and custom additions added by Regal, like the text representation of each line in the original policy. This is useful for authoring tests to assert linter rules work as expected. This is the built-in function equivalent of the `regal parse` command. If the `filename` provided ends with `_v0.rego`, the policy will be parsed as a Rego v0 module. ### `regal.last(array)` This built-in function is a much more performant way to express `array[count(array) - 1]`. This performance difference is almost always irrelevant in "normal" Rego policies, but can have a significant impact in linter rules where it's sometimes called thousands of times as part of traversing the input AST. ## Rego Library In addition to this, Regal provides many helpful functions, rules and utilities in Rego. Browsing the source code of the [regal.ast](https://github.com/open-policy-agent/regal/blob/main/bundle/regal/ast/ast.rego) package to see what's available is recommended! Note however that at this point in time, the Rego API is not considered stable, and breaking changes are likely to occur between versions. If you need stable versions of rules and functions found here, consider copying them into a library of your own, and use in your custom rules. Or engage with the Regal community and tell us what you need and depend on, and we'll try to take it into account, or at least help you find ways to make it work! ## Language server When using custom rules with the Regal language server, rules will be loaded from the `.regal/rules` directory relative to the selected workspace root. Custom rules will be loaded each time a lint is run for a file or the workspace. If your custom rules are broken, fixes will be used on the next linting run.