Open Policy Agent - v0.45.0

This release contains a mix of bugfixes, optimizations, and new features.

Improved Decision Logging with nd_builtin_cache

OPA has several non-deterministic built-ins, such as rand.intn and http.send that can make debugging policies from decision log results a surprisingly tricky and involved process. To improve the situation around debugging policies that use those built-ins, OPA now provides an opt-in system for caching the inputs and outputs of these built-ins during policy evaluation, and can include this information in decision log entries.

A new top-level config key is used to enable the non-deterministic builtin caching feature, as shown below:

nd_builtin_cache: true

This data is exposed to OPA's decision log masking system under the /nd_builtin_cache path, which allows masking or dropping sensitive values from decision logs selectively. This can be useful in situations where only some information about a non-deterministic built-in was needed, or the arguments to the built-in involved sensitive data.

To prevent unexpected decision log size growth from non-deterministic built-ins like http.send, the new cache information is included in decision logs on a best-effort basis. If a decision log event exceeds the decision_logs.reporting.upload_size_limit_bytes limit for an OPA instance, OPA will reattempt uploading it, after dropping the non-deterministic builtin cache information from the event. This behavior will trigger a log error when it happens, and will increment the decision_logs_nd_builtin_cache_dropped metrics counter, so that it will be possible to debug cases where the cache information is unexpectedly missing from a decision log entry.

Decision Logging Example

To observe the change in decision logging we can run OPA in server mode with nd_builtin_cache enabled:

opa run -s --set=decision_logs.console=true,nd_builtin_cache=true

After sending it the query x := rand.intn("a", 15) we should see something like the following in the decision logs:

{..., "msg":"Decision Log", "nd_builtin_cache":{"rand.intn":{"[\"a\",15]":3}}, "query":"assign(x, rand.intn(\"a\", 15))", ..., "result":[{"x":3}], ..., "type":"openpolicyagent.org/decision_logs"}

The new information is included under the optional nd_builtin_cache JSON key, and shows what arguments were provided for each unique invocation of rand.intn, as well as what the output of that builtin call was (in this case, 3).

If we send the query x := rand.intn("a", 15); y := rand.intn("b", 150)" we can see how unique input arguments get recorded in the cache:

{..., "msg":"Decision Log", "nd_builtin_cache":{"rand.intn":{"[\"a\",15]":12,"[\"b\",150]":149}}, "query":"assign(x, rand.intn(\"a\", 15)); assign(y, rand.intn(\"b\", 150))", ..., "result":[{"x":12,"y":149}], ..., "type":"openpolicyagent.org/decision_logs"}

With this information, it's now easier to debug exactly why a particular rule is used or why a rule fails when non-deterministic builtins are used in a policy.

New Built-in Function: regex.replace

This release introduces a new builtin for regex-based search/replace on strings: regex.replace.

See the built-in functions docs for all the details

This implementation fixes #5162 and was authored by @boranx.

object.union_n Optimization

The object.union_n builtin allows easily merging together an array of Objects.

Unfortunately, as noted in #4985 its implementation generated unnecessary intermediate copies from doing pairwise, recursive Object merges. These pairwise merges resulted in poor performance for large inputs; in many cases worse than writing the equivalent operation in pure Rego.

This release changes the object.union_n builtin's implementation to use a more efficient merge algorithm that respects the original implementation's sequential, left-to-right merging semantics. The object.union_n builtin now provides a 2-3x improvement in speed and memory efficiency over the pure Rego equivalent.

Tooling, SDK, and Runtime

• cli: Fix doubled CLI hints/errors. (#5115) authored by @ivanphdz
• cli/test: Add capabilities flag to test command. (authored by @ivanphdz)
• fmt: Fix blank lines after multiline expressions. (authored by @jaspervdj)
• internal/report: Include heap usage in the telemetry report.
• plugins/logs: Improve error message when decision log chunk size is greater than the upload limit. (#5155)
• ir: Make the internal/ir package public as ir.

Rego

• ast/parser+formatter: Allow 'if' in rule 'else' statements.
• ast/schema: Add support for recursive json schema elements. (#5166) authored and reported by @liamg
• ast/schema: Fix race condition in parsing with reused references.(authored by @liamg)
• internal/gojsonschema: Fix race condition in SetAllowNet. (#5187) authored and reported by @liamg
• ast/compiler: Rewrite declared variables in function calls and recursively rewrite local variables in with clauses. (#5148) authored and reported by @liu-du
• ast: Skip rules when parsing a body (or query) to help improve ambiguous parsing cases.

Topdown

• topdown/object: Rework object.union_n to use in-place merge algorithm. (reported by @charlesdaniels)
• topdown/jwt_decode_verify: Ensure exp and nbf fields are numbers when present. (#5165) authored and reported by @charlieflowers
• topdown: Fix InterQueryCache only dropping one entry when over the size limit. (authored by @vinhph0906)
• topdown+builtins: Block all ND builtins from partial evaluation.
• topdown/builtins: Add Rego Object support for GraphQL builtins to improve composability.
• topdown/json: Fix panic in json.filter on empty JSON paths.
• topdown/sets_bench_test: Add intersection builtin tests.
• topdown/tokens: Protect against nistec panics. (#5128)

Documentation

• Add IR to integration docs.
• Added Gloo Edge Tutorial with examples. (authored by @Parsifal-M)
• Updated examples for CLI commands.
• Updated section on performance metrics (authored by @hutchins)
• docs/annotations: Add policy example and a link to the policy reference. (#4937) authored by @Parsifal-M
• docs/policy-language: Be more explicit about future keywords.
• docs/security: Fix token authz example. (authored by @pigletfly)
• docs: Update generated CLI docs. (authored by @charlieflowers)
• docs: Update mentions of #development to #contributors. (authored by @charlieflowers)

Website + Ecosystem

• website/security: Style improvements. (authored by @orweis)

Miscellaneous

• ci: Add prealloc linter check and linter fixes.
• ci: Add govulncheck to Nightly CI.
• build/wasm: Use golang1.16 go:embed mechanism.
• util/backoff: Seed from math/rand source.

• version: Use runtime/debug.BuildInfo.

• Dependency bumps, notably:

• build: bump golang 1.19.1 -> 1.19.2
• build(deps): bump golang.org/x/net
• build(deps): bump internal/gqlparser to v2.5.1
• build(deps): bump tj-actions/changed-files from 29.0.3 -> 32.0.0
• deps(build): bump wasmtime-go 0.36.0 -> 1.0.0 (authored by @Parsifal-M)