If you are using the vitess-operator and want to remain on MySQL 5.7, you are required to use the vitess/lite:v16.0.0-mysql57 Docker Image, otherwise the vitess/lite:v16.0.0 image will be on MySQL 80.

However, if you are running MySQL 8.0 on the vitess-operator, with for instance vitess/lite:v15.0.2-mysql80, considering that we are bumping the patch version of MySQL 80 from 8.0.23 to 8.0.30, you will have to manually upgrade:

1. Add innodb_fast_shutdown=0 to your extra cnf in your YAML file.
2. Apply this file.
3. Wait for all the pods to be healthy.
4. Then change your YAML file to use the new Docker Images (vitess/lite:v16.0.0, defaults to mysql80).
5. Remove innodb_fast_shutdown=0 from your extra cnf in your YAML file.
6. Apply this file.

<a id="flag-deletions-and-deprecations">Flag Deletions and Deprecations

With the removal of the vtctld UI, the following vtctld flags have been deprecated:
- --vtctld_show_topology_crud: This was a flag that controlled the display of CRUD topology actions in the vtctld UI. The UI is removed, so this flag is no longer necessary.

The following deprecated flags have also been removed:
- --enable_realtime_stats
- --enable_vtctld_ui
- --web_dir
- --web_dir2
- --workflow_manager_init
- --workflow_manager_use_election
- --workflow_manager_disable

The following flags were removed in v16:
- --enable_semi_sync
- --backup_storage_hook, use one of the builtin compression algorithms or --external-compressor and --external-decompressor instead.
- --init_populate_metadata, since we have deleted the local_metadata and shard_metadata sidecar database tables.

The flag --disable-replication-manager is deprecated and will be removed in a future release.

The VTBackup flag --backup_storage_hook has been removed, use one of the builtin compression algorithms or --external-compressor and --external-decompressor instead.

VTOrc is now a required component of Vitess starting from v16. If the users want Vitess to manage replication, then they must run VTOrc.
Replication manager is removed from vttablets since the responsibility of fixing replication lies entirely with VTOrc now.
The flag disable-replication-manager is deprecated and will be removed in a future release.

The tablet throttler can now be configured dynamically. Configuration is now found in the topo service, and applies to all tablets in all shards and cells of a given keyspace.
It is possible to enable or disable throttling, and to change the throttling threshold as well as the throttler query.

Please note that this feature is considered experimental in this release. For backwards compatibility v16 still supports vttablet-based command line flags for throttler configuration.

--query-timeout allows you to specify a timeout for queries. This timeout is applied to all queries.
It can be overridden by setting the query_timeout session variable.
Setting it as query comment directive with QUERY_TIMEOUT_MS will override other values.

--vreplication-parallel-insert-workers=[integer] enables parallel bulk inserts during the copy phase
of VReplication (disabled by default). When set to a value greater than 1 the bulk inserts — each
executed as a single transaction from the vstream packet contents — may happen in-parallel and
out-of-order, but the commit of those transactions are still serialized in order.

Other aspects of the VReplication copy-phase logic are preserved:

1. All statements executed when processing a vstream packet occur within a single MySQL transaction.
2. Writes to _vt.copy_state always follow their corresponding inserts from within the vstream packet.
3. The final commit for the vstream packet always follows the corresponding write to _vt.copy_state.
4. The vstream packets are committed in the order seen in the stream. So for any PK1 and PK2, the write to _vt.copy_state and commit steps (steps 2 and 3 above) for PK1 will both precede the _vt.copy_state write and commit steps of PK2.

Other phases, catchup, fast-forward, and replicating/"running", are unchanged.

--queryserver-config-pool-conn-max-lifetime=[integer] allows you to set a timeout on each connection in the query server connection pool. It chooses a random value between its value and twice its value, and when a connection has lived longer than the chosen value, it'll be removed from the pool the next time it's returned to the pool.

The flag --throttler-config-via-topo switches throttler configuration from vttablet-flags to the topo service. This flag is false by default, for backwards compatibility. It will default to true in future versions.

Tablet throttler configuration is now supported in topo. Updating the throttler configuration is done via vtctldclient UpdateThrottlerConfig and applies to all tablet in all cells for a given keyspace.

Examples:

# disable throttler; all throttler checks will return with "200 OK"
$ vtctldclient UpdateThrottlerConfig --disable commerce

# enable throttler; checks are responded with appropriate status per current metrics
$ vtctldclient UpdateThrottlerConfig --enable commerce

# Both enable and set threshold in same command. Since no query is indicated, we assume the default check for replication lag
$ vtctldclient UpdateThrottlerConfig --enable --threshold 5.0 commerce

# Change threshold. Does not affect enabled/disabled state of the throttler
$ vtctldclient UpdateThrottlerConfig --threshold 1.5 commerce

# Use a custom query
$ vtctldclient UpdateThrottlerConfig --custom_query "show global status like 'threads_running'" --check_as_check_self --threshold 50 commerce

# Restore default query and threshold
$ vtctldclient UpdateThrottlerConfig --custom_query "" --check_as_check_shard --threshold 1.5 commerce

See https://github.com/vitessio/vitess/pull/11604

The Backup command now supports --incremental_from_pos flag, which can receive a valid position or the value auto. For example:

$ vtctlclient -- Backup --incremental_from_pos "MySQL56/16b1039f-22b6-11ed-b765-0a43f95f28a3:1-615" zone1-0000000102
$ vtctlclient -- Backup --incremental_from_pos "auto" zone1-0000000102

When the value is auto, the position is evaluated as the last successful backup's Position. The idea with incremental backups is to create a contiguous (overlaps allowed) sequence of backups that store all changes from last full backup.

The incremental backup copies binary log files. It does not take MySQL down nor places any locks. It does not interrupt traffic on the MySQL server. The incremental backup copies complete binlog files. It initially rotates binary logs, then copies anything from the requested position and up to the last completed binary log.

The backup thus does not necessarily start exactly at the requested position. It starts with the first binary log that has newer entries than requested position. It is OK if the binary logs include transactions prior to the requested position. The restore process will discard any duplicates.

Normally, you can expect the backups to be precisely contiguous. Consider an auto value: due to the nature of log rotation and the fact we copy complete binlog files, the next incremental backup will start with the first binay log not covered by the previous backup, which in itself copied the one previous binlog file in full. Again, it is completely valid to enter any good position.

The incremental backup fails if it is unable to attain binary logs from given position (ie binary logs have been purged).

The manifest of an incremental backup has a non-empty FromPosition value, and a Incremental = true value.

• --restore_to_pos: request to restore the server up to the given position (inclusive) and not one step further.
• --dry_run: when true, calculate the restore process, if possible, evaluate a path, but exit without actually making any changes to the server.

Examples:

$ vtctlclient -- RestoreFromBackup  --restore_to_pos  "MySQL56/16b1039f-22b6-11ed-b765-0a43f95f28a3:1-220" zone1-0000000102

The restore process seeks a restore path: a sequence of backups (handles/manifests) consisting of one full backup followed by zero or more incremental backups, that can bring the server up to the requested position, inclusive.

The command fails if it cannot evaluate a restore path. Possible reasons:

• there's gaps in the incremental backups
• existing backups don't reach as far as requested position
• all full backups exceed requested position (so there's no way to get into an ealier position)

The command outputs the restore path.

There may be multiple restore paths, the command prefers a path with the least number of backups. This has nothing to say about the amount and size of binary logs involved.

The RestoreFromBackup --restore_to_pos ends with:

• the restored server in intentionally broken replication setup
• tablet type is DRAINED

A new vexplain command has been introduced with the following syntax:

VEXPLAIN [ALL|QUERIES|PLAN] explainable_stmt

This command will help users look at the plan that vtgate comes up with for the given query (PLAN type), see all the queries that are executed on all the MySQL instances (QUERIES type), and see the vtgate plan along with the MySQL explain output for the executed queries (ALL type).

The formats VTEXPLAIN and VITESS for EXPLAIN queries are deprecated, and these newly introduced commands should be used instead.

Views sharded support is released as an experimental feature in v16.0.0.
Views are not enabled by default in your Vitess cluster, but they can be turned on using the --enable-views flag on VTGate, and --queryserver-enable-views flag on VTTablet.

Compression benchmarks have been added to the mysqlctl package.

The benchmarks fetch and compress a ~6 GiB tar file containing 3 InnoDB files using different built-in and external compressors.

Here are sample results from a 2020-era Mac M1 with 16 GiB of memory:

$ go test -bench=BenchmarkCompress ./go/vt/mysqlctl -run=NONE -timeout=12h -benchtime=1x -v
goos: darwin
goarch: arm64
pkg: vitess.io/vitess/go/vt/mysqlctl
BenchmarkCompressLz4Builtin
    compression_benchmark_test.go:310: downloading data from https://www.dropbox.com/s/raw/smmgifsooy5qytd/enwiki-20080103-pages-articles.ibd.tar.zst
    BenchmarkCompressLz4Builtin-8                  1        11737493087 ns/op        577.98 MB/s             2.554 compression-ratio
    BenchmarkCompressPargzipBuiltin
    BenchmarkCompressPargzipBuiltin-8              1        31083784040 ns/op        218.25 MB/s             2.943 compression-ratio
    BenchmarkCompressPgzipBuiltin
    BenchmarkCompressPgzipBuiltin-8                1        13325299680 ns/op        509.11 MB/s             2.910 compression-ratio
    BenchmarkCompressZstdBuiltin
    BenchmarkCompressZstdBuiltin-8                 1        18683863911 ns/op        363.09 MB/s             3.150 compression-ratio
    BenchmarkCompressZstdExternal
    BenchmarkCompressZstdExternal-8                1        10795487675 ns/op        628.41 MB/s             3.093 compression-ratio
    BenchmarkCompressZstdExternalFast4
    BenchmarkCompressZstdExternalFast4-8           1        7139319009 ns/op         950.23 MB/s             2.323 compression-ratio
    BenchmarkCompressZstdExternalT0
    BenchmarkCompressZstdExternalT0-8              1        4393860434 ns/op        1543.97 MB/s             3.093 compression-ratio
    BenchmarkCompressZstdExternalT4
    BenchmarkCompressZstdExternalT4-8              1        4389559744 ns/op        1545.49 MB/s             3.093 compression-ratio
    PASS
    cleaning up "/var/folders/96/k7gzd7q10zdb749vr02q7sjh0000gn/T/ee7d47b45ef09786c54fa2d7354d2a68.dat"

This is an internal refactor and should not change the behavior of Vitess as seen by users.