We added options to the TabletPicker that allow for specifying a cell preference in addition to making the default behavior to give priority to the local cell and any alias it belongs to. We are also introducing a new way to select tablet type preference which should eventually replace the in_order: hint currently used as a prefix for tablet types. The signature for creating a new TabletPicker now looks like:

func NewTabletPicker(
    ctx context.Context,
    ts *topo.Server,
    cells []string,
    localCell, keyspace, shard, tabletTypesStr string,
    options TabletPickerOptions,
) (*TabletPicker, error) {...}

Where ctx, localCell, option are all new parameters.

option is of type TabletPickerOptions and includes two fields, CellPreference and TabletOrder.
CellPreference: "PreferLocalWithAlias" (default) gives preference to vtgate's local cell, or "OnlySpecified" which only picks from the cells explicitly passed in by the clientTabletOrder`: "Any" (default) for no ordering or random, or "InOrder" to use the order specified by the client

When using TLS with vtgr, we now default to TLS 1.2 if no other explicit version is configured. Configuration flags are provided to explicitly configure the minimum TLS version to be used.

Prior to v17 Vitess incorrectly combined stats for VTGate Execute and Prepare operations under a single stats key (Execute). In v17 Execute and Prepare operations generate stats under independent stats keys.

Here is a (condensed) example of stats output:

{
  "VtgateApi": {
    "Histograms": {
      "Execute.src.primary": {
        "500000": 5
      },
      "Prepare.src.primary": {
        "100000000": 0
      }
    }
  },
  "VtgateApiErrorCounts": {
    "Execute.src.primary.INVALID_ARGUMENT": 3,
    "Execute.src.primary.ALREADY_EXISTS": 1
  }
}

Prior to v17, it was possible to create a keyspace with invalid characters, which would then be inaccessible to various cluster management operations.

Keyspace names are restricted to using only ASCII characters, digits and _ and -. TopoServer's GetKeyspace and CreateKeyspace methods return an error if given an invalid name.

Prior to v17, it was possible to create a shard name with invalid characters, which would then be inaccessible to various cluster management operations.

Shard names are restricted to using only ASCII characters, digits and _ and -. TopoServer's GetShard and CreateShard methods return an error if given an invalid name.

The CLI flags below were mistakenly added to vtctld and vtctldclient in v15. In v17, they are no longer present in those binaries.

• --compression-engine-name
• --compression-level
• --external-compressor
• --external-compressor-extension
• --external-decompressor

Prior to v17 the builtin Backup Engine does not use read buffering for restores, and for backups uses a hardcoded write buffer size of 2097152 bytes.

In v17 these defaults may be tuned with, respectively --builtinbackup-file-read-buffer-size and --builtinbackup-file-write-buffer-size.

• --builtinbackup-file-read-buffer-size: read files using an IO buffer of this many bytes. Golang defaults are used when set to 0.
• --builtinbackup-file-write-buffer-size: write files using an IO buffer of this many bytes. Golang defaults are used when set to 0. (default 2097152)

These flags are applicable to the following programs:

• vtbackup
• vtctld
• vttablet
• vttestserver

Add a new builtin/xtrabackup flag --manifest-external-decompressor. When set the value of that flag is stored in the manifest field ExternalDecompressor. This manifest field may be consulted when decompressing a backup that was compressed with an external command.

This feature enables the following flow:

1. Take a backup using an external compressor
   Backup --compression-engine=external \ --external-compressor=zstd \ --manifest-external-decompressor="zstd -d"
2. Restore that backup with a mere Restore command, without having to specify --external-decompressor.

This flag was introduced in v16 and defaulted to false. In v17 it defaults to true, and there is no need to supply it.

Note that this flag overrides --enable-lag-throttler and --throttle-threshold, which now give warnings, and will be removed in v18.

Backup metrics

Metrics related to backup operations are available in both Vtbackup and VTTablet.

BackupBytes, BackupCount, BackupDurationNanoseconds

Depending on the Backup Engine and Backup Storage in-use, a backup may be a complex pipeline of operations, including but not limited to:

• Reading files from disk.
• Compressing files.
• Uploading compress files to cloud object storage.

These operations are counted and timed, and the number of bytes consumed or produced by each stage of the pipeline are counted as well.

Restore metrics

Metrics related to restore operations are available in both Vtbackup and VTTablet.

RestoreBytes, RestoreCount, RestoreDurationNanoseconds

Depending on the Backup Engine and Backup Storage in-use, a restore may be a complex pipeline of operations, including but not limited to:

• Downloading compressed files from cloud object storage.
• Decompressing files.
• Writing decompressed files to disk.

These operations are counted and timed, and the number of bytes consumed or produced by each stage of the pipeline are counted as well.

Vtbackup metrics

Vtbackup exports some metrics which are not available elsewhere.

DurationByPhaseSeconds

Vtbackup fetches the last backup, restores it to an empty mysql installation, replicates recent changes into that installation, and then takes a backup of that installation.

DurationByPhaseSeconds exports timings for these individual phases.

Example

A snippet of vtbackup metrics after running it against the local example after creating the initial cluster

(Processed with jq for readability.)

{
  "BackupBytes": {
    "BackupEngine.Builtin.Source:Read": 4777,
    "BackupEngine.Builtin.Compressor:Write": 4616,
    "BackupEngine.Builtin.Destination:Write": 162,
    "BackupStorage.File.File:Write": 163
  },
  "BackupCount": {
    "-.-.Backup": 1,
    "BackupEngine.Builtin.Source:Open": 161,
    "BackupEngine.Builtin.Source:Close": 322,
    "BackupEngine.Builtin.Compressor:Close": 161,
    "BackupEngine.Builtin.Destination:Open": 161,
    "BackupEngine.Builtin.Destination:Close": 322
  },
  "BackupDurationNanoseconds": {
    "-.-.Backup": 4188508542,
    "BackupEngine.Builtin.Source:Open": 10649832,
    "BackupEngine.Builtin.Source:Read": 55901067,
    "BackupEngine.Builtin.Source:Close": 960826,
    "BackupEngine.Builtin.Compressor:Write": 278358826,
    "BackupEngine.Builtin.Compressor:Close": 79358372,
    "BackupEngine.Builtin.Destination:Open": 16456627,
    "BackupEngine.Builtin.Destination:Write": 11021043,
    "BackupEngine.Builtin.Destination:Close": 17144630,
    "BackupStorage.File.File:Write": 10743169
  },
  "DurationByPhaseSeconds": {
    "InitMySQLd": 2,
    "RestoreLastBackup": 6,
    "CatchUpReplication": 1,
    "TakeNewBackup": 4
  },
  "RestoreBytes": {
    "BackupEngine.Builtin.Source:Read": 1095,
    "BackupEngine.Builtin.Decompressor:Read": 950,
    "BackupEngine.Builtin.Destination:Write": 209,
    "BackupStorage.File.File:Read": 1113
  },
  "RestoreCount": {
    "-.-.Restore": 1,
    "BackupEngine.Builtin.Source:Open": 161,
    "BackupEngine.Builtin.Source:Close": 322,
    "BackupEngine.Builtin.Decompressor:Close": 161,
    "BackupEngine.Builtin.Destination:Open": 161,
    "BackupEngine.Builtin.Destination:Close": 322
  },
  "RestoreDurationNanoseconds": {
    "-.-.Restore": 6204765541,
    "BackupEngine.Builtin.Source:Open": 10542539,
    "BackupEngine.Builtin.Source:Read": 104658370,
    "BackupEngine.Builtin.Source:Close": 773038,
    "BackupEngine.Builtin.Decompressor:Read": 165692120,
    "BackupEngine.Builtin.Decompressor:Close": 51040,
    "BackupEngine.Builtin.Destination:Open": 22715122,
    "BackupEngine.Builtin.Destination:Write": 41679581,
    "BackupEngine.Builtin.Destination:Close": 26954624,
    "BackupStorage.File.File:Read": 102416075
  },
  "backup_duration_seconds": 4,
  "restore_duration_seconds": 6
}

Some notes to help understand these metrics:

• BackupBytes["BackupStorage.File.File:Write"] measures how many bytes were read from disk by the file Backup Storage implementation during the backup phase.
• DurationByPhaseSeconds["CatchUpReplication"] measures how long it took to catch-up replication after the restore phase.
• DurationByPhaseSeconds["RestoreLastBackup"] measures to the duration of the restore phase.
• RestoreDurationNanoseconds["-.-.Restore"] also measures to the duration of the restore phase.

VTTablet Error Count

VReplication publishes the VReplicationStreamState status which reports the state of VReplication streams. For example, here's what it looks like in the local cluster example after the MoveTables step:

"VReplicationStreamState": {
  "commerce2customer.1": "Running"
}

Prior to v17, this data was not available via the Prometheus backend. In v17, workflow states are also published as a Prometheus gauge with a state label and a value of 1.0. For example:

# HELP vttablet_v_replication_stream_state State of vreplication workflow
# TYPE vttablet_v_replication_stream_state gauge
vttablet_v_replication_stream_state{counts="1",state="Running",workflow="commerce2customer"} 1

In order to prevent SUPER privileged users like root or vt_dba from producing errant GTIDs on replicas, all the replica MySQL servers are initialized with the MySQL
global variable super_read_only value set to ON. During failovers, we set super_read_only to OFF for the promoted primary tablet. This will allow the
primary to accept writes. All of the shard's tablets, except the current primary, will still have their global variable super_read_only set to ON. This will make sure that apart from
MySQL replication no other component, offline system or operator can write directly to a replica.

A new flag, --schema-change-reload-timeout has been added to timeout the reload of the schema that Vttablet does periodically. This is required because sometimes this operation can get stuck after MySQL restarts, etc. More details available in the issue https://github.com/vitessio/vitess/issues/13001.

Online DDL's strategy now accepts --cut-over-threshold (type: duration) flag.

This flag stand for the timeout in a vitess migration's cut-over phase, which includes the final locking of tables before finalizing the migration.

The value of the cut-over threshold should be high enough to support the async nature of vreplication catchup phase, as well as accommodate some replication lag. But it mustn't be too high. While cutting over, the migrated table is being locked, causing app connection and query pileup, consuming query buffers, and holding internal mutexes.

Recommended range for this variable is 5s - 30s. Default: 10s.

The noblob binlog row image is now supported by the MoveTables and Reshard VReplication workflows. If the source
or target database has this mode, other workflows like OnlineDDL, Materialize and CreateLookupVindex will error out.
The row events streamed by the VStream API, where blobs and text columns have not changed, will contain null values
for those columns, indicated by a length:-1.

Gen4 planner was made default in v14 for SELECT queries. In v15 UPDATE and DELETE queries were moved to Gen4 framework.
With this release INSERT queries are moved to Gen4.

Clients can move to old v3 planner for inserts by using V3Insert planner version with --planner-version vtgate flag or with comment directive /*vt+ planner=<planner_version> for individual query.

• The deprecated automation and automationservice protobuf definitions and associated client and server packages have been removed.
• Auto-population of DDL revert actions and tables at execution-time has been removed. This is now handled entirely at enqueue-time.
• Backwards-compatibility for failed migrations without a completed_timestamp has been removed (see https://github.com/vitessio/vitess/issues/8499).
• The deprecated Key, Name, Up, and TabletExternallyReparentedTimestamp fields were removed from the JSON representation of TabletHealth structures.
• The MYSQL_FLAVOR environment variable is no longer used.
• The --enable-query-plan-field-caching/--enable_query_plan_field_caching vttablet flag was deprecated in v15 and has now been removed.

• Flag vtctld_addr has been deprecated and will be deleted in a future release. This affects vtgate, vttablet and vtcombo.
• The flag schema_change_check_interval used to accept either a Go duration value (e.g. 1m or 30s) or a bare integer, which was treated as seconds.
  This behavior was deprecated in v15.0.0 and has been removed.
  schema_change_check_interval now only accepts Go duration values. This affects vtctld.
• The flag durability_policy is no longer used by vtctld. Instead it reads the durability policies for all keyspaces from the topology server.
• The flag use_super_read_only is deprecated and will be removed in a later release. This affects vttablet.
• The flag queryserver-config-schema-change-signal-interval is deprecated and will be removed in a later release. This affects vttablet.
  Schema-tracking has been refactored in this release to not use polling anymore, therefore the signal interval isn't required anymore.

In vttablet various flags that took float values as seconds have updated to take the standard duration syntax as well.
Float-style parsing is now deprecated and will be removed in a later release.
For example, instead of --queryserver-config-query-pool-timeout 12.2, use --queryserver-config-query-pool-timeout 12s200ms.
Affected flags and YAML config keys:

• degraded_threshold
• heartbeat_interval
• heartbeat_on_demand_duration
• health_check_interval
• queryserver-config-idle-timeout
• queryserver-config-pool-conn-max-lifetime
• queryserver-config-olap-transaction-timeout
• queryserver-config-query-timeout
• queryserver-config-query-pool-timeout
• queryserver-config-schema-reload-time
• queryserver-config-schema-change-signal-interval
• queryserver-config-stream-pool-timeout
• queryserver-config-stream-pool-idle-timeout
• queryserver-config-transaction-timeout
• queryserver-config-txpool-timeout
• queryserver-config-txpool-idle-timeout
• shutdown_grace_period
• unhealthy_threshold

These stats are deprecated in v17.

| Deprecated stat | Supported alternatives |
|-|-|
| backup_duration_seconds | BackupDurationNanoseconds |
| restore_duration_seconds | RestoreDurationNanoseconds |