We're very pleased to present you restic 0.15.0!

In this version a new rewrite command has been implemented. This allows for removing unwanted data from existing snapshots, for example if one discovers that some files were unintentionally backed up, such as files with sensitive content or files that take up a lot of space and don't need to be backed up. Please see the corresponding documentation for more information and usage examples.

restic is distributed as a standalone binary: download the correct file for your operating system and architecture, extract the file and just run it. If you run into any issues, please report them at the GitHub issue tracker or visit the forum. If you already have restic >= 0.9.4, you can use restic self-update to get the latest version in a secure way.

The binaries released with each restic version are reproducible, which means that you can reproduce a byte identical version from the source code for that release. Instructions on how to do that in the Developer Documentation.

Changelog for restic 0.15.0 (2023-01-12)

The following sections list the changes in restic 0.15.0 relevant to restic users. The changes are ordered by importance.

Summary

Fix #2015: Make mount return exit code 0 after receiving Ctrl-C / SIGINT
Fix #2578: Make restore replace existing symlinks
Fix #2591: Don't read password from stdin for backup --stdin
Fix #3161: Delete files on Backblaze B2 more reliably
Fix #3336: Make SFTP backend report no space left on device
Fix #3567: Improve handling of interrupted syscalls in mount command
Fix #3897: Fix stuck copy command when -o <backend>.connections=1
Fix #3918: Correct prune statistics for partially compressed repositories
Fix #3951: Make ls return exit code 1 if snapshot cannot be loaded
Fix #4003: Make backup no longer hang on Solaris when seeing a FIFO file
Fix #4016: Support ExFAT-formatted local backends on macOS Ventura
Fix #4085: Make init ignore "Access Denied" errors when creating S3 buckets
Fix #4100: Make self-update enabled by default only in release builds
Fix #4103: Don't generate negative UIDs and GIDs in tar files from dump
Chg #2724: Include full snapshot ID in JSON output of backup
Chg #3929: Make unlock display message only when locks were actually removed
Chg #4033: Don't print skipped snapshots by default in copy command
Chg #4041: Update dependencies and require Go 1.18 or newer
Enh #14: Implement rewrite command
Enh #79: Restore files with long runs of zeros as sparse files
Enh #1078: Support restoring symbolic links on Windows
Enh #1734: Inform about successful retries after errors
Enh #1866: Improve handling of directories with duplicate entries
Enh #2134: Support B2 API keys restricted to hiding but not deleting files
Enh #2152: Make init open only one connection for the SFTP backend
Enh #2533: Handle cache corruption on disk and in downloads
Enh #2715: Stricter repository lock handling
Enh #2750: Make backup file read concurrency configurable
Enh #3029: Add support for credential_process to S3 backend
Enh #3096: Make mount command support macOS using macFUSE 4.x
Enh #3124: Support JSON output for the init command
Enh #3899: Optimize prune memory usage
Enh #3905: Improve speed of parent snapshot detection in backup command
Enh #3915: Add compression statistics to the stats command
Enh #3925: Provide command completion for PowerShell
Enh #3931: Allow backup file tree scanner to be disabled
Enh #3932: Improve handling of ErrDot errors in rclone and sftp backends
Enh #3943: Ignore additional/unknown files in repository
Enh #3955: Improve backup performance for small files

Details

Bugfix #2015: Make mount return exit code 0 after receiving Ctrl-C / SIGINT

To stop the mount command, a user has to press Ctrl-C or send a SIGINT signal to restic. This used to cause restic to exit with a non-zero exit code.

The exit code has now been changed to zero as the above is the expected way to stop the mount command and should therefore be considered successful.

[#2015](https://github.com/restic/restic/issues/2015) [#3894](https://github.com/restic/restic/pull/3894)

Bugfix #2578: Make restore replace existing symlinks

When restoring a symlink, restic used to report an error if the target path already existed. This has now been fixed such that the potentially existing target path is first removed before the symlink is restored.

[#2578](https://github.com/restic/restic/issues/2578) [#3780](https://github.com/restic/restic/pull/3780)

Bugfix #2591: Don't read password from stdin for backup --stdin

The backup command when used with --stdin previously tried to read first the password, then the data to be backed up from standard input. This meant it would often confuse part of the data for the password.

From now on, it will instead exit with the message Fatal: cannot read both password and data from stdin unless the password is passed in some other way (such as --restic-password-file, RESTIC_PASSWORD, etc).

To enter the password interactively a password command has to be used. For example on Linux, mysqldump somedatabase | restic backup --stdin --password-command='sh -c "systemd-ask-password < /dev/tty"' securely reads the password from the terminal.

[#2591](https://github.com/restic/restic/issues/2591) [#4011](https://github.com/restic/restic/pull/4011)

Bugfix #3161: Delete files on Backblaze B2 more reliably

Restic used to only delete the latest version of files stored in B2. In most cases this worked well as there was only a single version of the file. However, due to retries while uploading it is possible for multiple file versions to be stored at B2. This could lead to various problems for files that should have been deleted but still existed.

The implementation has now been changed to delete all versions of files, which doubles the amount of Class B transactions necessary to delete files, but assures that no file versions are left behind.

[#3161](https://github.com/restic/restic/issues/3161) [#3885](https://github.com/restic/restic/pull/3885)

Bugfix #3336: Make SFTP backend report no space left on device

Backing up to an SFTP backend would spew repeated SSH_FX_FAILURE messages when the remote disk was full. Restic now reports "sftp: no space left on device" and exits immediately when it detects this condition.

A fix for this issue was implemented in restic 0.12.1, but unfortunately the fix itself contained a bug that prevented it from taking effect.

[#3336](https://github.com/restic/restic/issues/3336) [#3345](https://github.com/restic/restic/pull/3345) [#4075](https://github.com/restic/restic/pull/4075)

Bugfix #3567: Improve handling of interrupted syscalls in mount command

Accessing restic's FUSE mount could result in "input/output" errors when using programs in which syscalls can be interrupted. This is for example the case for Go programs. This has now been fixed by improved error handling of interrupted syscalls.

[#3567](https://github.com/restic/restic/issues/3567) [#3694](https://github.com/restic/restic/issues/3694) [#3875](https://github.com/restic/restic/pull/3875)

Bugfix #3897: Fix stuck copy command when -o <backend>.connections=1

When running the copy command with -o <backend>.connections=1 the command would be infinitely stuck. This has now been fixed.

[#3897](https://github.com/restic/restic/issues/3897) [#3898](https://github.com/restic/restic/pull/3898)

Bugfix #3918: Correct prune statistics for partially compressed repositories

In a partially compressed repository, one data blob can exist both in an uncompressed and a compressed version. This caused the prune statistics to become inaccurate and e.g. report a too high value for the unused size, such as "unused size after prune: 16777215.991 TiB". This has now been fixed.

[#3918](https://github.com/restic/restic/issues/3918) [#3980](https://github.com/restic/restic/pull/3980)

Bugfix #3951: Make ls return exit code 1 if snapshot cannot be loaded

The ls command used to show a warning and return exit code 0 when failing to load a snapshot. This has now been fixed such that it instead returns exit code 1 (still showing a warning).

[#3951](https://github.com/restic/restic/pull/3951)

Bugfix #4003: Make backup no longer hang on Solaris when seeing a FIFO file

The backup command used to hang on Solaris whenever it encountered a FIFO file (named pipe), due to a bug in the handling of extended attributes. This bug has now been fixed.

[#4003](https://github.com/restic/restic/issues/4003) [#4053](https://github.com/restic/restic/pull/4053)

Bugfix #4016: Support ExFAT-formatted local backends on macOS Ventura

ExFAT-formatted disks could not be used as local backends starting from macOS Ventura. Restic commands would fail with an "inappropriate ioctl for device" error. This has now been fixed.

[#4016](https://github.com/restic/restic/issues/4016) [#4021](https://github.com/restic/restic/pull/4021)

Bugfix #4085: Make init ignore "Access Denied" errors when creating S3 buckets

In restic 0.9.0 through 0.13.0, the init command ignored some permission errors from S3 backends when trying to check for bucket existence, so that manually created buckets with custom permissions could be used for backups.

This feature became broken in 0.14.0, but has now been restored again.

[#4085](https://github.com/restic/restic/issues/4085) [#4086](https://github.com/restic/restic/pull/4086)

Bugfix #4100: Make self-update enabled by default only in release builds

The self-update command was previously included by default in all builds of restic as opposed to only in official release builds, even if the selfupdate tag was not explicitly enabled when building.

This has now been corrected, and the self-update command is only available if restic was built with -tags selfupdate (as done for official release builds by build.go).

[#4100](https://github.com/restic/restic/pull/4100)

Bugfix #4103: Don't generate negative UIDs and GIDs in tar files from dump

When using a 32-bit build of restic, the dump command could in some cases create tar files containing negative UIDs and GIDs, which cannot be read by GNU tar. This corner case especially applies to backups from stdin on Windows.

This is now fixed such that dump creates valid tar files in these cases too.

[#4103](https://github.com/restic/restic/issues/4103) [#4104](https://github.com/restic/restic/pull/4104)

Change #2724: Include full snapshot ID in JSON output of backup

We have changed the JSON output of the backup command to include the full snapshot ID instead of just a shortened version, as the latter can be ambiguous in some rare cases. To derive the short ID, please truncate the full ID down to eight characters.

[#2724](https://github.com/restic/restic/issues/2724) [#3993](https://github.com/restic/restic/pull/3993)

Change #3929: Make unlock display message only when locks were actually removed

The unlock command used to print the "successfully removed locks" message whenever it was run, regardless of lock files having being removed or not.

This has now been changed such that it only prints the message if any lock files were actually removed. In addition, it also reports the number of removed lock files.

[#3929](https://github.com/restic/restic/issues/3929) [#3935](https://github.com/restic/restic/pull/3935)

Change #4033: Don't print skipped snapshots by default in copy command

The copy command used to print each snapshot that was skipped because it already existed in the target repository. The amount of this output could practically bury the list of snapshots that were actually copied.

From now on, the skipped snapshots are by default not printed at all, but this can be re-enabled by increasing the verbosity level of the command.

[#4033](https://github.com/restic/restic/issues/4033) [#4066](https://github.com/restic/restic/pull/4066)

Change #4041: Update dependencies and require Go 1.18 or newer

Most dependencies have been updated. Since some libraries require newer language features, support for Go 1.15-1.17 has been dropped, which means that restic now requires at least Go 1.18 to build.

[#4041](https://github.com/restic/restic/pull/4041)

Enhancement #14: Implement rewrite command

Restic now has a rewrite command which allows to rewrite existing snapshots to remove unwanted files.

[#14](https://github.com/restic/restic/issues/14) [#2731](https://github.com/restic/restic/pull/2731) [#4079](https://github.com/restic/restic/pull/4079)

Enhancement #79: Restore files with long runs of zeros as sparse files

When using restore --sparse, the restorer may now write files containing long runs of zeros as sparse files (also called files with holes), where the zeros are not actually written to disk.

How much space is saved by writing sparse files depends on the operating system, file system and the distribution of zeros in the file.

During backup restic still reads the whole file including sparse regions, but with optimized processing speed of sparse regions.

[#79](https://github.com/restic/restic/issues/79) [#3903](https://github.com/restic/restic/issues/3903) [#2601](https://github.com/restic/restic/pull/2601) [#3854](https://github.com/restic/restic/pull/3854) https://forum.restic.net/t/sparse-file-support/1264

Enhancement #1078: Support restoring symbolic links on Windows

The restore command now supports restoring symbolic links on Windows. Because of Windows specific restrictions this is only possible when running restic with the SeCreateSymbolicLinkPrivilege privilege or as an administrator.

[#1078](https://github.com/restic/restic/issues/1078) [#2699](https://github.com/restic/restic/issues/2699) [#2875](https://github.com/restic/restic/pull/2875)

Enhancement #1734: Inform about successful retries after errors

When a recoverable error is encountered, restic shows a warning message saying that it's retrying, e.g.:

Save(<data/956b9ced99>) returned error, retrying after 357.131936ms: ...

This message can be confusing in that it never clearly states whether the retry is successful or not. This has now been fixed such that restic follows up with a message confirming a successful retry, e.g.:

Save(<data/956b9ced99>) operation successful after 1 retries

[#1734](https://github.com/restic/restic/issues/1734) [#2661](https://github.com/restic/restic/pull/2661)

Enhancement #1866: Improve handling of directories with duplicate entries

If for some reason a directory contains a duplicate entry, the backup command would previously fail with a node "path/to/file" already present or nodes are not ordered got "path/to/file", last "path/to/file" error.

The error handling has been improved to only report a warning in this case. Make sure to check that the filesystem in question is not damaged if you see this!

[#1866](https://github.com/restic/restic/issues/1866) [#3937](https://github.com/restic/restic/issues/3937) [#3880](https://github.com/restic/restic/pull/3880)

Enhancement #2134: Support B2 API keys restricted to hiding but not deleting files

When the B2 backend does not have the necessary permissions to permanently delete files, it now automatically falls back to hiding files. This allows using restic with an application key which is not allowed to delete files. This can prevent an attacker from deleting backups with such an API key.

To use this feature create an application key without the deleteFiles capability. It is recommended to restrict the key to just one bucket. For example using the b2 command line tool:

b2 create-key --bucket <bucketName> <keyName> listBuckets,readFiles,writeFiles,listFiles

Alternatively, you can use the S3 backend to access B2, as described in the documentation. In this mode, files are also only hidden instead of being deleted permanently.

[#2134](https://github.com/restic/restic/issues/2134) [#2398](https://github.com/restic/restic/pull/2398)

Enhancement #2152: Make init open only one connection for the SFTP backend

The init command using the SFTP backend used to connect twice to the repository. This could be inconvenient if the user must enter a password, or cause init to fail if the server does not correctly close the first SFTP connection.

This has now been fixed by reusing the first/initial SFTP connection opened.

[#2152](https://github.com/restic/restic/issues/2152) [#3882](https://github.com/restic/restic/pull/3882)

Enhancement #2533: Handle cache corruption on disk and in downloads

In rare situations, like for example after a system crash, the data stored in the cache might be corrupted. This could cause restic to fail and required manually deleting the cache.

Restic now automatically removes broken data from the cache, allowing it to recover from such a situation without user intervention. In addition, restic retries downloads which return corrupt data in order to also handle temporary download problems.

[#2533](https://github.com/restic/restic/issues/2533) [#3521](https://github.com/restic/restic/pull/3521)

Enhancement #2715: Stricter repository lock handling

Previously, restic commands kept running even if they failed to refresh their locks in time. This could be a problem e.g. in case the client system running a backup entered the standby power mode while the backup was still in progress (which would prevent the client from refreshing its lock), and after a short delay another host successfully runs unlock and prune on the repository, which would remove all data added by the in-progress backup. If the backup client later continues its backup, even though its lock had expired in the meantime, this would lead to an incomplete snapshot.

To address this, lock handling is now much stricter. Commands requiring a lock are canceled if the lock is not refreshed successfully in time. In addition, if a lock file is not readable restic will not allow starting a command. It may be necessary to remove invalid lock files manually or use unlock --remove-all. Please make sure that no other restic processes are running concurrently before doing this, however.

[#2715](https://github.com/restic/restic/issues/2715) [#3569](https://github.com/restic/restic/pull/3569)

Enhancement #2750: Make backup file read concurrency configurable

The backup command now supports a --read-concurrency option which allows tuning restic for very fast storage like NVMe disks by controlling the number of concurrent file reads during the backup process.

[#2750](https://github.com/restic/restic/pull/2750)

Enhancement #3029: Add support for credential_process to S3 backend

Restic now uses a newer library for the S3 backend, which adds support for the credential_process option in the AWS credential configuration.

[#3029](https://github.com/restic/restic/issues/3029) [#4034](https://github.com/restic/restic/issues/4034) [#4025](https://github.com/restic/restic/pull/4025)

Enhancement #3096: Make mount command support macOS using macFUSE 4.x

Restic now uses a different FUSE library for mounting snapshots and making them available as a FUSE filesystem using the mount command. This adds support for macFUSE 4.x which can be used to make this work on recent macOS versions.

[#3096](https://github.com/restic/restic/issues/3096) [#4024](https://github.com/restic/restic/pull/4024)

Enhancement #3124: Support JSON output for the init command

The init command used to ignore the --json option, but now outputs a JSON message if the repository was created successfully.

[#3124](https://github.com/restic/restic/issues/3124) [#3132](https://github.com/restic/restic/pull/3132)

Enhancement #3899: Optimize prune memory usage

The prune command needs large amounts of memory in order to determine what to keep and what to remove. This is now optimized to use up to 30% less memory.

[#3899](https://github.com/restic/restic/pull/3899)

Enhancement #3905: Improve speed of parent snapshot detection in backup command

Backing up a large number of files using --files-from-verbatim or --files-from-raw options could require a long time to find the parent snapshot. This has been improved.

[#3905](https://github.com/restic/restic/pull/3905)

Enhancement #3915: Add compression statistics to the stats command

When executed with --mode raw-data on a repository that supports compression, the stats command now calculates and displays, for the selected repository or snapshots: the uncompressed size of the data; the compression progress (percentage of data that has been compressed); the compression ratio of the compressed data; the total space saving.

It also takes into account both the compressed and uncompressed data if the repository is only partially compressed.

[#3915](https://github.com/restic/restic/pull/3915)

Enhancement #3925: Provide command completion for PowerShell

Restic already provided generation of completion files for bash, fish and zsh. Now powershell is supported, too.

[#3925](https://github.com/restic/restic/pull/3925)

Enhancement #3931: Allow backup file tree scanner to be disabled

The backup command walks the file tree in a separate scanner process to find the total size and file/directory count, and uses this to provide an ETA. This can slow down backups, especially of network filesystems.

The command now has a new option --no-scan which can be used to disable this scanning in order to speed up backups when needed.

[#3931](https://github.com/restic/restic/pull/3931)

Enhancement #3932: Improve handling of ErrDot errors in rclone and sftp backends

Since Go 1.19, restic can no longer implicitly run relative executables which are found in the current directory (e.g. rclone if found in .). This is a security feature of Go to prevent against running unintended and possibly harmful executables.

The error message for this was just "cannot run executable found relative to current directory". This has now been improved to yield a more specific error message, informing the user how to explicitly allow running the executable using the -o rclone.program and -o sftp.command extended options with ./.

[#3932](https://github.com/restic/restic/issues/3932) https://pkg.go.dev/os/exec#hdr-Executables_in_the_current_directoryhttps://go.dev/blog/path-security

Enhancement #3943: Ignore additional/unknown files in repository

If a restic repository had additional files in it (not created by restic), commands like find and restore could become confused and fail with an multiple IDs with prefix "12345678" found error. These commands now ignore such additional files.

[#3943](https://github.com/restic/restic/pull/3943) https://forum.restic.net/t/which-protocol-should-i-choose-for-remote-linux-backups/5446/17

Enhancement #3955: Improve backup performance for small files

When backing up small files restic was slower than it could be. In particular this affected backups using maximum compression.

This has been fixed by reworking the internal parallelism of the backup command, making it back up small files around two times faster.

[#3955](https://github.com/restic/restic/pull/3955)