Invisible files and Apple's Finder, tar, rsync, and GNU tar

Preface:
I know most individuals do not use the tar utility, but
this problem affects not only tar backups but also the
Apple Finder's Drag-and-Drop, Copy-and-Paste, AirDrop,
and the Universal Control's Drag-and-Drop.

File: README.Visibility.txt

Title: Invisible files and Apple's Finder, tar, rsync, and GNU tar

Date: 2022/04/22

Purpose:
Test the Finder's Drag-and-Drop, AirDrop, Copy-and-Paste,
and Universal Control's Drag-and-Drop for copying visible
and invisible files.
Test the Apple tar, GNU tar, and rsync utilities for copying
visible and invisible files.

Précis:
The short and sweet version:
Apple's Finder and Apple's tar utility miss some invisible
files when copying/backing up/transferring!
The GNU tar and rsync utilities do NOT miss these
invisible files.


Machines:
2020 13" M1 MacBook Pro (MacBookPro17,1),
2021 16" M1 Max MacBook Pro (MacBookPro18,2)

OS: macOS Monterey 12.3.1 (latest)

Programs:
Apple's Finder: version 12.3,
Apple's /usr/bin/rsync: version 2.6.9 protocol version 29
Apple's /usr/bin/tar: bsdtar 3.5.1 - libarchive 3.5.1 zlib/1.2.11 liblzma/5.0.5 bz2lib/1.0.8
GNU's /opt/homebrew/bin/gtar: tar (GNU tar) 1.34
(all of these are the latest versions available)

Below are the test files: some visible, some invisible, some hidden, some not hidden:
(I apologize for the formatting of the following listings. I do not know how to ensure
that the columns/spacings are obeyed in the following listings.)

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
dirac:Test-visibility-files solouki$ ls -alOR@
total 43392
drwxr-xr-x 13 solouki staff - 416 Apr 22 22:09 .
drwxr-xr-x 57 solouki staff - 1824 Apr 23 00:38 ..
-rwxr-xr-x 1 solouki staff - 187 Apr 12 2020 ._invisible-type-2.bin
-rw-r--r-- 1 solouki staff - 54 Apr 21 06:12 ._invisible-type-2.txt
-rw-r--r-- 1 solouki staff - 43 Apr 18 09:31 .invisible-type-1-made-visible.txt
-rwxr-xr-x 1 solouki staff - 187 Apr 12 2020 .invisible-type-1.bin
-rw-r--r-- 1 solouki staff - 43 Apr 18 09:31 .invisible-type-1.txt
-rw-r--r--@ 1 solouki staff - 7209398 Apr 18 12:29 Screen Shot 2022-04-18 at 12.29.24 PM.png
-rw-r--r-- 1 solouki staff - 14970880 Apr 21 12:00 test.tar
com.apple.FinderInfo 32
com.apple.lastuseddate#PS 16
com.apple.metadata:kMDItemIsScreenCapture 42
com.apple.metadata:kMDItemScreenCaptureGlobalRect 76
com.apple.metadata:kMDItemScreenCaptureType 48
drwxr-xr-x 12 solouki staff - 384 Apr 22 22:11 Temp
-rw-r--r-- 1 solouki staff - 36 Apr 22 01:33 hidden-made-visible.txt
-rw-r--r-- 1 solouki staff hidden 23 Apr 22 01:27 hidden.txt
-rw-r--r-- 1 solouki staff - 14970880 Apr 21 12:00 test.tar
-rw-r--r-- 1 solouki staff - 24 Apr 18 09:40 visible.txt

./Temp:
total 43392
drwxr-xr-x 12 solouki staff - 384 Apr 22 22:11 .
drwxr-xr-x 13 solouki staff - 416 Apr 22 22:09 ..
-rwxr-xr-x 1 solouki staff - 187 Apr 12 2020 ._invisible-type-2.bin
-rw-r--r-- 1 solouki staff - 54 Apr 21 06:12 ._invisible-type-2.txt
-rw-r--r-- 1 solouki staff - 43 Apr 18 09:31 .invisible-type-1-made-visible.txt
-rwxr-xr-x 1 solouki staff - 187 Apr 12 2020 .invisible-type-1.bin
-rw-r--r-- 1 solouki staff - 43 Apr 18 09:31 .invisible-type-1.txt
-rw-r--r--@ 1 solouki staff - 7209398 Apr 18 12:29 Screen Shot 2022-04-18 at 12.29.24 PM.png
com.apple.FinderInfo 32
com.apple.lastuseddate#PS 16
com.apple.metadata:kMDItemIsScreenCapture 42
com.apple.metadata:kMDItemScreenCaptureGlobalRect 76
com.apple.metadata:kMDItemScreenCaptureType 48
-rw-r--r-- 1 solouki staff - 36 Apr 22 01:33 hidden-made-visible.txt
-rw-r--r-- 1 solouki staff hidden 23 Apr 22 01:27 hidden.txt
-rw-r--r-- 1 solouki staff - 14970880 Apr 21 12:00 test.tar
-rw-r--r-- 1 solouki staff - 24 Apr 18 09:40 visible.txt
dirac:Test-visibility-files solouki$
----------------------------------------------------------------------

As you will notice in the above listing, there are `dot' files
(normally invisible from the Finder but are visible if the AppleShowAllFiles flag
is set for the Finder, i.e., if you execute: then the Finder shows these files as `grayed-out' names).

You will also notice there are `hidden' files
(files whose hidden flag has been set with: then the Finder shows these files as `grayed-out' if the AppleShowAllFiles
flag is also set, otherwise these files are hidden
from the Finder).

There is also a screenshot file with metadata.

You will also notice that there are `dot-underscore' files.
These files are not only invisible to the Finder, but they are not
shown by setting the AppleShowAllFiles flag. These files are not
only invisible to the Finder, they are also not copied by any Finder
techniques! Thus if you attempt to copy these files using the
Finder's Drag-and-Drop, AirDrop, Universal Control's Drag-and-Drop,
or Copy-and-Paste, these files will NOT be copied or transferred.

In addition, Apple's tar utility will NOT backup the `dot-underscore'
files, even if they are listed directly in the tar include file!
However, the GNU tar utility will backup these `dot-underscore'
files! Both Apple's tar and the GNU tar utilities now, under macOS
12.3.1, have "Cannot stat: No such file or directory" errors when
pattern matching regular expressions in the `--files-from=' option.
This used to work in earlier macOS versions, but pattern matching in
tar includes does not work in macOS 12.3.1. Intriguingly, regular
expression pattern matching does work in the ``--exclude-from='
option -- so there is an inconsistency between the include and
exclude options for pattern matching!

(I tried multiple different ways of including these `dot-underscore' files in
the Apple tar backups, including directly naming them. None of these
attempts was successful.)

Apple's rsync utility does copy/backup/transfer the `dot-underscore' files.
It does not copy the `hidden' flag, however, just the file.
(Thus there is an inconsistency between Apple's tar and rsync
utilities, with Apple's tar missing the `dot-underscore' files while
Apple's rsync properly copying the `dot-underscore' files.)

In Summary:
If you wish to transfer/backup/copy all files, especially including
any `dot-underscore' files, do NOT use Finder nor Apple's tar
utility which will miss these invisible files.
(In other words, Drag-and-Drop, Copy-and-Paste, AirDrop,
Universal Control, or Apple's tar will not transfer/backup/copy
all files.)
Instead, do use the GNU tar or Apple's rsync utilities to
transfer/backup/copy all files including the invisible
`dot-underscore' files.

Addendum:
While Apple's tar utility allows the `--files-from=' and
`--exclude-from=' options to be in any order, the GNU tar utility
requires the `--exclude-from=' option to precede the `--files-from='
option on the commandline.

EDIT: If anyone is interested in performing their own tests, I have a
bash shell script that performs the Apple tar and GNU tar tests.