Recent comments posted to this site:
On debian buster, copying to or from tahoe seems to alway cause an attempt to start a new tahoe process, even if one is already running, resulting in error messages like this on second or subsequent attempts:
copy P5273249.JPG STARTING '/home/jk/.tahoe-git-annex/3677c81a-5cd0-4817-9ad3-a7c29b74a7b7'
starting node in '/home/jk/.tahoe-git-annex/3677c81a-5cd0-4817-9ad3-a7c29b74a7b7'
Another twistd server is running, PID 20860
This could either be a previously started instance of your application or a
different application entirely. To start a new one, either run it in some other
directory, or use the --pidfile and --logfile parameters to avoid clashes.
Not a massive problem as everything seems to work.
Joey, I feel like I have asked it before, but can't quickly search it up, so asking again (sorry if repeating). By
To attach metadata to a particular path, rather than a particular key, use .gitattributes .
you meant attaching some other metadata, not really "git-annex" metadata, right?
Or there is a way to specify metadata key=value
pairs to be attached to annexed files, e.g. smth like
*_scans.json annex.metadata=(distribution-restrictions=sensitive)
?
"When versioning is not enabled, this risks data loss" -- maybe, add an option to always import before exporting? I don't want to enable versioning since that permanently loses the chance of dropping any objects (even if they're stored elsewhere).
Also, versioning on S3 buckets can be suspended/resumed; not sure if git-annex uses current bucket versioning state or the remote configuration.
@Ilya, importing before exporting does not avoid losing any data that gets written at just the wrong time. Versioning is the only way to guarantee no data loss.
If you enable versioning for the bucket and then disable it or don't version some items, git-annex won't notice and if you then do an export I'm afraid you again risk losing data.
I did not write that, Ilya did. I agree that it was confusing that it was talking about some entirely different kind of metadata, and have reworded it.
I am implementing a special remote using https://github.com/Lykos153/AnnexRemote
I found that the annexremote leaves the readline()
loop once it receives an empty line from git-annex over stdin (https://github.com/Lykos153/AnnexRemote/blob/master/annexremote/annexremote.py#L410). Given that the protocol description says nothing about empty lines this may or may not be sensible. However, I also found that git-annex does (sometimes?) sends empty lines. Here is an excerpt:
% git annex --debug copy --to inm7 down/cope1.feat/report_log.html
...
[2019-05-14 07:45:43.350358695] git-annex-remote-inm7[1] <-- TRANSFER STORE MD5E-s48796--33a9884cc35c891f693958b9dd7fccd6.html .git/annex/objects/mj/Kx/MD5E-s48796--33a9884cc35c891f693958b9dd7fccd6.html/MD5E-s48796--33a9884cc35c891f693958b9dd7fccd6.html
<<INCOMING ''
(repeated a total of 22 times)
...
<<INCOMING 'TRANSFER STORE MD5E-s48796--33a9884cc35c891f693958b9dd7fccd6.html .git/annex/objects/mj/Kx/MD5E-s48796--33a9884cc35c891f693958b9dd7fccd6.html/MD5E-s48796--33a9884cc35c891f693958b9dd7fccd6.html\n'
where '<<INCOMING' reports anything that is read via readline()
from the special remote's stdin
. So 22 empty lines are read from
stdin
before the TRANSFER STORE
reported by git-annex to have been sent actually appears.
If I drive the remote implementation "by hand" I don't see anything wrong, and no non-protocol output:
% git-annex-remote-inm7
VERSION 1
EXTENSIONS INFO
EXTENSIONS
PREPARE
GETCONFIG dataset_uuid
VALUE dda42d6c-7231-11e9-a901-0050b6902ef0
PREPARE-SUCCESS
CHECKPRESENT MD5E-s48796--33a9884cc35c891f693958b9dd7fccd6.html
DIRHASH MD5E-s48796--33a9884cc35c891f693958b9dd7fccd6.html
VALUE mj/Kx/
CHECKPRESENT-FAILURE MD5E-s48796--33a9884cc35c891f693958b9dd7fccd6.html
TRANSFER STORE MD5E-s48796--33a9884cc35c891f693958b9dd7fccd6.html .git/annex/objects/mj/Kx/MD5E-s48796--33a9884cc35c891f693958b9dd7fccd6.html/MD5E-s48796--33a9884cc35c891f693958b9dd7fccd6.html
DIRHASH MD5E-s48796--33a9884cc35c891f693958b9dd7fccd6.html
VALUE mj/Kx/
TRANSFER-SUCCESS STORE MD5E-s48796--33a9884cc35c891f693958b9dd7fccd6.html
If git-annex runs it, I see empty lines appearing only after CHECKPRESENT-FAILURE
.
Which purpose do these empty lines serve? Under what conditions would an empty response from git-annex occur?
Thx!
stdin
was connected to the SSH process -- which confused all involved parties.