How can I get better timestamp server error messages when signing binaries?
SignTool Error: The specified timestamp server either could not be reached or
returned an invalid response.
SignTool Error: An error occurred while attempting to sign: ...
Let's start out with a short rant:
As far as error messages go, this is somewhere between but-they-tried, stupid and plain insulting.
signtool.exe needs to talk to the timestamp server via HTTP, so WTF are we not getting told what exactly failed? This is such a lazy error message. "Could not be reached or returned an invalid response." - oh, my.
Maybe I should be content that the tool at least tells me that the error was with the timestamp server, but that does get us only so far.
/rant over
We will get temporary signing errors from http://timestamp.digicert.com from time to time. (Every odd month maybe.)
The thing is, our build scripts already try to circumnavigate this:
- We sign our binaries in blocks of 100 files, that is each
signtool.execall will receive 100 files on the command line. - Before even starting to use signtool, I separately verify the HTTP connection to
http://timestamp.digicert.com. (While that page will answer with a 404 for a simple GET request, that does indicate it is reachable.) - For all cases where we recorded this error, signtool will only record errors for some of the 100 passed files.
Our latest log looks something like this:
[2023-01-24 12:58:11.038] : (INFO ) Sign *all* 1337 binaries at <~redacted~>: This will be SLOW ...
...
[2023-01-24 12:58:11.334] : OK: HEAD request to 'http://timestamp.digicert.com' returned http status code NotFound/404 (any HTTP status code is fine).
[2023-01-24 12:58:11.335] : #100 ...
[2023-01-24 12:58:11.354] : Invoke-SignTool with 100 files ...
[2023-01-24 12:58:11.359] : Executing '&"C:\Program Files (x86)\Windows Kits\10\bin\10.0.18362.0\x64\signtool.exe" sign /v /sm /s My /n "~certname~" /fd SHA256 /tr http://timestamp.digicert.com /td SHA256 "..."' ...
The following certificate was selected:
Issued to: ~certname~
Issued by: DigiCert Trusted G4 Code Signing RSA4096 SHA384 2021 CA1
Expires: ~...
SHA1 hash: ~...
Done Adding Additional Store
SignTool Error: The specified timestamp server either could not be reached or
returned an invalid response.
SignTool Error: An error occurred while attempting to sign: ...
x repeat
...
Successfully signed: .\~....dll
Successfully signed: .\~....dll
x repeat
...
Number of files successfully Signed: 50
Number of warnings: 0
Number of errors: 50
SignTornadoBinaries error: (ERROR) Exec (exit code <1>): NONZERO exit code
Waiting for 10 seconds...
Retrying...
[2023-01-24 13:00:13.022] : (INFO ) Sign *all* 1287 binaries at <~redacted~>: This will be SLOW ...
[2023-01-24 13:00:13.319] : OK: HEAD request to 'http://timestamp.digicert.com' returned http status code NotFound/404 (any HTTP status code is fine).
[2023-01-24 13:00:13.320] : #100 ...
[2023-01-24 13:00:13.338] : Invoke-SignTool with 100 files ...
[2023-01-24 13:00:13.343] : Executing '&"C:\Program Files (x86)\Windows Kits\10\bin\10.0.18362.0\x64\signtool.exe" sign /v /sm /s My /n "~certname~" /fd SHA256 /tr http://timestamp.digicert.com /td SHA256 ...
The following certificate was selected:
Issued to: ~certname~
....
SignTool Error: The specified timestamp server either could not be reached or
returned an invalid response.
SignTool Error: An error occurred while attempting to sign: .\x64-vc143\....dll
...
Number of files successfully Signed: 50
Number of warnings: 0
Number of errors: 50
SignTornadoBinaries error: (ERROR) Exec (exit code <1>): NONZERO exit code
Waiting for 10 seconds...
Retrying...
[2023-01-24 13:02:16.970] : (INFO ) Sign *all* 1237 binaries at <~redacted~>: This will be SLOW ...
[2023-01-24 13:02:17.283] : OK: HEAD request to 'http://timestamp.digicert.com' returned http status code NotFound/404 (any HTTP status code is fine).
[2023-01-24 13:02:17.284] : #100 ...
[2023-01-24 13:02:17.301] : Invoke-SignTool with 100 files ...
[2023-01-24 13:02:17.306] : Executing ...
Number of files successfully Signed: 50
Number of warnings: 0
Number of errors: 50
Note that this 100-files-at-once normally works multiple times a week, sometimes multiple times a day (depending on our build schedule -- only production builds are signed.)
As we also can see here, here, the error was reported for exactly the first 50 files, and the second 50 files are then signed. Our retry is 3x with 10 sec wait, so after that the build just aborts: As far as I am concerned, this 50/50 split is a "clear" indication that this is not about a flaky network or some load issue on the digicert server or whatever.
Do note however, that I've also seen logs with 99/1 or maybe 80/20.
So, to get back to an actual question:
Is there any way to get more detailed errors from signtool?
Barring that, is there an alternative toolchain (some powershell commandlets maybe?) that will let me sign my binaries and give me useful error messages?
Aside: If you happen to know if a different number of files to pass to signtool at once may result in a more stable setup, please let me know.
Case in point:
In my case above, the 50/50 failure ratio came from our misbehaving firewall. It would block every odd HTTP Timestamp Protocol request with a HTTP/403.
If signtool would log the HTTP response including the content body, because otherwise I wouldn't have known that the response came from the firewall instead of from the actual target server then analyzing the problem would have been instantaneous!
You can try Jsign, if the timestamp server fails it can retry after a delay and/or fallback to other servers. The syntax looks like this:
(disclaimer: I'm the author of this project)