ERR_CONNECTION_RESET error when trying to update the configuration of some old Jenkins jobs

Question

Some of the older Jenkins jobs on a build server cannot be updated via the web GUI. I can rollback to older configs using things like the job config history plugin, but normal updates cause it to say something in the bottom of the browser like, Uploading (79%), progress up to 9x% and eventually it errors out in Chrome with ERR_CONNECTION_RESET.

It doesn't do it for other old jobs, only a small handful. I ran diffs of their config.xml and nothing stood out.

I also already uninstalled the MultiSCM plugin which apparently has caused a similar issue in the past.

I'm not getting any sort of warning or error in the Jenkins log itself when these updates fail.

I've also updated Jenkins to latest LTS as well as all the plugins.

Any idea what's going on or how to fix it?

In JavaMelody, it sees the POST failing, but doesn't give any indication why:

Answer 1

This was so annoying to debug due to not knowing the origin of the error. None of the jenkins logs were helpful between FINEST and SEVERE. After working with our lead of devops for a couple days we couldn't find anything on the system or server settings that changed the behavior.

I ran across someone who had a similar issue on GitHub saying that for one of their plugins, it would break if there was an extra EOL symbol. That got me thinking, could this be a parsing error for plugin input?

First I disabled everything that wasn't a build step. This fixed nothing. Then I started removing one plugin type at a time in the Build Step section and eventually narrowed it down to Windows Batch Command build steps. Through trial and error I determined it was 1 of 3 batches and debugged it down to finally a single line. This was the part of the batch where it was hidden:

@echo off
SetLocal EnableDelayedExpansion EnableExtensions
REM ...
call MsBuild.exe %SLN_FILE% /p:Configuration="%Build_Configuration%" /p:Platform="x86" -m -nodeReuse:false>nul 2>&1
if not "%errorlevel%"=="0" (
    echo Preliminary build initially failed. Running with console output next
    call MsBuild.exe /t:rebuild %SLN_FILE% /p:Configuration="%Build_Configuration%" /p:Platform="x86" -m -nodeReuse:false
    if not "!errorlevel!"=="0" echo Preliminary x86 build failed. Exiting early... & exit /b 1
)
REM ...
exit /b 0

Putting on my parse-error thinking cap, I refactored this,

if not "!errorlevel!"=="0" echo Preliminary x86 build failed. Exiting early... & exit /b 1

into this,

if not "!errorlevel!"=="0" (
    echo Preliminary x86 build failed
    echo Exiting early
    exit /b 1
)

And it started working again. So it's probably a parsing bug on the Jenkins-consuming side, but the actual bug origin is probably the Windows Batch Command module when its input is being formatted for escaping special characters.

FYI! :)