The line is basically programmability. If a device runs “firmware” that can’t be changed, that’s really just a part of the design of the hardware you bought, so the fact that you can’t see or modify the source code is irrelevant- even if you could, it wouldn’t give you any more control over the hardware. If it runs firmware that can be changed, it’s a programmable computer, and by running proprietary firmware, you’re giving up control you would otherwise have over your computer.
The FSF’s position is roughly that software (instructions that run on programmable computers) ought to be free as a matter of ethics, but that hardware is different, basically because of the difficulty and resources necessary to “copy” hardware. See Free Hardware and Free Hardware Designs for Stallman’s thoughts. Basically, he (and the FSF) don’t consider “free hardware” to be a coherent concept- just “free hardware designs”. So using FSF terminology, it doesn’t make sense to speak of free or open-source hardware.
The FSF basically draws the line at easy programmability. If a device was designed to be reprogrammed, what it’s running is software, which should be free, otherwise you’re ceding control over that computer to the copyright holders of the software it runs- and that doesn’t just include your host CPU, it includes every computer you own, including other ones on your motherboard. If a device was not meant to be easily reprogrammed, then the “software” it runs is really just a permanent part of the design, and the FSF doesn’t really care about this type of firmware being “free”. Devices with firmware in Linux are clearly able to be easily reprogrammed- you just change the firmware code- so firmware that runs on these devices meets the FSF’s definition of software.
They’re not saying devices shouldn’t be programmable- they’re saying if they are, users need to have the right to control the software that runs on said device.
For the FSF it’s not about what’s more “secure” or less complex, but rather whether there’s freedom to be had. In the one case, having rights to view and modify the source gives the user additional freedoms they otherwise don’t have, e.g. to apply security patches as you point out. In the other case, it doesn’t, because even if they could view and modify the source, they can’t make changes to their device with that ability even in principle. Note that, in general, the law does not prohibit users from modifying hardware that they own, so that’s not an issue of freedom. But software is.
Having a circuit diagram for a chip you own is cool, but it doesn’t allow you to change an IC- you don’t have the practical ability to do that. Likewise if the “firmware” is permanent- viewing the “source” is maybe technically interesting but grants you zero additional abilities. If it’s defective, it’s the same as buying a defective IC. Tough luck.
You can choose not to care about the additional freedom you get if you’re allowed to view and modify firmware for programmable devices you own, and that’s fine. It’s typically not meaningfully important for me either. But it’s absolutely not an arbitrary line.
Why do you say it matters where the firmware is being executed? Is it less important to control less powerful computers you own? Why should the code suddenly be unimportant if it’s not executing on your main processor?