Resolved! Very long post.
Well, yesterday it appears that we found the problem.
In our last episode, the Cummins mechanics had replaced the injectors, which were badly carboned up, but that didn't resolve the basic rough running and fuel knock problem. Further testing pointed to a short in the wiring harness.
After speaking with the Cummins marine manager we decided to have the mechanic (Josh) come to the boat again armed with a new wiring harness, plus I asked him to send along a new fuel pump actuator and fuel rail pressure sensor just in case. He was very willing to order those parts in "just in case".
A day later I get a call from the marine manager telling me that his mechanics had been discussing it more as well as talking with all the non-marine guys at the shop. One of them had related an experience with the same engine but in in a road commercial application. This mechanic said that he had spent 4, 14 hour days working on this engine, doing the same things my mechanic had done, chasing odd codes, replacing the injectors, wiring harness, fuel rail pressure sensor and more. For some reason he finally took the belt off the alternator and the engine ran great. The marine manager said that he was going to send a new alternator with Josh and the first thing he was going to do was take the alternator belt off.
So yesterday that is what we did. Took the alternator belt off and started the engine. The engine did run a little better, less fuel knock, smoother idle etc but still not like it should. Put the belt on again and the engine ran a bit worse. However, the problem has been variable over the past 3 1/2 months so I told Josh that I didn't think that was the right track. Sometimes it would run better than others. So we decided that was a fail and replaced the fuel pump actuator. This had no affect.
Josh, being much smarter than I, said that he wanted to go back to the alternator idea before he replaced the fuel rail sensor then the wiring harness. He decided to completely disconnect the alternator.
I shut down the engine power solenoid which cut the engine off from the start battery. Again, since he is much smarter than I am, checked the positive on the alternator with a multimeter before wrenching on the connections and said that he still had 12.85V to the alternator. This of course made sense since the alternator is wired to the house bank. I went into the aft lazarette and disconnected the 250amp fuse protecting the alternator wiring.
With the alternator disconnected, the engine started and ran perfectly.
We reconnected the alternator and the problem returned full force. Josh did this two more times just to be sure (he really wanted this problem to go away and didn't want to have to come back).
Sitting in the lazarette staring at the electrical panel I had an idea. A while back, I had changed the start battery charging from the existing Blue Seas ACR to a prior Balmar Duo Charge. The boat has three of the Balmar Duo Chargers, one for the genset battery, one for the thruster/windlass/crane bank, and one for the start battery that had been bypassed in favor of the ACR which looks like it had been added later. I changed back to the Duo Charger since I think it is actually a better way to charge an AGM start battery with a wet cell house bank than an ACR. However, something about that Duo Charger gives the electronic engine indigestion.
I then recalled something that Tony Athens wrote in another thread here on TF a while ago;
With electronic engines, WE NEVER let the engines talk to each other except in an emergency situation and a full understanding what the current issue is. The only common thing is ONE NEG for all with redundancy across all engines.
Being relatively stupid, this didn't make any sense to me. However, my recent experience demonstrates why Tony takes this approach. Very odd things can happen.
So, I have no idea why the Duo Charge creates the issue. It could be related to the health of the the start battery. When it is completely isolated, maybe the Duo Charge doesn't provide enough current to keep the volts up? Maybe it is too much? This shouldn't be the case if the Balmar Duo Charge is operating correctly. The ACR combines the batteries when a charge voltage is applied, either from the genset, dock charger, or the alternator. The engine runs fine with this combined bank.
There is plenty of power in the start battery, it fires the engine up easily. It is just when the engine is running that there is a problem. At least now I know what was causing the problem and have an engine that is running well. I can try to answer the questions as to why the Balmar Duo Charge was creating the issue another time.