I am not sure if this follows with electronics, but I know in normal electrical wiring it is never good to connect a positive to a negative. You are bringing the current full circle which can possibly over time short out your electronics. Like I said though, I am not sure if this applies to wiring with electronics.
I know it sounds like it makes sense to go positive to negative that way you have a positive and negative open on each end. Have you tried going positive to positive and negative to negative then tie those directly to your battery holder. You might be able to pull a full charge to your batteries. You can test this with a multi-meter to see what the draw is.