There wouldn't be any legality issues with that. In fact there's no reason you can't carry a rifle and a shotgun at the same time. Of course, there is the orange requirement, as you said, so just make sure you put your orange on when you switch from hunting ducks to hunting deer.