I agree that this is a tall order, but how about something like this...
To determine the relative denitrification of these two pathways in a 'typical' healthy reef aquarium system (once the overall rate of denitrification in such a system is first determined), would require the removal of as many of the phototrophic macro organisms as feasible from the system. Once completed, then the remaining denitrification would be accomplished primarily by the microorganisms. Keeping all other parameters (especially feeding) constant, then the difference between the denitrification rate of the whole system minus the denitrification rate of just the micro organisms would give the rate of denitrification of the phototrophic organisms. This could provide a general baseline determining which of the two groups actually contributes more to a 'typical' reef system's denitrification.
Removal of the phototrophic macro organisms from a typical reef aquarium is a hurdle. But a reef aquarium pre set up with holes drilled in the live rock, and all the phototrophic organisms on frag plugs, would perhaps be the easiest way to allow removal.
Soft corals and gorgonians are the most well known families that produce allelopathic substances. But many stony corals are now known to have their own allelopathic substances, too. When one considers that many successful fully stocked 'mixed reef' systems with potentially toxic species thrive with little, or even no chemical filtration, something quite extraordinary is going on. Some authors have hypothesized that the corals 'get used to their nasty neighbors' (habituation), others that the allelopathic substances are neutralized by 'something' in the system. Thought is, could that 'something' be bacteria that utilize the allelopathic chemicals as an energy source?
To prove something like this would require two aquarium setups, one with a full robust microbiome and one sterile. Place a few coral species together (but not physically touching) that are known to be toxic/don't have resistance to the other's toxin in both setups and observe the results.
Ralph.