I absolutely think you can set up a tank as an ecosystem. But conditional on a number of factors:
1. You need a lot of rock so that microfauna have a place to live and hide. And as a place for sponges and worms and tunicates to attach. Seeding the tank with live rock seems the way to go in my opinion. Of course a large amount of rock doesn't always make for a beautifully aqua-scaped appearance.
2. Minimal number of fish seems best. Too many fish will de-populate the microfauna and de-stabilize the ecosystem you are trying to maintain.
3. Fish that can graze on what grows in the tank will greatly reduce the food you have to add to the tank. Heavy import and export of nutrients can make it difficult to maintain stable microfauna populations. Microfauna populations can grow out of control and/or crash rather quickly and easily. Reducing the food input and keeping fish that manage the microfauna populations helps keep the system in balance.
4. A large refugium helps greatly.
5. Embracing algae growth (to a point) helps greatly. Algae growth in the main tank and the refugium can be a valuable source of food for the fish and the microfauna. And a place for microfauna to live. Your tank may consequently not be as visually attractive as you might hope. But it will certainly be more interesting.
6. Recognize that your tank almost certainly cannot maintain a
self-sustaining ecosystem. Just not enough tank volume. So you will have to feed and replenish the bottom of the food chain. I gave up on adding phyto directly to the tank and instead add phyto to glass gallon jugs full of copepods. I can screen out some of the copepods weekly and maintain a constant source for the tank without trying blanket the whole tank with sufficient amounts of phytoplankton. I also buy amphipods once or twice a year to add to the tank. I also add a lot of liquid food to the tank and carbon dose to boost bacterial populations. You have to be committed to feeding and maintaining all the small creatures you never really see.
7. You need microfauna in a variety of sizes. The more "levels" of microfauna present in your system the quicker your system can break down carbon waste and convert the waste into food for the bottom levels of the food chain. The goal is to cycle the nutrients in your tank efficiently to maintain some manner of ecosystem stability.
8. You need to recognize your tank will be carrying a much higher biological load than other tanks. So water movement and aeration to get carbon dioxide out of the water is much more critical. Your tank might also crash much more quickly in case of a power outage. You will need back up measures in place.
9. It takes time to achieve stability. I always see these "Clean-up Crew Packages" for sale that include like 100 or more hermit crabs. I can't imagine how any tank can support 100 hermit crabs. After 30+ years I know my system can comfortably support about 4 or 5 hermit crabs. If I add too many, the population just dies back to 4 or 5. If I stay at 4-5 they live for years and years and are a valuable part of the ecosystem. And that is just hermit crabs. Trying to figure out what works for the whole system is a slow and often frustrating process.
But I love at least trying to make it work. It certainly seems to be a very healthy environment for the fish and corals. I wish you the best of luck!!!
