I'm no expert but we talked a lot about small tank with hobbyist at our lfs :
Choose fishes that are not "swimmers". Therefore avoid big wrasses, pelagic or big triggers and Acanthurus tangs.
Instead, opt for fish that are more reef associated such as Zebrasoma tangs (still a bit big for less than 100g), small wrasses or small tetraodonts (the family with puffers, triggers).
Avoid fish that are aggressive or predatory toward inhabitants of the same small tanks. The stress will be constant. For instance potter angels can nip at corals. Angelfish in general tend to be coral nippers. Some people are lucky, some are less.
Angelfish are rock grazer so they eat pretty much anything palatable growing on them : small inverts, polyps, tunicates (sea squirts), sponges, algae, etc. Some angels are less nippy than other : C. loricula (flame angelfish), despite looking less 'endemic' and being quite common in reeftanks but more people have good track records with them. Plus they are less expensive. However people reported reef compatibility with their Centropyges potteri and loricula, so try if you are feeling lucky.
Some suggestions : if you can put your hand on a Cirripectes blenny,
Cirripectes vanderbilti is endemic to Hawaii and has ton of personality. Unfortunately very few wholeseller have them, and I have yet to see it at a LFS. You can substitute it with an Ember blenny (
Cirripectes stigmaticus) which is much more common.
Blennies are fun, will eat algae, and don't take up too much precious swimming space for the rest of the fishes.
If you really want an agressive Tetraodont you can also go for a
Canthigaster jactator. They swim much less than a medium sized trigger, are less aggressive toward other fishes. Also they are quite endemic to hawaii and have a lot of personality. They are still invertebrate killing machine.
Then you have the classic small fishes such as fire Dartfishes (
Nemateleotris magnifica).
I'm not competent for the butterflyfish part however.
