Splitting a neural network into specific Expert sub-networks. Creates a Expert Router to route to a specific segment. Similar to Multi-Agent, but at a granular scale Inspired by the human brain, where certain parts of the brain do certain things.
Architecture
- Input goes to the router
- Router assigns that input to one or more experts based off relevance (Top K Routing)
- Each relevant expert processes the input independently
- Outputs are combined by weighted averaging