1. [Publications](/index.php/publications)
2. Equivariant Architectures for Learning in Deep Weight Spaces

 # Equivariant Architectures for Learning in Deep Weight Spaces

  ![](/sites/default/files/styles/wide/public/publications/sym.png?itok=fklnzlH-)

 Designing machine learning architectures for processing neural networks in their raw weight matrix form is a newly introduced research direction. Unfortunately, the unique symmetry structure of deep weight spaces makes this design very challenging. If successful, such architectures would be capable of performing a wide range of intriguing tasks, from adapting a pre-trained network to a new domain to editing objects represented as functions (INRs or NeRFs). As a first step towards this goal, we present here a novel network architecture for learning in deep weight spaces. It takes as input a concatenation of weights and biases of a pre-trained MLP and processes it using a composition of layers that are equivariant to the natural permutation symmetry of the MLP’s weights: Changing the order of neurons in intermediate layers of the MLP does not affect the function it represents. We provide a full characterization of all affine equivariant and invariant layers for these symmetries and show how these layers can be implemented using three basic operations: pooling, broadcasting, and fully connected layers applied to the input in an appropriate manner. We demonstrate the effectiveness of our architecture and its advantages over natural baselines in a variety of learning tasks.

 ## Authors

Aviv Navon (Bar-Ilan University)

 Aviv Shamsian (Bar-Ilan University)

Idan Achituve (Bar-Ilan University)

Ethan Fetaya (Bar-Ilan University)

[Gal Chechik](/index.php/person/gal-chechik)

[Haggai Maron](/index.php/person/haggai-maron)

 ## Publication Date

Monday, March 6, 2023

 ## Published in

[ICML 2023](https://openreview.net/pdf?id=SCU1xlr9Y4)

 ## Research Area

[Artificial Intelligence and Machine Learning ](/index.php/research-area/machine-learning-artificial-intelligence)

 ## External Links

[Project page](https://avivnavon.github.io/DWSNets/)

[Arxiv](https://arxiv.org/abs/2301.12780)