We collected papers from ICLR, NEURIPS, ICCV, CVPR, ICML, IJCAI, EMNLP, ACL and AAAI. ICLR 2022 has papers from the review stage.
PS: Since most AI paper preprints are readily available in Arxiv now, what effect do you think it has if the paper is published in a popular conference?
[D] Is there an open-source implementation of the Retrieval-Enhanced Transformer (RETRO)?[reddit]/r/MachineLearning
Wondering if there's an open-source implementation of the Retrieval-Enhanced Transformer (RETRO) [arxiv link]?
I imagine that it may take some time to make its way to production libraries like Huggingface Transformers due to the external memory component. Anyone heard of people working on this?
Did you know you could use JAX to write Keras layers? It's as simple as a call to `jax2tf.convert()`. XLA is the shared language that makes this works. [twitter]
Mixture of Experts layers (MoEs) enable efficient scaling of language models
through conditional computation. This paper presents a detailed empirical study
of how autoregressive MoE language models scale in comparison with dense models
in a wide range of settings: in- and out-of-domain language modeling, zero- and
few-shot priming, and full fine-tuning. With the exception of fine-tuning, we
find MoEs to be substantially more compute efficient. At more modest training
budgets, MoEs can match the performance of dense models using $\sim$4 times
less compute. This gap narrows at scale, but our largest MoE model (1.1T
parameters) consistently outperforms a compute-equivalent dense model (6.7B
parameters). Overall, this performance gap varies greatly across tasks and
domains, suggesting that MoE and dense models generalize differently in ways
that are worthy of future study. We make our code and models publicly available
for research use.
