My lab has about 100k to spend on an entire computer setup[reddit]/r/MLQuestions
My lab has about 100k USD to spend on an entire computer setup. We are mainly focusing on large language models (GPT/BERT/etc.) and knowledge graphs. I'm not too familiar with actual computer setups (GPUS/fans/power/CPUS/etc.) and was hoping that someone here will have some good ideas on what to buy / where to look at / suggestions.
[R] NormFormer: Improved Transformer Pretraining with Extra Normalization[reddit]https://arxiv.org/abs/2110.09456/r/MachineLearningIn general, we have shown that adding small numbers of learnable parameters in the right places in our architectures can alleviate certain issues in current state of the art networks
‹›
During pretraining, the Pre-LayerNorm transformer suffers from a gradient magnitude mismatch: gradients at early layers are much larger than at later layers. These issues can be alleviated by our proposed NormFormer architecture, which adds three normalization operations to each layer: a Layer Norm after self attention, head-wise scaling of self-attention outputs, and a Layer Norm after the first fully connected layer. The extra operations incur negligible compute cost (+0.4% parameter increase), but improve pretraining perplexity and downstream task performance for both causal and masked language models ranging from 125 Million to 2.7 Billion parameters. For example, adding NormFormer on top of our strongest 1.3B parameter baseline can reach equal perplexity 24% faster, or converge 0.27 perplexity better in the same compute budget. This model reaches GPT3-Large (1.3B) zero shot performance 60% faster. For masked language modeling, NormFormer improves fine-tuned GLUE performance by 1.9% on average. Code to train NormFormer models is available in fairseq.
2021-10-18: Discovering and Achieving Goals via World Models https://arxiv.org/abs/2110.09514v1By proposing a challenging benchmark and the first agent to achieve meaningful performance on these tasks, we hope to stimulate future research on unsupervised agents, which we believe are fundamentally more scalable than traditional agents that require a human to design the tasks and rewards for learning
How can artificial agents learn to solve many diverse tasks in complex visual
environments in the absence of any supervision? We decompose this question into
two problems: discovering new goals and learning to reliably achieve them. We
introduce Latent Explorer Achiever (LEXA), a unified solution to these that
learns a world model from image inputs and uses it to train an explorer and an
achiever policy from imagined rollouts. Unlike prior methods that explore by
reaching previously visited states, the explorer plans to discover unseen
surprising states through foresight, which are then used as diverse targets for
the achiever to practice. After the unsupervised phase, LEXA solves tasks
specified as goal images zero-shot without any additional learning. LEXA
substantially outperforms previous approaches to unsupervised goal-reaching,
both on prior benchmarks and on a new challenging benchmark with a total of 40
test tasks spanning across four standard robotic manipulation and locomotion
domains. LEXA further achieves goals that require interacting with multiple
objects in sequence. Finally, to demonstrate the scalability and generality of
LEXA, we train a single general agent across four distinct environments. Code
and videos at https://orybkin.github.io/lexa/
Perception is fundamentally compositional. We make sense of novel stimuli (things we're not used to) by parsing its elements, then composing them following a known template. The left image and the right image are the same, rotated. Note how you interpret the upside-down face. [twitter]
A number of fields, most prominently speech, vision and NLP have been disrupted by deep learning technology. A natural question is: "which application areas will follow next?". My prediction is that the physical sciences will experience an unprecedented acceleration by combining the tools of simulation on HPC clusters with the tools of deep learning to improve and accelerate this process. Together, they form a virtuous cycle where simulations create data that feeds into deep learning models which in turn improves the simulations. In a way, this is like building a self-learning computational microscope for the physical sciences. In this talk I will illustrate this using two recent pieces of work from my lab: molecular simulation and PDE solving. In molecular simulation we try to predict molecular properties or digitally synthesize molecules with prescribed properties. We have built a number of equivariant graph neural networks to achieve this. Partial differential equations (PDEs) are the most used mathematical model in natural sciences to describe physical processes. Intriguingly, we find that PDE solvers can be learned from data using graph neural networks as well, which has the added benefit that we can learn a solver that can generalize across PDEs and different boundary conditions. Moreover, it may open the door to ab initio learning of PDEs directly from data.
About the speaker:
Prof. Dr. Max Welling is a research chair in Machine Learning at the University of Amsterdam and a Distinguished Scientist at MSR. He is a fellow at the Canadian Institute for Advanced Research (CIFAR) and the European Lab for Learning and Intelligent Systems (ELLIS) where he also serves on the founding board. His previous appointments include VP at Qualcomm Technologies, professor at UC Irvine, postdoc at U. Toronto and UCL under supervision of prof. Geoffrey Hinton, and postdoc at Caltech under supervision of prof. Pietro Perona. He finished his PhD in theoretical high energy physics under supervision of Nobel laureate prof. Gerard ‘t Hooft.
2021-10-18: Understanding Dimensional Collapse in Contrastive Self-supervised Learning https://arxiv.org/abs/2110.09348v1We provided the theoretical understanding of this phenomenon and showed that there are two mechanisms causing dimensional collapse: strong augmentation and implicit regularization
‹›
Self-supervised visual representation learning aims to learn useful
representations without relying on human annotations. Joint embedding approach
bases on maximizing the agreement between embedding vectors from different
views of the same image. Various methods have been proposed to solve the
collapsing problem where all embedding vectors collapse to a trivial constant
solution. Among these methods, contrastive learning prevents collapse via
negative sample pairs. It has been shown that non-contrastive methods suffer
from a lesser collapse problem of a different nature: dimensional collapse,
whereby the embedding vectors end up spanning a lower-dimensional subspace
instead of the entire available embedding space. Here, we show that dimensional
collapse also happens in contrastive learning. In this paper, we shed light on
the dynamics at play in contrastive learning that leads to dimensional
collapse. Inspired by our theory, we propose a novel contrastive learning
method, called DirectCLR, which directly optimizes the representation space
without relying on a trainable projector. Experiments show that DirectCLR
outperforms SimCLR with a trainable linear projector on ImageNet.
2021-10-13: Object DGCNN: 3D Object Detection using Dynamic Graphs https://arxiv.org/abs/2110.06923v1In this section, we provide results when the model is distilled with 1, 6, and 20 epochs
‹›
3D object detection often involves complicated training and testing
pipelines, which require substantial domain knowledge about individual
datasets. Inspired by recent non-maximum suppression-free 2D object detection
models, we propose a 3D object detection architecture on point clouds. Our
method models 3D object detection as message passing on a dynamic graph,
generalizing the DGCNN framework to predict a set of objects. In our
construction, we remove the necessity of post-processing via object confidence
aggregation or non-maximum suppression. To facilitate object detection from
sparse point clouds, we also propose a set-to-set distillation approach
customized to 3D detection. This approach aligns the outputs of the teacher
model and the student model in a permutation-invariant fashion, significantly
simplifying knowledge distillation for the 3D detection task. Our method
achieves state-of-the-art performance on autonomous driving benchmarks. We also
provide abundant analysis of the detection model and distillation framework.
If youre advertising a machine learning or AI scholarship or job on Twitter, please consider announcing it to @QueerinAI @AiDisability @black_in_ai @Khipu_AI @DeepIndaba @_LXAI @WiMLworkshop @women_in_ai and other groups who care about diversity and inclusion. Thanks https://twitter.com/NandoDF/status/1450751832238739464
