I feel like I'm going a little crazy so I wanted to ask this sub as it seems like there are a fair number of people here in relatively mature ML environments. I used to work at a company that had a fairly old-school development workflow (as in, modern in 2015). Individual DS/ML researchers would have fairly beefy personal machines (32+GB of RAM, big CPU, xx80 or two if necessary). We were not doing any groundbreaking DL research or anything but working on training data of usually a few GB. We used docker but it was more of an engineering thing than anything I dealt with day to day. Personally I never had trouble with just sharing environment files. We would use some shared HPC resources but again, it wasn't anything too sophisticated (and imo we didn't need anything like that).
Now, I work at a company that is supposedly more "modern" using the cloud and docker-first (similar scale of data and problem type), but the I find the workflow soooo slow and hard to deal with. Here, we do all the development inside cloud docker containers themselves. But it seems like there are 25 micro-steps to even get started on a project and so much overhead for such little gain. Okay let me spin up instance, ssh, pull docker image, mount shared drive, forward port, git clone, etc etc etc. Can't use PyCharm or even VS Code since the dev environment is inside a docker on a remote machine (I know you can technically do this with VS Code, but it's finnicky imo). Every day I am dealing with micro-problems arising from this. And at the end, when it comes time to deploy, I still need to ping MLOps guys to do anything useful with the image. I feel like I spend half my day on this and half my day on the actual problem at hand. What exactly is the point of all this? Are we even saving money on all these cloud instances being spun up and down?
Anyway, I asked my boss about this, and he doesn't see a problem since everyone else on the team is a linux ninja and just has a million little aliases and copy/paste cli routines to do things, and knows all the gotchas and they get work done, so why is it a problem? Why am I a special snowflake who NEEDS to use modern IDE like PyCharm rather than being happy with a simple notebook? Does everyone on modern ML teams work like this? Does this org need to hire more MLOps/Engineers or get one of these fancy cloud solutions? Seems a lot more expensive than just specing out a beefy desktop machine where everything is dead simple and easy to manage. This company is not Uber or Facebook, we are not working at global scale or preparing for hyper-growth.