Here’s an idea: Developers should go through a proficiency test before they are allowed to use LLMs to generate code. Think of it as an AI driver’s license.
This is similar to how software shops would ask a certain amount of apprenticeship from junior engineers before they would get push access or review credentials on a repository. Similarly, before we allow somebody to run loose with the code generating power of an LLM, we should make sure they know how to wield it safely.
Using an LLM for learning and doing research should of course be unlimited. Just that people shouldn’t spend their time trying to one-shot their assignments. That way of working is way too tempting for somebody who doesn’t know better.
Both sides would benefit from this arrangement.
Company#
It’s obvious that LLMs function as an amplifier. Those who are skilled, can wield them to become better and much more effective. Those who aren’t, will use the tool to paper over their gaps in knowledge and effective delivery.
A company will always demand more of its employees and that all but ensures that the technology will be abused to meet and surpass commitments no matter what externalities that will bring.
Mario Zechner: “All of this compounds into an unrecoverable mess of complexity.” —Thoughts on slowing the fuck down
To utilize the productivity increases of LLMs without having your engineers destroy the coherence of your codebase, some gating will go a long way. Along with the gate, there should be a clear path how to pass it. That would be either through a competency test of some sort or when you hit a predefined career level.
Engineer#
For the engineer the equation is even more clear cut. Prompting an LLM and doomscrolling while you wait for the output is a direct path to deskill yourself out of the industry.
Mitchell Hashimoto: “This is how you get dumber btw, true even before AI” — https://x.com/mitchellh/status/2031776788532379996
If you’re not continuously adding value in intent, knowledge and nous to the LLM loop, you are just a mindless driver, not that different from a Ralph. As an engineer, you need to be in the code and on top of the LLM. You need to know exactly what is happening, what it’s doing, why and how in many cases the suggested solution is very much deficient.
For a junior this is more or less impossible. Spending your time telling the LLM to “fix lint” until your change passes CI and then hoping somebody else will approve your code to merge, is a losing proposition.
A junior engineer needs to invest to get to a point from where they can use the LLM effectively. This will only happen if they work on their craft diligently and reflectively. The research and learning capabilities of the LLM are perfect for this. Also if there’s no expectation that you will one-shot thousands of lines of code every day, you will get the space to invest in getting better.
A driver’s license for LLMs is not a perfect solution but it could remove a lot of the sharp edges of the usage we are seeing right now. At the very least it should make clear that not all LLM usage is equal.
