Github Copilot Productivity

"One good dev with Github Copilot ships what entire teams did five years ago." - This is one example of some bold claims that have been made about Github Copilot.

Discussions about AI software development tools can be polarizing - most software engineers pride themselves on their ability to produce well thought-out code after all. It's difficult to get real feedback from engineers about how useful AI tools could be if they were given a fair shake.

This is my assessment of how best to use Github Copilot as a software developer, where I think the tool is lacking, and what kind of realistic productivity gains I think are possible with it.

The experience

Copilot is a VSCode plugin first and foremost, which makes it already a pretty opinionated tool. I should also lead with the caveat that I am a diehard Emacs user - I need a very compelling reason to leave my preferred text editing interface for VSCode.

While it's not the only product to provide LLM assistance to programmers, Copilot has done a lot to lead the direction of development that similar tools have taken. I'd classify its behaviour into two categories: code completion and 'conversational programming'. The 'code completion' side is pretty straightforward - it's like auto-complete but with more code suggested at once. The 'conversational programming' experiences are a little more varied, but I basically think of them as an optimization over copy-and-pasting code in an out of a conversation with an LLM.

Auto-complete

Let's start with the easiest feature to analyze. Copilot will suggest whole chunks of text, which, depending on the situation, can be either very helpful or very annoying.

For example:

Or, if I 'enter' to a newline, that completion becomes:

neither of which not being quite what I had in mind, but close enough to what I wanted that it's not hard for me to correct.

In my opinion, code-completion, at least as implemented in Copilot, is too rough a tool for me to want to use all the time. Maybe it's due to habit from years of writing code 'the good old fashioned way', but I find having to constantly double check Copilot's suggestions a little cognitively disorienting. Outside of some specific scenarios, I'd rather just type something out the way I wanted it.

Some of my peers enjoy using the code-completion feature of Copilot, and I suspect it's because they don't have such a strong relationship with (an admittedly peculiar) text editing workflow the way I do. Believe it or not, I actually have written tons of production Java with no autocomplete at all - so I clearly have a bias around auto-complete but hear me out:

Ultimately, I see this kind of code-completion as a 'last-mile' efficiency gain - I think by the time code-completion is useful, you know what you want to write and it's merely a matter of getting those characters into the file as fast as possible. If auto-complete helps you get ideas out faster, then by all means have at it, but I wouldn't labour under the illusion that this is going to make you a 10x faster developer than your coworkers.

Conversational programming

This is a style of LLM use for programming that I am more excited about. Copilot offers a couple ways to write code with prompts: 1) a 'multi-file' edit, and 2) an 'inline' edit.

Multi-file edits

Multi-file edits are a lot like giving instructions to another dev and having them execute the changes. In theory, if anything was going to deliver on the '10x productivity' promise, this would probably be it.

My prompt:

The proposed changes:

A proposed diff:

In my above example, the proposed changes seemed to miss the fact that there was already some event-handling infrastructure that should have been leveraged. This is all well-and-good if you are prepared to spend some time iterating on the changes that Copilot proposes, but can quickly become annoying when you have a clear idea of what the changes need to look like already.

Multi-file edits delegate a lot of decision-making to the LLM, and for now, I'm not convinced that this is the best way to use it when coding. My opinion is that an LLM is going to be much more effective at writing very specific code from very specific instructions, which leads us to the Copilot feature I actually use: inline prompting.

Inline Prompting

In 'inline prompting', instead of describing to the LLM a change that is distributed over multiple files, we restrict the change to the current file. Note that we can still add other files to the prompt though.

I find this to be the sweet spot of how I want to prompt the LLM - I basically describe in English exactly what I want to happen, using actual function / class names, accept the LLM output, then clean it up manually. I can't perfectly explain why, but keeping my prompts close to the actual code that should be written, along with being able to see surrounding code as the suggested code is written seems to keep me in a good flow state. Typically I would use this approach to hack out the gist of a change from a 'big-picture' headspace, then go back through in detail and clean everything up from a 'nitty-gritty details' perspective.

Code explanations

Copilot offers the ability to 'explain' some selections of code, or to ask questions against a subset of the codebase. Since this is a problem that I have spent a lot of time thinking about, I am pretty underwhelmed by the Copilot implementation of this functionality. I find the Copilot chat responses are generally a regurgitation of the code details itself, when the answer I'm looking for requires some higher-level analysis.

By the way - if this is something you're interested in, I'm looking for input on how teams want to use code analysis tools.

In Summary

Some really promising workflows are possible with Copilot. I've laid out how I think about Copilot's current features and how they have worked for me, but certainly others will use the tool differently. Ultimately I think the most important question is:

"How much more productive could I be if I became familiar with Copilot?"

Here's my take:

I think the major benefit from using Copilot is to avoid having to leave a 'higher-level' headspace in order to wire things up and compile. I think if it's leveraged properly, developers can avoid burning out their attention too quickly on small level details in order to stay in a high-level design state longer. My estimate is that using Copilot can translate into an extra hour per day of focus time for a dev writing code.

What do you think? Am I underselling the potential of Copilot?
I'd love to hear what you think about AI tools for software developers in general, if you have a minute to leave some feedback for the 2025 Software Engineering AI Tools Report.

Until next time,

Liam