June 23rd - AI: Anti-Hype Edition

How to begin? Perhaps a brief rant followed by an admission?

Brief Rant

Is anyone else beyond sick of seeing AI shoved into absolutely everything, like it's the new sriracha? Why did my dishwasher need AI included in it?!?! What's next, are they going to shove AI in my toothpaste?

Admission

On the first attempt of writing this post I followed a workflow similar to the one discussed below, using AI to scaffold the body of the post, but this lead to an obvious outcome. The whole thing read like it was written by AI(duh). This was also blatantly hypocritical given my rants at the beginning and frustration over all the AI slop out there. There was only one path forward, I trashed the original and rewrote it the "old school" way. I'm writing directly in markdown, will use spellcheck, and have some human reviewers but no AI writing.

Mindset

Let me get back on track here, since this is supposed to be about how I use AI in the context of software engineering. To me AI is purely another tool in my ever expanding toolbox; sitting comfortably on the shelf next to many other tools.

The usefulness I find in AI as a tool is predominately faster "googling" and non-deterministic automation.

With it I am able to research an error, bug, or potential solution via multiple sources in a more rapid fashion than manual searching. Is it perfect? No. This is clear from one of my most frequent prompts being:

Can you please provide links to the documentation you are referencing?

Best use cases I've found

Documentation This one is kinda related to the next item in the list but I wanted to call it out specifically. If you need to write documentation for a chunk of code or create a readme for a project you're new to, AI can be great. The documentation it creates still requires careful review but significantly faster and less brain melting than writing documentation completely by hand.

Scaffolding I rarely use fancy IDE's that have features for spitting out templated or boilerplate code so I leverage AI for scaffolding out the code that I have already handwritten in the past; once scaffolded the code can be reviewed with a fine tooth comb and have any adjustments made that are needed.

Research The context within which I operate often varies massively day to day (sometimes hour to hour); writing a script one moment, followed by debugging an API, followed by tweaking firewall rules, and then deploying some infra. This makes it extremely valuable to be able to do things like

• Getting a summary of a topic pulled from multiple sources; whether net new or something I've worked with previously but need a refresher on.
• Searching notes for a prior solution to a similar problem.

Rubber Duck Sometimes there is no human available and due to the non-deterministic nature of LLM responses they can be used to think through potential solutions or weird edge cases.

Learning Tool Basically the polar and principally opposite idea to "vibe-coding". I never blindly accept a line of code or a change proposed by an AI tool. If I don't understand what the code does, it's not getting committed. I will stop and have the tool explain character by character, over and over, until I fully understand it.

AI in my workflow

Currently my workflow looks like this:

1. Have a real problem to solve. Not some tutorial exercise or hypothetical that has no real world use.

2. Delve through the dark caverns of my brain searching for prior experience with a related problem and come up with some potential solutions. This is required, brain must be exercised regularly.

3. When a fellow human isn't available, I'm sick of talking to myself, or more feedback is needed than what my dog can provide I'll use an AI tool as a rubber duck. I'll use it to rip apart the potential solutions from step 2 or surface potential solutions that may not have occurred to me.

4. Once a path forward is decided on, think through how to break the work into workable chunks. Repeat this step as many times as needed to not feel overwhelmed.

5. Scaffolding - I really enjoying working with computers so if there are things for a solution that I have memorized, can quickly scaffold myself while enjoying my creamy keyboard, or something I want to learn I will do it without any AI involvment. The biggest constraint here is the amount of time available for a task. Sometimes you have a problem that needs to be solved as quickly as possible for whatever reason and in those situations I will use any tool available. Given infinite time I would spend time on every char, tracing every bug, perfectly optimize resource usage, or creating the perfect user experience. But unfortunately we don't have the watches from Clockstoppers, yet!

6. Scaffolding with AI - Describe the current piece of the problem being attacked in as much detail as possible. Being hyper specific with what you want will reduce the frustration that future you feels, sometimes.

   • Carefully read every single character that the AI proposes before any actions are approved.
   • If there is something that is new or doesn't make sense, pause, and get explanation character by character until 100% understanding is achieved.

7. Only when 100% understanding is achieved will the proposed action be allowed to be taken. This is for two main reasons. First, because I have been burned in the past by not following this rule. Second, it allows me to continue to gain knowledge along the way.

8. Review all of the generated code with a flea comb to find errors, vulnerabilities, edge cases, and adherence to specifications - triple check EVERYTHING.

9. Fix anything that is wrong and revise things that I don't like. For example I can't stand "magic" code that someone with no context of the project can't easily understand. If some random engineer can't there is a high probability that future you won't understand either.

My Current "Won't Do's"

• Trust all actions for a session. I review every single action the agent wants to perform.
• Allow agents to operate autonomously. If I need things automated I will build deterministic automation that can be trusted to run without constant oversight and have predictable outcomes.
• Blindly trusting information that is provided. Always follow up with "Please provide references" and actually vet those references.

Non-Determinism in Practice

For the non-deterministic automation I'm referring to tasks like "scaffold me a readme for this Github action I created" or "create an api endpoint based on this existing one". I'm open to at least trying AI tools to assist with things that don't depend on repeatable logic that is guaranteed to perform the exact same actions every single time. Often times I will use an AI tool to assist in scaffolding the code to build deterministic automation. There are many situations where the risk of having an agent do something completely off the wall based on a prompt is simply not worth it. The idea of the "DevOps" or "SRE" agents with the access required to resolve a production incident from a novel root cause gives me the heebs.

Take as an example (an overly simple one) the prompt:

create a local branch, fix error X, and push those changes

This could be executed in fundamentally different ways depending on the model being used, the context the agent has, or which way the wind is blowing. Even when you specify in the agents config, the specific way that it is supposed to complete specific tasks there is no guarantee that it will follow the exact same steps every single time.

The Sands of Time

AI can speed up the completion of tasks that don't require critical thinking or deterministic outcomes; things like drafting a readme for a chunk of code. That being said reviewing AI generated code is a time suck which requires you to be constantly on guard. AI will confidently propose things that are total garbage and your knowledge/experience is the final safe guard.

The Obsolescence Narrative

I may be biased from working in software engineering but the constant stream of redacted that "AI" will make software engineers(and related roles) obsolete is not occurring anytime soon based on the capabilities I have seen. There is a massive gap between vibe coding an app that would be at best considered a POC and successfully engineering a piece of software that can run for years on end. Software that can scale when needed, stable under load, adapt when a capability is missing, and valuable to the user.

It should also not be ignored that most of those hyping the capabilities of "AI" have a direct financial interest in the products success. That's not a reason to blindly dismiss the technology, but it is a reason to be skeptical of the marketing spiel.

Closing thoughts

AI tools are interesting but hardly the magical thing they are hyped as. Do I use it? Yeah, where it makes sense; but if it disappeared tomorrow I'd be perfectly fine.

Always say "Please" and "Thank you" when prompting AI, you know, just in case... oh and never stop learning!