From Software Engineer to AI Engineer: 8 Lessons From Under the Hood at Skai

• Becoming an AI engineer takes more than learning new tools. Understanding how AI models work is essential to building reliable, effective applications.
• AI models have limitations engineers need to account for. Data bias, tokenization, and context overload can all affect the quality of AI-generated outputs.
• Thoughtful engineering decisions improve performance and efficiency. Prompt caching, memory management, and model selection can significantly reduce costs and latency.
• The transition to AI engineering is an ongoing learning process. At Skai, hands-on experimentation and knowledge sharing are helping engineers build the skills needed for AI-driven development.

Anyone who has ever committed to going to the gym for a few months knows this feeling. For weeks, you show up, do the reps, and track your progress. But when you look in the mirror, it feels like nothing is changing. Then one morning, you realize your body has completely transformed. It’s almost like Tobey Maguire’s Peter Parker waking up after the spider bite in the original 2002 film, suddenly discovering he has a six-pack.

But as any athlete or engineer knows, the overnight transformation is a myth.

Here at Skai, many of us experienced something similar. We looked up from our IDEs and realized that, seemingly overnight, we had transitioned from traditional software engineers into AI engineers. Of course, it didn’t happen overnight. We spent months experimenting with different types of LLMs, while learning alongside our AI Champions and colleagues as we figured out what worked.

And now that we have these powerful capabilities in our hands, Uncle Ben’s (RIP) famous words feel particularly relevant: “With great power comes great responsibility.” Because becoming an AI engineer means more than knowing how to prompt a model. It means understanding what’s happening under the hood.

That thinking inspired an internal learning initiative I launched with my manager, Noa Weiss, called Under the Hood. Through short monthly sessions, engineers share practical AI concepts, challenge assumptions, and explore how these technologies apply to our everyday work.

In our first session, I covered eight foundational concepts that I believe every engineer working with AI should understand.

1. Data Bias: The Left-Handed Writer Paradox

Here’s a fun experiment. Ask an image generation model to create a picture of a left-handed person writing in a notebook next to a full glass of wine. You might get a right-handed person and a half-empty glass instead.

Why? AI models learn patterns from their training data. When something is less common, like a left-handed writer, the models gravitate toward the more familiar alternative. I call this distribution drag: the tendency for AI outputs to favor common patterns.

The same problem can appear in code generation. Ask a model to solve a unique edge case for your specific business logic, and it will gravitate to a familiar solution that may not quite fit.

The engineering lesson: When working with unusual requirements or edge cases, provide explicit constraints and examples rather than assuming the model will understand what makes the situation different.

2. Tokenization: The Strawberry Problem

Ask an AI model how many times the letter “r” appears in “strawberry.” Today, you’ll probably get the correct answer: three. But just a few months ago, models struggled with questions like this.

That’s because language models don’t process words the way humans do. They break text into smaller units called tokens, which never correspond to individual letters. When an AI assistant gets this question right, it may be using a code execution tool to count the letters rather than relying on language generation.

This distinction matters when building applications that require precision.

The engineering lesson: Use deterministic code for tasks like character counting, string manipulation, and exact formatting. AI can help generate the code, but it shouldn’t always be responsible for executing the logic itself.

3. Temperature: The Wheel of Fortune

Developers often describe temperature as an AI model’s “creativity setting.” That’s a useful shortcut, but here’s a better way to think about it.

Imagine a Wheel of Fortune where each wedge represents a possible next word. At lower temperatures, the model strongly favors the most likely options. At higher temperatures, less likely options have a better chance of being selected.

Here’s the catch: lower temperature doesn’t guarantee a correct answer. A model can produce a wrong answer, even when its output is highly predictable.

The engineering lesson: Use lower temperatures when consistency matters or when aiming for structured or predictable outputs, but don’t confuse consistency with no-hallucinations.  A low temperature cannot eliminate hallucinations, so the validation of the output is still a must.

4. Context Management: The Quiet Meeting Room vs. the Stadium

Imagine trying to have a productive conversation in a quiet meeting room with five people. Now imagine having that same conversation in a stadium filled with 100,000 screaming fans.

That’s the difference between giving an AI model focused, relevant context and overwhelming it with unnecessary information.

Every Transformer model operates under a fixed Attention Budget that sums mathematically to 1.0 (100%). As you stuff more tokens into the context window, the relative attention weight available for any single token shrinks, because every token demands a share of the attention.

During our own engineering work at Skai, I discovered that more than 40% of the lines in one of our codebases were Markdown documentation files. That’s a lot of potential noise for an AI coding agent to process.

When developers provide outdated documentation, temporary notes, and massive logs, important instructions can become harder for the model to identify. We’re now exploring ways to keep that context cleaner, including an agent designed to help maintain our documentation.

The engineering lesson: Treat context as a limited resource. Keep documentation current, remove unnecessary files and comments, and give AI agents the information they actually need.

5. Structured Outputs: Solving Calculus in Rhyme

Engineers love structured outputs because they make AI responses easier to integrate into applications. For example, requiring an AI model to return valid JSON helps ensure that another system can process its response.

But imagine being asked to solve a complicated calculus problem while delivering every step as perfectly rhyming poetry. Is your head spinning? Good.

Depending on the model and implementation, imposing strict formatting requirements too early can make complex tasks harder.

When imposing strict formatting from the first token, you deprive it of auto-regressively generating thought tokens (Chain of Thought) that help it reason through the problem.

The engineering lesson: For difficult reasoning tasks, test whether separating the reasoning and formatting stages improves results. One approach is to solve the problem first, then convert the answer into the required structure.

6. Prompt Caching: How One Small Change Reduced Costs

This was one of the most practical lessons from our experience at Skai.

Many AI providers offer prompt caching, which allows systems to reuse previously processed portions of a prompt instead of processing the same information repeatedly. But caching only works when the reusable content in the cache hits require character-level exact Prefix Matching.

In one of our early implementations, we placed dynamic information near the beginning of our system prompts. That small decision disrupted caching and increased processing costs.

After changing how we organized those prompts, one pull request reduced our cost per LLM request by more than 90%. Not bad for a relatively small change!

The engineering lesson: Put stable instructions and reusable information at the beginning of prompts. Keep changing inputs toward the end to improve caching opportunities.

7. AI Agent Memory: Beyond Chat History

When developers first think about giving an AI agent memory, the obvious solution is to save previous conversations and feed them back into the model. Unfortunately, that approach gets messy quickly.

Long conversation histories consume context budget, increase latency, and make it harder for an agent to identify relevant information. A better approach is to build an active memory pipeline to extract useful facts, store them efficiently, and retrieve only what’s needed for the current task.

For example, instead of storing an entire conversation, an agent might remember one useful preference: “User prefers Python over Java.” Techniques such as hybrid search (combining semantic and keyword matching), and vector deduplication can help make that memory more useful over time.

The engineering lesson: Design agent memory around active extraction and retrieval pipelines rather than raw conversation history.

8. Model Selection: Choosing the Right Tool for the Job

Not every AI task requires the biggest, most powerful model. Think about generating a conversation title versus debugging a complicated production issue. The first task is relatively simple. The second may require deeper reasoning, multiple steps, and more computing resources.

Using an expensive reasoning model for every task can increase latency and costs without delivering meaningful benefits.

It is crucial to understand the difference between standard fast-path models and “Thinking Models” (Inference-Time Compute) that allocate extra runtime tokens to self-correct before answering.

The engineering lesson: Match the model to the task. Use lightweight models for straightforward orchestration operations and reserve more advanced reasoning capabilities for complex decision nodes.

Putting Our Knowledge to the Test

Of course, we couldn’t finish our first Under the Hood session without a little friendly competition. We opened Kahoot, an interactive quiz platform where participants answer questions in real time, and challenged everyone to a 20-question competition covering what we’d learned.

The energy was incredible, and I’m sure the prize I promised the winner had absolutely nothing to do with it. More importantly, seeing engineers debate why a prompt failed or why a cache missed showed that these concepts were starting to click.

What’s Next for AI Engineering at Skai?

Becoming an AI engineer isn’t about memorizing the latest buzzwords or copying prompts from the internet. It’s about understanding why models behave the way they do, recognizing their limitations, and using that knowledge to build better systems.

Our first Under the Hood session was just the beginning. We’re continuing to explore these concepts together, with more engineers taking the stage to share what they’re learning with my tutoring in the process.

Because while the transition to AI engineering might feel like it happened overnight, the real transformation happens through continuous experimentation, collaboration, and learning. And personally, I can’t wait to see what’s under the hood next.

---

Frequently Asked Questions