BuildGuild

August 19 meeting notes

Topic: GitHub Copilot optimization, context engineering, model selection, and agent governance

• Quality-First Copilot Optimization: Kenedy explained that teams should optimize GitHub Copilot for reliable, maintainable code and developer ROI rather than minimizing token counts, using deliberate planning, frequent testing, and controlled agent usage to avoid compounding errors.
• ROI Measurement: Kenedy recommended evaluating Copilot by the developer time and effort saved, not simply by token consumption. They cited a legacy-application example in which Copilot reduced an estimated two-week reverse-engineering and repair effort to an afternoon, while noting that teams should also identify shifted bottlenecks such as code review.
• Compounding Errors: Kenedy explained that a small initial implementation error can compound across multiple agent turns until the resulting code is unusable. Developers should verify that each generated feature behaves as expected before allowing Copilot to build further on it.
• Frequent Testing: Kenedy advised testing after implementing each feature or function instead of waiting until the end of a session. Manual tests, automated tests, or Copilot-generated tests can detect buggy changes before they become the foundation for additional changes.
• Controlled Scaling: Teams should start with interactive, one-to-one work in the IDE to refine instructions and assess output quality. Once results are trusted and consistent, they can move to background agents, the CLI, or cloud agents while keeping changes small.
• Context And Conversation Management: Context engineering means supplying enough relevant information for accurate work without overwhelming the context window. Separate sessions, focused files, and explicit task boundaries help maintain context.
• Context Balance: Provide as little context as possible but as much as required. An entire codebase can make prioritization harder, while omitting one clarifying file can force incorrect assumptions.
• Context Biases: Task switches can create a lost-in-the-middle effect, while recency bias in longer conversations can displace original instructions. Use one chat per task and avoid growing beyond approximately 70% of the context window.
• Session Resetting: Start new sessions for new tasks, compact long conversations, or save useful context to a Markdown file for later reference. Input and cached-input tokens generally cost less than output tokens.
• Saved Markdown Context: Saving important conversation details into a Markdown file preserves context across sessions. Reusing the file adds mainly input tokens; narrowing references to specific files or highlighted code can improve results and reduce unnecessary usage.
• Model Selection And Token Economics: Model choice is a major cost lever. Match model capability to task complexity, using lightweight models for implementation, reasoning models for planning or complex debugging, and Auto mode when uncertain.
• Task-Based Selection: Lightweight and mid-tier models are generally sufficient for routine development, while higher-reasoning models help with architecture planning, complex multi-system debugging, synchronous work, and large modernization efforts.
• Plan Then Implement: Use a higher-reasoning model to create a detailed plan, then switch to a lightweight model to implement it directly.
• Auto Mode: Auto considers availability, outages, task complexity, and token optimization, and provides a 10% discount. Teams should still learn from the models Auto selects because capabilities and pricing change frequently.
• Cost Drivers: Output tokens are substantially more expensive than input or cached-input tokens. A low-tier model can become more expensive for a complex task if it requires many additional iterations.
• Prompt And Instruction Engineering: Precise prompts, explicit stop conditions, known-file references, concise Copilot instructions, and reusable skills or agents reduce unnecessary work and improve determinism.
• Precise Prompts: Replace vague requests such as “fix the bug” with the specific issue, observed behavior, and desired behavior.
• Stop Signals: State when Copilot must stop, such as “fix the bug and then stop” or “implement and test the feature, but do not commit or push.”
• Known Context: Attach or highlight relevant files, code sections, skills, or instructions directly instead of making Copilot spend reasoning tokens locating them.
• Copilot Instructions: Use instructions for non-negotiable coding, compliance, language, escalation, file-specific standards, and preferred response formats. Keep them concise and review them as models and projects evolve.
• Reusable Skills: Skills should provide capabilities or project-specific guidance that Copilot would not otherwise have. Avoid redundant or excessive skills and maintain them over time.
• Agent Configuration And Tool Governance: Custom agents, MCPs, scoped instructions, prompt files, subagents, and CLI tooling can organize work but also consume context and tokens, so teams should enable only what each workflow needs.
• Custom Agents: Custom agents can package models, tools, MCPs, and skills for defined workflows. Orchestrator agents can call other agents without enabling every tool for every session.
• MCP Management: MCPs can expand the context window and cause unnecessary tool use. Disable unused tools, enable only selected capabilities, or place MCPs inside dedicated custom agents.
• CLI And Subagents: The CLI is generally more token-efficient than MCPs for external services. Subagents can improve quality by dividing work but may increase spend; batch tool calls where possible.
• Preferred Formats: Markdown is generally the most agent-friendly format. PDFs and Word documents can be used, but choose formats compatible with the selected model.
• Scripted Comparisons: For comparing Excel files, ask Copilot to create and run a comparison script instead of traversing and reasoning over the spreadsheets directly.
• Architecture Guardrails: Clear project architecture helps Copilot place code correctly, reduces misses, and provides navigational guardrails that reduce corrective iterations.
• Copilot Memory And Future Enablement: Copilot memory and CLI guidance may support personalization, while customer prompts and data are not used to train the models.
• Data Protection: The language model does not store conversations for training or train on customer data, prompts, or related inputs.
• Follow-Up Materials: Share the slides, notes, recording, blog posts, and related guides after the session, and invite participants to bring their prompts, skills, agents, and customizations to a future session.

Follow-up tasks

• Meeting Materials: Send the recording and session notes to participants after Sean returns to the office. (Kenedy)
• Copilot Memory Availability: Confirm when Copilot memory reaches general availability and whether it is enabled for the participants’ organization. (Kenedy)
• Future Session Preparation: Bring existing Copilot customizations, skills, agents, and effective prompts to the next session for potential community sharing. (Participants)