GitHub Copilot - AI Impact
Configuration guide for the AI Coding Assistant Impact module: GitHub Copilot data source setup, token scopes, and team scorecard mapping.
The AI Coding Assistant Impact module in Oobeya enables organizations to measure the adoption, engagement, and real-world impact of AI coding assistants, such as GitHub Copilot. By integrating with your GitHub environment, Oobeya provides visibility from organization-level usage down to team-level insights.
1. Prerequisites
Before configuring the AI Impact module:
Ensure GitHub Copilot is enabled in your GitHub organization.
Verify that you have admin rights to generate a Personal Access Token (PAT) with the required scopes.
2. Enable GitHub Copilot Integration
Navigate to Integrations → Coding Assistant in Oobeya.
Select GitHub Copilot and click "Install".

3. Configure Data Source
Go to Data Sources and add a Data Source for GitHub Copilot.

When adding GitHub Copilot as a data source:
Data Source Name: Assign a descriptive name (e.g., GitHub Copilot Org).
API Token: Provide a GitHub PAT with the following scopes:
Required Token Scopes
repo
repo:status
repo_deployment
public_repo
repo:invite
security_event
admin:org
read:org
manage_billing:copilot
read:enterprise
(required only for GH enterprise usage)
Once the token is entered, click Test Connection → Update to finalize.
4. Map GitHub Copilot to Oobeya Teams
To ensure GitHub Copilot usage data flows correctly into Oobeya’s team dashboards, you need to map GitHub organizations and teams to Oobeya teams.
Go to Insights → Teams.
Open a Level-2 team that you want to configure.
Navigate to the Scorecard tab and click the Update button in the top-right corner.
Progress through the steps until you reach the GitHub Copilot step.
Select:
Data Source: Choose your GitHub Copilot data source.
Organizations: Pick the GitHub organization(s) linked with Copilot.
Teams: Map GitHub Copilot teams to the corresponding Oobeya team.

5. Viewing Results
Once configured, the AI Impact module provides analytics from the Org level → Team level → User level:
Organization-Level Metrics
Overall adoption rate
Active vs engaged users
Copilot license utilization
Team-Level Metrics
Adoption and engagement breakdown per team
Most active teams
Teams with low or no activity
Comparison between the actual contributions and AI contributions
User-Level Metrics
Individual adoption and engagement
Suggestions accepted vs rejected
Inactive users in the last 7/30 days
6. Dashboard Components
Copilot Adoption Over Time – Track licensed, active, and engaged users.
Engagement & Acceptance Trends – Measure quality of usage (accepted vs suggested completions).
Usage by Language & IDE – Identify where Copilot delivers the most value.
Feature Usage Insights – Breakdown of completions, chat, and PR integrations.
Team & User Metrics – Detect top users, low activity users, and training needs.
Comparison Between Actual Contributions and AI Contributions
Learn more about the module:
Measuring the Impact of AI Coding Assistants7. Troubleshooting
Connection Fails?
Ensure token scopes are correct.
Verify the token is valid and not expired.
Data Missing for Some Teams?
Confirm GitHub teams are properly mapped to Oobeya teams.
Low Engagement Metrics?
Ensure developers have Copilot enabled in their IDEs.
Provide enablement sessions to improve adoption.
Last updated
Was this helpful?