Buildkite

Buildkite is a CI/CD platform. Connect a Buildkite organization to Oleria to see who belongs to it, which teams they’re in, who’s an organization administrator, and which tokens can act on the organization or register build agents. Oleria reads this data from the Buildkite REST and GraphQL APIs with an API access token, or on the Enterprise plan with short-lived tokens from OAuth Token Exchange. You can then review over-provisioned access and take action directly from Oleria. One Oleria integration covers one Buildkite organization. To connect another organization, add another integration.

What Oleria discovers

Once connected, Oleria continuously discovers and maps the following from your Buildkite organization:

Accounts - every member of the organization, with name, email, role, when they joined, and when Buildkite last saw them. Organization administrators are marked as administrators. Oleria also shows how each member can sign in: password, two-factor authentication, SSO, or an API access token they own.
- MFA - a member with two-factor authentication turned on is shown with MFA required. A member who must sign in with SSO and doesn’t have two-factor authentication gets MFA from your identity provider, so isn’t shown as needing to enroll. Otherwise, when the organization enforces two-factor authentication, members without it are shown as needing to enroll.
- SSO - shown from the member’s SSO mode, required or optional, when the organization has an enabled SSO provider.
- Machine accounts - Buildkite bot users and machine users are shown as machine accounts.
- Licenses - complimentary members are shown with a free license.
Teams - every team appears as a group, with its members. Team maintainers are shown as owners of the team. Oleria also records each team’s privacy setting, default member role, and what team members are allowed to do, such as creating pipelines or deleting packages.
Roles - the two organization roles, Organization Administrator and Organization Member, and who holds each.
Organization - the organization itself, with whether it enforces two-factor authentication and whether SSO is enabled. Oleria also records whether the organization is public, whether Teams is enabled, its API IP allowlist, and how long inactive API tokens stay valid.
Non-human identities (NHIs):
- API access tokens - every user’s API access token that can access the organization, shown as a personal access token linked to the member who owns it. Each token shows its scopes, creation date, expiry, last-used date, and the IP address it was last used from. Expired tokens are shown as disabled.
- Agent tokens - unclustered agent tokens that aren’t revoked, and whether each one is public.
- Cluster agent tokens - the agent tokens of every cluster, with the cluster name, allowed IP addresses, and expiry.
- Registry tokens - the tokens of every package registry, with the registry name.
- Portals - every portal, as one identity for its long-lived service token, with the portal’s slug, description, allowed IP addresses, whether users can invoke it, and when it was created. Buildkite runs only the portal’s saved GraphQL operations with this token, with administrator-level permissions.
When Buildkite records who created a token or portal, Oleria links it to that member.
Secrets - the Buildkite secrets in each cluster, shown as credentials with their key, description, access policy, and when they were last updated and last read. Oleria never reads secret values.
Activity - events from the Buildkite audit log, on the Enterprise plan, such as:
- members joining, leaving, or changing role, and organization invitations
- team changes, team membership changes, and changes to teams’ access to pipelines, registries, and test suites
- API access token, agent token, cluster token, and registry token creation, changes, and revocation
- secret creation, changes, and reads
- SSO provider and organization security setting changes
- two-factor authentication turned on or off, password resets, and user impersonation
Each event shows who made it, the IP address and user agent, and the API access token that was used, if any. Events made by Buildkite agents, API applications, or Buildkite itself are shown with a machine account as the actor. Events Oleria doesn’t recognize are kept as custom events.

Oleria refreshes members, teams, roles, and NHIs on its regular sync schedule.

Oleria doesn’t import pipelines, builds, clusters, registries, test suites, or their settings. Oleria reads clusters and registries only to find their tokens and secrets.

Prerequisites

A Buildkite organization. Activity requires the Enterprise plan, because Buildkite’s audit log is available only on Enterprise. OAuth Token Exchange also requires the Enterprise plan, and Buildkite must enable it for your organization.
A Buildkite organization administrator to create the API access token or the Token Exchange App. Oleria acts with the access of the token’s owner, or of the subject user you choose for OAuth Token Exchange. Buildkite shows the organization’s full list of API access tokens and its audit log only to administrators.
Admin access to the Oleria workspace.

Buildkite issues API access tokens to individual users, not to the organization, and has no service accounts. The integration acts as the member who created the token, or as the subject user for OAuth Token Exchange. If that member is removed from the organization, or the API access token expires or is revoked, discovery stops until you update the integration. If the member loses administrator access, API access tokens and activity stop appearing.

Permissions

With the API access token method, Oleria uses one token for both discovery and remediation actions, and the token’s scopes decide what Oleria can read and change. With OAuth Token Exchange, the Token Exchange App’s Grantable Scopes decide it. The scopes are the same for both methods:

Scope	Used for
`graphql`	Members and their MFA and SSO status, teams, API access tokens, unclustered agent tokens, organization security settings, and activity. Required. In Buildkite, this is the token’s GraphQL API setting: Read-only is enough for discovery.
`read_organizations`	The organization. Required.
`read_teams`	Team members. Required.
`read_clusters`	Clusters, to find cluster agent tokens and secrets. Required.
`read_registries`	Package registries, to find registry tokens. Required.
`read_secrets_details`	Optional. Cluster secrets. Without it, Oleria doesn’t sync secrets.
`read_audit_events`	Optional. Activity, on the Enterprise plan. Without it, Oleria syncs no activity.
`read_portals`	Optional. Portals. Without it, Oleria doesn’t sync portals.
`write_organizations`	Optional. Required for Disable user accounts, Delete user, Add user to role, Remove user from role, and disabling machine users.
`write_organization_invitations`	Optional. Required for Disable user accounts, Delete user, Enable user accounts, and Create user.
`write_teams`	Optional. Required for Add user to group, Remove user from group, and Create group.
`write_clusters`	Optional. Required to revoke cluster agent tokens.
`write_registries`	Optional. Required to delete registry tokens.
`write_portals`	Optional. Required to delete portals.

To revoke API access tokens and unclustered agent tokens with the API access token method, set the token’s GraphQL API setting to Full access. A Read-only token can run GraphQL queries but not the mutations Buildkite uses for revocation. With OAuth Token Exchange, Oleria requests only the read scopes for discovery, and adds an action’s write scopes only when it runs that action. Grant all three optional read scopes, read_secrets_details, read_audit_events, and read_portals. If the app doesn’t grant all three, Oleria requests only the required scopes and skips secrets, activity, and portals. If the app doesn’t grant an action’s write scopes, that action fails. When you connect, Oleria rejects a connection that’s missing a required scope and names what it couldn’t sync. A missing optional scope doesn’t block the connection.

Choose an authentication method

Method	When to use
API access token (default)	Works on every plan. A long-lived token that an organization administrator creates in their personal settings.
OAuth Token Exchange (Enterprise plan)	Enterprise plan only, once Buildkite enables Token Exchange for your organization. Oleria signs a request with your private key and gets a short-lived token for the subject user, so there’s no long-lived token to rotate. Revoking API access tokens and unclustered agent tokens isn’t expected to work with this method. See Remediation actions.

Set up access in Buildkite

First, find the organization slug: the part of your organization’s URL after buildkite.com/. For example, for https://buildkite.com/acme-inc, the slug is acme-inc.

API access token
OAuth Token Exchange

Create the API access token

Sign in to Buildkite as an organization administrator. Select Personal Settings in the global navigation -> API Access Tokens -> New API Access Token, and enter your password if prompted. Then fill in the form:

Description - enter Oleria.
Credential Type - select Token.
Organization Access - select the organization you want to connect.
Token Expiry - choose an expiry that fits your rotation policy. Buildkite defaults to seven days, and discovery stops when the token expires.
REST API Scopes - select the scopes listed in Permissions.
GraphQL API - select Read-only, or Full access if Oleria should revoke API access tokens and unclustered agent tokens.
Allowed IP Addresses - optional. If you restrict the token to certain addresses, add Oleria’s IP addresses. Contact support@oleria.com for the current list.

Copy the token

Select Create New API Access Token and copy the token’s value. Buildkite shows it only once.

Generate a key pair

Generate an RSA 2048-bit or ECDSA P-256 key pair, for example with openssl genrsa -out private_key.pem 2048 and openssl rsa -in private_key.pem -pubout -out public_key.pem. Keep the private key unencrypted in PEM format. You enter it in Oleria. Convert the public key to a JWKS, and give every key in it a kid.

Create the Token Exchange App

Sign in to Buildkite as an organization administrator. Go to Organization Settings -> Integrations -> Token Exchange Apps, then select Create a Token Exchange App, or New App if the organization already has one. Then fill in the form:

Name and Description - for example, Oleria.
JWKS - your public key in JWKS format.
Grantable Scopes - the scopes listed in Permissions, including the write scopes for the actions Oleria should run.
Default Scopes - any subset of the grantable scopes. Oleria always asks for specific scopes, so it doesn’t use the defaults.
Allowed IPs for Token Exchange Requests and Allowed IPs for Minted Token Usage - optional. If you set them, add Oleria’s IP addresses. Contact support@oleria.com for the current list.
Max Token TTL (seconds) - keep the default or choose your own. Oleria gets a new token before the current one expires.

Select Create Token Exchange App, and copy the Client ID from the app’s page.

Choose the subject user

Choose the organization administrator Oleria acts as. They must be an active member of the organization with a verified email address.

If Token Exchange Apps doesn’t appear in your organization’s settings, contact Buildkite support to enable it.

If your organization restricts API access by IP address in its API security settings, an Enterprise plan feature, add Oleria’s IP addresses there too. The Buildkite interface changes over time. If these steps don’t match what you see, refer to Buildkite’s Managing API access tokens or OAuth Token Exchange documentation.

Connect Buildkite to Oleria

Open the integration

Complete the connection form

Fill in the connection form:

Field	Notes
Organization slug	Required. The slug from your organization’s URL, `https://buildkite.com/<slug>`.
Authentication method	API access token (default) or OAuth Token Exchange (Enterprise plan).
API access token	Required for the API access token method. The API access token you created.
Client ID	Required for OAuth Token Exchange. The Token Exchange App’s Client ID.
Private key	Required for OAuth Token Exchange. The PEM private key whose public key is in the app’s JWKS.
Subject user email	Required for OAuth Token Exchange. The verified email address of the organization administrator Oleria acts as.

Authenticate

Select Authenticate. Oleria checks that the token is valid, has the required scopes, and can reach the organization before it saves the integration. With OAuth Token Exchange, Oleria first gets a token from Buildkite, so a Buildkite error, such as a plan that doesn’t include Token Exchange or a subject user who isn’t an active member, appears here. The integration is named after your Buildkite organization.

Edit the connection

You can replace the API access token of an existing integration, for example when you rotate it, or the Client ID, Private key, and Subject user email of an OAuth Token Exchange integration. You can’t change the Organization slug or the Authentication method. To connect a different organization, add a new integration.

Verify the integration

Confirm the Buildkite integration appears on the Connected tab of your Oleria Integrations page with a status of Healthy. After the first sync completes, check that members, teams, roles, and NHIs appear in your Oleria workspace, and that organization administrators are marked as administrators.

With the API access token method, when the token expires or is revoked, discovery stops. Create a new token and enter it by editing the integration. The Duplicate button on Buildkite’s API Access Tokens page creates a new token with the same permissions.

Remediation actions

Oleria runs actions with the integration’s API access token. With OAuth Token Exchange, it requests a new token with the action’s scopes for each action. Before an action runs, Oleria checks that the token has the scopes the action needs.

Action	What it does	Revert
Disable user accounts	Removes the member from the organization, because Buildkite members have no disabled state. Before the removal, Oleria records the member’s role, SSO mode, and team memberships. If the user isn’t a member but has a pending invitation, Oleria revokes the invitation instead.	Supported - invites the user again by email with their previous role, SSO mode, and teams. Not supported when Oleria couldn’t record the member’s teams, for example because the member has no email or is in more than 100 teams, or when the action revoked an invitation.
Delete user	Removes the member from the organization, like Disable user accounts, without recording their role, SSO mode, or teams. If the user isn’t a member but has a pending invitation, Oleria revokes the invitation instead.	Not supported - run Enable user accounts to invite the user again.
Enable user accounts	Invites the user to the organization by email as an Organization Member, with SSO required. If the user is still a member or already has a pending invitation, nothing changes.	Not supported - run Disable user accounts to remove the user or revoke the invitation.
Create user	Invites the user to the organization by email as an Organization Member, with SSO required. Buildkite adds members only by invitation. If the user already has a pending invitation, nothing changes.	Supported - revokes the invitation Oleria sent. Fails if the user already accepted it, or it was revoked or expired. Run Disable user accounts to remove a user who joined.
Create group	Creates a visible team. New members join as team members, and team members can’t create pipelines, test suites, or registries, or delete registries or packages.	Supported - deletes the team, but only while it has no members.
Add user to role	Makes the member an organization administrator.	Supported - restores the previous role.
Remove user from role	Changes an organization administrator to an organization member.	Supported - restores the previous role.
Add user to group	Adds the member to a team as a team member.	Supported - removes the membership, unless it no longer exists or its team role changed after the action.
Remove user from group	Removes the member from a team.	Supported - adds the member back with their previous team role, member or maintainer.
Disable NHI accounts	Revokes the token. An API access token loses its access to the organization. An unclustered agent token is revoked with the reason `Revoked by Oleria`. A cluster agent token is revoked, and a registry token is deleted. A portal is deleted, because Buildkite has no API to revoke a portal’s token. A machine user is removed from the organization.	Not supported - Buildkite can’t restore a revoked token or a deleted portal. Create a new token or portal in Buildkite, or invite the machine user again.

Each action needs these scopes:

Action	Required scopes
Disable user accounts, Delete user	`write_organizations`, `write_organization_invitations`
Enable user accounts	`read_organizations`, `write_organization_invitations`
Create user	`write_organization_invitations`
Create group	`read_teams`, `write_teams`
Add user to role, Remove user from role	`write_organizations`
Add user to group, Remove user from group	`write_teams`
Disable NHI accounts - API access token or unclustered agent token	`graphql`, with the GraphQL API setting at Full access
Disable NHI accounts - cluster agent token	`read_clusters`, `write_clusters`
Disable NHI accounts - registry token	`read_registries`, `write_registries`
Disable NHI accounts - portal	`read_portals`, `write_portals`
Disable NHI accounts - machine user	`write_organizations`

Oleria can’t tell from the token’s scopes whether its GraphQL access is Read-only or Full access. With a Read-only token, Buildkite rejects the revocation, and Oleria reports a missing permission. With OAuth Token Exchange, Buildkite blocks every GraphQL mutation it doesn’t explicitly permit for token exchange tokens, so expect revoking API access tokens and unclustered agent tokens to fail. Revoke them in Buildkite instead. Every other action is unaffected. The integration’s Capabilities tab shows Disable NHI accounts as available only when the token has graphql, write_clusters, and write_registries, even if you only revoke one kind of token. Oleria refuses these actions and sends no change to Buildkite:

Removing the member Oleria acts as, or removing their administrator role. That’s the owner of the integration’s API access token, or the subject user for OAuth Token Exchange. Removing the member would also disconnect the integration. Oleria also refuses these actions when it can’t confirm who that member is.
Revoking the API access token the integration connects with.
Granting or removing any role other than Organization Administrator. Every member already holds Organization Member.
Running Disable user accounts, Delete user, or Enable user accounts on a token. Use Disable NHI accounts instead.
Disabling official Buildkite bots, or the API applications and agents that appear in activity. Buildkite has no API to disable them.
Revoking an unclustered agent token that isn’t among the first 500 Buildkite returns. Revoke it in Buildkite instead.

Keep these points in mind:

If the target is already in the requested state, the action succeeds without making a change, and there’s nothing to revert.
Enable user accounts, Create user, and reverting Disable user accounts send an invitation. The user gets access only after accepting it. Enable user accounts and Create user set SSO to required. If the user didn’t need SSO before, change their SSO mode back to optional in Buildkite after they accept.
Buildkite won’t remove a member if that would leave the organization without an administrator, so the action fails.

Known limitations

The connection is tied to one person. See Prerequisites.
Activity requires the Enterprise plan. On other plans, or when the token lacks read_audit_events, Oleria syncs no activity, and the rest of the organization still syncs. Oleria imports at most the last 90 days of earlier activity. Events made with the integration’s own API access token are skipped, so changes Oleria makes through remediation actions don’t appear as activity. With OAuth Token Exchange, Oleria can skip only events made with its current short-lived token, so changes it makes through remediation actions appear as activity by the subject user.
Members can’t be disabled. Buildkite has no disabled state for members, so every member is shown as enabled, and Disable user accounts removes the member instead.
Pending invitations aren’t shown. Users who haven’t accepted an invitation don’t appear as accounts.
No password-change dates. Buildkite’s API doesn’t report when a member last changed their password.
SSO status needs an enabled SSO provider. Buildkite reports a member’s SSO mode as required even when the organization has no SSO provider. Until a provider is enabled, Oleria shows SSO as not configured for every member.
License types aren’t known. Buildkite’s API doesn’t say which kind of seat a member holds. Only complimentary members are identified, with a free license.
Unclustered agent tokens are capped at 500. Buildkite returns them in a single list, so tokens beyond the first 500 aren’t shown and can’t be revoked from Oleria. Revoked agent tokens aren’t shown.
Secrets are inventory only. Oleria can’t disable or rotate Buildkite secrets.
Pipelines and resource access aren’t inventoried. Oleria shows team membership, but not the pipelines, registries, or test suites that teams can access.
Build agents aren’t inventoried. Connected agents appear only as actors in activity.
Event details are kept only for access changes. Oleria keeps the full details of member, team membership, team access, visibility, and cluster permission events. For other events, such as secret and token events, it keeps only the event type, actor, and target.
Token exchange limits. Revoking API access tokens and unclustered agent tokens is expected to fail, and secrets, activity, and portals sync only when the Token Exchange App grants all three optional read scopes. See Permissions.
Large organizations sync more slowly. Buildkite limits API requests per user, so the sync shares those limits with the token owner’s other API use.

Contact us

For questions about this integration, contact us at support@oleria.com.

Overview

Generic connectors

Identity providers

HR systems

Cloud infrastructure

Applications

Ticketing

Messaging

What Oleria discovers

Prerequisites

Permissions

Choose an authentication method

Set up access in Buildkite

Connect Buildkite to Oleria

Edit the connection

Verify the integration

Remediation actions

Known limitations

Contact us

​What Oleria discovers

​Prerequisites

​Permissions

​Choose an authentication method

​Set up access in Buildkite

​Connect Buildkite to Oleria

​Edit the connection

​Verify the integration

​Remediation actions

​Known limitations

​Contact us

What Oleria discovers

Prerequisites

Permissions

Choose an authentication method

Set up access in Buildkite

Connect Buildkite to Oleria

Edit the connection

Verify the integration

Remediation actions

Known limitations

Contact us