IBM TechXchange Korean CyberSecurity User Group (한국 사이버보안 사용자 그룹)

이 보고서의 세부 내용은 X-Force 레드팀 백서에서 자세히 보실 수 있습니다. 이 자료는 2022년 USA 블랙햇에서도 공개되어집니다. 
For full details on this research, see the X-Force Red whitepaper “Controlling the Source: Abusing Source Code Management Systems”. This material is also being presented at Black Hat USA 2022.

SCM (소스코드 관리 시스템) 은 엑티브 디렉토리와 같이 기업의 다른 중요한 시스템과 비교하였을 때 보안의 관점에서는 부차적인 시스템으로 여겨지지만, 기업에서 중요한 역할을 합니다. SCM 시스템은 많은 기업들이 소스를 관리하기 위해 많이 사용을 하고 있으며, 젠키스와 같은 CI/CD 데브옵스 파이프라인의 한 부분으로써 기업내의 다른 시스템들과 통합을 할 때 사용이 되어집니다. 이러한 SCM 시스템은 해커들이 소프트웨어 공급망을 공격을 할 수 있는 기회를 제공하고 있어서,이 시스템을 통해 기업 전체에 레터럴 내부이동을 하고, 권한을 상위로 올리곤 합니다. 

이 글은 SCM 시스템의 배경에 대해서 살펴보고 기업용 깃허브, 기업용 깃랩 그리고 빗버켓과 같이 인기있는 몇몇 SCM시스템들이 잘못 남용되어 사용되어지는 방법에 대해서 자세하게 다룰 예정입니다. 이러한 공격시나리오에는 정찰, 유저의 역할 조작, 저장소 탈취, 데브옵스 시스템 피봇공격, 유저행동 모방, 지속적인 접근권한을 유지하는 공격내용을 포함하고 있습니다. X-Force 레드팀의 소스코드관리 공격툴킷은 이러한 공격들이 어떻게 사용이 되는지에 대해서 보여주고 있으며, 추가적으로 이 SCM 시스템을 보호하기 위한 방어 가이드에 대한 대략적인 가이드를 제공 할 예정입니다. 

Source Code Management (SCM) systems play a vital role within organizations and have been an afterthought in terms of defenses compared to other critical enterprise systems such as Active Directory. SCM systems are used in the majority of organizations to manage source code and integrate with other systems within the enterprise as part of the DevOps pipeline, such as CI/CD systems like Jenkins. These SCM systems provide attackers with opportunities for software supply chain attacks and can facilitate lateral movement and privilege escalation throughout an organization.

This blog post will review a background on SCM systems, along with detailing ways to abuse some of the most popular SCM systems including GitHub Enterprise, GitLab Enterprise and Bitbucket to perform various attack scenarios. These attack scenarios include reconnaissance, manipulation of user roles, repository takeover, pivoting to other DevOps systems, user impersonation, and maintaining persistent access. X-Force Red’s source code management attack toolkit (SCMKit) will also be shown to perform and facilitate these attacks. Additionally, defensive guidance for protecting these SCM systems will be outlined.

배경 / Background

소스코드를 컴파일을 하면서, 소스코드를 추적하고 인터페이스를 하는 많은 방법들이 있습니다. 이 프로세스에서 사용되는 일반적으로 소스관리, 버전관리, 소스코드관리 용어들이 있습니다. '소스관리' 와 '버전관리' 용어들은 종종 유사한 의미로 혼용 되기도 합니다. 그러나 이 두개의 용어의 차이점은 소스관리는 소스코드의 변경사항을 추적하는데에 구체적으로 이야기하는 반면, 버전관리는 이진파일과 다른파일 들의 변동사항도 추적하는 것을 이야기 하곤합니다. 한가지 예를 들어보자면, 버전관리는 쌓아져 있는 실행파일들의 변화를 추적합니다. 반면에 소스관리는 C#의 열거형 타입과 실행가능한 컴파일된 C++파일의 변화를 추적할 수 있습니다. Git은 소스관리 도구로 인기가 있고,  Subversion은 버전관리에서 많이 사용이 됩니다. 

개발과정 일부에서, 실용적으로 소스관리를 하기위해서 소스코드관리 즉 SCM 또한 사용이 되어집니다. 이 SCM 시스템은 소스코드의 저장소에서 일어나는 변화를 추적하며, 또한 다양한 사람들이 동시에 섞인 소스코드를 서버에 올릴 때 일어나는 충돌사항을 해결할 수 있도록 도와줍니다. 이러한 SCM시스템들은  개발과정중 이러한 소스통합에 있어서 신뢰할만 하기 때문에 기업에서 이미 많이 도입을 해 왔습니다. 몇몇 인기있는 SCM 시스템인 기업용 깃허브, 기업용 깃랩, 빗버켓들이 이 방면에서 인기를 얻고 있습니다.  이렇게 SCM 시스템은 기업내에서 중요한 역할을 하고 있기 때문에, 오히려 잘못되어서 사용이 되어 질 수 있으며, 기업의 레터럴 내부이동과 소프트웨어 공급망 공격에 이용되어 질 수 있습니다.


There are many ways to interact with and track source code, along with compiled source code assets. Some of the common terms used in this process are source control, version control, and source code management. The terms “source control” and “version control” are often used interchangeably with each other. However, there are differences between these two terms. Source control is specifically for tracking changes in source code, whereas version control also includes tracking changes for binary files and other file types. An example of this would be version control tracking changes to compiled executables, whereas source control would be tracking the changes to the underlying C# or C++ source files that were compiled into that executable. Git is a popular source control tool, and Subversion is a popular version control tool.

To use source control in a practical manner as part of the development process, source code management (SCM) systems are used. These systems allow tracking changes to source code repositories and allow developers to resolve conflicts when merging code commits from multiple people concurrently. Some of these SCM systems are more popular than others and have been adopted by enterprises, as they integrate into the development process in a more reliable manner. Some of the popular SCM systems include GitHub Enterprise, GitLab Enterprise and Bitbucket. Due to the role and importance of these SCM systems, they can be abused to facilitate software supply chain attacks and lateral movement within an organization.

SCM 시스템과 데브옵스 파이프라인/ SCM Systems and the DevOps Pipeline

SCM systems are heavily used during the “build” phase of a project in the DevOps pipeline, as shown in the below diagram. All other phases depend on the source code that is developed and maintained within the SCM system. Once a source code project is ready to be compiled and built, it will get pushed to a Continuous Integration (CI) server. After that, it will be tested, scanned, and deployed for use in production.

DevOps Pipeline Diagram

소프트웨어 공급망 공격 / Software Supply Chain Attacks

An attack that has been gaining popularity recently is software supply chain attacks. In this attack, an attacker injects itself into the development process at one of the phases to deploy malicious code into production. This is typically performed in the “build” phase. For organizations that provide software to other organizations, this can enable the compromise of multiple organizations. One of the most notable software supply chain attacks was the SolarWinds breach, which impacted many organizations in the private and public sector.

The below diagram shows the opportunities an attacker has during the development process to implement a software supply chain attack. This research focuses on the highlighted areas of “B” and “C”, as it relates to the compromise of SCM systems. However, the compromise of these SCM systems can also lead to other scenarios such as “D” where an attacker can use an SCM system to compromise a build platform system.

Software Supply Chain Attack Opportunity Diagram

다른 데브옵스 시스템으로의 레터럴 내부이동  / Lateral Movement to Other DevOps Systems

SCM systems can be used as an initial access point to other DevOps systems that are used in separate phases of the DevOps lifecycle. Being able to pivot to the build system to compromise the CI/CD platform or pivoting to the package repository system to compromise the distribution platform are other scenarios where an attacker could perform a software supply chain attack. Two detailed examples of this are shown in the whitepaper.

공격 시나리오 / Attack Scenarios

The below attack scenarios are notable for an attacker to attempt against GitHub Enterprise, GitLab Enterprise, and Bitbucket. These have been useful as a part of X-Force Red’s Adversary Simulation engagements. This is not an exhaustive list of every single attack path available to execute on these SCM systems. The table, below, lists the attack scenarios that are detailed in the whitepaper. A few of these scenarios will be shown in subsequent sections.

Table of SCM attack scenarios

저장소 탈취 / Repository Takeover

Using administrative access (specifically site admin role), an attacker can give themselves write access to any repository within GitHub Enterprise. In the below example, we are attempting to view a repository in GitHub Enterprise that our compromised administrative user (adumbledore) does not have access to.

Viewing locked repository

Using the site admin role, you can choose to unlock the repository via the “Unlock” button shown below. This will unlock the repository for the user for two hours by default.

Viewing screen to unlock repository

You must provide a reason to unlock the repository, and this reason is logged along with the request.

Adding reason to unlocking repository

Now you can see we have successfully unlocked the repository, and it is unlocked for two hours for the adumbledore user account. Code within this repository can then be accessed or modified.

Showing repository has been unlocked

사용자 행위 모사 / User Impersonation

There are two options an attacker has if they have administrative access to GitLab Enterprise or GitHub Enterprise and would like to impersonate another user. The first option is to impersonate a user login via the web interface, and the second option is to create an impersonation token. This will be shown specifically with GitLab Enterprise.

Impersonate User Login

When viewing a user via the admin area in GitLab Enterprise, there is a button available in the top right-hand corner labeled “Impersonate”.

Impersonate user button in hpotter profile

After clicking the “Impersonate” button, you will be logged in as the user you are wanting to impersonate. In this instance, we are impersonating the hpotter user account.

Showing impersonation of hpotter

Impersonation Token

An attacker with admin access to GitLab Enterprise can also impersonate another user by creating an impersonation token. This can be performed via the web interface or the Users REST API. Using the web interface as an admin, you can navigate to the “Impersonation Tokens” section for the user account that you would like to impersonate; add the details for your token including name, expiration date, and scope of permissions.

Creating impersonation token

After you have created your impersonation token, the token value will be listed for use. The user that is impersonated cannot see this impersonation token when accessing GitLab Enterprise as themselves; it is only visible to other admin users.

Showing created impersonation token

An attacker can also create an impersonation token via the Users REST API as shown with the example curl command, below.

curl -k --request POST --header "PRIVATE-TOKEN: apiToken" --data "name=someName-impersonate" --data "expires_at=" --data "scopes[]=api" --data "scopes[]=read_user" --data "scopes[]=read_repository" --data "scopes[]=write_repository" --data "scopes[]=sudo" "https://gitlabHost/api/v4/users/userIDNumberToImpersonate/impersonation_tokens"

Output after creating impersonation token via API

CI/CD 파이프라인 수정하기  / Modifying CI/CD Pipeline

In Bitbucket, there is a feature called Bamboo that can be installed and configured to facilitate a CI/CD pipeline. If a repository is using a CI/CD pipeline with Bamboo, it will contain a directory named “bamboo-specs” within the root of the repository, along with a Bamboo configuration file. This configuration file will either be a YAML file (bamboo.yaml) or a Java spec file (pom.xml). If an attacker would like to discover any repositories that are configured with a CI/CD pipeline via Bamboo, they can search for “bamboo-specs” in either the web interface or REST API.

Discovering repos with CI/CD integration via Bamboo

If you have write access or admin access to a repository, the Bamboo configuration file can be modified. In this case, we are modifying the bamboo.yaml file to add our SSH key to the server where the Bamboo agent is running. This can be performed via the Git command line tool as well to commit the changes to the Bamboo configuration file.

Modifying Bamboo yaml file

This will immediately trigger the CI/CD pipeline to run as shown below.

Showing successful job status

When viewing the output from the pipeline, we can see our SSH key was added, and it printed the hostname of the server where the SSH key was added. At this point, we would be able to perform lateral movement to that server via our SSH key.

Viewing pipeline logs

지속적 접근권한을 유지하기 / Maintain Persistent Access

An attacker has three primary options in terms of maintaining persistent access to SCM systems. This can be performed either by creating a personal access token, impersonation token, or adding a public SSH key. All these options are detailed in the whitepaper.

The persistence option that will be shown is creating a personal access token in GitLab Enterprise. This can be performed via the web interface as a regular user or can be performed via the Users REST API as an administrator. The below screenshot shows creating a personal access token called “persistence-token”.

Creating personal access token for hpotter user

You can see the created personal access token and the token value, below.

Showing token value created

An attacker can also create a personal access token via the Users REST API as shown with the below example curl command. In GitLab Enterprise, this requires admin permissions.

curl -k --request POST --header "PRIVATE-TOKEN: apiToken" --data "name=hgranger-persistence-token" --data "expires_at=" --data "scopes[]=api" --data "scopes[]=read_repository" --data "scopes[]=write_repository" "https://gitlabHost/api/v4/users/UserIDNumber/personal_access_tokens"

Creating access token via API

소프트웨어 코드관리 공격 툴킷 / SCMKit

Background

At X-Force Red, we wanted to take advantage of the REST API functionality available in the most common SCM systems seen during engagements and add the most useful functionality in a proof-of-concept tool called SCMKit. The goal of this tool is to provide awareness of the abuse of SCM systems, and to encourage the detection of attack techniques against SCM systems. SCMKit will be shown at Black Hat USA 2022 Arsenal.

SCMKit allows the user to specify the SCM system and attack module to use, along with specifying valid credentials (username/password or API key) to the respective SCM system. Currently, the SCM systems that SCMKit supports are GitHub Enterprise, GitLab Enterprise and Bitbucket Server. The attack modules supported include reconnaissance, privilege escalation and persistence. Other functionality that are available in the non-public version of SCMKit were not included in consideration for defenders, such as user impersonation and built-in credential searching. SCMKit was built in a modular approach so that new modules and SCM systems can be added in the future by the information security community. The tool and full documentation are available on the X-Force Red GitHub. A few examples will be shown in the next sections.

정찰 / Reconnaissance

SCMKit has multiple modules available to perform reconnaissance of repositories, files, code, and other resources specific to various SCM systems such as GitLab Runners. The below example shows using the “codesearch” module in SCMKit. In this scenario, we are searching for any code in Bitbucket Server that contains “API_KEY” to try and discover API key secrets within source code.

Code search example for API key with SCMKit

There are several other reconnaissance modules that apply only to certain SCM systems. For example, there is a reconnaissance module to discover GitLab Runners that you have access to via the “runnerlist” module.

GitLab Runner reconnaissance example with SCMKit

Privilege Escalation

Another capability available in SCMKit is to add another regular user to the admin role. The example, below, shows adding a regular user under our control (hgranger, in this case) to the site admin role in GitHub Enterprise via the “addadmin” module.

Adding site admin example via SCMKit

You can see the change that took effect in GitHub Enterprise after performing the site admin addition via SCMKit, as the hgranger user is now a member of the site admins group.

Showing hgranger added as site admin

끈질긴 공격 / Persistence

There are two persistence modules within SCMKit that include the use of personal access tokens or SSH keys. This can be useful to maintain access to an SCM system. The below example shows creating an SSH key for the hgranger user account in Bitbucket via the “createsshkey” module.

Creating SSH key example with SCMKit

We can list all active SSH keys for a given user via the “listsshkey” module, as shown below. You will see the SSH key we added for the hgranger user listed.

Listing SSH keys example with SCMKit

방어시 고려사항 Defensive Considerations

소스코드 관리 공격툴킷 / SCMKit

There are multiple static signatures that can be used to detect the usage of SCMKit. These can be found in the Yara rule on the SCMKit repository.

A static user agent string is used when attempting each module in SCMKit. The user agent string is “SCMKIT-5dc493ada400c79dd318abbe770dac7c”. A Snort rule is provided on the SCMKit repository.

Additionally, any access tokens or SSH keys that are created in SCM systems using SCMKit will be prepended with “SCMKit-” in the name. This can be filtered in the respective SCM system to indicate an access token or SSH key was created using SCMKit.

소스코드 관리 시스템 / SCM Systems

Ensure that the below logs are being sent to your SIEM. This also lists the location of the logs on the server for each respective SCM system.

Logs of interest – GitHub Enterprise

Logs of interest – GitLab Enterprise

Logs of interest – Bitbucket Server

Below are the various filters you can apply to the logs to detect the attacks demonstrated in this research. Use these filters to build a baseline and detect anomalous activity in your environment.

Table of search queries for various attack types – GitHub Enterprise

Table of search queries for various attack types – GitLab Enterprise

Table of search queries for various attack types – Bitbucket Server

Additionally, the below items should be considered when configuring these SCM systems:

• Disable user impersonation
• Do not allow users to create personal access tokens or SSH keys with no expiration date
• Set automatic expiration date on all personal access tokens and SSH keys created/added
• Limit the number of administrative users. At minimum there should be two, and should not be more unless necessary
• Operate on a policy of least privilege in terms of access to repositories
• Require signed commits via GPG keys or S/MIME certificates
• Enable multi-factor authentication
• Ensure that code branches are deleted in a timely manner
• Require at least one approver for each code commit
• Increase logging level to detect reconnaissance where applicable

Conclusion

Source code management systems contain some of the most sensitive information in organizations and are a key component in the DevOps lifecycle. Depending on the role of an organization, compromise of these systems can lead to the compromise of other organizations. These systems are a high value to an attacker, and need more visibility from the information security community, as they are currently an afterthought compared to other systems such as Active Directory. It is X-Force Red’s goal that this research will bring more attention and inspire future research on defending these critical enterprise systems.

Acknowledgments

A special thank you to the below people for giving feedback on this research and providing whitepaper content review.

• Chris Thompson (@retBandit)
• Daniel Crowley (@dan_crowley)
• Dimitry Snezhkov (@Op_nomad)
• Patrick Fussell (@capt_red_beardz)
• Ruben Boonen (@FuzzySec)