Unfolding the Dramatic Turn of Events with the LiteLLM Compromise

LiteLLM Supply Chain Attack: What Happened and How to Respond Skip to content Free Trial Shedding The Lite: Unfolding The Dramatic Turn of Events with the LiteLLM Compromise Last updated: March 25, 2026 | 10 MIN Roni Kuznicki , Ilan Lidovski Security Developer, Software Engineer Get a Personal Demo Key Highlights What Happened: On March 24, 2026, the self-proclaimed “TeamPCP” threat actor compromised the PyPI publishing credentials for LiteLLM , a highly popular open-source library used to route requests across various LLM providers. They published two backdoored versions (1.82.7 and 1.82.8), with malicious code injected directly into the distributed wheels. When triggered, it orchestrated a three-stage attack: Harvesting credentials, attempting lateral movement across Kubernetes clusters, and installing a persistent systemd backdoor that polls for additional payloads. What’s The Blast Radius: LiteLLM is an extremely popular library, boasting an impressive 95 million monthly downloads (according to https://pypistats.org/packages/litellm). Version 1.82.6 is the most recent release known to be free of issues. What Is The Immediate Action: Check every development environment, be it local or in a CI/CD pipeline for the following installed packages: litellm:1.82.7, litellm:1.82.8, and work through the mitigation guidelines detailed below. The blast radius of the supply chain attacks that started in late February 2026 has reached PyPI. Following the intricate Github Actions supply chain attacks on Aquasec’s Trivy and Checkmarx’ `kics-github-action`, the threat actor ventures deep into a core library used as a gateway for AI models. One of the maintainers of LiteLLM notes that the breach originates from trivy used in their CI/CD pipelines. (https://news.ycombinator.com/item?id=47502858) Breach Timeline: Pre-incident (March 19–23): The threat actor group TeamPCP compromised the Trivy security scanner. LiteLLM’s CI/CD pipeline, which used Trivy, inadvertently exfiltrated the project’s PyPI publishing tokens to the attackers. 10:39 AM: The attackers use the stolen tokens to publish v1.82.7 of litellm to PyPI. This version contains a malicious payload hidden in proxy_server.py . 10:52 AM: A second malicious version, v1.82.8 , is published. It introduces a more aggressive delivery method using a .pth file ( litellm_init.pth ), which executes as soon as the Python interpreter starts. 11:48 AM: Security researcher Callum McMahon (FutureSearch) opens GitHub Issue #24512 after his development machine crashes due to a “fork bomb” side effect caused by the malicious .pth file. 12:44 PM: The attacker, still possessing maintainer access, attempts to suppress the discovery by closing the GitHub issue and flooding the thread with bot-generated comments. 1:38 PM: PyPI admins take emergency action and quarantine the entire litellm package , blocking all new downloads and removing the malicious versions. Filling the Gap: From CI/CD to the Developer What differentiates this attack from the earlier ones noted, is that instead of being executed in the CI/CD pipeline itself, it has reached into the developers’ machines and production environments. That means that the breach is no longer contained within the build process; it has successfully transitioned into a downstream execution attack. By poisoning a trusted library like LiteLLM, the threat actor shifted from attacking the factory to poisoning the product delivered to millions of users. Unfolding the Attack: A Three-Stage Payload Once a developer or a production server runs pip install litellm (grabbing versions 1.82.7 or 1.82.8), the malicious code hidden within the wheel’s distribution files executes automatically. The analysis of the backdoor reveals a sophisticated, multi-stage execution flow: Stage 1: Credential Theft The primary objective upon execution is environment discovery. The script immediately scans for environment variables and configuration files associated with cloud providers and AI services. This includes: AI Provider Keys: OpenAI, Anthropic, and Azure secrets stored in ENV variables. Cloud Metadata: AWS/GCP/Azure credentials used by the SDKs. Local Files: Extraction of .kube/config and .aws/credentials from the home directory. Stage 2: Lateral Movement (K8S Exploitation) If the infected library is running within a Kubernetes cluster, the code attempts to utilize the local service account token. It probes the internal K8s API to identify other pods, services, and secrets. The goal is to find a path to a more privileged node or a sensitive database within the cluster. Stage 3: Backdoor Persistence In this stage, the malware attempts to gain root or sudo privileges to install a persistent systemd unit. This service is designed to masquerade as a legitimate system process. Once active, it begins a “beaconing” process, reaching out to a Command & Control (C2) server controlled by TeamPCP to receive further instructions or secondary payloads. Indicators of Compromise: Category Indicator Description Affected Versions litellm: 1.82.7, 1.82.8 Malicious versions published to PyPI by TeamPCP. Malicious Files litellm_init.pth Located in site-packages/. Triggers execution on any Python startup (v1.82.8). Malicious Files litellm/proxy/proxy_server.py Contains obfuscated Base64 injection (v1.82.7). Persistence ~/.config/sysmon/sysmon.py The core backdoor script installed for local persistence. Persistence sysmon.service Systemd unit located in ~/.config/systemd/user/. Service Name System Telemetry Service The display name used by the malicious systemd unit. Exfiltration Domain models.litellm.cloud Primary C2/Exfiltration endpoint for the LiteLLM phase. C2 Domain checkmarx.zone Legacy C2 used in v1.82.7 and previous KICS attacks. Exfiltration Archive tpcp.tar.gz The temporary local filename used to bundle stolen secrets. K8s Workload node-setup-* Pods deployed to the kube-system namespace for lateral movement. K8s Images alpine:latest Used by the attacker to mount the host filesystem via privileged pods. Dead-Drop C2 github.com/tpcp-docs-* Attacker-controlled repositories used as fallback data drops. Conducting an Investigation: Beyond IoCs, general developer awareness is a powerful shortcut to mitigation. A defining characteristic of this attack is the massive RAM consumption caused by the litellm_init.pth launcher. Because it spawns a child process on every interpreter startup, it creates an exponential fork loop that can quickly freeze a machine. This is caused by a bug in the malware. If your team reports sudden, extreme system lag or memory exhaustion while working with LiteLLM, consider it a primary indicator of compromise. Recognizing this behavioral indicator early can save hours of investigation and help you roll back to version 1.82.6 immediately. Moreover, it is highly advised to check if you’ve upgraded or installed litellm on, or after the 24th of March. Run `pip show litellm` and check for the version, and also check for related uv caches. Next up, to get a wider view of the entire organization’s stance on the specific malicious pinned versions, we’ll demonstrate utilizing the Context Intelligence Graph to pin down developers in the organization that were potentially compromised. First up, we can query all the users that cloned the specific repository that was associated with the malicious dependencies: This gives us visibility to who cloned or comthe repository which is related to the pinned malicious dependency. The power and control this query gives us is immensely important, as it doesn’t matter when the user cloned the repository that contained the malicious package, even if it was remotely mitigated to another version, since as soon as they execute a command like `poetry install` the execution mechanism described earlier will start exfiltrating secrets. To pinpoint potentially compromised accounts, security teams should pivot to their logging platforms (e.g., CrowdStrike) and hunt for outbound requests to models.litellm.cloud and checkmarx.zone. This step further helps isolate which specific users were exposed to these indicators. Moreover, search for K8S pods in your environment that have been deployed with the metadata name containing “node-setup-*”, as in the malware code itself, it is assigned “node-setup-{node_name[:35]}”, and the namespace assigned is “kube-system” Mitigation Guidelines: Securing Your Environment If you’ve recently interacted with LiteLLM, speed is of the essence. Follow these steps to secure your infrastructure: 1. Immediate Version Rollback Force a downgrade to the last known safe version. Do not simply uninstall; ensure your requirements.txt or pyproject.toml is strictly pinned to the latest known safe version. pip install litellm==1.82.6 2. Audit and Purge CI/CD Pipelines: Clear all build caches in GitHub Actions, GitLab CI, or Jenkins. If you use self-hosted runners, consider them compromised and rebuild the underlying images. Local Environments: Search for the persistence unit. Check for any suspicious services in /etc/systemd/system/ or ~/.config/systemd/user/ . 3. Rotate Every Secret Assume any secret accessible to your LiteLLM environment is compromised. This is the most labor-intensive but critical step: Rotate all LLM API keys (OpenAI, Anthropic, etc.). Invalidate and reissue AWS/GCP/Azure IAM keys. Rotate Kubernetes service account tokens. 4. Network Egress Filtering The TeamPCP backdoor relies on reaching out to a C2 server. Implement strict egress filtering on your production clusters and developer VPCs. If a pod doesn’t need to talk to the open internet, it shouldn’t be able to. Where Cycode Comes In: Neutralizing Supply Chain Threats with Cimon The LiteLLM compromise proves a sobering point: traditional security perimeters are no longer enough. When a trusted library is subverted, the attack moves faster than most manual response teams can react. This is exactly why we developed Cimon , Cycode’s innovative solution for software supply chain security. Runtime Prevention and “Learning Mode” While the backdoor relies on reaching out to Command & Control (C2) servers to receive instructions, Cimon provides a critical line of defense at the execution level. By integrating into your CI/CD pipelines and build environments, Cimon monitors the network behavior of every process. Runtime Blocking: Cimon can automatically prevent access to malicious or unauthorized endpoints in real-time during the build process. Even if a poisoned version of LiteLLM is installed, its attempt to exfiltrate data would be instantly blocked. Behavioral Detection: Using its advanced Learning Mode , Cimon establishes a baseline of “normal” behavior for your builds. When a library suddenly deviates from that baseline, such as LiteLLM attempting to communicate with an unknown IP address or a host name in a different jurisdiction – Cimon flags the anomaly immediately, giving you visibility on anomalous behavior. Cimon can be easily configured to run in your CI/CD pipelines (Github, Gitlab, Harness, and more) and provide the build hardening required. Example using Github Actions: jobs: build-and-push: runs-on: ubuntu-latest steps: - uses: cycodelabs/cimon-action@v0 with: client-id: ${{ secrets.CIMON_CLIENT_ID }} secret: ${{ secrets.CIMON_SECRET }} prevent: true allowed-hosts: > objects.githubusercontent.com proxy.golang.org storage.googleapis.com allowed-ips: > 34.121.34.97 When the run finishes, the following report is generated and sent to the Cycode platform, where it can be conveniently shown in the Inventory page, or the Context Intelligence Graph itself. Followed by the data of the network connections attempted by the build itself: These build reports are also sent to the Cycode platform for a centralized location to query for finding in the graph: Continuous Visibility and Threat Intelligence with Context Intelligence Graph Securing the present is only half the battle, and visibility on the state of existing code is critical. Cycode runs constant Threat Intelligence feeds that are updated moments after a breach like the LiteLLM incident is confirmed. Our Context Intelligence Graph allows queries throughout your entire codebase and dependency tree for the specific pinned malicious versions (1.82.7 and 1.82.8). Instead of manually auditing hundreds of repositories, Cycode gives you a “single pane of glass” view of exactly where the risk lives, who introduced it, and its impact across the organization. In an era where the tools we use to secure our code are being turned against us, Cimon and the Cycode platform provide the visibility and automated enforcement needed to restore trust in the software supply chain. Originally published: March 25, 2026 Listen to the Blog Post 00:00 / 00:00 Breach Timeline: Filling the Gap: From CI/CD to the Developer Unfolding the Attack: A Three-Stage Payload Indicators of Compromise: Conducting an Investigation: Mitigation Guidelines: Securing Your Environment Where Cycode Comes In: Neutralizing Supply Chain Threats with Cimon related use cases CI/CD Security & Source Control SAST - Static Application Security Testing ASPM - Application Security Posture Management RELATED CONTENT Agentic Appsec Has Arrived AI Governance: From Visibility to Enforcement Across the Developer Surface Introducing AI Security: A Dedicated Violation Category for AI Risk in Application Security Start Securing the 10x Developer Today Discover the power of Cycode for your team. Get a Demo Platform SAST – Static Application Security Testing Next-Gen SCA – Software Composition Analysis Secrets Scanning ASPM – Application Security Posture Management Source Code Leakage Detection Source Control & CI/CD Security Infrastructure as Code (IaC) Security Container Security Scanning Cycode AI – Achieve the Impossible ASPM Marketplace – Connectors & Integrations Application Security Testing (AST) ConnectorX – Ingest & understand your security posture Application Security Platform for the AI Era Code Scanning Software Cimon – Build Hardening and Artifact Integrity Resource center ASPM Book State of ASPM 2025 Blog AppSec Accelerated Solution Briefs Analyst Research AppSec Best Practices Cygives ASPM – Guide ASPM University Integrations COMPANY About Us Customers Partners Press & Media Security & Trust Events Careers HIRING Contact Us COMPARE Veracode Snyk GitHub Advanced Security Checkmarx legal Terms Of Use Privacy Policy Cookie Policy Status Page Sitemap