On March 24, 2026, two versions of the litellm Python package were found to contain malicious code — the latest move in a coordinated, multi-ecosystem supply chain campaign by a threat actor known as TeamPCP. The group had spent the preceding weeks targeting the tools developers use to secure their code, then using that access to poison the applications those developers ship. Earlier in March, they had compromised Aqua Security’s Trivy vulnerability scanner and Checkmarx’s KICS static analysis tool using nearly identical methods. LiteLLM was the next link in the chain.
LiteLLM is downloaded roughly 3.4 million times per day. It acts as a universal proxy to over 100 large language model APIs — OpenAI, Anthropic, Google Gemini, Azure, AWS Bedrock — and is increasingly deployed as a centralized LLM gateway holding API keys for every model provider an organization uses. That combination of ubiquity and privilege made it the most consequential target in the campaign to date.
The two malicious versions were live on PyPI for approximately three hours before quarantine. At that download volume, three hours was enough to reach an estimated 500,000 machines and compromise more than 1,000 enterprise cloud environments. One of those environments belonged to Mercor, a $10 billion AI recruiting startup whose subsequent breach drew a Lapsus$ extortion campaign, a class-action lawsuit, and a suspension of work from Meta.
What We Know
The Attack Did Not Start With LiteLLM
To understand the LiteLLM compromise, you have to start five days earlier, with Trivy.
In late February 2026, a threat actor operating under the handle MegaGame10418 exploited a misconfigured pull_request_target workflow in Aqua Security’s Trivy CI pipeline. Unlike the standard pull_request trigger, pull_request_target runs with write access to base repository secrets — meaning an external contributor can invoke it against an organization’s own credentials. The attacker used this to exfiltrate the aqua-bot Personal Access Token.
Aqua disclosed the incident on March 1 and rotated credentials, but their own post-incident analysis noted the rotation “wasn’t atomic and attackers may have been privy to refreshed tokens.” That gap was decisive.
On March 19 at 17:43 UTC, TeamPCP used still-valid credentials to force-push 76 of 77 release tags in the trivy-action repository to a malicious commit. They also hit all seven tags in setup-trivy. Git tags are mutable pointers — an attacker with write access can silently repoint a tag like @0.24.0 to any commit they choose:
git tag -f 0.24.0 <malicious-commit>
git push -f origin refs/tags/0.24.0
The GitHub release page shows no change: same name, same dates, same description. Every CI/CD pipeline referencing the action by tag now pulled and executed backdoored code. CrowdStrike confirmed the malicious entrypoint.sh was 17,592 bytes versus the legitimate version’s 2,855 bytes.
On March 23, TeamPCP ran the same playbook against Checkmarx’s KICS infrastructure scanner. On March 24, they came for LiteLLM.
How LiteLLM Was Compromised
LiteLLM used Trivy as a security scanner inside its CI/CD pipeline, referenced by mutable tag rather than a pinned commit SHA. When that pipeline ran after March 19, the compromised Trivy action executed a credential theft operation before the real scan — so the workflow appeared to complete normally with clean output.
The malicious entrypoint.sh enumerated GitHub Actions runner processes, read /proc/{PID}/environ for environment variables, and filtered for JSON structures matching "name":{"value":"...","isSecret":true}. It exfiltrated the PYPI_PUBLISH token, encrypted it, and posted it to scan.aquasecurtiy[.]org — a typosquatted domain mimicking Aqua Security’s legitimate site. The real Trivy scan then ran and produced expected output.
With LiteLLM’s PyPI publishing token in hand, TeamPCP bypassed the official CI/CD workflow entirely and published two malicious versions:
- v1.82.7 at 10:39 UTC — payload embedded in
litellm/proxy/proxy_server.py, executing on any import oflitellm.proxy - v1.82.8 at 10:52 UTC — added
litellm_init.pthto site-packages, firing on every Python interpreter startup, including when pip or an IDE’s language server starts Python
Both existed only in the PyPI wheel, not in the GitHub repository. The attacker poisoned the build process rather than committing to source control — Git history reviews would have found nothing. Researchers who decoded the payload found two earlier development iterations embedded as commented-out blobs in proxy_server.py, tracing the toolset’s evolution back to at least December 2025.
The Payload
Once on a compromised system, the payload ran in three stages. It first swept for credentials across the full stack: SSH keys, .env files, shell history, CI/CD configs, AWS/GCP/Azure credential files, Kubernetes secrets and service account tokens, Docker config, and cryptocurrency wallet files. It queried AWS Secrets Manager and SSM Parameter Store directly via IMDSv2.
Collected data was encrypted, bundled as tpcp.tar.gz, and POSTed to https://models.litellm[.]cloud/ — registered one day before the attack. The same RSA public key used here appears in the Trivy and KICS payloads, the strongest technical link across all three operations. Full technical analysis of the payload is available from Trend Micro.
Finally, the payload established persistence via a backdoor at ~/.config/sysmon/sysmon.py and a systemd user service that polls https://checkmarx[.]zone/raw every five minutes for a rotating remote execution URL. On Kubernetes, it attempted to deploy a privileged pod to every node in kube-system to install the backdoor on the underlying host.
How the Community Responded — and How TeamPCP Tried to Stop Them
The compromise was discovered not by a security vendor but by Callum McMahon at FutureSearch, testing a Cursor MCP plugin that pulled in LiteLLM as a transitive dependency. His machine became unresponsive due to RAM exhaustion — an unintended side effect of the .pth mechanism creating an accidental fork bomb. McMahon published his findings and they spread to r/LocalLLaMA, r/Python, and Hacker News within the hour.
When community members filed GitHub issue #24512, TeamPCP launched a coordinated suppression campaign. In a 102-second window, 88 bot comments from 73 previously compromised developer accounts flooded the issue — 76% overlapping with the botnet used during the Trivy disclosure. Using the compromised krrishdholakia maintainer account, the attackers closed the issue as “not planned.” The community opened a parallel issue and continued on Hacker News, where the thread reached 324 points.
PyPI quarantined the packages at approximately 13:38 UTC. The LiteLLM team rotated all credentials, moved maintainer accounts to new identities, and engaged Mandiant for forensics. A clean version, v1.83.0, was released March 30 on an entirely new CI/CD pipeline.
Downstream Impact: Mercor, Lapsus$, and an Evolving Picture
The three-hour exposure window had consequences that are still developing.
On April 2, Mercor — a $10 billion AI recruiting startup used by OpenAI, Anthropic, and Meta — confirmed it was “one of thousands of companies” affected. Lapsus$ claimed to have exfiltrated 4TB of data through a compromised Tailscale VPN credential: 939GB of platform source code, a 211GB user database, and 3TB of storage including video interviews and passport scans for more than 30,000 contractors. Passwords can be reset; biometrics cannot. Meta paused work with Mercor indefinitely. A class-action complaint was filed April 1 in the Northern District of California.
Wiz researchers confirmed that Lapsus$ is now working in collaboration with TeamPCP, with TeamPCP stating on BreachForums that it intends to chain compromises into ransomware campaigns. Reports also place TeamPCP behind a breach of Cisco’s internal development environment, and CERT-EU has attributed a European Commission breach to the same campaign. The .pth technique from v1.82.8 has since been explicitly referenced in a copycat npm campaign targeting Namastex Labs packages. Zscaler’s ThreatLabz documented five major supply chain attacks in March 2026 alone, attributing the surge in part to TeamPCP’s infrastructure being reused across targets.
Several threads remain open. The full victim count is unknown — the FBI’s Cyber Division Assistant Director warned that downstream breach disclosures should be expected in the weeks ahead. The exact mechanism by which TeamPCP retained access after Aqua’s March 1 credential rotation has not been formally explained. And whether training data from Mercor’s AI lab clients was exposed, and what that means for model integrity, remains unconfirmed.
What Can the Industry Learn?
AI Popularity Has Outrun AI Security Scrutiny
LiteLLM went from a niche developer utility to critical production infrastructure for thousands of organizations in roughly eighteen months. Stripe, Netflix, Google ADK, and hundreds of enterprise teams standardized on it as their LLM routing layer because it solved a real problem quickly and the community vouched for it. That is how open source adoption works. It is also how you end up with a single Python package sitting between your applications and every AI API key you own, without anyone having formally asked: what happens if this is compromised?
The same pattern is visible across the AI stack. A small cluster of libraries (LiteLLM, LangChain, DSPy, MLflow, CrewAI) have become load-bearing infrastructure for organizations that would never have accepted comparable concentration risk in their database or authentication layers. These tools arrived with AI credibility: adopted by recognizable companies, endorsed by researchers, downloaded by the millions. That reputation functions as a substitute for security scrutiny, but also as irreresistable target for attackers.
The Mercor breach shows what concentrated AI infrastructure means in practice. Mercor’s data was not the direct target — it was reachable because a credential stolen somewhere in the blast radius gave Lapsus$ a foothold. The blast radius of a supply chain compromise is no longer bounded by what the package directly touches. It extends to every organization whose AI operations run through it.
Security Tools Are Now the Attack Surface
Every tool in this campaign — Trivy, KICS, LiteLLM — was targeted because it sits in automated pipelines with elevated trust and broad credential access. TeamPCP didn’t need a zero-day. They found a security tool running without a pinned dependency, exploited a credential rotation gap, and worked down the chain.
Mutable Tags Are an Unresolved Risk
Referencing GitHub Actions by tag — @0.24.0, @v2 — is a bet that nobody with write access to the upstream repository will repoint those tags. That is a bet on human behavior, not a cryptographic guarantee. 76 of 77 Trivy release tags were silently repointed to malicious code, and the GitHub release page showed no change. Pinning by commit SHA is not exotic — it is what actions/checkout itself recommends. The gap between knowing that and doing it across an organization’s pipeline estate is where this attack found its opening.
Hash Verification Doesn’t Catch This Class of Attack
Standard pip integrity checks — including --require-hashes — detect tampering after a package is published. They cannot detect malicious content published with legitimate credentials. The litellm_init.pth file in v1.82.8 had a valid, correct hash. The detection gap is not in hash verification; it is in the absence of behavioral inspection of what .pth files actually do at install time.
Stolen Credentials Don’t Stay With the Original Attacker
Security researchers have linked Lapsus$ — and, per Palo Alto Networks’ Unit 42, ransomware gangs CipherForce and Vect — to TeamPCP’s stolen access pool, though the exact operational relationship remains unconfirmed. The pattern that is clear: credentials harvested in a supply chain compromise don’t stay with the original attacker. They move, they get sold, and they surface weeks later against targets that had no direct exposure to the original package.
What Organizations Need to Do
Immediate Actions
The malicious versions were published on March 24. If you are reading this weeks later and have not yet checked your environment, that is not a reason to skip this section — it is a reason to take it more seriously. The persistence mechanism installed by v1.82.8 survives package removal. A backdoor planted six weeks ago and never found is still running. The credentials that left your environment in a tpcp.tar.gz bundle on March 24 may already have been sold, may already be in use, and the systems they open may have nothing to do with LiteLLM. Work through the checks below regardless of when you are reading this.
Check your installed version.
pip show litellm | grep Version
If the output shows 1.82.7 or 1.82.8, treat the system as compromised. Do not just upgrade — the payload may have already run.
Check for persistence artifacts.
ls -la ~/.config/sysmon/sysmon.py 2>/dev/null && echo "BACKDOOR FOUND"
systemctl --user status sysmon.service 2>/dev/null
ls /tmp/tpcp.tar.gz /tmp/session.key /tmp/payload.enc 2>/dev/null
Check for malicious .pth files.
find $(python3 -c "import site; print(' '.join(site.getsitepackages()))") \ -name "*.pth" -exec grep -l "base64\|subprocess\|exec" {} \;
Check for network indicators.
Look for outbound traffic to models.litellm[.]cloud, checkmarx[.]zone, or scan.aquasecurtiy[.]org (note the typo in the last one).
Check Kubernetes.
kubectl get pods -A | grep "node-setup-"
Rotate all credentials.
SSH keys, cloud access keys, API keys, database passwords, Docker registry credentials, Kubernetes tokens, and cryptocurrency wallet seed phrases. Audit AWS Secrets Manager and SSM Parameter Store — the payload queries these directly via IMDSv2.
Safe versions.
Upgrade to v1.83.0 or pin to ≤ v1.82.6. Users of the official LiteLLM Proxy Docker image (ghcr.io/berriai/litellm) were not affected.
Longer-Term Hardening
The immediate actions above address the LiteLLM incident specifically. The items below address the class of attack — because the same techniques that worked here are already being reused elsewhere, and the next campaign will not announce itself by name. These are not post-incident cleanup tasks; they are the changes that determine whether a future TeamPCP-style attack finds the same footholds or hits a wall.
Pin GitHub Actions by commit SHA.
Replace tag references like aquasecurity/[email protected] with the full commit SHA. Tags are mutable; SHAs are not. This single change would have blocked the entire Trivy phase of the campaign.
Treat CI/CD runners as production hosts.
Apply the same endpoint detection, anomaly alerting, and credential scoping to pipeline infrastructure as to production services. CrowdStrike caught this activity on affected runners — but only where that monitoring had been extended to CI/CD. Microsoft Security’s post-incident guidance covers the specific behavioral signatures to look for.
Scope pipeline secrets tightly.
A PyPI publishing token has no business in the same environment as cloud credentials. Had LiteLLM’s pipeline isolated its publishing token, TeamPCP would have gained one lever rather than many.
Inspect what gets installed, not just what’s in source.
The payload in v1.82.8 existed only in the published wheel. Source-code reviews and Git history would have found nothing. Package-level behavioral inspection — what .pth files actually do at install time — is a different discipline that requires different tooling. Cycode’s breakdown of the attack goes deeper on what that tooling gap looks like in practice.
