How I built a bulletproof CI/CD for my LLM Python library

Source: Dev.to | Dec 24, 2025 (UTC)

When building an open-source library that integrates with multiple LLM providers (OpenAI, Anthropic, Google), reliability matters. Users expect upgrades to be safe and predictable.

This post describes the CI/CD setup I use for llm-api-adapter. The key idea is simple: test not only the code, but the actual published package.

The Strategy: two pipelines, three stages

I use a dual-pipeline setup aligned with GitHub Flow:

• Dev pipeline — runs on every push to dev. Its job is early feedback and validating the distribution process.
• Main pipeline — runs on main and version tags. Its job is stable, repeatable releases to PyPI.

Most CI setups stop at unit or integration tests. This one goes further by validating the artifact installed from TestPyPI.

1. Dev pipeline: pre-flight validation

The dev workflow is where most of the safety guarantees come from.

Stage A: Unit & Integration tests

• Executed with pytest
• Tests are separated via markers (unit, integration)
• Fast feedback on logic and provider integration

Stage B: publish to TestPyPI

After tests pass, the package is built and published to TestPyPI.

This step catches issues that tests alone cannot:

• Incorrect pyproject.toml
• Missing files in the source distribution
• Broken dependency declarations

Stage C: E2E tests from TestPyPI

This is the critical part of the pipeline.

The job:

1. Waits for TestPyPI to index the new release
2. Installs the package from TestPyPI, not from source
3. Pulls dependencies from the real PyPI
4. Runs real end-to-end tests using live API keys

pip install --index-url https://test.pypi.org/simple/ \ --extra-index-url https://pypi.org/simple \ llm-api-adapter

Enter fullscreen mode

Exit fullscreen mode