Python Dependency Manager Companion

If you want to use S3 object storage to fetch your packages, you can use the `s3://` protocol as a channel. ```toml title="pixi.toml" [workspace] # ... channels = ["s3://my-bucket/custom-channel"] ``` In the bucket, your objects need to adhere to the standard conda repository structure: ``` my-bucket/ custom-channel/ noarch/ repodata.json ... linux-64/ repodata.json ... ``` Pixi supports two ways to configure access to your S3 bucket: 1. Using AWS credentials from environment variables or AWS configuration files, like in any other AWS tool 2. Using pixi's configuration in combination with storing the credentials in pixi's authentication storage !!! warning "" These two options are mutually exclusive! Specifying `s3-options` (see below) will deactivate the AWS credentials fetching. You can either use the AWS credentials from the conventional locations (by not specifying `s3-options`) or from pixi's authentication storage (by specifying `s3-options`). ## Using AWS configuration You can specify `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` in your environment variables for Pixi to use them. You can also specify `AWS_CONFIG_FILE` and `AWS_PROFILE` to use a custom AWS configuration file and profile. ```cfg title="aws.config" [profile conda] sso_account_id = 123456789012 sso_role_name = PowerUserAccess sso_start_url = https://my-company.awsapps.com/start sso_region = eu-central-1 region = eu-central-1 output = json ``` ```bash title="CLI usage" $ export AWS_CONFIG_FILE=/path/to/aws.config $ export AWS_PROFILE=conda $ aws sso login Attempting to automatically open the SSO authorization page in your default browser. If the browser does not open or you wish to use a different device to authorize this request, open the following URL: https://my-company.awsapps.com/start/#/device Then enter the code: DTBC-WFXC Successfully logged into Start URL: https://my-company.awsapps.com/start $ pixi search -c s3://my-s3-bucket/channel my-private-package # ... ``` ```yaml title="ci.yml" jobs: ci: runs-on: ubuntu-latest steps: - name: Check out repository uses: actions/checkout@v4 # temporary credentials via OIDC - name: Log in to AWS uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::123456789012:role/github-poweruser aws-region: eu-central-1 - name: Set up pixi # AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are set by aws-actions/configure-aws-credentials uses: prefix-dev/setup-pixi@v0.8.3 ``` ## Using Pixi's Configuration You can specify the `workspace.s3-options` in your `pixi.toml` file. This might be useful when you want to use a custom S3-compatible host and not AWS's configuration. ```toml title="pixi.toml" [workspace.s3-options.my-bucket] endpoint-url = "https://my-s3-host" region = "us-east-1" force-path-style = false ``` !!!note "" You need to configure this per bucket you use, i.e. use `[workspace.s3-options.<bucket-name>]`. ```bash $ pixi auth login --aws-access-key-id=... --aws-secret-access-key=... s3://my-s3-bucket Authenticating with s3://my-s3-bucket $ pixi search my-private-package # ... ``` You can also specify the `s3-options` in your [Pixi configuration](../reference/pixi_configuration.md). ```toml title="Global configuration" [s3-options.my-bucket] endpoint-url = "https://my-s3-host" region = "us-east-1" force-path-style = false ``` ```yaml title="ci.yml" jobs: ci: runs-on: ubuntu-latest steps: - name: Check out repository uses: actions/checkout@v4 # temporary credentials via OIDC - name: Log in to AWS uses: aws-actions/configure-aws-credentials@v4 id: aws with: role-to-assume: arn:aws:iam::123456789012:role/github-poweruser aws-region: eu-central-1 - name: Set up pixi # AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are set by aws-actions/configure-aws-credentials uses: prefix-dev/setup-pixi@v0.8.3 with: auth-s3-access-key-id: ${{ steps.aws.outputs.aws-access-key-id }} auth-s3-secret-access-key: ${{ steps.aws.outputs.aws-secret-access-key }} auth-s3-session-token: ${{ steps.aws.outputs.aws-session-token }} auth-host: s3://my-s3-bucket ``` ## Public S3 Buckets Public buckets that don't need authentication can be used by just specifying the endpoint as a regular `https` URL. For example, on AWS, you might have a bucket that is publicly accessible via `https://my-public-bucket.s3.eu-central-1.amazonaws.com`. ```toml title="pixi.toml" [workspace] channels = ["https://my-public-bucket.s3.eu-central-1.amazonaws.com/channel"] ``` Note that for this, you need to configure your S3 bucket in such a way that it allows public access. On AWS, you need the `GetObject` and `ListBucket` permissions for this. Here is an example policy for AWS S3: ```json title="Bucket policy" { "Version": "2012-10-17", "Statement": [ { "Sid": "PublicReadGetObject", "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::my-public-bucket/*" }, { "Sid": "PublicReadListBucket", "Effect": "Allow", "Principal": "*", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::my-public-bucket" } ] } ``` Cloudflare R2 also supports public buckets through a Cloudflare-managed `r2.dev` subdomain or a custom domain under your control, see [here](https://developers.cloudflare.com/r2/buckets/public-buckets/). ## S3-Compatible Storage Many other cloud providers offer S3-compatible storage APIs. You can use them with Pixi by specifying the `s3-options` in your manifest file. ### MinIO ```toml title="pixi.toml" endpoint-url = "https://minio.example.com" region = "us-east-1" force-path-style = true ``` ### Cloudflare R2 ```toml title="pixi.toml" endpoint-url = "https://<account-id>.eu.r2.cloudflarestorage.com" region = "WEUR" force-path-style = false ``` ### Wasabi ```toml title="pixi.toml" endpoint-url = "https://s3.de-1.wasabisys.com" region = "de-1" force-path-style = false ``` ### Backblaze B2 ```toml title="pixi.toml" endpoint-url = "https://s3.us-west-004.backblazeb2.com" region = "us-west-004" force-path-style = true ``` ### Google Cloud Storage Note Pixi also supports `gcs://` URLs. ```toml title="pixi.toml" endpoint-url = "https://storage.googleapis.com" region = "us-east-1" force-path-style = false ``` ### Hetzner Object Storage ```toml title="pixi.toml" endpoint-url = "https://fsn1.your-objectstorage.com" region = "US" force-path-style = false ``` ## Uploading to S3 You can upload to S3 using `rattler-build upload s3`. For more information, see [rattler-build's documentation](https://rattler.build/latest/authentication_and_upload/#s3). ### Re-indexing S3 buckets After Uploading new Packages Every time you upload new packages to your package repository, the `repodata.json` file needs to be updated. This is done automatically for conda package servers like anaconda.org or prefix.dev. For S3 buckets, on the other hand, we need to do this manually since an S3 bucket is only a storage system and not a package server. To re-index an S3 bucket, you can use the `rattler-index` package which is available on [conda-forge](https://prefix.dev/channels/conda-forge/packages/rattler-index). ```shell pixi exec rattler-index s3 s3://my-s3-bucket/my-channel \ --endpoint-url https://my-s3-host \ --region us-east-1 \ --force-path-style \ --access-key-id <access-key-id> \ --secret-access-key <secret-access-key> ```