Error: Failed to download metadata for repo 'amazonlinux': Cannot prepare internal mirrorlist: No URLs in mirrorlist

Question

It appears that my AL2023 EC2 instance DNF broke somehow (us-east-1, t3.micro). It's been a while since applied upgrades but now `sudo dnf makecache` results in an error:
```
] sudo dnf makecache
Amazon Linux 2023 repository                                                                                                                                                                                                                            4.7 kB/s | 160  B     00:00
Error: Failed to download metadata for repo 'amazonlinux': Cannot prepare internal mirrorlist: No URLs in mirrorlist
Amazon Linux 2023 Kernel Livepatch repository                                                                                                                                                                                                           5.0 kB/s | 172  B     00:00
Error: Failed to download metadata for repo 'kernel-livepatch': Cannot prepare internal mirrorlist: No URLs in mirrorlist
Ignoring repositories: amazonlinux, kernel-livepatch
Metadata cache created.
```
The same error is shown when running ` sudo dnf repolist enabled --verbose`, with `--verbose` option present. Without verbose it shows two repos enabled: `amazonlinux` and `kernel-livepatch`.
Command `sudo dnf repolist all --verbose` also shows the error, and lists all repos as disabled:
```
sudo dnf repolist all --verbose
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, release-notification, repoclosure, repodiff, repograph, repomanage, reposync, supportinfo
DNF version: 4.14.0
cachedir: /var/cache/dnf
Amazon Linux 2023 repository                                                                                                                                                                                                                            4.7 kB/s | 160  B     00:00
Error: Failed to download metadata for repo 'amazonlinux': Cannot prepare internal mirrorlist: No URLs in mirrorlist
Amazon Linux 2023 Kernel Livepatch repository                                                                                                                                                                                                           4.9 kB/s | 172  B     00:00
Error: Failed to download metadata for repo 'kernel-livepatch': Cannot prepare internal mirrorlist: No URLs in mirrorlist
Ignoring repositories: amazonlinux, kernel-livepatch
Repo-id            : amazonlinux
Repo-name          : Amazon Linux 2023 repository
Repo-status        : disabled
Repo-mirrors       : https://al2023-repos-us-east-1-de612dc2.s3.dualstack.us-east-1.amazonaws.com/core/mirrors/2023.3.20240108/x86_64/mirror.list
Repo-expire        : 172800 second(s) (last: unknown)
Repo-filename      : /etc/yum.repos.d/amazonlinux.repo

Repo-id            : amazonlinux-debuginfo
Repo-name          : Amazon Linux 2023 repository - Debug
Repo-status        : disabled
Repo-mirrors       : https://al2023-repos-us-east-1-de612dc2.s3.dualstack.us-east-1.amazonaws.com/core/mirrors/2023.3.20240108/debuginfo/x86_64/mirror.list
Repo-expire        : 21600 second(s) (last: unknown)
Repo-filename      : /etc/yum.repos.d/amazonlinux.repo

Repo-id            : amazonlinux-source
Repo-name          : Amazon Linux 2023 repository - Source packages
Repo-status        : disabled
Repo-mirrors       : https://al2023-repos-us-east-1-de612dc2.s3.dualstack.us-east-1.amazonaws.com/core/mirrors/2023.3.20240108/SRPMS/mirror.list
Repo-expire        : 21600 second(s) (last: unknown)
Repo-filename      : /etc/yum.repos.d/amazonlinux.repo

Repo-id            : kernel-livepatch
Repo-name          : Amazon Linux 2023 Kernel Livepatch repository
Repo-status        : disabled
Repo-mirrors       : https://al2023-repos-us-east-1-de612dc2.s3.dualstack.us-east-1.amazonaws.com/kernel-livepatch/mirrors/al2023/x86_64/mirror.list
Repo-expire        : 172800 second(s) (last: unknown)
Repo-filename      : /etc/yum.repos.d/kernel-livepatch.repo

Repo-id            : kernel-livepatch-source
Repo-name          : Amazon Linux 2023 Kernel Livepatch repository - Source packages
Repo-status        : disabled
Repo-mirrors       : https://al2023-repos-us-east-1-de612dc2.s3.dualstack.us-east-1.amazonaws.com/kernel-livepatch/mirrors/al2023/SRPMS/mirror.list
Repo-expire        : 21600 second(s) (last: unknown)
Repo-filename      : /etc/yum.repos.d/kernel-livepatch.repo
Total packages: 0
```
If I download the first mirror list manually from the URL above (EC2 network connectivity works fine), I get a file containing just one line:
```
https://al2023-repos-us-east-1-de612dc2.s3.dualstack.us-east-1.amazonaws.com/core/guids/8f5098a368baae8fc9d160c812bde9270bf82532e1552dedfc94f590b8a04145/x86_64/
```
I am not sure how to find out what file(s) my DNF is trying to access, if any. But directory listing is disabled there, so one gets 403 when attempting to GET it.

Some additional info from my troubleshooting in case it's helpful:
```
] ls /etc/dnf/vars/
awsdomain  awsregion  dualstack  mirrorlist  repoguid
] cat /etc/dnf/vars/repoguid
46ff4933b89b948580f3b223b826fee3c1830b85885db3f7f90502c0ac99698c
] cat /etc/dnf/vars/awsregion
us-east-1
] cat /etc/dnf/vars/awsdomain
amazonaws.com

] cat /etc/yum.repos.d/amazonlinux.repo
[amazonlinux]
name=Amazon Linux 2023 repository
mirrorlist=https://al2023-repos-$awsregion-de612dc2.s3$dualstack.$awsregion.$awsdomain/core/mirrors/$releasever/$basearch/$mirrorlist
priority=10
enabled=1
repo_gpgcheck=0
type=rpm
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-amazon-linux-2023

// (skipped two more entries: amazonlinux-source and amazonlinux-debuginfo, both enabled=0)

] cat /etc/cloud/cloud.cfg.d/10_aws_dnfvars.cfg
# ### DO NOT MODIFY THIS FILE! ###
# This file will be replaced if cloud-init is upgraded.
# Please put your modifications in other files under /etc/cloud/cloud.cfg.d/
#
# Note that cloud-init uses flexible merge strategies for config options
# https://cloudinit.readthedocs.org/en/latest/topics/merging.html

write_metadata:
  # Fill in yum vars for the region and domain
  - path: /etc/dnf/vars/awsregion
    data:
      - identity: region
      - "default"
  - path: /etc/dnf/vars/awsdomain
    data:
      - metadata: services/domain
      - "amazonaws.com"

# vim:syntax=yaml expandtab

]  find /var/cache/dnf -type f
/var/cache/dnf/expired_repos.json
/var/cache/dnf/packages.db
/var/cache/dnf/.gpgkeyschecked.yum
/var/cache/dnf/tempfiles.json

] sudo yum reinstall -y system-release
Amazon Linux 2023 repository                                                                                                                                                                                                                            2.6 kB/s | 160  B     00:00
Error: Failed to download metadata for repo 'amazonlinux': Cannot prepare internal mirrorlist: No URLs in mirrorlist
Amazon Linux 2023 Kernel Livepatch repository                                                                                                                                                                                                           4.4 kB/s | 172  B     00:00
Error: Failed to download metadata for repo 'kernel-livepatch': Cannot prepare internal mirrorlist: No URLs in mirrorlist
Ignoring repositories: amazonlinux, kernel-livepatch
Installed package system-release-2023.3.20240108-0.amzn2023.noarch (from amazonlinux) not available.
Installed package system-release-2023.6.20250128-0.amzn2023.noarch (from amazonlinux) not available.
```
Any ideas how to fix this, short of re-creating the EC2 instance from scratch? (I'd like to avoid that if possible, since there was a lot of setup done for a web server)

Stewart Smith · Answer

Most likely is that there is an issue in network connectivity from your instance to the S3 repositories.

Amazon Linux repositories are mirrored both in a CDN and in S3 buckets local to each AWS region. The AMIs come with the "amazon-linux-repo-s3" package installed, which provides the configuration you show here. The container images, and onprem images, default to using the "amazon-linux-repo-cdn" package, which points the repository configuration at the CDN. This allows instances running in EC2 to talk to S3 efficiently, within region, without requiring any external traffic.

For EC2 instances (i.e. with the S3 repository configuration) on boot, cloud-init talks to IMDS (the Instance Metadata Service) to find out which region the instance is running in, and write that to a DNF variable, which is then substituted in the URL in the repository configuration. If you have disabled access to IMDS, then this will fail. If you look at your cloud-init logs, then this should be apparent if it's occurring.

Let's unpack the variables used in the `mirrorlist=` line in the repository configuration:
```
https://al2023-repos-$awsregion-de612dc2.s3$dualstack.$awsregion.$awsdomain/core/mirrors/$releasever/$basearch/$mirrorlist
```
The `$awsregion` comes from `/etc/dnf/vars/awsregion` and thus points to the S3 bucket that will be in the region the instance is running in, plus is used again to resolve to an S3 endpoint that is within region. The `$dualstack` variable is `/etc/dnf/vars/awsregion` and dictates if the S3 dualstack (i.e. IPv4 and IPv6) endpoints are used. In some restrictive VPC configurations, I *believe* this can end up being an issue, so you may want to see if the non-dualstack endpoints work (IIRC just making the `/etc/dnf/vars/awsregion` file empty should do the trick, which you can try manually or have cloud-init do it for you very early on in boot). The `awsdomain` variable is pretty self-explanatory, `$basearch` means you access the repositories for your CPU architecture, `$mirrorlist` just means `mirror.list`, and `$releasever` is substituted for the version of the `system-release` package and is how the [Deterministic Updates through Versioned Repositories](https://docs.aws.amazon.com/linux/al2023/ug/deterministic-upgrades.html) feature works.

The *content* of the `mirror.list` that is fetched is indeed a single URL (although is not guaranteed to be, and should be considered an implementation detail). The way that Amazon Linux repositories are implemented under the hood is to have this level of indirection so that the update to a `mirror.list` file is the single atomic operation required to release a version of the OS, and as long as the content is present before the `mirror.list` is updated, no instance should ever get a `404` error during our release process.

re:Post Agent · Answer

It appears that your Amazon Linux 2023 EC2 instance is experiencing issues with its DNF (Dandified YUM) package manager, specifically related to repository metadata. The problem seems to be that DNF cannot retrieve the mirror list for the Amazon Linux repositories. Here are some steps you can try to resolve this issue:

1. First, check and ensure that the content of the /etc/dnf/vars/awsregion file is correct. It should contain the region where your EC2 instance is running (in your case, us-east-1).

2. Verify that the /etc/dnf/vars/repoguid file contains a valid GUID. This file should be automatically populated, but you might want to check if it's present and not empty.

3. Try clearing the DNF cache:
   sudo dnf clean all

4. Manually update the system release package:
   sudo amazon-linux-extras install system-release

5. If the above steps don't work, you might need to manually edit the /etc/yum.repos.d/amazonlinux.repo file. Check if the mirrorlist URL is correct and matches your region.

6. Ensure that your EC2 instance has proper network connectivity, especially to S3 endpoints. Check your security groups and network ACLs to make sure they're not blocking outbound traffic to S3.

7. If you're using a custom VPC, verify that you have a route to the S3 gateway endpoint for your region.

8. As a last resort, you could try to manually download the mirror list file and place it in the appropriate location, but this would only be a temporary fix.

If none of these steps resolve the issue, you might need to consider more drastic measures like attaching the EBS volume to another instance to fix the repository configuration files, or potentially recreating the instance. However, before doing so, it would be advisable to create an AMI of your current instance to preserve your setup.  
**Sources**  
[Unable to install services on my instance. Take Two. | AWS re:Post](https://repost.aws/questions/QU2c7eUP0HTqexa9YUp1TJIQ/unable-to-install-services-on-my-instance-take-two)  
[Not able to install MYSQL on EC2 | AWS re:Post](https://repost.aws/questions/QUG_6UULT_Q_aeJ1uhxv03Yg/not-able-to-install-mysql-on-ec2)  
[Installing security updates | AWS re:Post](https://repost.aws/questions/QUB787OakJS_C6rD6zN1AyvA/installing-security-updates)

Error: Failed to download metadata for repo 'amazonlinux': Cannot prepare internal mirrorlist: No URLs in mirrorlist

Add your answer

Relevant content