The Wayback Machine - https://web.archive.org/web/20201116234209/https://github.com/coreos/rpm-ostree/issues/2251
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFE: implement a weekly DNF-countme-compatible query #2251

Open
mattdm opened this issue Jun 20, 2020 · 7 comments
Open

RFE: implement a weekly DNF-countme-compatible query #2251

mattdm opened this issue Jun 20, 2020 · 7 comments
Assignees
Labels

Comments

@mattdm
Copy link

@mattdm mattdm commented Jun 20, 2020

I added a note at coreos/fedora-coreos-tracker#86 but figured I'd make a pointer here too. We now have a feature in DNF where systems ask to be counted once per week, with some very basic data (underlying Fedora OS version, OS variant ("coreos"), architecture, and an age bucket). This is documented in dnf.conf:

Determines whether a special flag should be added to a single, randomly chosen > metalink/mirrorlist query each week. This allows the repository owner to estimate the number of systems consuming it, by counting such queries over a week's time, which is much more accurate than just counting unique IP addresses (which is subject to both overcounting and undercounting due to short DHCP leases and NAT, respectively).

The flag is a simple "countme=N" parameter appended to the metalink and mirrorlist URL, where N is an integer representing the "longevity" bucket this system belongs to. The following 4 buckets are defined, based on how many full weeks have passed since the beginning of the week when this system was installed: 1 = first week, 2 = first month (2-4 weeks), 3 = six months (5-24 weeks) and 4 = more than six months (> 24 weeks). This information is meant to help distinguish short-lived installs from long-term ones, and to gather other statistics about system lifecycle.

I know this isn't an exact match with what you're looking for, but having data in a way that is an exact match would be very helpful for me. We already have the DNF code in place in F32, and the first-pass backend is in progress. (Right now, we're getting information extracted from the logs, so we have a delay, but it's great for monthly reports, etc.. Future implementation may get info directly from the proxies in closer to real time.)

@miabbott miabbott transferred this issue from coreos/fedora-coreos-pinger Oct 9, 2020
@miabbott miabbott added the jira label Oct 9, 2020
@travier
Copy link
Member

@travier travier commented Oct 9, 2020

This issue was moved to rpm-ostree as I am working on adding DNF Count Me support here for the following reasons:

  • rpm-ostree already "knows" how to read yum repo files.
  • Enabling support in rpm-ostree will enable support in Fedora CoreOS, Silverblue and IoT.
  • This is a smaller first step to implementing full metrics collection for Fedora CoreOS (and other ostree based systems).
@travier
Copy link
Member

@travier travier commented Oct 9, 2020

I'm looking at the DNF Count Me feature and wondering: If we are running on a rpm-ostree based system with no package installed / overlayed, will we ever query the fedora.repo metalink when doing upgrades with rpm-ostree?

Unfortunately, I don't think so, and we do not run regular dnf make-cache services by default thus we need a different logic to enable periodic reporting.

I'm currently looking at creating a rpm-ostree countme command to be called weekly from a systemd timer.

@travier
Copy link
Member

@travier travier commented Oct 9, 2020

Moreover, updates usually only happen once every two weeks for Fedora CoreOS for example thus we would not report every week.

@travier travier added the enhancement label Oct 9, 2020
@jlebon
Copy link
Member

@jlebon jlebon commented Oct 9, 2020

I'm looking at the DNF Count Me feature and wondering: If we are running on a rpm-ostree based system with no package installed / overlayed, will we ever query the fedora.repo metalink when doing upgrades with rpm-ostree?

Unfortunately, I don't think so

You're correct. A really nice/fundamental feature of the design is that if you don't package layer, it's basically pure OSTree (this also has an unexpected downside: #1696).

Having it be a separate command makes sense to me. In the pure OSTree case, it'd be nice if it were just a single GET since we don't actually need the full repodata, though work might be needed in libdnf/librepo for that (or I guess we could circumvent that whole stack and just HEAD the repomd.xml ourselves?).

@travier
Copy link
Member

@travier travier commented Oct 12, 2020

The current logic implemented in DNF is private (Repo-private.hpp, Repo.cpp) and called only when repos are updated (dnf-repo.cpp) which is not ideal if we want to only make this a single GET request.

I'm currently working on a standalone implementation in Rust based on the C++ logic in DNF.

@lucab
Copy link
Member

@lucab lucab commented Oct 14, 2020

From an out-of-band chat: this relies on re-implementing a bunch of libdnf logic, including config parsing. That assumes that config parsers (esp. INI ones) are equivalent, which is rarely the case. A cleaner approach would be exposing an interface to perform the HEAD (or at least to extract metalink/countme details) from libdnf and consuming it here.

@cgwalters
Copy link
Member

@cgwalters cgwalters commented Oct 14, 2020

Or perhaps even simpler, add support to just fetch the metalink but do nothing with it to libdnf.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.