The Wayback Machine - https://web.archive.org/web/20220320195156/https://github.com/cli/cli/issues/4888
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No indication on how to bypass --limit #4888

Open
GadgetSteve opened this issue Dec 11, 2021 · 7 comments
Open

No indication on how to bypass --limit #4888

GadgetSteve opened this issue Dec 11, 2021 · 7 comments

Comments

@GadgetSteve
Copy link

@GadgetSteve GadgetSteve commented Dec 11, 2021

Describe the feature or problem you’d like to solve

A number of options have a -L or --limit option with defaults (20 or 30) and can be supplied with a --limit n where n is a positive integer option, e.g.:

gh issue list --state=open --limit=2000 --json=number

However, if the user needs to get a list of all of the open issue numbers they need to either guess how many there might be or supply a very high number.

Proposed solution

Either have a --nolimit flag or allow either --limit=0 or --limit=-1 to mean no limit. This would allow things like pre-populating a list of all open tickets for pre-commit, etc. hooks to use to validate commit messages include only open ticket numbers without possibly doing multiple calls during the commit message processing.

Alternatively, a documented mechanism to get more items when piping output could be added - to me this seems more complex to both implement and use.

Additional context

@mislav
Copy link
Member

@mislav mislav commented Dec 13, 2021

Hi, thank you for the feature request, but it is by design that you must always specify the limit for fetching issues, pull request, repositories, or basically any unbounded set of data on GitHub.

Having a "no limit" setting would be convenient but would also be tricky if people started using it in their workflows or scripts without giving it a second thought. For example, fetching all issues without limit from this repository would likely complete in under a minute, but fetching all issues from some other repositories I belong to could potentially take many minutes. For example, the VS Code repository currently has 124,827 issues. Since we can only ever fetch 100 issues per request, we would need 1,249 sequential requests to fetch all of these issues. Now, if you take into account the overhead of making a single API request to GitHub, then multiply that to a thousand, then also consider rate limit restrictions, you can start to understand why unbounded fetching of data over the network is rarely a good idea.

So, if you need to fetch all issues from a large repository, specify a large --limit number to encompass all data that you are interested in. Specifying a large number as a limit forces you to consider what would happen if there actually were enough records to fill that limit.

However, we could document this restriction better, I agree! I'm going to re-classify this as a documentation issue.

@GadgetSteve
Copy link
Author

@GadgetSteve GadgetSteve commented Dec 13, 2021

If there were a simple to use continue or start-from mechanism documented this would allow a work-around.

@mislav
Copy link
Member

@mislav mislav commented Dec 13, 2021

@GadgetSteve Possibly, but I'm not sure how gh would display pagination info in its UI. Exposing lower-level pagination info doesn't seem like a great fit for gh right now, and if you need a way to continue paginating based on a previously fetched set, perhaps you should drop down to the GitHub API level manually where you have access to page parameter in REST API or cursor pagination in the GraphQL API?

query ($endCursor: String) {
	repository(owner: "cli", name: "cli") {
		issues(states: OPEN, first: 100, after: $endCursor) {
			nodes {
				number
				title
			}
			pageInfo {
				hasNextPage
				endCursor
			}
		}
	}
}

@GadgetSteve
Copy link
Author

@GadgetSteve GadgetSteve commented Dec 18, 2021

Since API returns a flag (hasNextPage) to say that there is more data including this in the response along with the endCursor and adding a --startfrom option could potentially allow a last line to be added to the output text something along the lines of:

More entries to list if needed use --startfrom=<endCursor>

The json response could include a last entries with fields of:
'NextPage:endCursor`

@mislav
Copy link
Member

@mislav mislav commented Dec 20, 2021

could potentially allow a last line to be added to the output text something along the lines of:

That sounds good in theory, but how would be output this line in machine-readable mode and expect that current scripts won't trip up over it thinking it's an issue record?

The json response could include a last entries with fields

That also sounds good in theory, but the top-level data type of our --json response is a JSON array. We can only add more records to the array but not extra fields without changing the JSON array to a JSON object, which would break backwards-compatibility with JSON consumers.

@peteos123

This comment has been minimized.

@peteos123

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants