Issue downloading csv format using API

Project: TWI 2021 Public Saves.

Currently has 2 entries to download. They download when asking for page 1. They download again when asking for page 2. And again when asking for page 3.

The program works by asking for 500 entries per page until it gets no entries, with a maximum number of asks. (It also filters by date uploaded, but that has no bearing on this.)

Perhaps there is a better way to detect end of data when downloading csv?

We’re using csv format because it is easier to do diagnostics on the content.

However, I have a test utility that uses json instead, and it is working. It gets the 2 entries when asking for page 1, and gets nothing when asking for page 2 and so quits.

We are in between seasons right now, so I haven’t exercised csv downloads in the past several weeks, so I don’t know when it changed. I was testing today in response to the notice yesterday of recent changes.

Thanks

Hi Homer,

many thanks for reporting this. It looks like the page parameter is not working as expected when fetching the csv fomat, we will look into it.

A workaround (until it gets fixed) could be doing a pre-request for the json and look at the content of the meta property:

meta": {
"total": 2,
"per_page": 50,
"current_page": 1,
"last_page": 1,
"from": 1,
"to": 1,
"newest": "2021-02-25T15:16:48.000Z",
"oldest": "2021-02-25T15:16:38.000Z"
}

last_page will tell you when to stop fetching. Passing a per_page=1 will keep the request size to a minimum.

Or, as you said, using a high per_page parameter to get the whole dataset

Our season is mid-May to mid-September, so I’ll watch for developments. Meantime, I’ll implement one of the work-arounds. Maybe even switch over to json. Thanks for the quick response.

We just pushed a quick fix, please let us know if everything works as expected now.

Yes, working now. Thanks!

1 Like