Project: TWI 2021 Public Saves.
Currently has 2 entries to download. They download when asking for page 1. They download again when asking for page 2. And again when asking for page 3.
The program works by asking for 500 entries per page until it gets no entries, with a maximum number of asks. (It also filters by date uploaded, but that has no bearing on this.)
Perhaps there is a better way to detect end of data when downloading csv?
We’re using csv format because it is easier to do diagnostics on the content.
However, I have a test utility that uses json instead, and it is working. It gets the 2 entries when asking for page 1, and gets nothing when asking for page 2 and so quits.
We are in between seasons right now, so I haven’t exercised csv downloads in the past several weeks, so I don’t know when it changed. I was testing today in response to the notice yesterday of recent changes.
many thanks for reporting this. It looks like the
page parameter is not working as expected when fetching the csv fomat, we will look into it.
A workaround (until it gets fixed) could be doing a pre-request for the json and look at the content of the
last_page will tell you when to stop fetching. Passing a
per_page=1 will keep the request size to a minimum.
Or, as you said, using a high
per_page parameter to get the whole dataset
Our season is mid-May to mid-September, so I’ll watch for developments. Meantime, I’ll implement one of the work-arounds. Maybe even switch over to json. Thanks for the quick response.
We just pushed a quick fix, please let us know if everything works as expected now.
Yes, working now. Thanks!
Would you be able to share your code to download csv format using the api? I tried using the python example EpiCollect5 showed but I was getting JSON format.
Thank you @Epicollect5, the JSON format is working so well. I am just sticking to the JSON format. I use this function jsonlite::from_json(rawToChar(content(res1) to convert it straight to an R object.
Thanks for the tutorial.