Project Name (if applicable, otherwise just type n/a)
PVBR_RS_aftosa
Question
Hi,
I use Power BI to process data from my Epicollect using the API.
It’s been a week that I’ve been receiving a 429 error: Too Many Requests. This has never happened before, despite dealing with large datasets.
Has Epicollect made any changes in the past few weeks? Other BIs are experiencing the same issue.
I confirm I have read the User Guide at docs.epicollect.net
on
I confirm I have searched for an answer in this community.
on
Yes, Epicollect5 has implemented standard rate limits on its API to ensure fair usage and maintain optimal system performance.
These limits have been in place for some time and are clearly documented here:
Epicollect5 API Rate Limiting
In your specific case, it appears that you are making hundreds of requests for a project containing 20,000 entries while only requesting 50 entries per request.
For example:
/api/export/entries/pvbr-rs-aftosa?form_ref=3406c921427d4cb5a9ae18d15cfef3de_634d7dad03bde&parent_form_ref=&branch=&branch_ref=&branch_owner_uuid=&parent_uuid=&uuid=&input_ref=&per_page=50&sort_by=created_at&sort_order=DESC&map_index=0&filter_by=&filter_from=&filter_to=&format=json&headers=true&title=&page=9
At this rate, to retrieve all 20,000 entries, you would need approximately 400 requests (20,000 / 50 = 400 ).
To optimize this process, you could consider increasing the per_page parameter to a higher value, such as 200. With this adjustment, you would only need approximately 100 requests to retrieve the entire dataset (20,000 / 200 = 100 ).
You can experiment with higher values, the per_page upper limit is 1000.
Additionally, it seems that you are requesting the entire dataset every time. Is this intentional? If not, you might want to filter your requests to retrieve only new or updated entries. This can be done by applying date filters, as explained in the Epicollect5 documentation:
Filtering Entries
By using these optimizations, you can reduce the number of API requests, improving efficiency while also staying within the rate limits.
es, I request all the data in order to maintain a historical dataset for future consultations.
I tried per page = 1000, but the problem is still the same.
We cannot reproduce the issue we are afraid.
Also be aware the rate limit is by ip address.
Screenshot from Insomnia:
This is our system log listing your requests to the Epicollect5 API:
log.txt (52.0 KB)
It appears that while there are numerous successful requests, the system eventually kicks you out.
We kindly ask that you review your scripts to ensure everything functions as intended.
I’ll try to review my scripts, and then I’ll come back to let you know if I was able to solve the problem. Thank you for your help so far.
How can I increase the paramenter “per page”? Always that I use the link, it came automatically per50
let
Fonte = Json.Document(Web.Contents(“
https://five.epicollect.net/api/export/entries/pvbr-rs-aftosa?form_ref=3406c921427d4cb5a9ae18d15cfef3de_634d7dad03bde”)),
#“meta” = Fonte[meta]
in
#“meta”
The URL you’re using is:
https://five.epicollect.net/api/export/entries/pvbr-rs-aftosa?form_ref=3406c921427d4cb5a9ae18d15cfef3de_634d7dad03bde
Problem:
You are not setting a per_page parameter, so it uses the default (50).
Solution:
You need to add &per_page=XXX to the URL.
For example, if you want 500 entries per page, change your Power Query like this:
let
Fonte = Json.Document(Web.Contents("https://five.epicollect.net/api/export/entries/pvbr-rs-aftosa?form_ref=3406c921427d4cb5a9ae18d15cfef3de_634d7dad03bde&per_page=500")),
#"meta" = Fonte[meta]
in
#"meta"
Summary:
Just add &per_page=500 at the end of the URL.
To get further code assistance we recommend AI tools like ChatGPT, Gemini and Copilot.
It appears that no additional details have been provided regarding the current thread.
In the absence of further information, we are unable to proceed with the discussion.
If you have any additional details to share or questions to ask, please feel free to start a new thread.
Thank you for your understanding.
