Why is the API returning sometimes an integer, sometimes a string in the JSON response metadata?

I’m pulling JSON and the “per_page” is giving an int on the first page:
{“total”:77,“per_page”:50,“current_page”:1,“last_page”:2,“from”:1,“to”:2,“newest”:“2023-01-17T21:44:44.000Z”,“oldest”:“2021-10-13T23:22:50.000Z”},

Then on the second page it’s giving a string:
“total”:77,“per_page”:“50”,“current_page”:2,“last_page”:2,“from”:2,“to”:2,“newest”:“2023-01-17T21:44:44.000Z”,“oldest”:“2021-10-13T23:22:50.000Z”},

This is a real annoyance as this means I have to disable any kind of type checking to get it to work if I want to unmarshal the JSON in Go.

This is also happening with other fields as well. But only fields generated by the epi API, such as pages, mapping, etc. It’s going to be a game of whack -a-mole trying to fix them, if there’s no way to figure out why they are different across pages.

What’s the deal?

1 Like

We cannot reproduce the issue we are afraid.

See →

https://five.epicollect.net/api/export/entries/ec5-demo-project?page=1

https://five.epicollect.net/api/export/entries/ec5-demo-project?page=2

https://five.epicollect.net/api/export/entries/ec5-demo-project?page=3

…and so on. per_page is always an integer.

Can you provide a sample public project where this is happening so we can double-check?

Thank you

Unfortunately it’s not public and the data are sensitive/not mine to publicize!

Not sure how to figure out I suppose.

However, using Go and Postman, you can see page 1 in an int:

image

and page 2 is a string:

image

page 3 is a string:

image

And so on. Seems to be all but page one, from what I can see so far.

Thanks

You could easily clone that project and add some fake entries for testing purposes.

However, from the screenshots above, we assume the project you are having issues with is
BATI Fish Health MOWI_Updated.

We tried on our side and it is still all good (using Postman)

page=1

page=2

page=3

You could try a different environment, like https://stackblitz.com/ to exclude any issue with your Postman/Go settings/scripts.

Also, have a look at the “raw” view in Postman:

Screenshot 2023-02-16 at 11.51.14

"meta":{"total":1911,"per_page":50,"current_page":3,"last_page":39,"from":3,"to":39,"newest":"2023-02-14T21:56:11.000Z","oldest":"2021-10-13T22:14:14.000Z"}

Huh, well that’s useful! I thought it was private, but that’s not my concern anyhow.

This is the link to the project:

https://five.epicollect.net/project/bati-fish-health-mowi-updated

The API get link to a page that shows the issue, which includes headers, etc.:

https://five.epicollect.net/api/export/entries/bati-fish-health-mowi-updated%20?form_ref=5adfcd605e054c0c9a750640df7e34e0_609c46a2bd078&parent_form_ref=&branch=&branch_ref=5adfcd605e054c0c9a750640df7e34e0_609c46a2bd078_613fc459209b0&branch_owner_uuid=&parent_uuid=&uuid=&input_ref=&per_page=50&sort_by=created_at&sort_order=DESC&map_index=1&filter_by=&filter_from=&filter_to=&format=json&headers=true&title=&page=2

I’m still getting random strings in returns. It’s consistent with per_page, but now I’m getting some with some entered values.

Raw view on postman still shows string and int variability:

image

I’m also using stock Postman settings, i.e. not changing anything, even trying a different machine and network. Obviously can’t rule out Go as it would be looking at my source code, so we’ll ignore Go, but the int/string variability is consistent across Postman sessions and networks.

Even trying another API tester such as Testfully is the same:

image

Update:

Ok, so I’ve realized why this is. If you use the “next” and “previous” links that the API gives you, it changes the values to strings because it looks like it’s pulling from the links themselves (if you look at the 2nd link above, you can see per_page is in the link - this is generated by the API).

Great, found it. However, this doesn’t really make much sense. Why is the API not converting these values to JSON correctly?

Update 2
I’m now getting a float that’s being converted to a string on page 28 for “Length_cm” and “Weight_g”:

https://five.epicollect.net/api/export/entries/bati-fish-health-mowi-updated%20?form_ref=5adfcd605e054c0c9a750640df7e34e0_609c46a2bd078&parent_form_ref=&branch=&branch_ref=5adfcd605e054c0c9a750640df7e34e0_609c46a2bd078_613fc459209b0&branch_owner_uuid=&parent_uuid=&uuid=&input_ref=&per_page=50&sort_by=created_at&sort_order=DESC&map_index=1&filter_by=&filter_from=&filter_to=&format=json&headers=true&title=&page=28

image

That should be a float val. It is for previous entries:

image

This is also happening for non-API generated links, e.g. https://five.epicollect.net/api/export/entries/bati-fish-health-mowi-updated?map_index=1&form_ref=5adfcd605e054c0c9a750640df7e34e0_609c46a2bd078&branch_ref=5adfcd605e054c0c9a750640df7e34e0_609c46a2bd078_613fc459209b0&page=28

Also with both map_index=1 and the Epi default mapping.

Update 3

This is also happening with 6 fields on the “NetPen_” int val on page 38:

image

Update 4

The same result occurs when you download these manually from the download area (as JSON) on the web page for the project.

Seems very inconsistent!

The projects are private, no worries. We just tested the responses on our side, privately.

Thanks for spotting this, we are now able to reproduce the per_page issue using the prev and next autogenerated links.
We will look into it.

Sounds good.

As I’ve updated in my post, it’s also happening in other fields as well, ones that are input fields.

This is also happening on these input fields (things that aren’t per_page, etc.) on any link you try - even non-API generated links.

This basically means pulling data for this project will require a lot of manual work. Even then, it seems that it isn’t consistent and will change. Seems like a race condition or something.

It’s interesting because I could have sworn that the float and int vals I mention were working yesterday. Maybe I’m miss-remembering. Either way, it is very inconsistent and type checking is now useless!

Same with another project “bati-edna-survey” (I’ve bypassed the per_page not unmarshalling into the correct type):

This is also just using a +1 for the page as well, instead of using next:

image
image

That’s also just showing the first occurrence per page where they are converting to string. So, there are likely many more.

Without the bypass for per_page, using next:

So, the API is pretty much unusable at the moment.

Any idea if these issues will be fixed?

We have more projects starting and I will have to advise moving to another platform if it can’t be fixed/it will take a long time, as it breaks import pipelines.

I’m not trying to be demanding or whatever, just wondering if it’s something that will have any attention given to it, so that our projects can move forward! In addition to having maintainable new projects.

Thanks

Alright, so it looks like it’s not going to be looked into.

For anyone else coming here with this issue, there is a workaround, not a fix, where you can make the types generic and manually create your own conversions in Golang if you use interfaces, you can likely do this in whatever other language you’re using (this is also working in a C# backend service we’re running).

It just takes some manual set up and some basic knowledge of type conversions. However, this of course means that some assumptions need to be made and type safety is no longer quite so safe, in addition whatever is causing these issues could be causing incorrect data being exposed from Epicollect’s API.

Hopefully this will be corrected on Epicollect’s actual API, but their lack of response looks as though it’s not a priority for them.

1 Like

Was this ever actually fixed? Or was it just marked as fixed?

This issue is marked as fixed.