The DHS Program User Forum: Application Programming Interface (API)

Home » Dissemination and Capacity Strengthening » Application Programming Interface (API) » Record Limits

Show: Today's Messages :: Show Polls :: Message Navigator

Record Limits [message #4242]

Fri, 24 April 2015 14:05

jd.weaver
Messages: 5
Registered: April 2015
Location: Seattle, WA

Member

All,

I am trying to access DHS family planning data for the Track20 (track20.org) countries, including sub-national regions. However, I seem to be running into record limits. What are the API record limits?

The full API call url is here:
http://api.dhsprogram.com/rest/dhs/data?breakdown=subnationa l&indicatorIds=34202002,34202003,34202004,34202005,34202 006,34202007,34202008,34202009,34202010,34202011,34202012,34 202013,34202014,34202019,34202020,34202021,34202022,34202023 ,34202024,34202029,34202030,34202035,34202037&countryIds =BD,BJ,BF,CD,CI,ET,GN,IA,ID,KE,ML,MR,NI,NG,PK,PH,SN,TZ,TG,UG &surveyIds=BD2000DHS,BD2004DHS,BD2007DHS,BD2011DHS,BJ199 6DHS,BJ2001DHS,BJ2006DHS,BJ2012DHS,BF1993DHS,BF1999DHS,BF200 3DHS,BF2010DHS,CD2007DHS,CD2013DHS,CI1994DHS,CI1998DHS,CI201 2DHS,ET2011DHS,GN2005DHS,GN2012DHS,IA1993DHS,IA1999DHS,IA200 6DHS,ID1997DHS,ID2003DHS,ID2007DHS,ID2012DHS,KE1993DHS,KE199 8DHS,KE2003DHS,KE2008DHS,ML1996DHS,ML2001DHS,ML2006DHS,ML201 2DHS,MR2000DHS,NI1992DHS,NI1998DHS,NI2006DHS,NI2012DHS,NG199 0DHS,NG2003DHS,NG2008DHS,NG2013DHS,PK1991DHS,PK2006DHS,PK201 2DHS,PH1998DHS,PH2003DHS,PH2008DHS,PH2013DHS,SN1993DHS,SN199 7DHS,SN2005DHS,SN2010DHS,TZ1996DHS,TZ1999DHS,TZ2004DHS,TZ201 0DHS,TG1988DHS,TG1998DHS,UG1995DHS,UG2000DHS,UG2006DHS,UG201 1DHS&f=html

Report message to a moderator

Re: Record Limits [message #6805 is a reply to message #4242]

Fri, 17 July 2015 16:18

ychoi
Messages: 4
Registered: April 2014

Member

In case you haven't found the answer yet:

- Default record limit is 100: "perpage: Indicates maximum number of rows returned. Maximum is 1000 for non-authenticated users, and 5000 for authenticated users. Default is 100." (http://api.dhsprogram.com/#/api-advancedqueries.cfm)
- So. you'd add "perpage=1000" in your code to increase the number of record.
- If you want to become an autheticated/registered user (to increase the number of record even more), just email API. They were fast in response! More here: http://api.dhsprogram.com/#/introdevelop.html

Report message to a moderator

Re: Record Limits [message #9569 is a reply to message #6805]

Mon, 18 April 2016 15:39

nchakraborty
Messages: 6
Registered: April 2016

Member

What is the best way to get all of the records needed, if you are exceeding the record limit per page? For example, if I want data for several indicators, for breakdown=all, for all countries, is there something more efficient than having multiple API calls, such as:

" http://api.dhsprogram.com/rest/dhs/data?breakdown=all&in dicatorIDs=FP_SRCM_W_PUB,FP_SRCM_W_PRV,FP_SRCM_W_SHP,FP_SRCM _W_CHH,FP_SRCM_W_FRR,FP_SRCM_W_OSR,FP_SRCM_W_OTH,FP_SRCM_W_D KM,FP_SRCM_W_MIS,FP_CUSM_W_MOD,FP_CUSA_W_MOD,FP_CUSU_W_MOD&a mp;perpage=1000"

" http://api.dhsprogram.com/rest/dhs/data?breakdown=all&in dicatorIDs=FP_SRCM_W_PUB,FP_SRCM_W_PRV,FP_SRCM_W_SHP,FP_SRCM _W_CHH,FP_SRCM_W_FRR,FP_SRCM_W_OSR,FP_SRCM_W_OTH,FP_SRCM_W_D KM,FP_SRCM_W_MIS,FP_CUSM_W_MOD,FP_CUSA_W_MOD,FP_CUSU_W_MOD&a mp;perpage=1000&page=2"

" http://api.dhsprogram.com/rest/dhs/data?breakdown=all&in dicatorIDs=FP_SRCM_W_PUB,FP_SRCM_W_PRV,FP_SRCM_W_SHP,FP_SRCM _W_CHH,FP_SRCM_W_FRR,FP_SRCM_W_OSR,FP_SRCM_W_OTH,FP_SRCM_W_D KM,FP_SRCM_W_MIS,FP_CUSM_W_MOD,FP_CUSA_W_MOD,FP_CUSU_W_MOD&a mp;perpage=1000&page=3"

etc.?

Report message to a moderator

Re: Record Limits [message #9570 is a reply to message #9569]

Mon, 18 April 2016 16:00

Trevor-DHS
Messages: 803
Registered: January 2013

Senior Member

A few suggestions:
1) You can set perpage=3000 without using an API key.
2) If you request an API key you can set perpage=5000
3) Write your code to access the API in a loop that will loop through the pages until you have returned all of the data you need.

Report message to a moderator

Re: Record Limits [message #9591 is a reply to message #9570]

Tue, 19 April 2016 15:17

nchakraborty
Messages: 6
Registered: April 2016

Member

Thank you, Trevor.

The STATA loop I wrote is shared below for others. The list of indicators requested has been truncated.

/*API allows 3000 records at a time for a user, but you can select subsequent pages of records
The code below loops through requests for additional pages of records, without knowing
how many pages there will be. It then appends all of the pages, each saved as a dataset
into 1 file */

local i=1
while `i'<1000 {

clear
gen str9 surveyid=""
gen str30 group=""
gen str30 grouplabel=""
gen str30 indicatorID=""
gen str10 value=""
gen str30 country=""

#delimit;
insheetjson surveyid country indicatorID group grouplabel value
using " http://api.dhsprogram.com/rest/dhs/data?breakdown=all&su rveyYear=2010,2011,2012,2013,2014&indicatorIDs=FP_CUSM_W _MOD,FP_CUSA_W_MOD,FP_CUSU_W_MOD&perpage=3000&page=` i'",
table(Data) col(SurveyId CountryName IndicatorId CharacteristicCategory CharacteristicLabel Value);

#delimit cr

save "page`i'.dta", replace

if _N<3000 {
exit
}
else {
local i=`i'+1
}
}

while `i'>1 {
local i=`i'-1
append using "page`i'.dta"
}

Report message to a moderator

Previous Topic:	getting indicator by only selected background characteristics
Next Topic:	Trends in vaccination coverage and factors for complete vaccination

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sun Dec 1 07:15:46 Coordinated Universal Time 2024