The DHS Program User Forum      
Discussions regarding The DHS Program data and results
Home » Dissemination and Capacity Strengthening » Application Programming Interface (API)  » Record Limits
Record Limits [message #4242] Fri, 24 April 2015 14:05 Go to next message
jd.weaver is currently offline  jd.weaver
Messages: 5
Registered: April 2015
Location: Seattle, WA
Member
All,

I am trying to access DHS family planning data for the Track20 (track20.org) countries, including sub-national regions. However, I seem to be running into record limits. What are the API record limits?

The full API call url is here:
http://api.dhsprogram.com/rest/dhs/data?breakdown=subnationa l&indicatorIds=34202002,34202003,34202004,34202005,34202 006,34202007,34202008,34202009,34202010,34202011,34202012,34 202013,34202014,34202019,34202020,34202021,34202022,34202023 ,34202024,34202029,34202030,34202035,34202037&countryIds =BD,BJ,BF,CD,CI,ET,GN,IA,ID,KE,ML,MR,NI,NG,PK,PH,SN,TZ,TG,UG &surveyIds=BD2000DHS,BD2004DHS,BD2007DHS,BD2011DHS,BJ199 6DHS,BJ2001DHS,BJ2006DHS,BJ2012DHS,BF1993DHS,BF1999DHS,BF200 3DHS,BF2010DHS,CD2007DHS,CD2013DHS,CI1994DHS,CI1998DHS,CI201 2DHS,ET2011DHS,GN2005DHS,GN2012DHS,IA1993DHS,IA1999DHS,IA200 6DHS,ID1997DHS,ID2003DHS,ID2007DHS,ID2012DHS,KE1993DHS,KE199 8DHS,KE2003DHS,KE2008DHS,ML1996DHS,ML2001DHS,ML2006DHS,ML201 2DHS,MR2000DHS,NI1992DHS,NI1998DHS,NI2006DHS,NI2012DHS,NG199 0DHS,NG2003DHS,NG2008DHS,NG2013DHS,PK1991DHS,PK2006DHS,PK201 2DHS,PH1998DHS,PH2003DHS,PH2008DHS,PH2013DHS,SN1993DHS,SN199 7DHS,SN2005DHS,SN2010DHS,TZ1996DHS,TZ1999DHS,TZ2004DHS,TZ201 0DHS,TG1988DHS,TG1998DHS,UG1995DHS,UG2000DHS,UG2006DHS,UG201 1DHS&f=html
Re: Record Limits [message #6805 is a reply to message #4242] Fri, 17 July 2015 16:18 Go to previous messageGo to next message
ychoi is currently offline  ychoi
Messages: 4
Registered: April 2014
Member
In case you haven't found the answer yet:

- Default record limit is 100: "perpage: Indicates maximum number of rows returned. Maximum is 1000 for non-authenticated users, and 5000 for authenticated users. Default is 100." (http://api.dhsprogram.com/#/api-advancedqueries.cfm)
- So. you'd add "perpage=1000" in your code to increase the number of record.
- If you want to become an autheticated/registered user (to increase the number of record even more), just email API. They were fast in response! More here: http://api.dhsprogram.com/#/introdevelop.html


Re: Record Limits [message #9569 is a reply to message #6805] Mon, 18 April 2016 15:39 Go to previous messageGo to next message
nchakraborty
Messages: 4
Registered: April 2016
Member
What is the best way to get all of the records needed, if you are exceeding the record limit per page? For example, if I want data for several indicators, for breakdown=all, for all countries, is there something more efficient than having multiple API calls, such as:

" http://api.dhsprogram.com/rest/dhs/data?breakdown=all&in dicatorIDs=FP_SRCM_W_PUB,FP_SRCM_W_PRV,FP_SRCM_W_SHP,FP_SRCM _W_CHH,FP_SRCM_W_FRR,FP_SRCM_W_OSR,FP_SRCM_W_OTH,FP_SRCM_W_D KM,FP_SRCM_W_MIS,FP_CUSM_W_MOD,FP_CUSA_W_MOD,FP_CUSU_W_MOD&a mp;perpage=1000"

" http://api.dhsprogram.com/rest/dhs/data?breakdown=all&in dicatorIDs=FP_SRCM_W_PUB,FP_SRCM_W_PRV,FP_SRCM_W_SHP,FP_SRCM _W_CHH,FP_SRCM_W_FRR,FP_SRCM_W_OSR,FP_SRCM_W_OTH,FP_SRCM_W_D KM,FP_SRCM_W_MIS,FP_CUSM_W_MOD,FP_CUSA_W_MOD,FP_CUSU_W_MOD&a mp;perpage=1000&page=2"

" http://api.dhsprogram.com/rest/dhs/data?breakdown=all&in dicatorIDs=FP_SRCM_W_PUB,FP_SRCM_W_PRV,FP_SRCM_W_SHP,FP_SRCM _W_CHH,FP_SRCM_W_FRR,FP_SRCM_W_OSR,FP_SRCM_W_OTH,FP_SRCM_W_D KM,FP_SRCM_W_MIS,FP_CUSM_W_MOD,FP_CUSA_W_MOD,FP_CUSU_W_MOD&a mp;perpage=1000&page=3"

etc.?
Re: Record Limits [message #9570 is a reply to message #9569] Mon, 18 April 2016 16:00 Go to previous messageGo to next message
Trevor-DHS is currently offline  Trevor-DHS
Messages: 527
Registered: January 2013
Senior Member
A few suggestions:
1) You can set perpage=3000 without using an API key.
2) If you request an API key you can set perpage=5000
3) Write your code to access the API in a loop that will loop through the pages until you have returned all of the data you need.
Re: Record Limits [message #9591 is a reply to message #9570] Tue, 19 April 2016 15:17 Go to previous message
nchakraborty
Messages: 4
Registered: April 2016
Member
Thank you, Trevor.

The STATA loop I wrote is shared below for others. The list of indicators requested has been truncated.

/*API allows 3000 records at a time for a user, but you can select subsequent pages of records
The code below loops through requests for additional pages of records, without knowing
how many pages there will be. It then appends all of the pages, each saved as a dataset
into 1 file */


local i=1
while `i'<1000 {

clear
gen str9 surveyid=""
gen str30 group=""
gen str30 grouplabel=""
gen str30 indicatorID=""
gen str10 value=""
gen str30 country=""

#delimit;
insheetjson surveyid country indicatorID group grouplabel value
using " http://api.dhsprogram.com/rest/dhs/data?breakdown=all&su rveyYear=2010,2011,2012,2013,2014&indicatorIDs=FP_CUSM_W _MOD,FP_CUSA_W_MOD,FP_CUSU_W_MOD&perpage=3000&page=` i'",
table(Data) col(SurveyId CountryName IndicatorId CharacteristicCategory CharacteristicLabel Value);


#delimit cr

save "page`i'.dta", replace

if _N<3000 {
exit
}
else {
local i=`i'+1
}
}

while `i'>1 {
local i=`i'-1
append using "page`i'.dta"
}
Previous Topic: getting indicator by only selected background characteristics
Next Topic: Trends in vaccination coverage and factors for complete vaccination
Goto Forum:
  


Current Time: Sun Sep 24 17:05:06 Eastern Daylight Time 2017