The DHS Program User Forum: Dataset access » Bulk download only Standard DHS datasets?

Home » Data » Dataset access » Bulk download only Standard DHS datasets?

Show: Today's Messages :: Show Polls :: Message Navigator

Switch to threaded view of this topic

Create a new topic

Submit Reply

Bulk download only Standard DHS datasets? [message #13323]

Wed, 18 October 2017 14:56

archju01 is currently offline

archju01
Messages: 2
Registered: October 2017
Location: Washington, DC

Member

The bulk download function is great and has greatly improved our ability to access only the newest datasets for each country of interest. However, my current project is interested in some variables present in the standard DHS but not the MIS. When selecting the most recent survey, the bulk download list does not seem to allow for only the most recent standard DHS. This affects Ghana, Kenya, and others. Is there a way to exclude MIS survey data? Thanks in advance.

Report message to a moderator

Re: Bulk download only Standard DHS datasets? [message #13325 is a reply to message #13323]

Thu, 19 October 2017 10:05

admin is currently offline

admin
Messages: 50
Registered: November 2012

Senior Member
Administrator

Hello, Yes we can add an option to download most recent by survey type: "Most Recent DHS"; "Most Recent SPA"; "Most Recent AIS", and "Most Recent MIS". We will include this in our next set of updates to the bulk download tool which will likely happen within the new few weeks.
Thank you for the suggestion!

DHS UserForum Administrator

We now have repositories of code written in Stata and SPSS available on Github. Please reference these code repositories as a resource for code for matching or calculating DHS indicators. The code repositories can be found at:

https://github.com/DHSProgram/DHS-Indicators-Stata
https://github.com/DHSProgram/DHS-Indicators-SPSS

[Updated on: Tue, 31 October 2017 13:07]

Report message to a moderator

Re: Bulk download only Standard DHS datasets? [message #13327 is a reply to message #13325]

Thu, 19 October 2017 11:30

archju01 is currently offline

archju01
Messages: 2
Registered: October 2017
Location: Washington, DC

Member

Thank you for being so quick to reply and accommodate. I look forward to using the new function when it's online.

Report message to a moderator

Re: Bulk download only Standard DHS datasets? [message #14346 is a reply to message #13323]

Mon, 26 March 2018 12:45

cportner is currently offline

cportner
Messages: 20
Registered: September 2014
Location: Seattle

Member

When is this feature going to roll out? It would be great if there was a way to download only DHS datasets (not just the most recent one).

Report message to a moderator

Re: Bulk download only Standard DHS datasets? [message #14347 is a reply to message #14346]

Mon, 26 March 2018 13:00

admin is currently offline

admin
Messages: 50
Registered: November 2012

Senior Member
Administrator

Yes, this feature is still on the list of items for next release. Should be available by middle of next month.

Sorry for the delay.

DHS UserForum Administrator

We now have repositories of code written in Stata and SPSS available on Github. Please reference these code repositories as a resource for code for matching or calculating DHS indicators. The code repositories can be found at:

https://github.com/DHSProgram/DHS-Indicators-Stata
https://github.com/DHSProgram/DHS-Indicators-SPSS

Report message to a moderator

Re: Bulk download only Standard DHS datasets? [message #14349 is a reply to message #14347]

Mon, 26 March 2018 17:37

cportner is currently offline

cportner
Messages: 20
Registered: September 2014
Location: Seattle

Member

Wonderful! Thank you for the quick reply.

In the meantime, is there a way to separate out which files are standard DHS and which files are other types? I am using a lot of countries for my current project, and I am trying to automate the read-in process as much as possible.

From what I can tell the files follow an identical naming pattern across standard DHS and MIS. Also, a quick look at the survey data reveals no indicator for which type of survey it was. Hence, it appears that the only option currently is to read in the data set, check whether all the variables I need are available, and drop the data set if they are not. Is there another way to do this?

Claus

Report message to a moderator

Re: Bulk download only Standard DHS datasets? [message #14357 is a reply to message #14349]

Tue, 27 March 2018 07:45

Bridgette-DHS is currently offline

Bridgette-DHS
Messages: 3199
Registered: February 2013

Senior Member

You can go to the survey search and use the "survey type" criteria to get a list of the surveys you are interested in. You have a complete breakdown of survey types here.

[Updated on: Tue, 27 March 2018 07:47]

Report message to a moderator

Re: Bulk download only Standard DHS datasets? [message #17639 is a reply to message #14357]

Wed, 01 May 2019 15:03

cportner is currently offline

cportner
Messages: 20
Registered: September 2014
Location: Seattle

Member

I tried the survey search function, but that does not appear to give you an option to download all the files directly. Would it be possible to expand the "Download Datasets Using a Download Manager" option to include only standard DHS surveys?

All the best,

Claus

Report message to a moderator

Re: Bulk download only Standard DHS datasets? [message #17641 is a reply to message #17639]

Wed, 01 May 2019 15:13

admin is currently offline

admin
Messages: 50
Registered: November 2012

Senior Member
Administrator

Hello Claus, I believe what you are asking for is the "All DHS" option in the download manager.

The DHS Program User Forum Administrator

We now have repositories of code written in Stata and SPSS available on Github. Please reference these code repositories as a resource for code for matching or calculating DHS indicators. The code repositories can be found at:

https://github.com/DHSProgram/DHS-Indicators-Stata
https://github.com/DHSProgram/DHS-Indicators-SPSS

Report message to a moderator

Re: Bulk download only Standard DHS datasets? [message #17645 is a reply to message #17641]

Wed, 01 May 2019 17:40

cportner is currently offline

cportner
Messages: 20
Registered: September 2014
Location: Seattle

Member

The problem with that one is that it also includes the once classified as "special DHS". For example, the two from the Dominican Republic with samples living in sugar cane planations are included if I use this approach. What I am looking for it an option to select *only* standard DHS surveys.

Report message to a moderator

Re: Bulk download only Standard DHS datasets? [message #17646 is a reply to message #17645]

Wed, 01 May 2019 17:58

admin is currently offline

admin
Messages: 50
Registered: November 2012

Senior Member
Administrator

I see. We will look into adding the option. In the meantime can you manually remove those datasets you don't want once downloaded?

The DHS Program User Forum Administrator

We now have repositories of code written in Stata and SPSS available on Github. Please reference these code repositories as a resource for code for matching or calculating DHS indicators. The code repositories can be found at:

https://github.com/DHSProgram/DHS-Indicators-Stata
https://github.com/DHSProgram/DHS-Indicators-SPSS

Report message to a moderator

Re: Bulk download only Standard DHS datasets? [message #17648 is a reply to message #17646]

Wed, 01 May 2019 21:10

cportner is currently offline

cportner
Messages: 20
Registered: September 2014
Location: Seattle

Member

That is essentially what I have been doing. The problem is that the download consists of 286 separated zip files, so I have tried to automate the process. What I have done is automatically drop any surveys that are missing a specific set of variables such as v602 and bidx_01 (or rather have all missing observations for these variables). This, however, does not get rid of some special DHS, such as the Dominican Republic ones on sugar plantation populations.

Report message to a moderator

Re: Bulk download only Standard DHS datasets? [message #17654 is a reply to message #17648]

Thu, 02 May 2019 08:34

admin is currently offline

admin
Messages: 50
Registered: November 2012

Senior Member
Administrator

I see the issue you are having. You could also parse the file name to remove or skip the files you dont want. Each file is coded with the country code and version number that uniquely identifies a survey.

That might be messy coding but it should work.

However I do agree adding an option to just select standard surveys is good idea and probably not too complicated. We will look into this.

The DHS Program User Forum Administrator

We now have repositories of code written in Stata and SPSS available on Github. Please reference these code repositories as a resource for code for matching or calculating DHS indicators. The code repositories can be found at:

https://github.com/DHSProgram/DHS-Indicators-Stata
https://github.com/DHSProgram/DHS-Indicators-SPSS

[Updated on: Thu, 02 May 2019 08:35]

Report message to a moderator

Re: Bulk download only Standard DHS datasets? [message #17656 is a reply to message #17654]

Thu, 02 May 2019 12:02

cportner is currently offline

cportner
Messages: 20
Registered: September 2014
Location: Seattle

Member

Thank you! It would be great if the option to just select standard DHS was available.

Until then, what I am currently doing is close to what you describe. I take the country code and version number from both the file name and from within the data set itself (since that helps catch/remove the individual Indian state files). The script uses those to select the relevant variables by phase.

Is there a way to identify whether a given survey comes from a special DHS or a standard DHS from within the data? It appears that the version numbering is in chronological order, so, for example, the Dominican Republic special DHS are named 5A and 6A because they came after the standard ones for the respective phases. Alternatively, is there a file that identifies surveys as either special or standard DHSs?

[Updated on: Thu, 02 May 2019 12:12]

Report message to a moderator

Re: Bulk download only Standard DHS datasets? [message #17678 is a reply to message #17656]

Mon, 06 May 2019 11:05

admin is currently offline

admin
Messages: 50
Registered: November 2012

Senior Member
Administrator

Hi Claus, I think what you are doing is the best workaround for now. We do plan to add an option for selecting standard only surveys. Not sure exactly when that will get done but not too distant future.

Thank you for the suggestion.

The DHS Program User Forum Administrator

We now have repositories of code written in Stata and SPSS available on Github. Please reference these code repositories as a resource for code for matching or calculating DHS indicators. The code repositories can be found at:

https://github.com/DHSProgram/DHS-Indicators-Stata
https://github.com/DHSProgram/DHS-Indicators-SPSS

Report message to a moderator

Switch to threaded view of this topic

Create a new topic

Submit Reply

Previous Topic:	Height and Weight for men and women
Next Topic:	Bulk Downloading of Datasets Using a Download Manager

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

PDF

]

Current Time: Wed Nov 27 17:14:36 Coordinated Universal Time 2024