[Eileen Kennedy] Welcome, we’re delighted to be doing this webinar on Evidence Use for Policies and Programs: Generating Evidence. We have four areas we’d like to discuss in this webinar, Why do we need evidence? How is evidence generated? Levels of Evidence and then Study Designs. Why do we need evidence? Put very very broadly typically one would want evidence for evaluation of impact, cost effectiveness, advocacy. I’ll use an example that illustrates each of these in the next slide. I have picked the Lancet series on maternal and child health, the first one in 2008, and in the next slide we see that through a series of systematic reviews the Lancet series came up with very strong evidence for uh implementation in what are called 36 high burden countries. Implementation of interventions that would improve maternal and birth outcomes, uh in- uh interventions for newborn babies, and uh interventions for infants and children. In the same 2008 series there also was a uh articulation of evidence uh for implementation in specific situations. So that was the evidence. Equally important and in the policy community uh in discussions in which I was involved, this was the the evidence that really created a lot of buzz, the fact that not only did we have for pregnant women young children, not only did we have have efficacious interventions but we could quantify the number of lives saved and we could also quantify cost savings in DALYs which is Disability Adjusted Life Years, basically healthy years accounting for premature mortality and disability. And it was that powerful combination of the impact evidence as well as the uh economic analysis that really created a buzz in the community and this led to, in the next slide as we see, Lancet’s series 2008 produced the evidence, it was used as the the key body of evidence for uh to the momentum for the Scaling Up Nutrition movement and uh research that Shibani and I have done in Nepal and Uganda as well as the work in Ethiopia with Save the Children, the Lancet series was cited as a key piece of evidence in each of those three countries in developing multi-sectoral nutrition plans. Lessons Learned. Evidence is important, but advocacy is key. Now the reason I say this is I can point to a lot of research over the years, over the past 50 years, really rigorous, high quality research that never was used and I think the difference with the Lancet series uh and feeding into the Scaling Up Nutrition movement, was there a fact that we had a series of champions for nutrition and this is critical.
The Scaling Up Nutrition movement was launched at the uh General Assembly meeting in September 2010. I had the honor of attending but more importantly there were high level officials who launched it, then Secretary Hillary Clinton of the US, Secretary Martin of uh Ireland uh and basically the message from Secretary Clinton and Secretary Martin was… [No Audio – Technical Difficulties] Okay basically, advocacy is key. So the um linking of evidence to advocacy is the uh in my uh uh experience, the formula for success uh and the message uh when SUN was launched is we know a lot, let’s put it into action. So the rest of the story, um SUN (the Scaling Up Nutrition movement) had uh has a two-pronged strategy, direct nutrition interventions where the 2008 Lancet series provided a solid body of evidence but the second prong of the strategy was nutrition sensitive approaches and the rationale was that in order to achieve agreed upon global targets, which Shibani will talk about in a minute, in order to achieve agreed upon global targets we need both direct as well as nutrition sensitive approaches. The um following Lancet series, uh in 2013, so the second Lancet series, put more emphasis on generating evidence on nutrition sensitive interventions uh but despite this increased focus there were still large gaps in evidence and in the next slide what we see is there have been a lot of meta-analyses of effects of, in particular nutrition agriculture strategies, and in 2014, Patrick Webb and I did a synthesis of these meta-analysis and I’d like to to look at some lessons learned in the next slide.
Here we review the individual uh meta-analyses that were done, I’m not going to go through each of the slides, but what’s very clear is where you go from Ruel to Birdie to Leroy and Frangillo, the kind of agricultural activities that were included in the meta-analysis varied, that’s number one. Some of the meta-analyses were systematic reviews some were not. The fourth column, number of studies retained for review as you can see we’re quite small and if we go to the next slide, uh third one down, Masset et al. in 2011, of the 7239 studies identified only 23 were considered of sufficient quality to be included in the systematic review, and what’s interesting in the Masset et al. column, the last column, uh quote “we found no evidence of impact on prevalence rates of stunting wasting and underweight among children.” um I would um report that that’s the wrong metric for looking at impact of agricultural evaluations and I think globally where we are is that stunting is seen as quite a good indicator of overall development but it’s a quite reductionist point of view to look at an agricultural intervention as and stunting as the outcome uh and the uh as a result of the various um impact measures for nutrition that were used, the conclusions of the lessons learned in the next slide from the Webb and Kennedy paper was the empirical evidence for plausible and significant outcomes remains disappointingly scarce, however, the absence of evidence should not be equated with evidence of no impact and what we really need to do is what I call unpackage what is meant by nutritional status. The analysis of agriculture nutrition have used varying indicators of outcome including nutrient density, diet diversity, diet as a whole (and that could be a household, at child level), anthropometric measures and the measure that is used really needs to be linked to the specific agricultural intervention under study. Now also we need to consider and I think this has rarely been done from the point of view of the particular agricultural strategy or intervention, are we looking at a short-term versus a medium- to longer-term metric. and that’s critical. Now the landscape landscape has changed a bit as we see in the next slide, where a year ago the UN declared a Decade of Action on Nutrition and this Decade of Action is being championed by FAO and WHO with help from other parts of the UN system as well as national level governments, as well as international NGOs. And what we see in the uh next slide is the 2017 Africa Nutrition Map, the large circles, and let’s go to Ethiopia in East Africa, the dominant um nutrition problems are still the classic under nutrition problems but if you look at the triangles in many many countries what we see emerging is problems of overweight and obesity and chronic diseases and that those uh problems of overweight and obesity are increasing at exponential rates.
So when the UN and others talk about malnutrition in all its forms, we’re talking about under nutrition, micronutrient deficiencies, as well as overweight and obesity and chronic diseases As a result of this, in the next slide what we see is when you look at stratifying nutrition problems by typologies of food systems and this comes out of the Global Nutrition Report where they had five food systems going from on the the left of my slide the industrialized food system down to the rural food system, what we see in the industrialized food system is uh very high rates of overweight in adults in the industrial, in the mixed, and transitioning food systems. Much lower rates in the food system classified as rural, however the flip side of this when we go to the next slide is that when we look at Vitamin A deficiency, stunting and anemia uh the opposite finding from the prior slide that you find in industrialized food systems, very low Vitamin A deficiency in children, low rates of stunting and lower rates of anemia in women. So that food systems are associated with a different mix of nutrition problems and this really speaks to the issue on the next slide of in an earlier slide I use the term unpackaged, really looking at the food system from the point of view of production all the way up to consumption and what are the effects at each stage of this system from production to storage to processing to retailing up to consumer behavior in affecting the food environments. And the food environments is the actual context that the consumer faces in making food choices and involves uh factors like food access, food availability, food promotion, information, and quality and safety and by unpackaging, we look at at each part of the food system where our points of entry for maximizing nutrition input – let me give you an example on the production system, maximizing nutrition effect or input might involve uh eliminating aflatoxin contamination. You’d have a different point of entry at the process and processing and packaging level so you maximize nutrition input and you minimize nutrition coming out of the system. For example storage, exchange and distribution, minimizing post harvest losses, makes a contribution to nutrition so really looking at the food system from the point of view of process indicators which when combined lead to or can lead to a significant impact on diets and ultimately possibly nutrition and health outcomes. And with that, I’d like to hand over to my colleague Shibani Ghosh to talk about other aspects of evidence. Thank you. [Shibani Ghosh] Thank you Eileen and um Good Afternoon everybody. I’m just going to take a different um take the presentation to a different place but um just as relevant to what Eileen has been speaking uh but before I go there I think Eileen has set the stage to talk about um um you know, what are the global issues um and what are the prevalences of the different nutrition conditions look like. I think it’s important to really just look at local nutrition targets for 2025 and these are a combination of the World Health Assembly targets which are focused really on under nutrition, as well as what is called as a Global Monitoring Framework for Preventing and Control of NCDs and as Eileen has mentioned this is something that is really an emerging issue in many transitioning countries, emerging economies as well as in developing economies. So you have very clear targets that have been set up for stunting, for wasting, for overweight, for anemia, exclusive breastfeeding, low birth weight, adult overweight, diabetes and adult obesity. So the question that always comes to mind, whether you’re in research or programming, is how do we track these targets? How do we know that we are achieving or not achieving these targets? And um this is a table that comes from, um again I’m using the source is actually the Global Nutrition Report, this is the older one, which is the 2014 one, um and not to go through this entire table, but what you’re looking at here is the global program against global nutrition targets as of 2014. And what you’re seeing on the uh uh basically the fifth column, which says on or off course, it’s going to be pretty much off course on all the different targets, uh and this is the global numbers okay, you can actually, you might actually want to know what’s happening at the country level and this is where it gets very sticky and that’s really where I want to focus on in terms of data and evidence. One of the biggest issues is when you go to the country level is, you may or may not have current data or you may have no data at all. So what you’re looking at here is a slide from Global Nutrition Report 2014, and I can assure you that this situation has not really changed as much. What you you’re seeing is the four major indicators of the nutrition targets which is reduced stunting, prevent overweight, cut wasting and halve anemia, and I don’t know how the colors are appearing on your end, but the dots that are appearing blue or it says um “no data,” basically that’s telling you that for the, in the case of reducing stunting, 84 countries did not have data as of 2014. So we have no way of knowing whether these countries are actually meeting their targets or not. This goes also on the other spectrum of uh nutrition which is prevention of overweight and in children 86 countries do not collect data that will even tell us what the prevalence of overweight is at that time point. So on, the only indicator that has data is essentially anemia. And you can see that the orange dots are telling you uh many of these countries out of the 193 that have ratified these nutrition targets are actually not on course, um so that’s something to think about that we really, one of the big questions and one of the big comments that comes up in pretty much many many many applied uh nutrition and programming meetings is that we really need data at the global level and at the country level and data that’s collected on a regular basis for us to understand how these countries are achieving their targets, or not. So I’m going to just shift gears a little bit and go in the direction of data for what and and go towards the the idea of generating evidence and how do we generate evidence. So data is needed, as I mentioned my first example was the global nutrition targets, that’s basically generating nutrition information in the form of surveillance that will allow you to follow populations over time to see whether you’re meeting your targets or not.
Now another reason you need data is to understand whether it’s at the country level, at the sub-national level, at a regional level, if the implementation of a policy initiative or one program or several programs are actually achieving their intended um outcome and that’s sort of slightly different from the idea of doing an impact evaluation. So the second bullet really focuses on this concept of implementation research which is not just if they have achieved the outcome, if they haven’t achieved the outcome, why they haven’t achieved the outcome, because it could be that there have to be certain tweaks that have to be made to the policy initiative or to the program. And in the fourth bullet I put in what really is the focus of a lot of applied nutrition research which is to look at new strategies and innovation and these are obviously linked to generate evidence that is going to guide policy but I consider this as a separate bullet in which you’re doing studies um that are testing efficacy or effectiveness of new innovations or new strategies that might eventually go down in the pipeline towards programmatic um programmatic activities or policy initiatives. So, irrespective of what your data is for, and irrespective of the type of your study, there are certain basic elements when you design and implement a study. So it doesn’t matter if you’re doing a large national cross-sectional survey or you’re doing a very small study in which you’re following one community or three communities the first thing is you really need to have a research question and or an objective if it’s a sort of national surveillance and as well as a hypothesis as to what is it that you are trying to um to study and obviously linked to that comes the fact that you need to have an outcome of interest. So in the case of and Eileen has pointed out that the idea of looking at stunting might be too reductionist but obviously that is one of the indicators and outcomes of interest that a lot of nutrition sensitive interventions and studies are using um in developing the research question. The key element in trying to answer your research question and studying your outcomes of interest is to study design. This is extremely imperative and we spend very little time on really deciding what the study design should be and even less time is spent on sampling strategies and sample size estimation. This is really, these are two very crucial points, we could go into each and every one of these and we can have a long uh discussion about it but there are really really important things besides the research question and hypothesis of your study design, your sampling strategy and sample size estimation. There are obviously statistical analysis that need to be conducted and these should, really this idea of doing your statistical plan while you are planning the study is is quite important because you may land up with data that you cannot utilize. And finally obviously you have study implementation. So just to drill down in a few of the elements of a study you have the research question. There are certain essential characteristics and some of the information that I’m using um is fully, I’m pulling it out of clinical research, but I think it has a lot of relevance for programmatic research or for applied nutrition research as well.
So obviously you have to have a research question that is feasible but at the same time it has to be interesting and innovative or novel as in it’s adding to the literature um but you have to also keep in mind that it has to be ethical and relevant or worth doing. And the same thing applies for in terms of your hypothesis, your hypothesis is always based on your research question and it should state very very clearly how, what type of a relationship are you looking at, what population are you going to um study, what variables are going to be studied, as well as what are you expecting, whether if for instance you’re looking at an intervention, what is the difference you’re expecting to see because of the implementation of that intervention? Most importantly any statement that you write in a hypothesis has to be measurable So this is just to give you this is a list of different study designs that are used in the applied realm and you have observational studies and uh many of you may be familiar with the ENGINE Birth Cohort study that was a prospective cohort study where we followed a group of pregnant women um and the outcome of interest was to look at the birth outcomes in their infants and to look at heights-for-age at 12 months of age. So if you’re following people over a period of time but it’s an observational study you’re not implementing an intervention so in this case ENGINE was being implemented around this study, the study itself was not in implementing any ENGINE intervention. You have cross-sectional studies for the DHS for example, the Demographic Health Survey would be classified as a cross-sectional study and then you have what is called as a case control study where you would have defined cases, let’s say you have you would really like to see um what is the difference between stunted children and non-stunted children in a population and then you would define the stunted cases and the non-stunted cases and do a measurement of one time point. um I think many of you will be familiar with the experimental study design such as the randomized control trial. uh you have also qualified experimental trials which are not uh randomized, they are usually non-random intervention assignments. In both cases you will have a treatment and a comparison group and you will have you need to measure pre- and post-test measurements so you can do a comparison before and after and usually the analysis is called difference in difference estimation. So this is just to give you a different schematic of epidemiological study designs, um so it kind of puts it a little bit more in a, in a total sense of what I, what what the different designs look at. So the first one is really a cross-sectional uh design where the exposure and outcome are measured at the same time. So you have the only um the only uh answer you can get out of this kind of a study is an association that are these two related to each other. You cannot make any causal inferences using a cross-sectional design. The second one is a case control and a retrospective study, again you’re not looking at causality here these three designs are essentially aimed at looking at ah is the outcome related to the exposure and you’re doing it in different ways.
So the central one is where you look at the outcome, let’s say you’re looking at something right now and you try to do a retrospective analysis and this is very hard to do and it’s and has less rigor where you go back into the exposure so if you had really good hospital data for instance you could um determine if the stunted kids what were their birth weights if the birth rates were recorded and in most cases they may not be recorded so this is why a retrospective study becomes really hard to implement. And then you have the prospective study where you follow a group of households or a group of women or a group of children for a specific exposure and then you see what happens to them um at a certain time point um you’re following them prospectively. And so to so what does that mean and in just taking it to where Eileen was coming from in the Lancet series, there is a hierarchy of research evidence, so you have different designs, you have different methods, you can implement a study, those designs have to be really linked to what your research question is but we know that systematic reviews which are a compilation of different studies are considered as the highest level of rigor and I’m going to go into some of the first three um slices of this pyramid so we can we can look at each of these and why these are considered at the highest level. So you have the systematic reviews, randomized control trials, and cohort studies, these are considered as the top three tiers of the hierarchy of research evidence. Case control studies are the next level, case series usually these are done in clinical settings in a hospital when doctors are doing research and they write up reports and lastly you have editorials and expert opinions and um in this case the author puts them at the bottom but these are often allowing us to think about uh future research ideas because you have experts who have um a lot of experience who are probably putting all the years of experience down into an editorial or an expert opinion. So I do not discount any of the lower tiers, I think they just help us build the case um to do more rigorous research. So I’m just going to go through the first three layers – the systematic review and the meta-analysis, now everybody is familiar with Lancet 2008 and 2013, and each and every paragraph in the Lancet 2008 papers that you will read, or 2013, for each and every nutrient that they reviewed or every approach or every intervention was the findings were based on systematic reviews and meta-analyses. So basically if you had those 10 interventions, each of them had their own systematic review and their own meta analysis. And so that’s a lot and a lot of different studies that were reviewed and compiled together so that in itself explains why a systematic review is considered as the highest level of evidence because you’re able to bring together a lot of different studies conducted by different groups which may have differing results but you’re able to come up with a common answer.
Now there are two things here to keep in mind, the systematic review is something that is a systematic way of reviewing published and unpublished data. In addition to that you could utilize the findings of those studies and then do a meta-analysis, which essentially requires a pretty advanced statistical procedures, so there are two different components to uh so the systematic review can be done by itself, you don’t necessarily have to do a meta-analysis, but generally a meta-analysis is always done along with a systematic review. So meta-analysis is essentially going to give you greater statistical power and the conclusion is going to be stronger than the analysis of a single study. Essentially it helps us to establish statistical significance if you have conflicting results. And in the case of zinc for instance there have been a lot of conflicting results about the relationship of zinc with linear growth.There hasn’t been a lot of conflicting results about zinc and diarrhea treatment for diarrhea so essentially any meta-analysis or systematic reviews if you read about zinc you will find that there is absolute recommendation that you can go ahead and use zinc as a treatment for diarrhea but you find very little evidence you the the the meta-analysis is not giving us the confidence to say that you should use zinc as a supplement for linear growth. It is connected to linear growth but it is not currently recommended as an intervention.
So these are some of the advantages and disadvantages as Eileen pointed out and that was a very good example of the Masset study which they actually found 7000 titles on their first um review and of that by the time they had whittled down through and defined which ones were the highest rigor studies they came down to 23. So 7000 to 23. That’s a lot of reading, that’s a lot of, it’s exhaustive to go through and a lot of usually these kind of research initiatives uh have a lot of graduate students fortunately for that um but so but they take time. The good thing is they’re less costly because they review all the prior studies um and generally less time is required than if you have to just consider one single study and they’re more reliable and accurate, but they are very time consuming as well. And as I’ve already mentioned about the meta analysis this is a very strong tool to bring together conflicting data and able to sort of say okay here you have 10 studies that showed no effect of Vitamin A and you have 10, 15 studies that showed an effect of Vitamin A if you bring them together and use statistical methods what is the net result and does that give the confidence to the experts that then Vitamin A supplementation is essential and necessary for uh for children under five years of age. So that’s just to give you an example, each and every nutrient in the Lancet series has been reviewed and treated through meta-analyses and systematic reviews And this is just to show you the different stages of a systematic review and a meta-analysis, um this in itself is a separate presentation by itself, but I’m just sort of sharing it with anybody who might be interested. There’s a it’s a very nice article that I’ve um used here and it’s a small brief article that this author goes through and outlines exactly what you need to do um to do a systematic review and a meta-analysis. Randomized Control Trials. I think again very many of you will be familiar, this is where you randomly assign participants um to an intervention or an experimental group versus a control group and the randomization can occur at an individual level or at a community level. Most of the work that we do in applied nutrition and in programmatic interventions is going to be at the community and the household level for any of those studies will be called as cluster randomized studies. Now a key element of a randomized study is that it has, you need to blind um people off the treatment that they’re receiving and you ideally should also be blinding the investigators so there is no bias generated. Unfortunately that’s not always possible, so you’ll find a lot of the times that the blinding is single where the participants are blinded from the treatments but not necessarily the investigators. And again there are advantages, this one of the biggest advantages, considered this is the golden standard in nutrition research, because if you randomly assign individuals communities or groups to receiving Treatment A versus Treatment B, what you find is that you will have very little confounding, you will have very little effect of other variables outside your intervention value.
The disadvantages are it’s an extremely expensive study to do, most randomized studies require intensive monitoring of compliance and there could be volunteer biases because people who agree to participate in the study might actually be interested in the study and that creates a bias. And a key element, because we use RCTs we all apply nutrition and programming we always consider RCTs as a a way of defining causal causality and in clinical medicine they actually are concerned of the fact that may not reveal causation because of the heterogeneity within a population. So if you’re dealing with a homogeneous population um of let’s say seeds or crops it’s very easy, you can definitely show causality, however when you’re dealing with human beings, each and every person, even whether you’re within the same community or the same tribe or the same clan, each individual is different from the other in the way they react to treatments to whether it’s medications, to supplementary foods, to supplements um and so the concern that is and this is a much higher level statistical concern is that it may not necessarily reveal causation. And the last point is there is there can be loss to follow up because intervention studies um are over a period of time um you might have a loss to follow up. And last but not the least which is what we have done in the ENGINE um program is the cohort study where you follow groups, in this case we followed pregnant women um prospectively and we did status evaluations every three months and while this is on the third year of.. [aside have they left the meeting? are they on? okay, sorry all right, because it just kept saying it’s coming in and on] Okay, sorry about that everybody, I just thought you had left the meeting. So essentially when this study is conducted um as as you progress along you’re going to measure the outcome and the advantage of this is that you can measure the outcome over a period of time and you can have um repeat measurements and will allow you to understand how the exposure is related to the outcome. So it sort of can be a very robust way of assessing the relationship of exposure to outcomes. I’m just going to skip over and talk a little bit about the how that relates to the ENGINE Birth Cohort study that was conducted. We used the cohort design to conduct an observational study where we created a quasi-experimental design where we were in ENGINE woredas and we matched it to a non-ENGINE woreda so it allowed us to see how these uh two populations, which is ENGINE-exposed population and a non-ENGINE-exposed population, um we observe them through their pregnancies and down to the time their child’s on 12 months of age. So we used a sort of a quasi-experimental approach and linked it to the observational study and our outcome of interest was length-for-age, height-for-age, and stunting, and the aim was every single data point that was collected was aimed at generating evidence that would guide future policy. And um just to give you an idea, and this is some, some of you have seen this for sure but just to give you the breath and scope of these kind of studies, this is the ENGINE Birth Cohort Study and we had seven time points of measurements, we went every three uh we went um basically at enrollment when the mother was pregnant and when she had, within three days after she gave birth, and then every three months after birth up to 12 months of age. And a key point that I wanted to point out was because we were comparing an ENGINE population to a non-ENGINE population, we had to estimate the sample size to make a comparison of these different groups but we also had to account for attrition, which is one of the big issues not just in RCTs but also in cohort studies so it’s a fairly large sample size because of that. What we were very fortunate with was, like in the last column as you can see is that we had such a well implemented study that the attrition was very very low, except in the last time point where we were at 80 percent, most of the time were almost at 90 percent of our participation which is which is a very high participation rate. So and the importance of thinking through observational studies um that are looking at a longer period of time is really illustrated in this graph which is my last slide here, where this is this is essentially identified as a stunting syndrome by Andrew Prentegrast and Gene Humphreys (they’re working on in Zimbabwe on the SHINE trial) and I think they’ve illustrated this very well where you can see that we tend to look at relationships in a linear fashion but in when it comes to stunting um it’s actually a cyclical um relationship, it’s sort of it’s you have to go from neonates to two years of age to school to puberty and adults and then you go back to conception.
So when you’re using study design it’s really important So when you’re looking at a um when you’re planning a study it’s really important to understand what your research question is and in this case if our research question is and as it was was to understand what’s happening in the first two years of life, it was really important to have a design that will allow us to not just address the first two years of life but what was happening in pregnancy. Now uh advantage of a lot of the times of these kind of studies which I didn’t bring up and it’s not in the advantages listed in any of them, is that particularly with cohort studies or RCTs um is that there is an opportunity because they have been exposed to an intervention or because you have followed them over a period of time, at some point it’s you you as researchers might be able to go back to those households and collect data at a later time point maybe when the children are in school or when they’re adults and there’s some classical studies that have been done which have shown some very very interesting results of early life interventions and one of them is in Guatemala and called the INCAP studies where they found the effect of the intervention of providing incaparina which is a complementary food to children, they found the effects of that in adulthood on work productivity and wages. So so in the end it I think the point that I want to make here is um that generating evidence is really important, we need a lot of data, we need surveillance, particularly for uh when you want to understand if you’re meeting those nutrition targets or not and it’s really really important to consider what your research question is when you’re planning and implementing a study. Thank you so much and I think we’ll be open for questions. [Meghan Kershaw] The first question that we received is “In the nutrition and food system impact pathway, one of the impact is social equity impact, what does it mean?” [Eileen] In economic terms, um people talk about winners and losers, and if you look at uh interventions generally, but agricultural interventions, early adopters tended to be uh less risk averse etc, so in the food system what that means is we’re trying to um provide uh access, uh quality, across the food system so you don’t have winners and losers. I mean the losers in large part tend to be lower income populations, uh sometimes certain subgroups women and children, so looking at the food system as a way of delivering sustainable diets in a way that no one has no one is left behind. [Meghan] “Can we do meta-analysis without systematic review? vice versa?” [Shibani] um a meta-analysis generally is linked to a systematic review because you do have to go through the steps of identifying those articles um and and find the um so you have in a way actually they go hand in hand. You can do a systematic review by itself without applying the statistical and analytical tools um so you can just do it it becomes it’s sort of a more systematic literature review uh and and then you add the meta-analysis to it. So I don’t think that those, you can’t do a meta-analysis without a systematic review.
Wedge type allocation? [Meghan] “Yeah, you say something about wedge type allocation, is it a design or?” [Shibani] Yes, um so yes, I would I would have liked to go in a little bit more into sort of the experimental designs and the wedge type is one of them which, in which you essentially, um particularly on a programmatic um application, what you can do is in your population um provide the intervention to one part of your population, whereas the other part serves as a comparison and then you roll out once you have enough time to study or do the research on the differences between the two um populations on a specific outcome. So um so yeah so the wedge type design is considered as an alternative to actually implementing a randomized control trial when you randomly assign individuals or communities or clusters to an intervention or not. [Meghan] Next question is, “what is the scope for real-time monitoring & evaluation to generate evidence?” [Shibani] There is absolutely a lot of scope for monitoring and evaluation data to be used as evidence. The key thing is um to ensure that your system is set up to generate the data. um I, I’m not familiar with um, I’m familiar with monitoring and evaluation frameworks and I think that they are a very good system to um, they look, they are very robust, the question is if the data is actually um collected against that framework. And if it is, then yes, it would be a very very good um source for generating evidence. As a researcher obviously then that goes into the realm of if you’re using that data it needs to be de-identified so that you’re exempt from IRB, so there are those kind of technical issues to consider but it definitely is a source. My key uh point would be is ensuring that the data is that is being collected is of good quality and its rigorously collected. [Eileen] I’d like to add something there, in a specific example, uh in the late 1980s, I was working with the government of Malawi on a monitoring evaluation system and uh there was a mealy bug infestation of the crop in northern Malawi and the the normal M&E uh wasn’t picking it up, but uh uh feedback from relatives in the north to the capital said, you know, they were being devastated, kids are dying, and so I think you know, uh getting back to your comment earlier about innovations, I mean I think the experiential data is also important uh where sometimes the quantitative data on a timely basis may not be indicating the nature of the situation. So I think you need to think again creatively about how you blend both. [Shibani] Right, right, yeah and I’m sorry I always have a quantitative hat on, but yes, you can actually also link your M&E to qualitative uh data collection so you can have both qualitative and quantitative data that would contribute towards generating evidence. But this is one thing that we’ve been at the Nutrition Innovation Lab, Patrick Webb and I would love to see an M&E system used more effectively because there is a lot of data generated. I wouldn’t know what the quality of the data is if we can ensure that the quality is of the highest level, that data would be extremely valuable for answering programmatic questions, particularly implementation research question.
[Meghan] The next question is, “What would be the best method to generate evidence for this particular project?” [Eileen] Growth through Nutrition? [Shibani] So I think the question I would have is, um generate evidence to answer what? So again we sort of come back to what would be the question um that needs to be answered relative to this project. Is it to uh determine if the project interventions are being implemented? So that would be an implementation research question. Or is it to understand if um the uh the project has achieved this outcome? [Eileen] Yeah and if I could answer Shibani, I totally agree, I also think it’s important to look at at uh um integration of components, um I know the presentation on the ag-nutrition sounded a bit negative, it was not meant to be that way. I mean there have been success stories in ag-nutrition, but let me take one, um bio fortification orange sweet potato, uh where not only did they adapt the cultivation but uh Vitamin A status of children improved. Now the reason I point this out is that was not simply a stand alone agricultural intervention, there was uh education of the mother on behavior change, there was income generating activities for the mother uh and there was a dose response issue, uh uh households and mothers who had more exposure, the effect was greater. So I think you need to look at all of these issues uh and not think of agriculture in isolation and what’s I think Growth through Nutrition uh the the overall framework is linking, you know, linking WASH, nutrition, agriculture, uh behavior change, and looking at um exposure and what at least the continuum of these complements of activities, what do you see on various outcomes? I mean yes, stunting being one of them but a behavior change related to diet, behavior change related to WASH. [Meghan] Next question is “What do you think is a more appropriate method or methods to generate evidence regarding agriculture impact on nutrition?” [Shibani] Is it a method or what it would be the right outcome? That would be my question. [Eileen] Well let me, let me just kind of, while somebody’s answering Shibani’s question, it really depends on how you’re defining nutrition. And I’m not meaning to be facetious and I think part of the problem in the ag-nutrition literature is the outcomes have been all over the map. uh if you have an intervention, uh say home gardens has been popular in some of what uh for example Marie Well in her uh meta-analysis uh analyzed, and why would you expect home gardens in the uh initial time period where you have your first harvest from the home garden, basically uh leafy green vegetables, why would you expect that, in a short period of time be related to anthropometric indicators and children? What you really should be looking at is uh did the um nutrient density of the child’s diet, if that’s the the uh objective of interest change, uh did overall consumption at the household level change? Did income change? So I think this gets back to defining what we mean by nutrition and it’s been a real frustration, the literature that nutrition is used very broadly without any identification of the specific metric that we need to use. [Shibani] Yeah and I think just to add on to what Eileen was uh has just said, um the the issue that um that occurs when you’re trying to link agriculture to nutrition, um is that it’s it’s a very long pathway and as if everybody is familiar with the UNICEF framework of the basic underlying and immediate causes, you should know that that’s when those are not causal uh pathways, those are uh those are proposed pathways, and in that itself you can see one of the issues is that it’s really hard to measure um that relationship from say poverty or even poverty alleviation because of agriculture, to nutrition status of the child because you could have so many other things that could confound that relationship and for an instance, if the child has diarrhea, if the child has EED, then because it’s poor WASH, you can invest a lot in agriculture but if with poor WASH you will land up with a lot of diarrhea and chronic inflammation that’s not going to allow that child um to thrive. And so I think the um one really needs to understand what the underlying biological issues are and think about looking at intermediate outcomes where as Eileen had said, nutrient density, dietary diversity, do these elements improve in these households? A key thing that we have been finding in some of our Asian um studies, uh for instance in Bangladesh, we have households where we’re working in the Feed the Future zone of influence and we have um we have surveyed 3000 households about three times and they’re all rural uh households, they’re all farmers, 90 of them report having farming income but all of them are as reliant on the market for their food as they are on their home production. So I think that that’s also an element that needs to be considered within the context of agriculture to nutrition and it’s not just home-based production, it’s also access to good quality food and produce in the market.
um It’s very interesting that we find that whether it’s vegetables or fruits or pulses or animal source foods, um there is an equal reliance on household production and purchases from the market. [Eileen] The the other issue is and I think it’s how we measure some of these factors, Shibani and I were talking about this, um in some work I did in southwestern Kenya we found no relationship between having a latrine at the household level and uh child morbidity or child growth. When we’re back to revisit this and I put this in the area of delivery science, what we found was a disproportionate share of households that had latrines weren’t using them, so we were asking the wrong question, it’s not “do you have latrine?,” “do you use it, if not why?” and then getting getting at that context which I think is where the uh, what I call, ethnographic data can be very helpful in explaining some of the quantitative findings that are being found. [muffled speaking] [Shibani] Okay, Are there any other questions? [Meghan] Yes, next question is um, “is that hierarchy, the hierarchy of evidence applies mainly for findings from quantitative, how do we systematically present qualitative?” [Shibani] Oh yes, that is a very good question and maybe that should be a topic for the next webinar. I don’t know the answer… you have some thoughts? [Eileen] Well yeah I mean I my uh I’ve been involved in a lot of qualitative work primarily in the US, uh and uh uh focus group type type of work, ethnographic kind of work, which uh we never put forward as being representative but it does it does look at issues through the lens of the intended beneficiary and it’s, it’s it’s very helpful. The one challenge you have is and I spoke about this early in the presentation is that um policy officials, finance kind of people, uh tend to be more persuaded by cost effectiveness analysis and the challenge with qualitative data which can be very valuable is how do you then merge that into a cost effectiveness analysis to say this works. um and I think we need a lot more work on that. [Shibani] So and and just I mean thinking this through from the perspective of the hierarchy of evidence, I think it’s not as much as the hierarchy of evidence because qualitative is um methods uh you have key informants, you have focus group discussion, um and um essentially all the methods that you have, did you really rely on the quality of um the facilitator and the communicator so it’s really the person who’s actually conducting these uh focus groups and and key informant interviews that should be able to synthesize um and then whoever is going to be analyzing the data is able to find the pattern. So I would not put the qualitative methods into the kind of pyramid approach, I would sort of, you really have to take a qualitative approach to assess the quality of data that’s coming from a qualitative study because it’s really it’s a it’s a very different lens you have to wear and that’s something that I immediately put on because I work with so much quantitative data.
The other thing that I feel is very important is um qualitative data um and quantitative data together, so like a mixed message approach, is a very very good way of going, um so it gives you the data that you need, as Eileen is saying, to support um your qualitative findings um with policymakers or people who are working in the Ministries of Finance, but at the same time the qualitative data might actually help you understand why you are seeing something or not seeing something. So um so yeah, I think that that would be my takeaway. [Meghan] Next question is “how realistic is it to use evidence generated from a single study or project for policy advocacy?” [Eileen] It depends where you I think where you are in the cycle. I mean that that gets back to the question we posed, when is evidence enough? um and again I’ll use an example from the US, um well actually, let me let me use a Vitamin A example, uh where um so it’s not so close to home. The original work out of Johns Hopkins, uh out of Indonesia, which uh suggested that um Vitamin A supplementation of preschool has resulted in decreased mortality. There’s a lot of disbelief because of that, but we’ve known for years that the Vitamin A deficiency, if left untreated was related to blindness so that was not, but this was the first time back in the 90s, there was a link between, 80s and 90s, a link between Vitamin A deficiency and mortality and there’s a lot of disbelief as it was the first study, the one study. uh What that generated was they didn’t, some people dismissed it but it generated uh number one, a uh an enormous amount of additional research on other studies which eventually wound up in a systematic review, uh Beaton and Martorell, showing that on average Vitamin A supplementation decreased mortality by 23 percent, but another part, that I think the rest of the story is if you look at the the Hopkins people, they were confident in their data so they didn’t just let it stand as one freestanding study, they actually used that information to adver- advocate for increased funding for Vitamin A research but also Vitamin A programs and we know what’s happened as a result of that. So again it depends, you know then of course, year after year there was more data generated but it’s tough in the beginning where you have a a single well done study that might be suggestive, you know, do you use it? Would you wait for more information? and I I think it’s um, if you’re confident in that piece of work hopefully use it for advocacy in an appropriate way.
Nature’s Way Sambucus Black Elderberry Gummies with Vitamin C and Zinc, 60 Gummies
Source: Save the Children in Ethiopia
Comments
Post a Comment