Evidence Use For Policy & Programs:Generating Evidence

[Eileen Kennedy] Welcome, we’re  delighted to be doing this webinar on Evidence Use for Policies and  Programs: Generating Evidence. We have four areas we’d like to discuss  in this webinar, Why do we need evidence? How is evidence generated? Levels  of Evidence and then Study Designs. Why do we need evidence? Put very very broadly  typically one would want evidence for evaluation of impact, cost effectiveness, advocacy. I’ll use  an example that illustrates each of these in the next slide. I have picked the Lancet series on  maternal and child health, the first one in 2008, and in the next slide we  see that through a series of systematic reviews the Lancet series  came up with very strong evidence for uh implementation in what are called 36 high burden  countries. Implementation of interventions that would improve maternal and birth outcomes, uh  in- uh interventions for newborn babies, and uh interventions for infants and children. In the  same 2008 series there also was a uh articulation of evidence uh for implementation in specific  situations. So that was the evidence. Equally important and in the policy community uh in  discussions in which I was involved, this was the the evidence that really created a lot of buzz,  the fact that not only did we have for pregnant women young children, not only did we have have  efficacious interventions but we could quantify the number of lives saved and we could also  quantify cost savings in DALYs which is Disability Adjusted Life Years, basically healthy years  accounting for premature mortality and disability. And it was that powerful combination of the impact  evidence as well as the uh economic analysis that really created a buzz in the community and  this led to, in the next slide as we see, Lancet’s series 2008 produced the evidence, it  was used as the the key body of evidence for uh to the momentum for the Scaling Up Nutrition  movement and uh research that Shibani and I have done in Nepal and Uganda as well as the  work in Ethiopia with Save the Children, the Lancet series was cited as a key piece  of evidence in each of those three countries in developing multi-sectoral nutrition plans.  Lessons Learned. Evidence is important, but advocacy is key. Now the reason I say this is  I can point to a lot of research over the years, over the past 50 years, really rigorous, high  quality research that never was used and I think the difference with the Lancet series  uh and feeding into the Scaling Up Nutrition movement, was there a fact that we had a series  of champions for nutrition and this is critical.

The Scaling Up Nutrition movement was launched at  the uh General Assembly meeting in September 2010. I had the honor of attending but more importantly  there were high level officials who launched it, then Secretary Hillary Clinton of the US,  Secretary Martin of uh Ireland uh and basically the message from Secretary Clinton  and Secretary Martin was… [No Audio – Technical Difficulties] Okay basically, advocacy is key. So the um linking  of evidence to advocacy is the uh in my uh uh experience, the formula for success uh and the  message uh when SUN was launched is we know a lot, let’s put it into action.  So the rest of the story, um SUN (the Scaling Up Nutrition movement)  had uh has a two-pronged strategy, direct nutrition interventions where the 2008  Lancet series provided a solid body of evidence but the second prong of the strategy was nutrition  sensitive approaches and the rationale was that in order to achieve agreed upon global targets, which  Shibani will talk about in a minute, in order to achieve agreed upon global targets we need both  direct as well as nutrition sensitive approaches. The um following Lancet series, uh  in 2013, so the second Lancet series, put more emphasis on generating evidence  on nutrition sensitive interventions uh but despite this increased focus there  were still large gaps in evidence and in the next slide what we see is there have  been a lot of meta-analyses of effects of, in particular nutrition agriculture strategies,  and in 2014, Patrick Webb and I did a synthesis of these meta-analysis and I’d like to to look  at some lessons learned in the next slide.

Here we review the individual uh meta-analyses  that were done, I’m not going to go through each of the slides, but what’s very clear is where you  go from Ruel to Birdie to Leroy and Frangillo, the kind of agricultural activities that  were included in the meta-analysis varied, that’s number one. Some of the meta-analyses  were systematic reviews some were not. The fourth column, number of studies retained  for review as you can see we’re quite small and if we go to the next slide, uh third one  down, Masset et al. in 2011, of the 7239 studies identified only 23 were considered of sufficient  quality to be included in the systematic review, and what’s interesting in the Masset  et al. column, the last column, uh quote “we found no evidence of impact  on prevalence rates of stunting wasting and underweight among children.” um I  would um report that that’s the wrong metric for looking at impact of agricultural  evaluations and I think globally where we are is that stunting is seen as quite a good indicator of  overall development but it’s a quite reductionist point of view to look at an agricultural  intervention as and stunting as the outcome uh and the uh as a result of the various um  impact measures for nutrition that were used, the conclusions of the lessons learned in the  next slide from the Webb and Kennedy paper was the empirical evidence for plausible and significant  outcomes remains disappointingly scarce, however, the absence of evidence should  not be equated with evidence of no impact and what we really need to do is what I call  unpackage what is meant by nutritional status. The analysis of agriculture nutrition have  used varying indicators of outcome including nutrient density, diet diversity, diet as a whole  (and that could be a household, at child level), anthropometric measures and the measure that is  used really needs to be linked to the specific agricultural intervention under study. Now  also we need to consider and I think this has rarely been done from the point of view of the  particular agricultural strategy or intervention, are we looking at a short-term versus  a medium- to longer-term metric. and that’s critical. Now the landscape landscape  has changed a bit as we see in the next slide, where a year ago the UN declared  a Decade of Action on Nutrition and this Decade of Action is being championed  by FAO and WHO with help from other parts of the UN system as well as national level  governments, as well as international NGOs. And what we see in the uh next slide is the  2017 Africa Nutrition Map, the large circles, and let’s go to Ethiopia in East Africa,  the dominant um nutrition problems are still the classic under nutrition problems but if you  look at the triangles in many many countries what we see emerging is problems of overweight  and obesity and chronic diseases and that those uh problems of overweight and obesity  are increasing at exponential rates.

So when the UN and others talk about malnutrition  in all its forms, we’re talking about under nutrition, micronutrient deficiencies, as well  as overweight and obesity and chronic diseases As a result of this, in the next  slide what we see is when you look at stratifying nutrition problems  by typologies of food systems and this comes out of the Global Nutrition Report  where they had five food systems going from on the the left of my slide the industrialized  food system down to the rural food system, what we see in the industrialized food system  is uh very high rates of overweight in adults in the industrial, in the mixed, and  transitioning food systems. Much lower rates in the food system classified as rural, however  the flip side of this when we go to the next slide is that when we look at Vitamin  A deficiency, stunting and anemia uh the opposite finding from the prior slide that  you find in industrialized food systems, very low Vitamin A deficiency in children, low rates of  stunting and lower rates of anemia in women. So that food systems are associated with  a different mix of nutrition problems and this really speaks to the issue on  the next slide of in an earlier slide I use the term unpackaged, really looking at  the food system from the point of view of production all the way up to consumption and  what are the effects at each stage of this system from production to storage to processing to  retailing up to consumer behavior in affecting the food environments. And the food environments is  the actual context that the consumer faces in making food choices and involves uh factors like  food access, food availability, food promotion, information, and quality and safety and by  unpackaging, we look at at each part of the food system where our points of entry for maximizing  nutrition input – let me give you an example on the production system, maximizing  nutrition effect or input might involve uh eliminating aflatoxin contamination. You’d  have a different point of entry at the process and processing and packaging level so you  maximize nutrition input and you minimize nutrition coming out of the system. For example  storage, exchange and distribution, minimizing post harvest losses, makes a contribution to  nutrition so really looking at the food system from the point of view of process indicators which  when combined lead to or can lead to a significant impact on diets and ultimately possibly nutrition  and health outcomes. And with that, I’d like to hand over to my colleague Shibani Ghosh to talk  about other aspects of evidence. Thank you. [Shibani Ghosh] Thank you Eileen and um Good  Afternoon everybody. I’m just going to take a different um take the presentation to a different  place but um just as relevant to what Eileen has been speaking uh but before I go there I think  Eileen has set the stage to talk about um um you know, what are the global issues um and what  are the prevalences of the different nutrition conditions look like. I think it’s important  to really just look at local nutrition targets for 2025 and these are a combination of the World  Health Assembly targets which are focused really on under nutrition, as well as what is called as  a Global Monitoring Framework for Preventing and Control of NCDs and as Eileen has mentioned this  is something that is really an emerging issue in many transitioning countries, emerging economies  as well as in developing economies. So you have very clear targets that have been set up for  stunting, for wasting, for overweight, for anemia, exclusive breastfeeding, low birth weight, adult  overweight, diabetes and adult obesity. So the question that always comes to mind, whether  you’re in research or programming, is how do we track these targets? How do we know that we are  achieving or not achieving these targets? And um this is a table that comes from, um again I’m  using the source is actually the Global Nutrition Report, this is the older one, which is the 2014  one, um and not to go through this entire table, but what you’re looking at here is the global  program against global nutrition targets as of 2014. And what you’re seeing on the uh  uh basically the fifth column, which says on or off course, it’s going to be pretty  much off course on all the different targets, uh and this is the global numbers okay, you can  actually, you might actually want to know what’s happening at the country level and this is where  it gets very sticky and that’s really where I want to focus on in terms of data and evidence. One of  the biggest issues is when you go to the country level is, you may or may not have current data  or you may have no data at all. So what you’re looking at here is a slide from Global Nutrition  Report 2014, and I can assure you that this situation has not really changed as much. What  you you’re seeing is the four major indicators of the nutrition targets which is reduced stunting,  prevent overweight, cut wasting and halve anemia, and I don’t know how the colors are appearing on  your end, but the dots that are appearing blue or it says um “no data,” basically that’s telling you  that for the, in the case of reducing stunting, 84 countries did not have data as of 2014. So we  have no way of knowing whether these countries are actually meeting their targets or not. This goes  also on the other spectrum of uh nutrition which is prevention of overweight and in children  86 countries do not collect data that will even tell us what the prevalence of overweight  is at that time point. So on, the only indicator that has data is essentially anemia.  And you can see that the orange dots are telling you uh many of these countries out of the 193 that  have ratified these nutrition targets are actually not on course, um so that’s something to think  about that we really, one of the big questions and one of the big comments that comes up in pretty  much many many many applied uh nutrition and programming meetings is that we really need data  at the global level and at the country level and data that’s collected on a regular basis for us to  understand how these countries are achieving their targets, or not. So I’m going to just shift gears  a little bit and go in the direction of data for what and and go towards the the idea of generating  evidence and how do we generate evidence. So data is needed, as I mentioned my first example  was the global nutrition targets, that’s basically generating nutrition information in the form  of surveillance that will allow you to follow populations over time to see whether  you’re meeting your targets or not.

Now another reason you need data is to  understand whether it’s at the country level, at the sub-national level, at a regional level,  if the implementation of a policy initiative or one program or several programs are actually  achieving their intended um outcome and that’s sort of slightly different from the idea of  doing an impact evaluation. So the second bullet really focuses on this concept of implementation  research which is not just if they have achieved the outcome, if they haven’t achieved the outcome,  why they haven’t achieved the outcome, because it could be that there have to be certain tweaks  that have to be made to the policy initiative or to the program. And in the fourth bullet  I put in what really is the focus of a lot of applied nutrition research which is  to look at new strategies and innovation and these are obviously linked to generate  evidence that is going to guide policy but I consider this as a separate bullet in which  you’re doing studies um that are testing efficacy or effectiveness of new innovations  or new strategies that might eventually go down in the pipeline towards programmatic um  programmatic activities or policy initiatives. So, irrespective of what your data is for,  and irrespective of the type of your study, there are certain basic elements when you design  and implement a study. So it doesn’t matter if you’re doing a large national cross-sectional  survey or you’re doing a very small study in which you’re following one community or three  communities the first thing is you really need to have a research question and or an objective  if it’s a sort of national surveillance and as well as a hypothesis as to what is it that you are  trying to um to study and obviously linked to that comes the fact that you need to have an outcome of  interest. So in the case of and Eileen has pointed out that the idea of looking at stunting might be  too reductionist but obviously that is one of the indicators and outcomes of interest that a  lot of nutrition sensitive interventions and studies are using um in developing the research  question. The key element in trying to answer your research question and studying your outcomes  of interest is to study design. This is extremely imperative and we spend very little time on  really deciding what the study design should be and even less time is spent on sampling strategies  and sample size estimation. This is really, these are two very crucial points, we could go into each  and every one of these and we can have a long uh discussion about it but there are really really  important things besides the research question and hypothesis of your study design, your sampling  strategy and sample size estimation. There are obviously statistical analysis that need to be  conducted and these should, really this idea of doing your statistical plan while you are planning  the study is is quite important because you may land up with data that you cannot utilize. And  finally obviously you have study implementation. So just to drill down in a few of the elements  of a study you have the research question. There are certain essential characteristics  and some of the information that I’m using um is fully, I’m pulling it out of clinical  research, but I think it has a lot of relevance for programmatic research or  for applied nutrition research as well.

So obviously you have to have a research  question that is feasible but at the same time it has to be interesting and innovative  or novel as in it’s adding to the literature um but you have to also keep in mind that it  has to be ethical and relevant or worth doing. And the same thing applies for in terms of your  hypothesis, your hypothesis is always based on your research question and it should state very  very clearly how, what type of a relationship are you looking at, what population are you going to  um study, what variables are going to be studied, as well as what are you expecting, whether if for  instance you’re looking at an intervention, what is the difference you’re expecting to see because  of the implementation of that intervention? Most importantly any statement that you  write in a hypothesis has to be measurable So this is just to give you this is a list of  different study designs that are used in the applied realm and you have observational studies  and uh many of you may be familiar with the ENGINE Birth Cohort study that was a prospective cohort  study where we followed a group of pregnant women um and the outcome of interest was to look at the  birth outcomes in their infants and to look at heights-for-age at 12 months of age. So if  you’re following people over a period of time but it’s an observational study you’re not  implementing an intervention so in this case ENGINE was being implemented around  this study, the study itself was not in implementing any ENGINE intervention. You  have cross-sectional studies for the DHS for example, the Demographic Health Survey  would be classified as a cross-sectional study and then you have what is called as a case  control study where you would have defined cases, let’s say you have you would really like  to see um what is the difference between stunted children and non-stunted children in a population  and then you would define the stunted cases and the non-stunted cases and do a measurement  of one time point. um I think many of you will be familiar with the experimental study  design such as the randomized control trial. uh you have also qualified experimental trials  which are not uh randomized, they are usually non-random intervention assignments. In both cases  you will have a treatment and a comparison group and you will have you need to measure pre- and  post-test measurements so you can do a comparison before and after and usually the analysis is  called difference in difference estimation. So this is just to give you a different  schematic of epidemiological study designs, um so it kind of puts it a little bit  more in a, in a total sense of what I, what what the different designs look at. So  the first one is really a cross-sectional uh design where the exposure and outcome  are measured at the same time. So you have the only um the only uh answer you can get out  of this kind of a study is an association that are these two related to each other. You cannot  make any causal inferences using a cross-sectional design. The second one is a case  control and a retrospective study, again you’re not looking at causality here these  three designs are essentially aimed at looking at ah is the outcome related to the exposure  and you’re doing it in different ways.

So the central one is where you look at the  outcome, let’s say you’re looking at something right now and you try to do a retrospective  analysis and this is very hard to do and it’s and has less rigor where you go back into the  exposure so if you had really good hospital data for instance you could um determine if the stunted  kids what were their birth weights if the birth rates were recorded and in most cases they may  not be recorded so this is why a retrospective study becomes really hard to implement. And then  you have the prospective study where you follow a group of households or a group of women or a  group of children for a specific exposure and then you see what happens to them um at a certain  time point um you’re following them prospectively. And so to so what does that mean and  in just taking it to where Eileen was coming from in the Lancet series, there  is a hierarchy of research evidence, so you have different designs, you have  different methods, you can implement a study, those designs have to be really linked  to what your research question is but we know that systematic reviews which  are a compilation of different studies are considered as the highest level of rigor and  I’m going to go into some of the first three um slices of this pyramid so we can we can look at  each of these and why these are considered at the highest level. So you have the systematic reviews,  randomized control trials, and cohort studies, these are considered as the top three tiers of  the hierarchy of research evidence. Case control studies are the next level, case series usually  these are done in clinical settings in a hospital when doctors are doing research and they write up  reports and lastly you have editorials and expert opinions and um in this case the author puts them  at the bottom but these are often allowing us to think about uh future research ideas because you  have experts who have um a lot of experience who are probably putting all the years of experience  down into an editorial or an expert opinion. So I do not discount any of the lower tiers, I  think they just help us build the case um to do more rigorous research. So I’m just going to go  through the first three layers – the systematic review and the meta-analysis, now everybody  is familiar with Lancet 2008 and 2013, and each and every paragraph in the Lancet 2008 papers  that you will read, or 2013, for each and every nutrient that they reviewed or every approach  or every intervention was the findings were based on systematic reviews and meta-analyses.  So basically if you had those 10 interventions, each of them had their own systematic  review and their own meta analysis. And so that’s a lot and a lot of different studies  that were reviewed and compiled together so that in itself explains why a systematic review is  considered as the highest level of evidence because you’re able to bring together a lot of  different studies conducted by different groups which may have differing results but you’re  able to come up with a common answer.

Now there are two things here to keep in  mind, the systematic review is something that is a systematic way of reviewing published and  unpublished data. In addition to that you could utilize the findings of those studies and then  do a meta-analysis, which essentially requires a pretty advanced statistical procedures, so  there are two different components to uh so the systematic review can be done by itself, you  don’t necessarily have to do a meta-analysis, but generally a meta-analysis is always  done along with a systematic review. So meta-analysis is essentially going  to give you greater statistical power and the conclusion is going to be stronger  than the analysis of a single study. Essentially it helps us to establish statistical  significance if you have conflicting results. And in the case of zinc for instance there have been a  lot of conflicting results about the relationship of zinc with linear growth.There hasn’t  been a lot of conflicting results about zinc and diarrhea treatment for diarrhea so essentially  any meta-analysis or systematic reviews if you read about zinc you will find that there is  absolute recommendation that you can go ahead and use zinc as a treatment for diarrhea but you find  very little evidence you the the the meta-analysis is not giving us the confidence to say that you  should use zinc as a supplement for linear growth. It is connected to linear growth but it is  not currently recommended as an intervention.

So these are some of the advantages and  disadvantages as Eileen pointed out and that was a very good example of the Masset  study which they actually found 7000 titles on their first um review and of that by the time  they had whittled down through and defined which ones were the highest rigor studies they came down  to 23. So 7000 to 23. That’s a lot of reading, that’s a lot of, it’s exhaustive to go through  and a lot of usually these kind of research initiatives uh have a lot of graduate students  fortunately for that um but so but they take time. The good thing is they’re less costly  because they review all the prior studies um and generally less time is required than  if you have to just consider one single study and they’re more reliable and accurate,  but they are very time consuming as well. And as I’ve already mentioned about the meta  analysis this is a very strong tool to bring together conflicting data and able to sort of  say okay here you have 10 studies that showed no effect of Vitamin A and you have 10, 15 studies  that showed an effect of Vitamin A if you bring them together and use statistical methods what is  the net result and does that give the confidence to the experts that then Vitamin A supplementation  is essential and necessary for uh for children under five years of age. So that’s just to give  you an example, each and every nutrient in the Lancet series has been reviewed and treated  through meta-analyses and systematic reviews And this is just to show you the different stages  of a systematic review and a meta-analysis, um this in itself is a separate presentation  by itself, but I’m just sort of sharing it with anybody who might be interested. There’s a  it’s a very nice article that I’ve um used here and it’s a small brief article that this  author goes through and outlines exactly what you need to do um to do a systematic review and a  meta-analysis. Randomized Control Trials. I think again very many of you will be familiar, this  is where you randomly assign participants um to an intervention or an experimental group versus a  control group and the randomization can occur at an individual level or at a community level. Most  of the work that we do in applied nutrition and in programmatic interventions is going to be at  the community and the household level for any of those studies will be called as cluster randomized  studies. Now a key element of a randomized study is that it has, you need to blind um people off  the treatment that they’re receiving and you ideally should also be blinding the investigators  so there is no bias generated. Unfortunately that’s not always possible, so you’ll find a lot  of the times that the blinding is single where the participants are blinded from the treatments  but not necessarily the investigators. And again there are advantages, this one of  the biggest advantages, considered this is the golden standard in nutrition research, because  if you randomly assign individuals communities or groups to receiving Treatment A versus Treatment  B, what you find is that you will have very little confounding, you will have very little effect of  other variables outside your intervention value.

The disadvantages are it’s an extremely expensive  study to do, most randomized studies require intensive monitoring of compliance and there  could be volunteer biases because people who agree to participate in the study might actually be  interested in the study and that creates a bias. And a key element, because we use RCTs we all  apply nutrition and programming we always consider RCTs as a a way of defining causal causality and  in clinical medicine they actually are concerned of the fact that may not reveal causation because  of the heterogeneity within a population. So if you’re dealing with a homogeneous population  um of let’s say seeds or crops it’s very easy, you can definitely show causality, however when  you’re dealing with human beings, each and every person, even whether you’re within the same  community or the same tribe or the same clan, each individual is different from the other  in the way they react to treatments to whether it’s medications, to supplementary foods, to  supplements um and so the concern that is and this is a much higher level statistical concern  is that it may not necessarily reveal causation. And the last point is there is there can be loss  to follow up because intervention studies um are over a period of time um you might have a loss  to follow up. And last but not the least which is what we have done in the ENGINE um program is  the cohort study where you follow groups, in this case we followed pregnant women um prospectively  and we did status evaluations every three months and while this is on the third year of..  [aside have they left the meeting? are they on? okay, sorry all right, because it just  kept saying it’s coming in and on] Okay, sorry about that everybody, I  just thought you had left the meeting. So essentially when this study is conducted  um as as you progress along you’re going to measure the outcome and the advantage of  this is that you can measure the outcome over a period of time and you can have um repeat  measurements and will allow you to understand how the exposure is related to the outcome.  So it sort of can be a very robust way of assessing the relationship of exposure to  outcomes. I’m just going to skip over and talk a little bit about the how that relates to the  ENGINE Birth Cohort study that was conducted. We used the cohort design to conduct an observational  study where we created a quasi-experimental design where we were in ENGINE woredas and we matched it  to a non-ENGINE woreda so it allowed us to see how these uh two populations, which is ENGINE-exposed  population and a non-ENGINE-exposed population, um we observe them through their pregnancies  and down to the time their child’s on 12 months of age. So we used a sort of a quasi-experimental  approach and linked it to the observational study and our outcome of interest was length-for-age,  height-for-age, and stunting, and the aim was every single data point that was collected was  aimed at generating evidence that would guide future policy. And um just to give you an idea,  and this is some, some of you have seen this for sure but just to give you the breath and scope  of these kind of studies, this is the ENGINE Birth Cohort Study and we had seven time points  of measurements, we went every three uh we went um basically at enrollment when the  mother was pregnant and when she had, within three days after she gave birth,  and then every three months after birth up to 12 months of age. And a key point that  I wanted to point out was because we were comparing an ENGINE population to a non-ENGINE  population, we had to estimate the sample size to make a comparison of these different groups but  we also had to account for attrition, which is one of the big issues not just in RCTs but also in  cohort studies so it’s a fairly large sample size because of that. What we were very fortunate  with was, like in the last column as you can see is that we had such a well implemented study  that the attrition was very very low, except in the last time point where we were at 80 percent,  most of the time were almost at 90 percent of our participation which is which is a very high  participation rate. So and the importance of thinking through observational studies um  that are looking at a longer period of time is really illustrated in this graph which is my  last slide here, where this is this is essentially identified as a stunting syndrome by Andrew  Prentegrast and Gene Humphreys (they’re working on in Zimbabwe on the SHINE trial) and I think  they’ve illustrated this very well where you can see that we tend to look at relationships in  a linear fashion but in when it comes to stunting um it’s actually a cyclical um relationship,  it’s sort of it’s you have to go from neonates to two years of age to school to puberty and  adults and then you go back to conception.

So when you’re using study design  it’s really important So when you’re looking at a um when you’re  planning a study it’s really important to understand what your research question is and  in this case if our research question is and as it was was to understand what’s  happening in the first two years of life, it was really important to have a design that will  allow us to not just address the first two years of life but what was happening in pregnancy. Now  uh advantage of a lot of the times of these kind of studies which I didn’t bring up and it’s  not in the advantages listed in any of them, is that particularly with cohort studies or RCTs  um is that there is an opportunity because they have been exposed to an intervention or because  you have followed them over a period of time, at some point it’s you you as researchers might  be able to go back to those households and collect data at a later time point maybe when the children  are in school or when they’re adults and there’s some classical studies that have been done which  have shown some very very interesting results of early life interventions and one of them is  in Guatemala and called the INCAP studies where they found the effect of the intervention of  providing incaparina which is a complementary food to children, they found the effects of that  in adulthood on work productivity and wages. So so in the end it I think the point that  I want to make here is um that generating evidence is really important, we need a lot of  data, we need surveillance, particularly for uh when you want to understand if you’re  meeting those nutrition targets or not and it’s really really important to consider  what your research question is when you’re planning and implementing a study. Thank you so  much and I think we’ll be open for questions. [Meghan Kershaw] The first question that we received is “In  the nutrition and food system impact pathway, one of the impact is social  equity impact, what does it mean?” [Eileen] In economic terms, um people talk  about winners and losers, and if you look at uh interventions generally,  but agricultural interventions, early adopters tended to be uh less risk averse  etc, so in the food system what that means is we’re trying to um provide uh access, uh  quality, across the food system so you don’t have winners and losers. I mean the losers  in large part tend to be lower income populations, uh sometimes certain subgroups women and children,  so looking at the food system as a way of delivering sustainable diets in a way  that no one has no one is left behind. [Meghan] “Can we do meta-analysis without  systematic review? vice versa?” [Shibani] um a meta-analysis generally is linked  to a systematic review because you do have to go through the steps of identifying those articles um  and and find the um so you have in a way actually they go hand in hand. You can do a systematic  review by itself without applying the statistical and analytical tools um so you can just do  it it becomes it’s sort of a more systematic literature review uh and and then you add the  meta-analysis to it. So I don’t think that those, you can’t do a meta-analysis  without a systematic review.

Wedge type allocation? [Meghan] “Yeah, you  say something about wedge type allocation, is it a design or?” [Shibani] Yes, um  so yes, I would I would have liked to go in a little bit more into sort of the  experimental designs and the wedge type is one of them which, in which you essentially,  um particularly on a programmatic um application, what you can do is in your  population um provide the intervention to one part of your population, whereas the  other part serves as a comparison and then you roll out once you have enough time to study or do  the research on the differences between the two um populations on a specific outcome. So um so  yeah so the wedge type design is considered as an alternative to actually implementing a  randomized control trial when you randomly assign individuals or communities or  clusters to an intervention or not. [Meghan] Next question is, “what is the scope  for real-time monitoring & evaluation to generate evidence?” [Shibani] There is absolutely a lot  of scope for monitoring and evaluation data to be used as evidence. The key thing is um to ensure  that your system is set up to generate the data. um I, I’m not familiar with um, I’m familiar  with monitoring and evaluation frameworks and I think that they are a very good system to um,  they look, they are very robust, the question is if the data is actually um collected against that  framework. And if it is, then yes, it would be a very very good um source for generating evidence.  As a researcher obviously then that goes into the realm of if you’re using that data it needs to  be de-identified so that you’re exempt from IRB, so there are those kind of technical issues to  consider but it definitely is a source. My key uh point would be is ensuring that the data is  that is being collected is of good quality and its rigorously collected. [Eileen] I’d like to add  something there, in a specific example, uh in the late 1980s, I was working with the government  of Malawi on a monitoring evaluation system and uh there was a mealy bug infestation of the  crop in northern Malawi and the the normal M&E uh wasn’t picking it up, but uh uh feedback from  relatives in the north to the capital said, you know, they were being devastated, kids are dying,  and so I think you know, uh getting back to your comment earlier about innovations, I mean I think  the experiential data is also important uh where sometimes the quantitative data on a timely basis  may not be indicating the nature of the situation. So I think you need to think again creatively  about how you blend both. [Shibani] Right, right, yeah and I’m sorry I always have a quantitative  hat on, but yes, you can actually also link your M&E to qualitative uh data collection so you can  have both qualitative and quantitative data that would contribute towards generating evidence. But  this is one thing that we’ve been at the Nutrition Innovation Lab, Patrick Webb and I would love to  see an M&E system used more effectively because there is a lot of data generated. I wouldn’t  know what the quality of the data is if we can ensure that the quality is of the highest  level, that data would be extremely valuable for answering programmatic questions,  particularly implementation research question.

[Meghan] The next question is,  “What would be the best method to generate evidence for this particular project?” [Eileen] Growth through Nutrition? [Shibani] So I think the question I would have is, um generate evidence to answer what? So again we  sort of come back to what would be the question um that needs to be answered relative to this  project. Is it to uh determine if the project interventions are being implemented? So that  would be an implementation research question. Or is it to understand if um the uh the project  has achieved this outcome? [Eileen] Yeah and if I could answer Shibani, I totally agree, I  also think it’s important to look at at uh um integration of components, um I know the  presentation on the ag-nutrition sounded a bit negative, it was not meant to be that way. I mean  there have been success stories in ag-nutrition, but let me take one, um bio fortification orange  sweet potato, uh where not only did they adapt the cultivation but uh Vitamin A status of children  improved. Now the reason I point this out is that was not simply a stand alone agricultural  intervention, there was uh education of the mother on behavior change, there was  income generating activities for the mother uh and there was a dose response issue, uh uh  households and mothers who had more exposure, the effect was greater. So I think you need to  look at all of these issues uh and not think of agriculture in isolation and what’s I think Growth  through Nutrition uh the the overall framework is linking, you know, linking WASH, nutrition,  agriculture, uh behavior change, and looking at um exposure and what at least the continuum of these  complements of activities, what do you see on various outcomes? I mean yes, stunting being one  of them but a behavior change related to diet, behavior change related to WASH. [Meghan]  Next question is “What do you think is a more appropriate method or methods to generate evidence  regarding agriculture impact on nutrition?” [Shibani] Is it a method or what it would be  the right outcome? That would be my question. [Eileen] Well let me, let me just kind of, while  somebody’s answering Shibani’s question, it really depends on how you’re defining nutrition. And I’m  not meaning to be facetious and I think part of the problem in the ag-nutrition literature  is the outcomes have been all over the map. uh if you have an intervention, uh say home  gardens has been popular in some of what uh for example Marie Well in her uh meta-analysis uh  analyzed, and why would you expect home gardens in the uh initial time period where you have  your first harvest from the home garden, basically uh leafy green vegetables, why would  you expect that, in a short period of time be related to anthropometric indicators and children?  What you really should be looking at is uh did the um nutrient density of the child’s diet, if  that’s the the uh objective of interest change, uh did overall consumption at the household level  change? Did income change? So I think this gets back to defining what we mean by nutrition and  it’s been a real frustration, the literature that nutrition is used very broadly without any  identification of the specific metric that we need to use. [Shibani] Yeah and I think just to  add on to what Eileen was uh has just said, um the the issue that um that occurs when you’re trying  to link agriculture to nutrition, um is that it’s it’s a very long pathway and as if everybody is  familiar with the UNICEF framework of the basic underlying and immediate causes, you should know  that that’s when those are not causal uh pathways, those are uh those are proposed pathways, and in  that itself you can see one of the issues is that it’s really hard to measure um that  relationship from say poverty or even poverty alleviation because of agriculture, to  nutrition status of the child because you could have so many other things that could confound that  relationship and for an instance, if the child has diarrhea, if the child has EED, then because it’s  poor WASH, you can invest a lot in agriculture but if with poor WASH you will land up with a lot of  diarrhea and chronic inflammation that’s not going to allow that child um to thrive. And so I think  the um one really needs to understand what the underlying biological issues are and think about  looking at intermediate outcomes where as Eileen had said, nutrient density, dietary diversity,  do these elements improve in these households? A key thing that we have been finding in some  of our Asian um studies, uh for instance in Bangladesh, we have households where we’re working  in the Feed the Future zone of influence and we have um we have surveyed 3000 households about  three times and they’re all rural uh households, they’re all farmers, 90 of them report having  farming income but all of them are as reliant on the market for their food as they are on their  home production. So I think that that’s also an element that needs to be considered within the  context of agriculture to nutrition and it’s not just home-based production, it’s also access  to good quality food and produce in the market.

um It’s very interesting that we find that whether  it’s vegetables or fruits or pulses or animal source foods, um there is an equal reliance  on household production and purchases from the market. [Eileen] The the other issue is and I  think it’s how we measure some of these factors, Shibani and I were talking about this, um in  some work I did in southwestern Kenya we found no relationship between having a latrine at the  household level and uh child morbidity or child growth. When we’re back to revisit this and  I put this in the area of delivery science, what we found was a disproportionate share of  households that had latrines weren’t using them, so we were asking the wrong question, it’s not “do  you have latrine?,” “do you use it, if not why?” and then getting getting at that context which I  think is where the uh, what I call, ethnographic data can be very helpful in explaining some of  the quantitative findings that are being found. [muffled speaking] [Shibani] Okay, Are there any other  questions? [Meghan] Yes, next question is um, “is that hierarchy, the hierarchy of evidence  applies mainly for findings from quantitative, how do we systematically  present qualitative?” [Shibani] Oh yes, that is a very good question and maybe  that should be a topic for the next webinar. I don’t know the answer… you have some  thoughts? [Eileen] Well yeah I mean I my uh I’ve been involved in a lot of qualitative work  primarily in the US, uh and uh uh focus group type type of work, ethnographic kind of work, which uh  we never put forward as being representative but it does it does look at issues through the lens  of the intended beneficiary and it’s, it’s it’s very helpful. The one challenge you have is and I  spoke about this early in the presentation is that um policy officials, finance kind of people, uh  tend to be more persuaded by cost effectiveness analysis and the challenge with qualitative data  which can be very valuable is how do you then merge that into a cost effectiveness analysis to  say this works. um and I think we need a lot more work on that. [Shibani] So and and just I mean  thinking this through from the perspective of the hierarchy of evidence, I think it’s not as much as  the hierarchy of evidence because qualitative is um methods uh you have key informants, you have  focus group discussion, um and um essentially all the methods that you have, did you really  rely on the quality of um the facilitator and the communicator so it’s really the person  who’s actually conducting these uh focus groups and and key informant interviews  that should be able to synthesize um and then whoever is going to be analyzing  the data is able to find the pattern. So I would not put the qualitative methods into  the kind of pyramid approach, I would sort of, you really have to take a qualitative approach  to assess the quality of data that’s coming from a qualitative study because it’s really it’s  a it’s a very different lens you have to wear and that’s something that I immediately put on  because I work with so much quantitative data.

The other thing that I feel is very important  is um qualitative data um and quantitative data together, so like a mixed message  approach, is a very very good way of going, um so it gives you the data that you need, as  Eileen is saying, to support um your qualitative findings um with policymakers or people who  are working in the Ministries of Finance, but at the same time the qualitative data might  actually help you understand why you are seeing something or not seeing something. So um so  yeah, I think that that would be my takeaway. [Meghan] Next question is “how realistic is it to use evidence generated from a single  study or project for policy advocacy?” [Eileen] It depends where you I think where you  are in the cycle. I mean that that gets back to the question we posed, when is evidence enough?  um and again I’ll use an example from the US, um well actually, let me let me use a Vitamin  A example, uh where um so it’s not so close to home. The original work out of Johns Hopkins,  uh out of Indonesia, which uh suggested that um Vitamin A supplementation of preschool  has resulted in decreased mortality. There’s a lot of disbelief because of  that, but we’ve known for years that the Vitamin A deficiency, if left untreated  was related to blindness so that was not, but this was the first time back in  the 90s, there was a link between, 80s and 90s, a link between Vitamin A  deficiency and mortality and there’s a lot of disbelief as it was the first study, the one  study. uh What that generated was they didn’t, some people dismissed it but it generated uh  number one, a uh an enormous amount of additional research on other studies which eventually  wound up in a systematic review, uh Beaton and Martorell, showing that on average Vitamin A  supplementation decreased mortality by 23 percent, but another part, that I think the rest of the  story is if you look at the the Hopkins people, they were confident in their data so they didn’t  just let it stand as one freestanding study, they actually used that information to adver-  advocate for increased funding for Vitamin A research but also Vitamin A programs and we  know what’s happened as a result of that. So again it depends, you know then of course, year  after year there was more data generated but it’s tough in the beginning where you have a a  single well done study that might be suggestive, you know, do you use it? Would you wait for more  information? and I I think it’s um, if you’re confident in that piece of work hopefully  use it for advocacy in an appropriate way.

Nature’s Way Sambucus Black Elderberry Gummies with Vitamin C and Zinc, 60 Gummies

Source: Save the Children in Ethiopia