Being picky: study selection for the systematic review

The OKIS systematic review will bring together the results of all the previous research on the benefits and any harms from using orthotic devices for knee instability in adults with neuromuscular disease. This will tell us what is currently known about the usefulness of the devices. In previous posts, Rocio described search terms and where to search how she went about identifying all the existing studies. In this post I describe what we’re doing with what she found.

Rocio’s searches of a wide range of databases gave us 4233 references to potentially relevant studies.  She put these references into a reference manager program called ‘Endnote’. Alison, Joanne and I sifted through the references to determine which were obviously irrelevant from the information given and which needed to be read as full papers to determine their relevance. To ensure nothing was missed, two of us looked at every reference independently. As a result we identified 485 papers that we need to read in full.

The next job is to evaluate each of the 485 studies against our predetermined review inclusion criteria for participants, interventions, outcomes and study designs as mentioned in a previous post. We will document our decision to include or exclude each study with a reason. Again, two of us will do this and where there is disagreement on the relevance of a study we will discuss it and either arrive at a consensus or seek advice from a third member of the project team. This process of study selection is one of the aspects that makes a review systematic and ensures transparency.

It is too early to say how many studies will ultimately be included in the OKIS review. The above process is time-consuming, but as systematic reviewers we have to be patient! We’ll have our results soon enough!

Blogger: Debra Fayter

Needles in Haystacks: Part 2: which haystacks should we look in?

Having decided what we are going to search for, the next important aspect of the searching is deciding where to look. In consultation with the review team we have decided which bibliographic databases and other resources should be searched and the words and phrases that will be used. We need to search different databases because they each include different literature.  Sometimes the content overlaps so we have to make sure we remove duplicate records from the search results, but it’s better to find something twice than miss it from a search.

A good systematic review must search at least Medline and Embase because these databases are the most comprehensive and important ones in health sciences.  Our search includes a number of other databases, for example CINAHL which covers a wide range of topics including nursing and allied health disciplines. We have also considered specialist databases, and are including the database PEDro which indexes the physiotherapy literature in our list of databases to search. We are also searching clinical trials registers to identify ongoing or unpublished studies. This should help make sure we find the most up to date research available.

Since starting this blog post I have now complete the searches.  I have identified 4233 bibliographic references for the reviewers to look through. Hidden amongst these will be the studies which might just help us find out which orthotics are most helpful for people with knee instability in neuromuscular disease and central nervous system disorders.

Blogger: Rocio Rodriguez-Lopez

Needles in Haystacks: Part 1: what sort of needles are we looking for?

For a good many of us, searching for information on a topic involves typing a word or phrase into Google and scrolling through the millions of hits! Searching for studies to include in a systematic review is a rather more rigorous process!  This two part blog will give you a flavour of what is involved.

Searches in a systematic review must aim to retrieve all the relevant evidence about the research question and follow a specific methodology, so that, if necessary, the search could be reproduced.  However, they also need to be conducted within the budget, time and resources available for the project. As Information Specialist for the project this is my responsibility.

My first job was to devise a search strategy that would be capable of identifying all the relevant studies. The medical literature is vast and indexed across a considerable number of bibliographic databases so it is important to be clear what you are searching for. Searches in a systematic review are based on the research question which is composed of four elements. For this review:

• Population is people with knee instability related to a neuromuscular disease or central nervous system disorder
• Intervention is orthotic devices
• Comparator is any used
• Outcomes  are any investigated

The main challenges of this project are the number of diseases and conditions of interest and the number of possible orthotic devices. Studies can be indexed very differently on different databases so it is important to think of all the terms that could be used to describe the devices. The clinical experts on our Advisory Group provided valuable input on the names of all the possible relevant orthotic devices, so we are confident that we will locate the relevant studies.

Because there are many, diverse neuromuscular diseases and central nervous system conditions, we could possibly miss some if we were to list them in our searches, even with clinical advice. It would make searches very long, unwieldy and time-consuming. Instead we plan to concentrate on finding studies that describe using an orthotic device to deal with knee instability. For the same reasons, the comparators and outcomes have not been included in the search.

Blogger: Rocio Rodriguez-Lopez

A complex problem

In an earlier post I described what makes a review 'systematic'. In considering all the relevant research, we have to use transparent and replicable methods. Ideally, a reader of our review should be able to reproduce the review and get the same results. An important part of our methods is to clearly specify the nature of the research to be included in the review, our ‘inclusion criteria’.

Inclusion criteria are set out using an acronym called PICOS which stands for Participants, Interventions, Comparators, Outcomes and Study designs. In a review looking at a new drug to improve survival in breast cancer, for example, the participants are those with breast cancer, the intervention is the new drug, the comparator the currently prescribed drug, the outcome survival and usually the study design will be randomised controlled trials (often abbreviated to RCTs) as these are the most rigorous form of medical research.

Rarely, however are systematic reviews so straightforward.  The OKIS review is particularly complex as our inclusion criteria are very broad.  The participants are people with neuromuscular disorders of which there are many types. Neuromuscular disorders can involve the structure of the muscle cells in conditions such as Muscular Dystrophy the neuromuscular junction where the nerves and muscles meet such as Myasthenia Gravis and the motor nerves in the arms, legs, neck and face as in Charcot-Marie Tooth Disease to give just a few examples. This review also includes neuromuscular disorders arising from central nervous system causes such as Stroke, Multiple Sclerosis and Post-Polio Syndrome. The common element in all these disorders for this review is knee instability.

Another complexity in this review is that there is more than one type of orthotic intervention that may be used to help with knee instability. Some orthoses are fixed below the knee supporting the ankle and foot. Others are longer and cover the knee, ankle and foot. For severe weakness they can also include a hip band. Each one has its own acronym! The range of possible comparators is equally diverse. And, you’ve guessed it, the range of outcomes is wide and varies according to the conditions investigated. Pain, walking ability, usage of the orthotic and patient satisfaction are just some of the outcomes of interest. Furthermore, this topic area is at an early stage of research so we cannot restrict our study design inclusion criteria to RCTs as these have not been conducted in great numbers. We will, therefore, consider a range of study designs.

The complexity of this topic will be challenging at every stage but we now know the terms we need to use as we start to search for all the relevant research...but that will have to be the subject of another blog post!

Blogger: Debra Fayter

What makes a review systematic?

The first of the three elements of the OKIS project is to carry out a systematic review of the existing evidence on orthotics for knee instability in adults. But what exactly is a systematic review? The short answer is that it’s a rigorous way of identifying, evaluating and summarising all the relevant research papers on a given topic. So far so good, but what makes a review ‘systematic’? It’s because it follows a strict protocol or plan where the research methods are set out in advance and are stuck to throughout the review.

The first step of a systematic review is to specify the research question in terms of the study participants, the intervention, the outcomes and the types of study to be included. This is essential information in the protocol. We then conduct a thorough search of electronic databases and other sources for all the relevant literature to answer the research question. We extract data from the relevant studies and assess their quality and finally synthesise the results. We’ll post information about what’s involved and what we find at each of these stages as we reach them.

At the end of this systematic review we should have an unbiased assessment of the existing evidence on the effectiveness of orthotic devices for knee instability. This will, we anticipate, show us what best practice is and where further research is needed to improve knowledge in this area.

Over the course of these blogs we’ll tell you more about the systematic review process and keep you up to date with how we’re getting on. I have conducted reviews on many topics but I’m particularly excited about this one as the results of it will be combined with the views and experiences of health professionals fitting orthotics and the views and experiences of those who have been fitted with an orthotic device for knee instability. This should make sure the project overall provides robust information useful to practitioners, patients and those commissioning and providing services.

Blogger: Debra Fayter

P.S. If you're keen, you can see the detailed protocol for this project on our funders' website: http://www.nets.nihr.ac.uk/projects/hta/133002 

And we've started!

It was great to get the project started - on the auspicious day of 1st April(!). Although it is the beginning it has felt like quite a journey getting here. The project was commissioned by the NIHR HTA programme back in Spring 2013 when they advertised an outline of the question they wanted addressed and invited proposals from researchers. But the HTA will have started exploring it as a possible topic for commissioning research on well before then. If you are interested in how HTA prioritises topics for research you can find out more information by following the links on the HTA Programme website.  Members of the public, health care professionals and policy makers can make suggestions to NIHR about any ideas they would like them to consider for funding. You can find details of how they identify research questions and access the on line suggestion form at http://www.nets.nihr.ac.uk/identifying-research.

The overall questions to be addressed by this research project are: what orthotic devices are currently in use in the NHS for knee instability for people with neuromuscular conditions, what are the costs involved and what further research is needed. An important aspect of our work is to find out what the most important issues are for patients. Our research project is intended as preparatory work to inform a different piece of research sometime in the future which will look at the clinical and cost-effectiveness of different types of orthotic management of the knee in people with neuromuscular disease.

We are starting with the systematic review of previous evidence. However we have also started planning the qualitative study of the views of orthotic users as we will need to apply for research ethics approval before we can start recruiting people to take part. We have also started planning exactly what we need to ask healthcare professionals delivering orthotic services and how we can make sure we get a good response rate to the survey. So please feel free to tell anyone relevant about this blog and our research project and help us build up a network.

Blogger: Catriona McDaid