Different Types of Research Contributions and Challenges in Defense

Writing and Research Strategies

 [Copied from the net in 2012. Source unknown]


Here is a brief reminder of how to write papers and what kinds of
papers to write. Please also see the other materials on our web site on paper
What matters most about a paper is what the reader gets out of it,
not how much work you put into it. That is, after reading your paper, the
reader should feel that he knows something new, has a better insight into
something, or has a better idea of how to do something.
Each paper you write should stick to one main idea, and that idea
should be clear enough that you can formulate it in a single sentence. If you
can’t tell someone else what the main contribution of your paper is in a single
sentence, you haven’t worked it out enough.
Writing papers doesn’t need to be all that hard because papers fall
into a small number of common categories. Most of them involve some problem
that needs to be solved or some observation that needs to be explained; for the
paper to be accepted, it’s important that this problem or observation is
interesting in the first place.
Solving problems that aren’t interesting to reviewers won’t get you
published. It is therefore important that you keep track of the literature and
figure out what topics are likely going to be of interest to reviewers.
In writing your paper, keep in mind that many reviewers are
graduate students like yourself. Think about how you approached your last few
paper reviews. How well did you yourself know the literature? Did you give the
authors the benefit of the doubt when you didn’t understand something? Did you
ask for more experiments?


Now let’s look at common different paper types. I have given these
papers short, catchy names because you should be able to identify clearly what
kind of paper you are writing.

reviewed the literature about solutions to problem X.” (Literature Review)

A literature review should pick a well-defined research area,
something that is neither too large nor too small. An area might be something
in which there are 100-200 research papers total that you need to read.
A literature review is a little bit like an experimental paper. It
usually starts with a question, then looks at data, and then draws its
conclusions. The question it asks is usually of the form “what are the unsolved
problems” or “where are there areas for improvement”. In the conclusions, you
should answer these questions. They form the basis for further research.
In fact, when you start on your thesis, you need to do a good
literature review anyway, and you need to incorporate it into your thesis. If
you can get this kind of review published, that’s really good. However,
publishable review papers are a lot of work and usually require upwards of a
hundred original references to be organized and analyzed. Nevertheless, you
should at least aim for a group seminar on your chosen topic.

looked at the best ways of solving problem X, benchmarked, and compared
them.” (Benchmark Paper)

A benchmarking paper asks the question: “given these standard
methods, how do they perform under different conditions?” It isn’t necessarily
for finding “the best” of the methods (usually, there is no uniformly best
method), but for identifying tradeoffs between different methods.

A significant component of a benchmarking paper is the datasets. If there are
standard datasets and standard algorithms, and they simply haven’t been
benchmarked against each other, then it’s easy to do that. Otherwise, you may
have to create your own benchmark dataset. This can be a lot of work, but it
can also be rewarding: if your dataset is novel and interesting, other people
will use it and cite your benchmarks.
This is often a good paper to write at the start of your thesis
work: you gain experience with the problem you are trying to solve, you
implement algorithms that you need to implement as control experiments later,
and you collect and/or create databases you can evaluate on. Papers like this
have the potential of getting highly cited. They are also conceptually fairly
simple and you are almost guaranteed to get a useful result. However, they are
a lot of work to do well.

came up with a new/better/interesting way for solving problem X.” (Method

This is perhaps the most common paper people write; it’s usually
the kind of paper that takes an engineering view of the world. You usually need
to show experimentally that your method is better than existing methods, at
least under some circumstances or on some kinds of data (rarely, a complexity
analysis may be sufficient). Here, “better” may mean one of several
  1. It yields lower error rates.
  2. It is faster.
  3. It is simpler to implement.
Additional useful distinguishing features can be the following, but
they are rarely sufficient by themselves:
  1. It is easier to understand.
  2. It is theoretically better
  3. It is available in open
    source form.
The required experiments make these kinds of papers similar to
benchmarking papers, but there are some differences. While the goal of a
benchmarking paper is to gain insights into existing methods (data sets
represent a wide range of conditions, authors are supposed to have no
preferences), the goal of the experiments of a “new method” paper is
to show that the method is actually better. But that also means that the reviewers
will take a more adversarial view of your paper, meaning, they will try to poke
holes into your arguments. Key questions they will ask are:
  1. Is the problem you’re solving
    interesting at all?
  2. Are the conditions under
    which your method is better than other methods too restrictive?
  3. Are the advantages of your
    method sufficient large?
  4. Does your method have so many
    parameters that it is hard to implement?
  5. Did you tune your method to
    the specific dataset, i.e. do the results generalize?
  6. Is the comparison fair? I.e.,
    did you unfairly bias the data or preprocessing in favor of your method?
Generally, when you write up your paper, you need to think about
the list of objections that reviewers might make, and you need to have answers
to them.
“I performed observations/measurements on
X” (Research Paper)
This is the more standard scientific research paper; you don’t
create something and you don’t invent a new method for doing something, instead
you observe and interpret existing systems. In some way, it is like a benchmark
These papers are, on balance, easier to write than method papers,
because it is usually easier to get publishable results out of observations.
There are a lot of opportunities for these kinds of papers in computer vision,
pattern recognition, social network analysis, etc.

have a new mathematical model/theory for observation X.” (Model Paper)

Mathematical modeling or theory is a very different approach from
experimental approaches, but the basic rules still apply: your result needs to
be novel, interesting, relevant, and useful somehow. Generally, mathematical
models or theories should make predictions. Some of those predictions, you can
compare against existing data, others may suggest new experiments.

“I found a new/different interpretation of observation
X.” (Response Paper)

These papers are generally responses to the literature, where you
read a paper and decide that its conclusions, model, or theory aren’t quite
right. These papers tend to be shorter and fairly easy to write (because
someone else has already done all the background research). They may involve
experiments illustrating the point you are trying to make. They are often good
papers to write, but they are difficult to plan for because, of course, you
don’t know when you’ll come across a paper you can respond to.

new framework/’theory’ for X.” (Framework Paper)

These papers often don’t involve any experiments, theory, or
serious review of the literature. Instead, they provide a view of a field,
observations, or data. If you hit the “Zeitgeist” just right, these
kinds of papers can be spectacularly successful and highly cited, in particular
if you have already made a name for yourself. However, most of them end up just
languishing at a conference.

for Ph.D.

Although you may be enrolled in a Ph.D. program, you may first need
to learn how to work with the literature and how to conduct experimental
research. At some other universities, this phase is delimited from the actual
start of the Ph.D. program by the submission and presentation of a formal Ph.D.
proposal. We don’t have that requirement at our university, but the two phases
really still exist.
  1. Seminars and
    Before you can write reviews and papers, perform
    benchmarks, and perform experiments, you need to learn these things. As
    part of our graduate program, you should learn reviewing as part of your
    seminar course, and benchmarking and/or experimenting is commonly a
    project topic. The seminar and project are the main time during which
    someone will look at your work and give you detailed feedback. They can do
    this because it’s area and research they understand well. Later, your
    advisor will be able to give you much less input and help on your Ph.D.
    project because they will know fewer of the details of your Ph.D. work.
  2. Collaborate. Another
    way of gaining experience with reviews and research work is to collaborate
    on research and papers with others. This may include detailed
    proof-reading of their papers, contributing to their writing, taking over
    experimental work from them using their methods, or applying special
    skills you have to their problems. When collaborating, think not only
    about the obvious benefit (like maybe another publication), but also what
    you can learn from the collaboration.
  3. Small
    If you need more experience or still don’t know what
    exactly you want to do for your Ph.D., in addition to collaborations, you
    can also take on smaller projects. These are projects similar to what your
    advisor might give you in a project course, but they are not formal parts
    of your course requirements and you don’t get ECTS points for them. You
    should still aim for getting publications out of them. There’s usually a
    long list of such small projects to choose from.
The previous steps are preparatory for your Ph.D. You really
shouldn’t spend too much time on them because you don’t want to spend too much
time on your Ph.D. At some point, when you have a reasonable handle on how to
work with the literature, how to run experiments, and what you want to do, you
need to pick a Ph.D. topic. That’s when your Ph.D. studies usually start.


Actual Ph.D. research, like all research, really means three steps:
  1. familiarizing yourself with
    the literature
  2. getting familiar with the
    research methods and tools
  3. conducting actual research
Note that you don’t usually start this process from zero;
presumably, you have already done a little bit of work in the area–maybe as
part of collaborations or projects–simply to determine that you actually are
interested in it.
Each of these steps should produce some output, in the form of a
review, a benchmark, and research contributions.
  1. Review/Proposal.
    Start by writing a review paper of the field your thesis is in. You should
    plan on presenting this at a seminar. You can actually do this for seminar
    credit. Sometimes, you can publish this as a review article, although that
    usually involves a lot more work. This should also give you a good idea of
    what the open problems are and how success is measured in your area. A
    good review with meaningful conclusions is basically equivalent to a Ph.D.
  2. Benchmark.
    Next, implement algorithms, collect datasets, and perform benchmarks. You
    can do this as part of a project course. If your results warrant it, this
    is fairly easy to publish at least at a conference. This will give you
    better understanding of the data and get you set up for your own
  3. Research
    There are actually two major kinds of research
    contributions: new methods or observational studies. New
    . Then you start implementing your own ideas and methods,
    benchmark them against the standard methods, and start writing up the
    results, resulting in several publications. Observational
    You can also make part or all of your thesis observational
    studies. You should consider this possibility seriously because it can be
    easier than “new methods” work. However, while for engineering and methods
    research, the primary deliverable is something that works better, for
    observational studies, scientific reasoning and understanding is more of a
    challenge. (The “benchmarks” for observational studies are replication of
    other people’s observations.)
This division and three step approach are useful, and it’s the way
researchers generally approach starting research in a new field. But it is not
formally required and many students end up working on a lot of different
projects and ideas and then at some point putting everything together into a
thesis. Each of project implicitly still requires a literature review,
benchmarks, and new contributions, but they happen in different orders and
perhaps aren’t made explicit in the form of publications. There is nothing wrong
with that if that works for you and if you get a good thesis out of it.
However, if you are limited in the amount of time you have for your Ph.D., if
you feel lost or are concerned about completing your Ph.D., then you should
consider picking a single topic and focusing on the three steps for research.

and Ph.D. Proposals

Some scholarship organizations will require you to submit a Ph.D.
proposal in order to get the scholarship. That is a useful starting point, but
it is rarely as detailed as a true Ph.D. proposal and literature review should
be (although the more detailed you make it, the less work and uncertainty you
will have later). Furthermore, after getting the scholarship, you may still
need to spend some time with seminars, projects, etc. to get more familiar with
research methodology.
This means that you shouldn’t assume that your Ph.D. proposal is
finished just because you submitted something with the scholarship application.
On the other hand, scholarship agencies are generally fairly flexible about
proposals and you can change them later as you learn more about a research

You Will Encounter

Of course, in real life, things don’t necessarily work as smoothly.
Common problems are:
  1. The literature
    seems to confusing and you can’t make a coherent review or presentation
    out of it.
    This means that you really don’t understand the field
    well enough yet to even start your research work. When you get serious
    about your Ph.D. thesis (or research in general), you should have a good answer
    to the question of “what are the open questions in your area”,
    “what are promising ways of attacking them”, and “what are
    the most common ways in which people have tried to solve these problems
    before and why have they failed.” As part of this, you should become
    more of an expert on your chosen area than your Ph.D. advisor.
  2. The
    benchmarking doesn’t yield meaningful results.
    One of the most
    important skills you need to learn as part of your Ph.D. is how to conduct
    meaningful experiments. This means asking the right questions, choosing
    the right datasets, performing the right measurements, and presenting the
    results in a meaningful and compelling way. Your Ph.D. advisor can help
    you and give you feedback, although it will largely be of the kind of
    feedback you would get from a reviewer: he will point out problems and
    limitations with your approach. Fixing those problems is your job. Your
    advisor may be able to give you a little input on that, but benchmarks and
    data sets are very hands-on problems, so that kind of input is limited.
  3. Your new
    method isn’t working better than the old one.
    Perhaps your idea
    just isn’t a good one, or maybe the dataset you are testing it on isn’t
    good for showcasing its advantages. This is the kind of thing your advisor
    can help most with: he can provide ideas for other, new approaches and
    suggest other ways of evaluating your method. However, you shouldn’t just
    abandon an approach if it isn’t working; you probably invested a lot of
    time in trying to make it work and you should try to learn from the
    experience and understand why it isn’t working. If you just keep trying
    one idea after another and never figure out why they are failing, you are
    going to make very slow progress. On the other hand, if you learn from
    your failures, even if you don’t get a publication out of it, you are
    still making progress.


Here are some other strategies you need to keep in mind:
  1. Talk with other people about
    your work. It doesn’t matter whether they are experts in your area or not.
    One of the most important parts about talking to other people is that you
    learn to explain your work better. If you are lucky, they will give you
    useful feedeback.
  2. When you talk to your
    advisor, have clear and concise explanations of what you have done, and
    have specific questions you ask. If you don’t, your advisor simply can’t
    help you very much.
  3. In addition to your primary
    work, do some other things: collaborate with others on projects
    (contributing the expertise you have obtained from your primary research),
    and also take on some small and easy additional side-projects.
  4. Pick your research area so
    that there are lots of opportunities for collaboration and additional

See Also: 

What is PhD?

Why PhD is Difficult: 

Starting with your PhD

Reading Research and Writing Your Research 
Qualitative Learning from a PhD



Leave a Reply

Your email address will not be published. Required fields are marked *