VFSS Quality Assessment Tool VFSS QA Tool

Rater Instructions and Project Overview

1. What This Study Is About

Thank you for taking the time to support this project. Your expertise is essential.

In this study, we ask you to rate the technical image quality of single VFSS frames (still images). The key question for every image is:

If this frame showed a clinically relevant moment, would the image quality be good enough for you to clearly see the important anatomy and contrast material to make a diagnostic judgment?

A few important points:

  • Please ignore whether the specific moment in the frame is clinically meaningful or shows much happening.
  • You are not asked to make a clinical decision based on the frame itself.
  • You are only asked to judge whether the imaging quality (contrast, sharpness, noise, artifacts, implants, etc.) is sufficient so that relevant structures and material would be visible if present.

In short:
Do not rate the content of the moment — rate only whether the image quality is good enough for diagnostic use.

Your ratings will be used to:

  • Calculate Mean Opinion Scores (MOS) for each image.
  • Assess inter‑rater reliability between experts.
  • Develop and validate VFSS‑specific image quality assessment models.
  • Test how useful these models are in practice, for example to automatically identify low‑quality frames before using them in AI-based analysis.

This project is a collaboration between:

  • Louisiana State University (LSU)
  • East Carolina University (ECU)
  • Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)

2. Your Task as a Rater

You will be asked to rate about 500 VFSS still images on a 1–5 Likert scale that reflects how suitable the image quality is for diagnostic purposes.

For each image, please consider:

“If this frame contained a critical swallowing event, would the quality be sufficient for me to clearly see the relevant anatomy and contrast material for clinical interpretation?”

Examples of what should influence your rating:

  • Contrast and brightness
  • Sharpness
  • Noise or graininess

You can add a brief optional comment, but comments are not required.

3. Rating Scale: 1–5 Likert (Image Quality for Diagnostics)

Please use the following interpretation of the scale:

  • 5 – Excellent quality for diagnostics
    • Image is very clear.
    • Relevant anatomy and contrast material would be easily visible.
    • Little to no noise, blur, or artifacts.
    • You would be fully comfortable relying on frames of this quality for clinical decisions.
  • 4 – Good quality for diagnostics
    • Overall clearly interpretable.
    • Minor imperfections (noise, small artifacts, slightly suboptimal contrast), but they do not meaningfully interfere with your ability to see relevant structures or material.
    • You would comfortably use frames of this quality in your routine practice.
  • 3 – Acceptable / borderline quality
    • You can interpret the image, but you clearly notice limitations (e.g., noise, suboptimal contrast, some blur or artifacts).
    • You could use frames of this quality for clinical decisions, but you would prefer better quality and might feel some uncertainty.
  • 2 – Poor quality
    • Clear limitations in image quality (e.g., strong noise, blur, very low contrast, substantial artifacts).
    • Important anatomy or contrast material would be difficult to see or distinguish.
    • You would be hesitant to rely on frames of this quality for diagnostic judgments.
  • 1 – Non‑diagnostic / unusable quality
    • Image quality is so poor that meaningful clinical interpretation is not feasible.
    • You would not use frames of this quality for diagnostic purposes.

If you are unsure between two categories, please choose the one that best reflects your real‑world comfort level with using images of that quality.

4. How to Use the Web Tool (vfss.anki.xyz)

4.1 Access and Login

  1. Open your web browser and go to: https://vfss.anki.xyz
  2. On the login page, enter your name as the username.
    • No password is required.
  3. After logging in, you will be taken to your personal user page.

4.2 Your User Page

On your user page, you will see:

  • A progress overview, showing how many images you have already rated.
  • A button labeled “Start annotating” to begin or continue rating.
  • A Revisit section where your ratings are later displayed in a table.

To get started:

  1. Click “Start Annotating”.
  2. You will be taken to the annotation view, where you can begin rating images.

4.3 Annotating Images

In the annotation view:

  • You will see one VFSS frame at a time.
  • Under or beside the image, there are five rating buttons (1–5).
  • There is also an optional comment field.

The typical workflow is:

  1. Look at the image and focus only on image quality, not on how informative the moment is.
  2. Choose the rating 1, 2, 3, 4, or 5 that best matches the quality descriptions above.
  3. If you like, add a short comment (optional).
  4. Click “Next” or press the Enter key to go to the next image.

You can also use keyboard shortcuts if you prefer:

  • Press 1–5 to select a rating.
  • Press Enter to confirm and move to the next image.

All your ratings and comments are saved automatically, so you do not need to worry about losing your work.

4.4 Reviewing and Adjusting Your Ratings

You do not need to finish all 500 images in one session. You can:

  • Stop at any time by closing the page.
  • Return later to https://vfss.anki.xyz.
  • Log in again with the same name and continue where you left off.

To review your ratings:

  1. Go to your user page (home view after login). You will see a table of all images and your ratings. From there, you can revisit images and adjust your ratings if needed.
  2. Or you can go to the Annotation View and click on the Revisit button.

When all ratings are complete, you will also be able to see, on the homepage, an overview of:

  • The ratings from all raters (aggregated), and
  • The Mean Opinion Scores (MOS) for each image.

5. Estimated Time Commitment

There are approximately 500 images to rate.

If we assume an average of about 10 seconds per image:

  • 500 images × 10 seconds ≈ 5000 seconds
  • 5000 seconds ≈ 83 minutes
  • This equals roughly 1.5 hours of total rating time.

You are welcome to spread this out over several shorter sessions (for example, three sessions of around 30 minutes). The tool will remember your progress automatically.

6. How Your Ratings Will Be Used

Your participation makes it possible to improve VFSS image analysis in a meaningful way. Specifically, your ratings will be used to:

  • Compute Mean Opinion Scores (MOS) for each image by averaging the scores from all raters.
  • Assess inter‑rater reliability, to understand how consistently different experts judge image quality.
  • Train and refine no‑reference image quality assessment models tailored to VFSS, which estimate image quality directly from the images.
  • Evaluate how useful these models are in practice, for example:
    • Automatically flagging or filtering out poor‑quality frames before using them in AI-based segmentation or analysis.
    • Providing a standardized image quality metric for future research and clinical studies.

Ultimately, the goal is to help ensure that automated and research workflows use images of sufficient diagnostic quality, leading to more reliable and robust VFSS analysis.

7. Thank You

Your time and expertise are greatly appreciated.
If you have questions about the task or encounter any issues while using the tool, please reach out to Martin (martin.reimer@fau.de).

Thank you again for contributing to this work!