Rule Based Annotations

A demo UI has been developed to facilitate the process of creating rule-based annotations on a map. This user-friendly interface allows users to visually interact with a map and define rules or criteria for data annotations.

The current state of the demo UI for rule-based annotations is considered a work in progress, and it is expected to evolve based on valuable feedback gathered during the pilot user testing phase.

The main underlying technology that will be power the annotations will be a database store and a restful API interface.

Backend GitHub

Initially the main way to interact with the annotations store, will be through the API or R.

The R package can be found here.

Seeding rulesets into our annotation store provides several benefits, especially when it comes to improving user experience. Seeding data can help new users get acquainted with the annotation system more quickly. By providing sample data or pre-existing annotations, users can learn how the system works and what is expected of them in terms of annotation tasks. This can reduce the learning curve and increase user engagement. New and pilot Users can be more productive when they have access to seeded data. They can start their annotation tasks immediately instead of filling in well-known information.

Before releasing the rule based annotations tool to the public, it will undergo internal and external piloting phase. In this phase, pilot users will be given a small annotations task within a group they are familiar. The pilot users will then be interviewed about their overall experience, particularly if there were any "rules" or annotations they wanted but make but could not.

After the internal and external review we plan to write a data paper that will present the annotations tool to wider audience. The internal and external pilot users will be encouraged to be co-authors on the paper. The "data" part of the data paper will be an export of rules that were created during the pilot phase.

One of the final tasks will be the development of data cleaning workflows using the collected annotations.

UI link

If you are a pilot user, you should already have your project ruleset permissions set up for you.

To make a rule

If vocabulary this doesn’t work for you, please pick the closest fitting, and request additional vocabulary in your feedback.

Create your rule.

While there is not absolute definition of a good rule and a bad one.

Good annotations usually have a few properties:

The Global Biodiversity Information Facility (GBIF) is a vital data infrastructure for researchers, conservationists, and policymakers across the globe. It aggregates and mediates access to extensive datasets of biodiversity occurrence records, thereby fostering scientific research, conservation efforts, and informed decision-making. Nevertheless, the quality of these records is pivotal for their "fitness for use", and data cleaning becomes an essential process to ensure their reliability and utility.

A competing viewpoint with regard to data cleaning is to "fix at source". Fixing GBIF occurrence data at the source, such as reaching out to data publishers to address issues and errors in their datasets, is an ideal approach in theory. However, in practice, this approach often encounters challenges, primarily because publishers may not respond to emails or communication attempts. It’s essential to bear in mind that rule-based annotations can contribute to rectifying data problems at their origin as well. Additionally, it is often the case that records do not need to be fixed, but merely are not acceptable for a certain application, such as species distribution mapping.

Automated solutions, like CoordinateCleaner, while valuable tools for data cleaning, may be considered incomplete in certain contexts due to their limited flexibility and potential to miss edge cases. A rule-based annotation system, on the other hand, allows users to make data quality decisions that fit their use case in a more granular way.

Annotation systems, like any software or tool, have the potential to become unusable when they become overly complicated.

One goal of a our rule-based annotation system is to make it accessible to a broad user base, including researchers, scientists, and casual users. If the system becomes overly complex, it can discourage potential users who may not have a deep technical background or a lot of time, but still have valuable feedback.

A rule-based annotation system, especially one used for annotating complex datasets like GBIF occurrence records, must strike a delicate balance between complexity and usability.

One of the key ways to increase usability and complexity is to introduce a controlled vocabulary.

Using a small controlled vocabulary over in an annotation system offers several advantages to downstream users. Finding the right level of granularity and flexibility within the controlled vocabulary is key to reaping the benefits while accommodating the specific needs of the annotation user.

A basic rule in our system looks like this.

rule → taxon in geo-polygon are controlled vocab

In our system a geo-polygon is a Well-Known Text (WKT) object. A geo-polygon could also be the name of a place that eventually maps to a WKT polygon (like a country code or GADM code).

A taxon in our system is going to be a GBIF taxonKey so rules are more likely to look like this in practice.

A ruleset is a collection of rules.

For example, a ruleset could be "Annotations of the Genus Leo", and it could look something like the table below.

A project is a collection of rulesets.

Projects are designed to allow for collaboration between users and logical grouping of rulesets. For example, a ruleset could focus on Lions, but be part of a bigger project about cleaning up Mammal occurrence records.

Note how a project can encode knowledge from other sources into a ruleset, such as iNaturalist atlases.

We hope that users will collaborate on a project that interests them and create rulesets that are widely beneficial to others within their research community.

Within a project, only users with access, granted by the project creator, will be able to create rules and rulesets. However, rules, rulesets, and projects will all be open and publicly available.

For downstream users, deciding which rule and rulesets to use might become challenging without some quality control. Currently, we imagine a simple upvote-downvote system on rule, ruleset, and perhaps project. With voting users could see what annotations are supported by the broader community, and create cleaning scripts that are only use annotations supported by the community.

Additionally, voting could provide protection against vandalism.

Annotating higher taxonomy is harder than annotating at the species level because you have to be confident, the annotation at the higher level fits all child taxa.

Given the distribution of Amphibians, a good rule for the high taxon Amphibians would be :

rule → Amphibians in Antarctica are Suspicious

One challenge is that is is hard to downcast annotations like "Native" to lower levels, since species of a big group tend not to be "Native" to exactly the same areas.

Inevitably, there are going to be rules created in our system that conflict. For example, a user might mark and area as "Native", while another user will mark the same area as "Suspicious".

In our rule-based system, unlike perhaps other platforms, we are not striving to create a single ground truth. We aim only to have a collection of useful opinions, and we leave it to the end user to decide what to do with the information.

We might consider using the preexisting vocabulary, although we are attempting to annotate land area (ranges) more than we are attempting annotate occurrence records.

https://registry.gbif.org/vocabulary/DegreeOfEstablishment/concepts

Below is the working controlled vocabulary for location-based annotations.

This vocabulary is meant to be a compromise between modeling species ranges and establishment means accurately, while not being overly complex.

The current vocabulary might change in the future. Namely, there has been some discussion introducing hierarchy such that perhaps certain terms map to present or absent for example.

Annotating land areas (and extensions) provide at least two advantages over annotating occurrences: