So a flag that recently turned up in validation is that “NA” is a categorical missingness values in the parts list, and “NA” is also Namibia’s ISO code in the zones and countries reference tables.
This, in theory, shouldn’t lead to a collision because the countries field in addresses should only be populated by values in the countries reference table. However, there are a number of fields in which we’ve allowed entries with a missingness part type to also populate these fields. I will try to go through and make some structural considerations so that “NA” the missingness indicator cannot be used in the country field, (opting instead for “NULL”, nr, etc.) to avoid possible confusion and data mistakes moving forward.
You could also decide to replace NA by Not Applicable so it solves the issue everywhere, not only for countries.
A very fair point! I think we’re trying to keep our missingness indicators more code-ready, and using terms as they’re used in various coding languages (like R), so I worry about making the “not applicable” to text-y might create some new challenges. Though I suppose challenges persist regardless of the chosen solution, haha.
This thread might be related to the other one about NA and Null (Order colmns in table part). In R for instance NA will be considered as missing rather than not applicable if I am not wrong. If you want to enforce not applicable then I think the only possibility is “Not applicable” in a character variable.
You may be right - I’ll have to take this one away for discussion.
On the nr value, “NR” is also the iso code for the Republic of Nauru, so we’ll have to use “NULL” until we finalize the decision on “NA” vs “Not Applicable”
One way to solve this would be to use N/A.
That could be functional, but it breaks the rule we’ve have about partIDs not using any special characters. Which may or may not be a big deal.
We could find missingness values that don’t collide with existing codes and are still valid partIDs. But the conflict isn’t really about which values we pick. It exists because several fields accept a domain set *plus* the missingness set (country = `countries` + missingness), and that is only safe while the two sets share **no** values. So we should be clear about the approach before choosing tokens.
The durable fix is a disjointness rule: the missingness set may not share a value with any set it is combined with. This is mechanically checkable against the current tables, and it fits our plan to make the LinkML schema the canonical home for dictionary validation rules.
One observation is key to keeping the fix cheap: partID and label are different. Country codes are fixed by an external standard, so they can’t and shouldn’t change – which means the missingness partIDs are the side that changes. And because partIDs are stable public identifiers, we wouldn’t rename them; we’d deprecate the colliding ones and add successors, which is an additive change.
Concrete proposal:
-
Adopt the disjointness rule and enforce it – in the LinkML schema, with a CI check on the dictionary tables in the meantime. Run it once now: it flags exactly two collisions today, NA (Namibia) and nr (Nauru).
-
Deprecate the NA and nr missingness parts and add successor partIDs – compounds of three or more characters that no two-letter code standard can collide with (e.g. naMissing, nrMissing; names open for discussion). Labels keep the familiar short forms for templates and display.
-
Keep the missingness in labels matched to what R, Python, and SQL users expect – we designed PHES-ODM to be Open Science-compatible, and the common platforms have fairly consistent, minimal missing-data conventions. That consideration belongs in the token discussion.
-
Add tooling guidance regardless of choice: three of the eight missingness values (NA, nan, null) are on pandas’ default NA-sentinel list, so a default read_csv silently converts them to NaN – and NULL is on the same list, so it doesn’t escape the problem. Dictionary tables should be read with na_filter off (or explicit empty na_values).
Also noting, though I don’t see it as a major issue: NR currently carries three meanings – missingness partID nr, Nauru’s ISO code, and a placeholder token in the wideNames templates (e.g. wat_sa_hFr_hMe_hUn_hAg_NR_value).
One issue to state up front: raw data files carry partIDs, not labels. So hand-entered CSVs would use the successor tokens (naMissing), while Excel templates and dropdowns continue to show the short familiar forms. During the deprecation window, tools reading a mix of old and new data need to accept both – the deprecated parts stay in the dictionary with status = depreciated, so both remain resolvable. In practice, the people most affected are those writing CSVs by hand rather than through templates
Thoughts on the proposal, particularly the deprecate-and-replace path and successor names?