Why did you cross nikext? It's still available, the vcs link just needs updating.

In T4085#4301589, @Nikerabbit wrote:

Why did you cross nikext? It's still available, the vcs link just needs updating.

Sorry, I did not find it so I assumed it is gone. You are welcome to do the honors.

Change 508295 had a related patch set uploaded (by Dvorapa; owner: Dvorapa):
[mediawiki/core@master] Add a new {{USERLANGUAGE}} variable for use in wikitext

https://gerrit.wikimedia.org/r/508295

Change 507328 had a related patch set uploaded (by Dvorapa; owner: Want):
[mediawiki/core@master] Add system message MediaWiki:Langcode

https://gerrit.wikimedia.org/r/507328

An we still lack the possibility of marking a specific page (with a margic syntax generating metadata, not content, like "[[Category:...]]") as being primarily in a specific language (independant of the user language, but that should NOT be inserted in pages marked for translations with the translation tools, which are marked automatically by the Translation tool and uses a specific page naming convention using "/langcode" suffixes/subpages, or some "langcode:" prefix or namespace, like on the OpenStreemap wiki).

This magic metadata keyword to insert in the wikicode of pages (or in transcluded templates), such as:

• {{PAGELANGUAGE:langcode}} or
• [[PAGELANGUAGE:langcode]] (like categories), or
• <meta name="lang" value="langcode"/> (using a generic metadata system for pages, not limited to languages)

should be natively implemented. It should also properly set the HTML metadata in the page header and/or in the MIME headers of HTTP query replies.

And the page language metadata would be overriden by a wiki-specific hook if there's a naming convention enforced by the translation tool: this hook would then detect the presence of the keyword in the wikipage content, and any difference would generate a warning (at least on the preview) and mark the parsed page with a tracking maintenance category, if language code set by the magic keyword is different from the language code set by the translation tool or enforced naming convention; so that we can either:

• remove the magic keyword from the page content (or its transcluded templates) so that the wiki-specific hook or translation extension will set the expected language itself; or
• keep this magic keyword, but move the page to use a different full pagename (with the correct language code or none) so that the wiki-specific hook or translation extension will not imply a different language.

The need of this keyword is for all other cases where a translation tool is not applicable (e.g. talk pages like "embassies", or because the page is not intended to be itself translated, or because its content is generated by templates using the user's prefered language only and is then completely "autotranslated", with the page possibly embedding a custom "View this page in..." language chooser to render it in one of the supported languages, independantly of the universal language selector which autoselects the prefered user's language throughout his navigation on the wiki, this custom language chooser also setting the correct HTML or HTTP headers in the current session, but not changing the user's prefered language for his navigation on the wiki site)

There is Special:PageLanguage. Anything on the page itself doesn't work, as the language needs to be known to parse the page, so it would be chicken and egg situation.

I'm not convinced this is needed to parse the page, only to generate its content. But is this related to conditional code like #if and #switch and with transclusion of Lua generated contents (that would then need to generate all lingusitic versions until a laguage filter is applied at end to purge the excluded section) ?

Anyway, there's still no evident way in the UI to mark a page with this special page (which is also too much restricted). Marking an individual page to use a specific language should be as open to anyone as being able to create or rename/move a page with a correct title in that language.

So the menu at top of pages that allows renaming a page should also include the option to change its language (this change should be also visible in the history and can be reverted by reviewers if needed)

An alternative is to integrate the language setting of a page in the same dialog for renaming/moving a page. And in the page editing dialog notably when creating one (insert that option just above the comment line before the submit button). In the same place we could also have other selectors for setting other page metadata (that may be needed for some wiki extensions, e.g. pages that contain input forms or that run some scripts, or for setting specific CSS stylesheets applicable to the page).

Change 507328 abandoned by Krinkle:
Add system message MediaWiki:Langcode

Reason:
This is not a stable implementation for the software and will be hard to maintain and verify over time. Also something about appropriate fallback for new languages and insurance that this exists for every recognised language (including those without translations), which would require a structure test at the very least.

I don't think we can improve this approach to a version that is acceptable, as such closing in favour of <https://gerrit.wikimedia.org/r/#/c/mediawiki/core/ /508295/> - which, if this is a feature we can support, offers a more maintainable implementation.

https://gerrit.wikimedia.org/r/507328

Change 507328 abandoned by Krinkle:

Just an afterthought, that maybe the combination of both possibilities could be beneficial as Urbanecm suggested in the abandoned change. Generate the message Want suggested for all languages by (translate? separate?) extension could be a second approach alternative to the magic word. Anyway this is a thing for performance and parsoid teams to decide

Please note: I'm seen https://www.mediawiki.org/wiki/Extension:LanguageCode that a realize two magic Words:

• {{USERLANGUAGE}} - for a display the user language
• {{PAGELANGUAGE}} - for a display the page language

I have not reviewed the source code, but maybe this is a solution for this issue.

By way, {{PAGELANGUAGE}} is also a very needed feature! See T59603 and T161976 about this issue.

It's listed in the description :) The issue is not having code that implements this, the issue is whether this feature is suitable (now and in the future) for MediaWiki core and/or deployment to Wikimedia sites.

Of cource ;) This is notice only. And also this is marker, that this issue still important and very needed.

There should not be any need to call the wiki parser from Lua just to get an information which is built in the Wikimedia API (in PHP) itself. The best way would be to expose this value directly in the environment offered by Scribunto.

Now for the access from wikipages, it should also not be ncessary to call a Lua module. But using the template expansion of "{{int:Lang}}" [which works if there are resources loaded in pages of a special namespace intended for storing translated texts, but that is non sense here as this is NOT translatable and it is just returning a rewriting of the page name that does not need to be stored as a page and its content should not be modifiable at all] should be deprecated.

However, there's not necessarily a need for a new magic keyword, if the "int:" parser function hook implements it internally instead of performing a template expansion from loaded pages. Here the parseer function hook implementing the "'int:" special namespace is written in PHP and can instantly use the Mediawiki API (in PHP) which exposes a PHP variable (which should have been "cleaned", i.e. should return already a canonicalized and validated code, that should then map to a supported code, even if the user is querying a page with an invalid value for the "?uselang=*" parameter; that varaible should also already canonicalize some known aliased codes, such as "zh-min-nan" to "nan", or "fra" and "fre" to "fr", using BCP47 rules, or "EN" to "en"; invalid codes specified in query parameters may be replaced by another "guessed" language, from the browser's "prefered language" environment, or at least the default language code of the wiki).

The whole code for getting and validating the user language (from HTTP query parameters, otherwise from user's preferences, otherwise from the browser's environment in the "AcceptLanguage:" MIME header, which also specifies a list of user-prefered fallbacks that coulmd be exposed as well as a separate variable containing an ordered array of validated codes) is part of Mediawiki core. It is at the root even before parsing any page (and there's not always a wiki page to parse when the user requests a page in the "Special:" namespace, which is implemented by Mediawiki hooks that won't parse any wiki code from any actually stored page; this is also true when rendering pages from the "api.php" endpoint). Then only, the wiki code of wiki pages is parsed and expanded (and later cached by this validated user language). The core will recompose the final page containing the cached parsed page and the rest of the UI. In all cases, the Mediawiki parser and the Lua parsers are only plugins loaded from the Mediawiki internal environemnt where all is ready:

It's up to the MediaWiki API to expose this variable (or varaibles, it it exposes not just the prefered user language, but also his own prefered fallbacks, which could also be stored in user preferences, as an ordered array of language codes that will however need to be revalidated by the core when it will load the user's preferences from the database if the user is logged in; the user preferences, users may not even need to set these preferences which should by default use the browser's settings exposed in the standard "AcceptLanguages:" header of HTTP but with an order specified by "q=*" numeric values qualifiers rather than just an ordered list; we don't need these values, just an ordered list of validated codes usable for the Mediawiki UI). So expose the main prefered language in the PHP implementation of the "int:" special namespace, and in Scribunto, and all is done! No need to load extra pages in wikis, and no longer any need for users to configure the prefered language for each wiki, as the Mediawiki UI by default will honor the browser's settings (in fact users may also want to REMOVE this language setting from their stored preferences, with the option "user language preferences of the current browser", or "application" if he's using a detected Mediawiki mobile app).

And there will be no need for admins to manage special pages on each wiki. users will still be able to use another language using the universal Language Selector (ULS), temporarily, or persistantly on a specific wiki if they wish to do so and are logged in. Users without accounts are not required to create an account to view and navigate anywhere on a wiki in their prefered language, using simply the language settings of their browser or app, and the server will never need to store anything (except in server logs whose access is highly restricted for privacy reasons) for non-logged in user.

For this reason, I think that exposing wgUserLanguage of PHP (or a subvariable in the MediaWiki environment) is the solution to go: simplest, fastest, most efficient (no need to cross multiple language parser barriers: PHP is already at the root of everything in Mediawiki, for the Scribunto extension and for the "int:" namespace hook if it implements "int:Lang" internally without loading any page from the database with an unnecessarily complex mechanism of fallbacks that should not be needed at all if the exposed variable is already canonicalized).

In general we would like to separate "page content" (in the "page content language") from user interface (in the "user interface language"). Currently much of that user interface is expressed in wikitext on sites like commons, but we'd like to reduce that over time. As mentioned before, wikitext in the user interface language complicates caching.

As a separate note, I'd like to encourage any new parser functions/magic words to use the {{#....}} syntax, as described in T204370: Behavior switch/magic word uniformity.

"As mentioned before, wikitext in the user interface language complicates caching."

But this is not new and this task won't change it! Caching is already performed by user language (if there's any use of it in the content, including when using the "{{int:*}}" syntax or using any parser function that requires it, or in most special pages when their content is cachable).

Caching is also controled by an expiration date that already depends as well on the use of current date functions and magic keyword (depending on their precision: "{{CURRENTYEAR}}" will limit the expiration date to the end of the year, "{{CURRENTDAY}}" will limit it to the end of the day ; "{{CURRENTMINUTE}}" would limit it to the end of the current minute, but caches also define a minimum caching period, probably set to a few minutes, to avoid too fast refreshes which could be costly for rendering servers, but whose exact duration will depend on the administrative settings of each wiki).

Yes, the legacy parser splits the user cache by user. Parsoid, however, does not: instead any per-user adjustments must happen as a post-processing pass. So we'd rather *not* see new features introduced which vary by user; these will all have to be reimplemented as post-processing passes later.

Just a note. This is a cool feature needed and used via extensions on many third party wikis. It will be sad to not have this feature at all via extensions in the future due to shortcomings or improvements, probably depends on how you see it, of the new parser. I very much suspect that every wiki using the Translate extension is likely to need this feature.

How does (or doesn't) parsoid currently handle {{int:}} syntax?

I don't think Translate users need this feature. Most cases should be covered by {{PAGELANGUAGE}} and the soon to be introduced {{TRANSLATIONLANGUAGE}}.

I don't think Translate users need this feature. Most cases should be covered by {{PAGELANGUAGE}} and the soon to be introduced {{TRANSLATIONLANGUAGE}}.

I was indeed a bit too general here I guess. Sorry for putting Translate into the same pot as multilingual wiki. ;)

The use case is to serve content depending on the language the user chose in their settings without having them to again choose the language on every page they are currently looking at. So after having a second peep, a bit late I guess, at the original description I see that {{int:Lang}} could do the same thing. Perhaps a bit more effort to set up initially but I do not mind here. Anyhow that's the use case I have in mind.

I also agree that {{int:Lang}} is already widely used. However, the "int:" naamespace by default performs a lookup of pages that must be created in the "Mediawiki:" namespace, named "Lang/*", with one page per language. This is the old method, and in my opinioln Mediawiki does not necessarily needs to loads hundreds of pages for hundreds of languages (plus their variant subcodes): the parser can implement a hook that can directly return the user language from the MediaWiki's API, without loading any page from the "Mediawiki:" namespace (so all pages "MediaWiki:Lang/*" could be simply deleted if they still exist, and protected from being recreated, or this namespace "Mediawiki;" would also have a hook implemented that locks creating/editing any page named "Lang" or its subpages).

The hook is very minimalist as it won't perform use any Media API, but will just look into variables that are already exposed by the internal API.

Why is it so complicate to implement and deploy this hook (which will also accelerate the rendering as there's no longer any need to load an external page from the database, just containing a plain-text code matching the subpage name???)

Why do then we need annother special magic keyword ? The "int:" namespace is already reserved and can be used for this. All what is needed is to implement a hook for pagename resolvers in the two existing namespaces: they can perfectly detect that the "base page name" is "Lang" and just return the language code seen in the requested subpagename (for the "MediaWiki:" namespace) or in the requested page name (in the "Int:" pseudonamespace). No need to pollute the main namespace with more magic keywords.

And for compatiblity with wikis that have already deployed various kooks for extra magic keywords, inform them they can deprecate these keywords, and convert them to a very basic template just containing {{int:Lang}} (only to make the transition, bots will be able to replace invokatiions to these templates).

Now all wikis will be unified with the same syntax {int:Lang}which is the most common in many widely used wikis.

In general I dont like at all the magic keywords using the template like syntax with an unqualified name, even if this name is capitalized: we have too many "magic keywords" using unqualified names: using a namespace prefix for them is highly preferable, and it is much more flexible: new additions of qualified keywords witll be safer, will avoid clashing with existing templates already used for incomaptible goals).

In general, many internationalization features could be implemented with the same hook in the "int:" namespace (which you can think as meaning "international" or "internal", and already reserved for the internal maintenance and management of Mediawiki itself).

As well {{TRANSLATIONLANGUAGE}} could just be {{translate:Lang}} using the "translate:" existing namespace.

Please cleanup with namespace separation. Pollution of the main/global namespace by magic keywords should be deprecated. Each Mediawiki extension should have a policy requiring them to use their own namespace for all their needs.

And please adopt a policy for the addition of new spacespaces: some wikis have a LOT of namespaces for normal pages. namespaces for wiki extensions should better use a prefix starting by "#". In fact I would even prefer that the "int:" pseudo-namespace was changed to "#int:" so the final prefered syntax would be {{#int:Lang}}. As well the internal special namespace for "Mediawiki:" whould better become #mediawiki: or shorter as #mw:.

This would also help avoiding clashes with interwiki codes (which are hard to detect as the list is long an extensible, and non-existing namespace currently resolved as part of the main namespace where the ":" character is allowed anywhere in the base page name. MediaWiki suffers from these old legacies made in a time when there was no policy at all about naming conventions. Many new magic keywords added in successive versions of MediaWiki have caused name clashes with exiting and unrelated pages or templates.

It's time to cleanup this nightmare (which would also simplify the parsers that constantly need to be updated, and that require long and costly maintenance on the wikis).

If you're trying to do a transparent redirect, then there are Parser hooks (see ParserOptions::getCurrentRevisionRecordCallback()) which might be more appropriate than setting up transclusion templates.

And yes, int: is also an issue: T85581: Parsoid page views: need to do something about {{int:}}. But we're trying to draw a line in the sand and avoid adding *new* things we can't support.

There's also T114640: make Parser::get_targetLanguage aware of multilingual wikis.

Change 508295 abandoned by TTO:

[mediawiki/core@master] Add a new {{USERLANGUAGE}} variable for use in wikitext

Reason:

Task was declined.

https://gerrit.wikimedia.org/r/508295

Change #508295 restored by Tim Starling:

[mediawiki/core@master] Add a new {{USERLANGUAGE}} variable for use in wikitext

https://gerrit.wikimedia.org/r/508295

Change #508295 merged by jenkins-bot:

[mediawiki/core@master] parser: Add a new {{USERLANGUAGE}} magic word for use in wikitext

https://gerrit.wikimedia.org/r/508295

Change #1079680 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[operations/mediawiki-config@master] Enable {{USERLANGUAGE}} on Commons and Meta

https://gerrit.wikimedia.org/r/1079680

Please could someone suggest wording for this proposed Tech News entry, including what documentation it should link to. Thanks.

Change #1079680 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable {{USERLANGUAGE}} on Commons and Meta

https://gerrit.wikimedia.org/r/1079680

Mentioned in SAL (#wikimedia-operations) [2024-10-17T02:18:42Z] <tstarling@deploy2002> Synchronized wmf-config/InitialiseSettings.php: T4085 Enable {{USERLANGUAGE}} on Commons and Meta (duration: 06m 34s)

@tstarling Thanks! Is there a documentation page#section that it can link to?

I added an entry to https://meta.wikimedia.org/wiki/Tech/News/2024/43

In T4085#10238970, @Quiddity wrote:

@tstarling Thanks! Is there a documentation page#section that it can link to?

I added an entry to https://meta.wikimedia.org/wiki/Tech/News/2024/43

It should have been in https://mediawiki.org/wiki/Help:Magic_words , but it seems there's nothing there yet.

In T4085#10239021, @jhsoby wrote:

In T4085#10238970, @Quiddity wrote:

@tstarling Thanks! Is there a documentation page#section that it can link to?

I added an entry to https://meta.wikimedia.org/wiki/Tech/News/2024/43

It should have been in https://mediawiki.org/wiki/Help:Magic_words , but it seems there's nothing there yet.

I added something there. Feel free to merge to another section, tweak, etc.

Change #1083394 had a related patch set uploaded (by XtexChooser; author: XtexChooser):

[mediawiki/core@master] Mention wgParserEnableUserLanguage in changelog

https://gerrit.wikimedia.org/r/1083394

Change #1083395 had a related patch set uploaded (by XtexChooser; author: XtexChooser):

[mediawiki/core@REL1_43] Mention wgParserEnableUserLanguage in changelog

https://gerrit.wikimedia.org/r/1083395

Change #1083395 merged by jenkins-bot:

[mediawiki/core@REL1_43] docs: Mention wgParserEnableUserLanguage in release notes

https://gerrit.wikimedia.org/r/1083395

Change #1083394 merged by jenkins-bot:

[mediawiki/core@master] docs: Mention wgParserEnableUserLanguage in release notes

https://gerrit.wikimedia.org/r/1083394