Repository navigation
Adapt the scripts to the 2026 upgrade of the website - #53
Open
cristianadam wants to merge 7 commits into
Open
cristianadam wants to merge 7 commits into
cristianadam wants to merge 7 commits into
Conversation
The Qt Help output is read in Qt Assistant and in Qt Creator, and the pass that built it was written when Assistant rendered with QTextBrowser, which follows no stylesheet and knows a small corner of CSS. So the pass flattened cppreference's stylesheet into `style` attributes, threw the classes away and rewrote the layout as nested tables, and the pages read like it. Both viewers render with litehtml now -- Assistant since Qt 6.2, Qt Creator since 4.11 -- which does follow a stylesheet, so give them the site's own and let them lay the pages out the way a browser does. The stylesheet. The pages keep their markup and their classes and link one file, which `commands/qch_css.py` builds out of `ext.css`, `site_modules.css`, the seven distinct `<style>` blocks that turn out to lie behind six thousand pages, and `qch-viewer.css` for what a help pane needs on top. The archive drops from forty-four megabytes to thirty-two on the way, most of it the navigation menus every page carried twice: litehtml has `:hover` and would have worked them, but a help pane is a poor place to reach for a menu and both viewers put a Contents and an Index beside the page already. A color scheme. Asked for `offline-simple.css`, both viewers serve `offline-dark.css` in its place when their own theme is dark, and `offline.css` when it is light; Qt Creator 21 does the same for a page that asks for `offline.css` outright. So the pages ask for the first and the light sheet is written under both of the others. The dark sheet is the light one followed by a counterpart for every color in it, mapped from the light color rather than written out by hand, so that it keeps up with cppreference on its own; the mapping holds a dark surface to WCAG AA against the body text and keeps chroma rather than HSL saturation, so that a pale tint does not come out vivid. The counterpart is written for the declaration the light cascade settles on and not for every declaration along the way: the derived rules go at the end, where they outrank the whole of the sheet above them, so a color that something further down takes back would be a color the dark page has and the light page has not -- `ext.css` gives inline `code` a background and `site_modules.css` makes it transparent again, and deriving from the first of those would put a grey box behind every type name in the archive. That is read across the sheets together, because the cascade runs across them too. Rules the site wrote for `print` are left to it. What no rule can reach is the color a wiki template wrote into a `style` attribute -- the green and red of the compiler-support tables -- so `preprocess_qch_html` names each of those with a class and the dark rules for them say `!important`, which litehtml honours over the attribute as the cascade asks. Type sizes. The site is written against a browser's 16px default and takes a fifth off its content to read at about 13px; a help pane starts from the font the reader chose for it, which is already the size they want to read at, so the fifth was coming off twice and the page came out small. Let the content sit at the reader's size, and pull the proportions around it towards the middle by raising each to a power. The sizes of a cascade multiply and a root distributes over a product, so compressing every declared proportion by the same exponent compresses the size each element ends up at by exactly that exponent, and nothing has to work out what an element inherits. It does have to be every declared proportion, the eleven thousand a wiki template wrote into a `style` attribute along with the ones in the sheets, or whatever hangs under one of those is a sixth smaller than the page around it. The content lands at 0.89 of the reader's size and a page title at 1.26 of it -- 14px and 20px of a 16px Help font -- which is the scale Qt's own documentation reads at, and the whole of it still follows the Help font size. Fonts. A font stack is worth its first name and no more: litehtml hands the whole list to the viewer, the viewer hands it to `QFont::setFamilies`, and that list is for glyph coverage rather than for choosing a face -- Qt resolves the first family and substitutes its own default when it is missing instead of trying the next. cppreference names DejaVu first throughout, which hardly anyone outside Linux has, so every code sample came out in a proportional font. Each stack is cut down to the first generic in it, those being the names qlitehtml answers with faces of its own; a stack that comes down to `sans-serif` asks to inherit instead, which is the reader's font at the root and still takes an element back out of a monospace rule further in. The `style` attributes are cut down with them: three thousand of those name Georgia first, which is no better installed than DejaVu. Courier, which is what a viewer gives us for `monospace`, draws larger than the proportional face beside it, so a type name in the middle of a sentence stands up out of the line; ask for 85% of it. That belongs to the face and not to the text, so it is taken where the monospace starts and not again inside it -- and monospace nests on every page, a `<pre>` inside a highlighted block inside a monospace cell being three deep. There is no way to ask CSS for "unless you are already inside one", so the pairs are written out. An element that is set proportionally goes back to the size it inherits for the same reason: the member names of the declaration tables carry a monospace rule and a proportional one on the same element, and the face they end up in is the reader's own, which needs no correcting. What a stylesheet cannot do is left to `preprocess_qch_html`. A MediaWiki template asks a cell to span 142 rows as a way of saying "all of them", and litehtml lays a table out as a flat run with no notion of the row groups a browser stops a `rowspan` at, so the markers that number the overloads ended up nowhere near them; say what we mean. litehtml has no `writing-mode` either, so the column headings of the compiler-support tables came out lying across one another: the stylesheet lays them out across and lets them wrap, and the row they were standing in no longer needs the height, which no selector can reach from the heading. Images. Only the HTML was carried into the archive, so every figure the pages draw on -- the diagrams of what `begin` and `end` point at, the inheritance charts of the exception types -- arrived broken. The ones the pages and the stylesheet ask for are taken now, the page icon among them, which is fewer than the site has. The skin images came down with their cache-busting query percent-encoded into the file name, and nothing renamed the files to match, so those references are mended on the way in. A raster image is handed the white background it was drawn against, without which its lettering disappears. A drawing does better than being plated, because the viewer tells it which page it is on: both Qt Assistant and Qt Creator put `class="dark"` on the root <svg> element when their own theme is dark. So `commands/qch_svg.py` gives each one something to answer with. Name every color it is drawn with by a class, state under that class the color it already had, and state the dark counterpart under `svg.dark`, mapped by the same palette the stylesheet is mapped by -- a dark color lightened the way the page's text is, a light one darkened the way a surface is, which carries a drawing over whole. The marks lighten, the boxes behind them darken, and the grey of a muted label stays the step off the lettering that it was. Nothing is taken away. A color stated as a presentation attribute is left where it stands, since a rule already outranks one and it is what a renderer that reads no CSS at all goes on drawing; only a color in a `style` attribute has to be moved, that outranking every rule. A viewer that does not know the trick never matches `svg.dark` and draws exactly what it drew before -- which is why the drawings are also handed a backdrop. They have no ground of their own; they were borrowing the page's, and the plate is now kept off them. Three things Qt's SVG renderer settles rather than the standard. It honours a class selector but not an attribute one, so the classes are written out rather than the colors matched where they stand. It applies no rule to the root <svg> element at all, so what the root is drawn in is restated on a <g> put around everything it holds -- and the root must not be given a class in any case, the viewer writing one onto it and a second `class` attribute beside the first not being XML. And it reads a gradient stop's color off the stop and never asks the stylesheet, so a gradient cannot be restyled; `fill: url(#...)` is honoured, so each one is copied into a dark twin and the reference is switched instead. The lettering asks for no color at all and takes the black SVG paints with when nothing says otherwise, so that is said outright where a mark would otherwise inherit it -- but not inside <defs> and the like, where a mark is stood up by a <use> somewhere else and takes its color from there, which is how the plot markers of the graphs are colored. The same pass gives a drawing the pixels to be drawn sharply. Qt rasterises an SVG at the size the file declares and takes no account of the ratio of the display, so on a Retina screen the bitmap is blown up to twice the pixels it was drawn with and a hairline diagram comes out soft; the viewer's own zoom multiplies on top of that. So declare the drawing at twice the size and leave what it is drawn at to the page, which sizes every one of these figures with a `width` and a `height` on the <img> element -- litehtml maps the two onto the CSS properties and only falls back on the file's own size when neither is given. Nothing moves; there is simply more of the drawing to draw, and a display that is not Retina scales it back down smoothly. Where a drawing states no `viewBox` the box it has been drawn in all along is stated, without which a larger `width` would only lay out a larger canvas. Against the sixty-four drawings the archive carries, Qt's renderer gives back the light rendering pixel for pixel, every one of them goes dark, and each is rasterised at twice the pixels the page lays it out in.
Builds the .qch on every push and attaches it to the release when a version is tagged, so the archive people actually download is built the same way every time rather than on whichever machine happened to have a qhelpgenerator on it. The raw copy of the website is the awkward part: it is not in the repository, it is 475 MB, and crawling en.cppreference.com on every run is what the README asks us not to do. It does not have to be crawled, though -- `make release` already packs `reference/` into `cppreference-doc-<version>.tar.xz`, so the workflow takes the site copy out of an earlier release's source archive, which is 13 MB and unpacks in about a minute. Refreshing the content for a real release is then a matter of running `make source` locally and publishing a new source archive; the workflow picks it up as the latest on its own. Which repository that archive comes from cannot just be this one. A fork has no releases until it makes one, so the run asks GitHub what this repository was forked from and works its way up until it finds a source archive, which keeps the workflow building in a fork while still preferring a release of its own once there is one. The dispatch inputs name a repository and a tag outright when some other content is wanted. An artifact and a release asset want different things. GitHub zips an artifact whatever it is handed and serves the result under the artifact's name, so the artifact is the .qch itself -- handing it the release archives makes a zip holding an archive holding the one file anybody wanted, compressed twice over. A release asset is served as it is, so the tar.xz and the zip `make release` writes are built only for a tag, where they are the real thing. qhelpgenerator comes from Ubuntu's `qt6-documentation-tools`, whose path is asked of dpkg rather than written out, Qt keeping the binary under /usr/lib/qt6/libexec and being free to move it. A tag that disagrees with VERSION in the Makefile stops the run, rather than publishing archives whose names do not match the release they hang under. The .qch is a SQLite database, so once built it is asked whether the pages, the images and both stylesheets arrived in it. pylint is left out of the checks, unlike `test.sh`: `config/pylintrc` predates pylint 3 and is rejected across the whole repository, which is its own change to make.
The site was upgraded on 17 April 2026 (MediaWiki 1.43): the wiki moved from /w/ to the root, the loader and the skin from /mwiki/ to the root as well, and the uploaded images to upload.cppreference.com/images/. The crawl started at a URL that is now a redirect. What the crawl cannot do anything about is that the site did not move with itself: the pages still carry <img> links to the old /mwiki/ path of every image, and MediaWiki:Common.css still names the old path of the navbar images, all of which answer 404. fixup_source.py reads the copy back, works out what it asks for and fetches those from where they live now, next to the files wget did get. /mwiki/ is left out of the crawl so that wget does not walk into the 404s on the way. The fonts and the favicon the site served from its root are simply gone, and are nowhere else to be had; nothing downloads them any more. export.py no longer runs: the site answers 403 to any request that names more than one page -- Special:Export and the API alike -- and a copy of the wiki source is not worth 19000 requests. Its user agent and its handling of the continuation the API has been sending since MediaWiki 1.26 are fixed anyway, so that it works against a wiki that allows it.
The archive is rearranged out of the layout the site has now: the pages sit at the root of en.cppreference.com, with the loader's style sheets beside them and the skin under /skins/ and /resources/, and the uploaded images come from upload.cppreference.com/images/. The loader no longer serves one style sheet for the whole site but one cut to measure for each page -- the skin, plus whatever extensions that page draws on -- which is the same rules in a dozen combinations. They all have the one name, so the one that asks for the most modules is kept and every page is pointed at it; if the combinations should ever stop being nested, the build says so rather than losing rules quietly. Links are taken as they are found: the loader links are built from the name they end up with rather than adjusted, since they are relative on a page whose style sheet wget got and absolute on one where it did not, and the images are linked to the old /mwiki/ path throughout. The style sheets name the version of each image the loader serves it with, which is part of the file name in the copy and was renamed away; the @font-face rules that point at the fonts the site no longer serves are dropped, which leaves the locally installed faces and the generic families to fall back on. Every page carries the New Relic agent inlined into its head, 80 kB of it, which is most of what a page weighs; it goes the way of the Google Analytics tag.
MediaWiki used to hand the page configuration to mw.config.set in the page; since 1.43 it assigns it to RLCONF for the resource loader to pick up. Nothing matched, so the map came out empty -- and with it the devhelp index and the contents of the .qch.
The parser hands back no element at all -- rather than raising -- for a document with nothing in it, and every call on the tree after that fails an assertion, which takes the whole run with it. The .qch pass has said so and carried the file through since it learned to read the pages; say so here as well. A copy packed on macOS is one way to come by such a file: tar writes the extended attributes of each file as an AppleDouble file beside it, and `._cpp.html` is picked up as a page like any other.
Since the upgrade every figure on the site is wrapped in a link to the page describing the file, which the pass that takes out the "About this image" badge mistook for one: it removes the element around such a link, and took 119 figures with it -- the diagrams of what begin and end point at, the inheritance charts, the plots of the math functions. A link the site marks as mw-file-description now has the link taken off it and the figure left where it is. Figures uploaded as bitmaps are offered at a few sizes for the viewer to pick from by how dense the display is; those links are carried over with the rest, and the .qch takes the files along.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #51.
The site was upgraded on 17 April 2026 (MediaWiki 1.43) and the download and preprocessing scripts have not worked since. This takes them through the move, and a copy taken with it builds all four outputs.
The crawl. The wiki moved from
/w/to the root, and the loader and skin out of/mwiki/, so the crawl starts athttps://en.cppreference.com/and the archive is rearranged out of the layout the site has now: pages at the root ofen.cppreference.com, the loader's stylesheets beside them, the skin under/skins/and/resources/, the uploaded images underupload.cppreference.com/images/.What the crawl cannot reach. The site did not move with itself: the pages still carry
<img>links to the old/mwiki/path of every image, andMediaWiki:Common.cssstill names the old path of the navbar images -- all 404 now. The newfixup_source.pyreads the copy back, works out what it asks for, and fetches those from where they live now;/mwiki/is left out of the crawl so wget does not walk into the 404s. The fonts and the favicon the site served from its root are simply gone and are nowhere else to be had, so the@font-facerules that point at them are dropped and the stacks fall back on the locally installed faces and the generic families.The loader no longer serves one stylesheet for the site but one cut to measure for each page -- the skin, plus whatever extensions that page draws on, which came to a dozen combinations in this copy. They all map to the one name, so the one asking for the most modules is kept and every page points at it; if the combinations ever stop being nested, the build says so rather than losing rules quietly.
The link map read the page name out of
mw.config.set, which 1.43 no longer emits -- it assignsRLCONFfor the resource loader instead. Nothing matched, so the map came out empty, and with it the devhelp index and the contents of the.qch. This one is easy to miss: the build still runs to completion.New Relic. Every page now carries the agent inlined into its head, 80 kB of it, which is most of what a page weighs. It goes the way of the Google Analytics tag: the raw copy is 985 MB and the processed one 432 MB.
export.pyno longer runs frommake source: the site answers 403 to any request naming more than one page --Special:Exportand the API alike -- and a copy of the wiki source is not worth 19000 requests. Its user agent and its handling of the continuation the API has been sending since MediaWiki 1.26 are fixed anyway, so it works against a wiki that allows it. Nothing in the build reads that XML.Checks
make allruns clean end to end on a copy taken on 12 October 2026: 7033 pages (6638 in the February copy), and the.qchhas 7128 files and 39056 index entries against 6641 and 39056 in v20250209. In a 400-page sample ofoutput/referenceall 54894 local links resolve, and everyurl()in the stylesheets the.qchcarries resolves too.python -m unittest discoverand flake8 pass; there are new tests for the loader and upload links and for the stylesheet rewriting.The version is deliberately not bumped here -- that and the copy of the website are yours to cut. A built archive and the source package it came from are at https://github.com/cristianadam/cppreference-doc/releases/tag/v20261012 if you want to look at the output without crawling the site yourself.