Program to Split Single-Page HTML Documentation

[Computer Nerd Kev]

I don't like browsing huge single HTML pages of documentation. Does
anyone know of a program or script (preferably for Linux) that can
scan a big software manual's single HTML page and automatically
break it up according to the contents section and the corresponding
anchor links?

Basically I want something to turn this:
http://www.gnu.org/software/coreutils/manual/coreutils.html

into this:
http://www.gnu.org/software/coreutils/manual/html_node/index.html

But without the Texinfo source like GNU software (usually) uses.
Just from the HTML itself. I also want it to output static HTML, so
no solutions using Javascript or browser add-ons.

One option might be to use csplit to break it up at common section
separator patterns, then a simple script renames the new files
according to their heading text. But I'd like to have HTML
navigation links, ideally including converting existing anchor
links inside the document.

A prime target would be the Raspberry Pi configuration
documentation, which has convinced me of the merit of multi-page
docs by how confusing it has become for me since they switched to a
single-page layout:
https://www.raspberrypi.com/documentation/computers/configuration.html

[Bud Frede]

Post by Computer Nerd Kev
I don't like browsing huge single HTML pages of documentation. Does
anyone know of a program or script (preferably for Linux) that can
scan a big software manual's single HTML page and automatically
break it up according to the contents section and the corresponding
anchor links?
http://www.gnu.org/software/coreutils/manual/coreutils.html
http://www.gnu.org/software/coreutils/manual/html_node/index.html
But without the Texinfo source like GNU software (usually) uses.
Just from the HTML itself. I also want it to output static HTML, so
no solutions using Javascript or browser add-ons.
One option might be to use csplit to break it up at common section
separator patterns, then a simple script renames the new files
according to their heading text. But I'd like to have HTML
navigation links, ideally including converting existing anchor
links inside the document.
A prime target would be the Raspberry Pi configuration
documentation, which has convinced me of the merit of multi-page
docs by how confusing it has become for me since they switched to a
https://www.raspberrypi.com/documentation/computers/configuration.html

I've never used it myself, but I seem to remember talking to someone in
the past who used htmldoc to do this kind of thing?

https://www.msweet.org/htmldoc/

[Computer Nerd Kev]

Post by Bud Frede

Post by Computer Nerd Kev
A prime target would be the Raspberry Pi configuration
documentation, which has convinced me of the merit of multi-page
docs by how confusing it has become for me since they switched to a
https://www.raspberrypi.com/documentation/computers/configuration.html

I've never used it myself, but I seem to remember talking to someone in
the past who used htmldoc to do this kind of thing?
https://www.msweet.org/htmldoc/

Thanks for that. I've been to that website before but it's not
until you look in the README that it mentions the option of HTML as
an output format as well as for input. In fact the "htmlsep"
format option does exactly what I wanted:

mkdir rpi_config
htmldoc -t htmlsep -d ./rpi_config 'https://www.raspberrypi.com/documentation/computers/configuration.html'

It did pass through a few broken relative links that pointed to
other pages at the website, but I think I can live with that.

[Bud Frede]

Post by Computer Nerd Kev

Post by Bud Frede

Post by Computer Nerd Kev
A prime target would be the Raspberry Pi configuration
documentation, which has convinced me of the merit of multi-page
docs by how confusing it has become for me since they switched to a
https://www.raspberrypi.com/documentation/computers/configuration.html

I've never used it myself, but I seem to remember talking to someone in
the past who used htmldoc to do this kind of thing?
https://www.msweet.org/htmldoc/

Thanks for that. I've been to that website before but it's not
until you look in the README that it mentions the option of HTML as
an output format as well as for input. In fact the "htmlsep"
mkdir rpi_config
htmldoc -t htmlsep -d ./rpi_config 'https://www.raspberrypi.com/documentation/computers/configuration.html'
It did pass through a few broken relative links that pointed to
other pages at the website, but I think I can live with that.

I'm glad I could help out! :-)

[Carlos E.R.]

Post by Computer Nerd Kev
I don't like browsing huge single HTML pages of documentation. Does
anyone know of a program or script (preferably for Linux) that can
scan a big software manual's single HTML page and automatically
break it up according to the contents section and the corresponding
anchor links?
http://www.gnu.org/software/coreutils/manual/coreutils.html
http://www.gnu.org/software/coreutils/manual/html_node/index.html
But without the Texinfo source like GNU software (usually) uses.
Just from the HTML itself. I also want it to output static HTML, so
no solutions using Javascript or browser add-ons.
One option might be to use csplit to break it up at common section
separator patterns, then a simple script renames the new files
according to their heading text. But I'd like to have HTML
navigation links, ideally including converting existing anchor
links inside the document.
A prime target would be the Raspberry Pi configuration
documentation, which has convinced me of the merit of multi-page
docs by how confusing it has become for me since they switched to a
https://www.raspberrypi.com/documentation/computers/configuration.html

Interesting. I find multi-page docs confusing, I prefer single page. I
can do searches on the whole thing.

No, sorry, I do not know a tool to do the splitting automatically. I
would simply use an html editor and save chunks manually. LO could
possibly do it. Otherwise, I would try Composer in Seamonkey.

[Helmut Richter]

No, sorry, I do not know a tool to do the splitting automatically. I would
simply use an html editor and save chunks manually. LO could possibly do it.
Otherwise, I would try Composer in Seamonkey.

Web pages are made by lots of tools which yield readable or unreadable
HTML code. Things that make code unreadable are

– CSS for distributing text over the screen (this has decreased a
little since Web designers reckon with smartphones that have no
screen in the classic sense, i.e. at least 25 cm wide). CSS that works
locally (fonts, sizes, ...) is less of a problem, you only have to make
sure to repeat the declarations in each portion,

– a converter from something else, e.g. Word or PDF, to HTML, especially if
it is expected to convert also vice versa,

– many tools working simultaneously (a CMS, a WYSIWYG editor, a
converter of text formats, macro techniques like SSI, and – most
prominently – server side scripting like PHP and others) not
knowing which other tools are underway as well.

If you look into the HTML code without understanding anything you
cannot expect any tool to understand more. In simple cases, a tool
like HTML Tidy (https://www.w3.org/People/Raggett/tidy/) might improve
the situation but a tool working on everything is hardly conceivable.

Consequently, the proposed tool HTMLdoc excludes virtually everything:

While it currently does not support many things in “the modern web”
such as Cascading Style Sheets (CSS), forms, full Unicode, and Emoji
characters, ...

--
Helmut Richter

[Computer Nerd Kev]

While it currently does not support many things in "the modern web"
such as Cascading Style Sheets (CSS), forms, full Unicode, and Emoji
characters, ...

Well no, it doesn't exclude most software documentation because for
whatever reason the HTML in much of that has remained relatively
sane. Some CSS is creeping in, but a tool that ignores it still
produces clear text with some formatting. So far I've tested it
with two documentation pages published in 2023 and it's understood
the HTML fine (except for that out-of-page link problem in the RPi
doc). Plus often I'm looking at docs published 10-20 years ago
anyway.

[Helmut Richter]

Post by Computer Nerd Kev

While it currently does not support many things in "the modern web"
such as Cascading Style Sheets (CSS), forms, full Unicode, and Emoji
characters, ...

Well no, it doesn't exclude most software documentation because for
whatever reason the HTML in much of that has remained relatively
sane. Some CSS is creeping in, but a tool that ignores it still
produces clear text with some formatting. So far I've tested it
with two documentation pages published in 2023 and it's understood
the HTML fine (except for that out-of-page link problem in the RPi
doc). Plus often I'm looking at docs published 10-20 years ago
anyway.

Out-of-page links are trivial: replace each link "#xyz" by "subpage#xyz".
It is known to which subpage each link belongs, at least if you go over
the text in two passes. This is a procedure which I apply to all my web
pages, which are written as one document, and split into pieces later.

Another automatic change is to close and reopen each open tag:

<tag1>
a...
<tag2>
b...
---------------------------- cut here --------------
c...
</tag2>
d...
</tag1>

becomes

<tag1>
a...
<tag2>
b...
</tag2>
</tag1>
--------------------------was cut here ---------------
<tag1>
<tag2>
c...
</tag2>
d...
</tag1>

It is not obvious which attributes should be repeated in the second copy.
Mostly it is safer to repeat them, at least "class" and "style" attributes
and "id" attributes if they are referred to in style sheets.

[Helmut Richter]

Post by Helmut Richter
Out-of-page links are trivial: replace each link "#xyz" by "subpage#xyz".
It is known to which subpage each link belongs, at least if you go over
the text in two passes. This is a procedure which I apply to all my web
pages, which are written as one document, and split into pieces later.

It might be interesting to see an example of the TOC (table of contents)
of such a split article (https://hhr-m.de/sw-fibel/contents.html). It
contains all anchors in the whole article, which are possible but not
necessarily used (except from the TOC, of course) link targets. The link
structure might be still better visible if you look into the source code
of that web page which is fairly readable.

Sorry, the article itself is in German, a tutorial for learning Swahili.

[Computer Nerd Kev]

Post by Helmut Richter

Post by Helmut Richter
Out-of-page links are trivial: replace each link "#xyz" by "subpage#xyz".
It is known to which subpage each link belongs, at least if you go over
the text in two passes. This is a procedure which I apply to all my web
pages, which are written as one document, and split into pieces later.

It might be interesting to see an example of the TOC (table of contents)
of such a split article (https://hhr-m.de/sw-fibel/contents.html). It
contains all anchors in the whole article, which are possible but not
necessarily used (except from the TOC, of course) link targets. The link
structure might be still better visible if you look into the source code
of that web page which is fairly readable.

I think you misunderstood the problem. Perhaps I should have
explained that I would prefer it to rewrite relative links to other
webpages as absolute links.

As it is, a link like this:
<a href="/documentation/computers/processors.html#bcm2835">BCM2835</a>

From here:
https://www.raspberrypi.com/documentation/computers/configuration.html

Doesn't work when conveted unless the processors.html page is also
saved locally. Seeing as the program saw the source URL, I would
have liked it to be smart enough to turn such relative links into
absolute links when the link distination is another webpage.

This has fixed many of those relative links which had a directory
path:
for page in *.html; do sed -i \
's/<a href="\//<a href="https:\/\/www.raspberrypi.com\//g' $page; done

Pre-processing the page to rewrite relative links to other pages in
the same directory when the path isn't in the href, before running
HTMLDOC, would fix the rest.

Such as this:
<a href="config_txt.html#video-options">

It's not a major complaint.

[23k.304]

Post by Computer Nerd Kev

Post by Helmut Richter

Post by Helmut Richter
Out-of-page links are trivial: replace each link "#xyz" by "subpage#xyz".
It is known to which subpage each link belongs, at least if you go over
the text in two passes. This is a procedure which I apply to all my web
pages, which are written as one document, and split into pieces later.

It might be interesting to see an example of the TOC (table of contents)
of such a split article (https://hhr-m.de/sw-fibel/contents.html). It
contains all anchors in the whole article, which are possible but not
necessarily used (except from the TOC, of course) link targets. The link
structure might be still better visible if you look into the source code
of that web page which is fairly readable.

I think you misunderstood the problem. Perhaps I should have
explained that I would prefer it to rewrite relative links to other
webpages as absolute links.
<a href="/documentation/computers/processors.html#bcm2835">BCM2835</a>
https://www.raspberrypi.com/documentation/computers/configuration.html
Doesn't work when conveted unless the processors.html page is also
saved locally. Seeing as the program saw the source URL, I would
have liked it to be smart enough to turn such relative links into
absolute links when the link distination is another webpage.
This has fixed many of those relative links which had a directory
for page in *.html; do sed -i \
's/<a href="\//<a href="https:\/\/www.raspberrypi.com\//g' $page; done
Pre-processing the page to rewrite relative links to other pages in
the same directory when the path isn't in the href, before running
HTMLDOC, would fix the rest.
<a href="config_txt.html#video-options">
It's not a major complaint.

Ummm ... are you trying to do this STATICALLY, on
pre-existing HTML, or DYNAMICALLY, as users actually
access the pages ???

For the first case, a little Python will do wonders.
Identify tags, where you need to insert the absolute
parts of the paths, do it. Python is great with text
strings.

The second case ... not as easy. There you'd be best
off using PHP. Yea, JavaScript will do it too, but
it's always something<dot>something<dot>something<dot>
something<dot>something<dot> ... functional but it
gets UGLY/OPAQUE real quick :-)

[Computer Nerd Kev]

Post by 23k.304

Post by Computer Nerd Kev
This has fixed many of those relative links which had a directory
for page in *.html; do sed -i \
's/<a href="\//<a href="https:\/\/www.raspberrypi.com\//g' $page; done
Pre-processing the page to rewrite relative links to other pages in
the same directory when the path isn't in the href, before running
HTMLDOC, would fix the rest.
<a href="config_txt.html#video-options">
It's not a major complaint.

Ummm ... are you trying to do this STATICALLY, on
pre-existing HTML, or DYNAMICALLY, as users actually
access the pages ???

Statically, as stated in the first post.

Also no users besides me, browsing locally with file:// URLs.

Post by 23k.304
For the first case, a little Python will do wonders.
Identify tags, where you need to insert the absolute
parts of the paths, do it. Python is great with text
strings.

Well Sed will do that too with a smarter regex (using selection
brackets), and I don't like Python for reasons we've already
argued about. As it happens there are so few links without an
absolute path in that doc (I thought there were none until browsing
around after that first try) that I figured it wasn't worth
bothering with. Other docs may sufficiently motivate me eventually.

[Helmut Richter]

Post by Computer Nerd Kev

Post by 23k.304

Post by Computer Nerd Kev
This has fixed many of those relative links which had a directory
for page in *.html; do sed -i \
's/<a href="\//<a href="https:\/\/www.raspberrypi.com\//g' $page; done
Pre-processing the page to rewrite relative links to other pages in
the same directory when the path isn't in the href, before running
HTMLDOC, would fix the rest.
<a href="config_txt.html#video-options">
It's not a major complaint.

Ummm ... are you trying to do this STATICALLY, on
pre-existing HTML, or DYNAMICALLY, as users actually
access the pages ???

Statically, as stated in the first post.

Yes, there is nothing dynamic in it.

I had still another idea: leave the links as is, and map them to the
correct URLs by a rewrite in the server. The table of necessary rewrite
statements is static but can be modified if necessary. I discarded that
because it is too complex for too little benefit, if any.

Post by Computer Nerd Kev
Also no users besides me, browsing locally with file:// URLs.

Of course, the rewrite is not an option in this case as there is no
server.

[23k.304]

Post by Helmut Richter

Post by Computer Nerd Kev

Post by 23k.304

Post by Computer Nerd Kev
This has fixed many of those relative links which had a directory
for page in *.html; do sed -i \
's/<a href="\//<a href="https:\/\/www.raspberrypi.com\//g' $page; done
Pre-processing the page to rewrite relative links to other pages in
the same directory when the path isn't in the href, before running
HTMLDOC, would fix the rest.
<a href="config_txt.html#video-options">
It's not a major complaint.

Ummm ... are you trying to do this STATICALLY, on
pre-existing HTML, or DYNAMICALLY, as users actually
access the pages ???

Statically, as stated in the first post.

Yes, there is nothing dynamic in it.
I had still another idea: leave the links as is, and map them to the
correct URLs by a rewrite in the server. The table of necessary rewrite
statements is static but can be modified if necessary. I discarded that
because it is too complex for too little benefit, if any.

Post by Computer Nerd Kev
Also no users besides me, browsing locally with file:// URLs.

Of course, the rewrite is not an option in this case as there is no
server.

It's also more "hidden" ... which can getcha when you,
or somebody, has to debug/rewrite in a couple of years.

[23k.304]

Post by Computer Nerd Kev

Post by 23k.304

Post by Computer Nerd Kev
This has fixed many of those relative links which had a directory
for page in *.html; do sed -i \
's/<a href="\//<a href="https:\/\/www.raspberrypi.com\//g' $page; done
Pre-processing the page to rewrite relative links to other pages in
the same directory when the path isn't in the href, before running
HTMLDOC, would fix the rest.
<a href="config_txt.html#video-options">
It's not a major complaint.

Ummm ... are you trying to do this STATICALLY, on
pre-existing HTML, or DYNAMICALLY, as users actually
access the pages ???

Statically, as stated in the first post.
Also no users besides me, browsing locally with file:// URLs.

Post by 23k.304
For the first case, a little Python will do wonders.
Identify tags, where you need to insert the absolute
parts of the paths, do it. Python is great with text
strings.

Well Sed will do that too with a smarter regex (using selection
brackets), and I don't like Python for reasons we've already
argued about. As it happens there are so few links without an
absolute path in that doc (I thought there were none until browsing
around after that first try) that I figured it wasn't worth
bothering with. Other docs may sufficiently motivate me eventually.

I absolutely hate regex. To me a few clear lines of Python
are much more agreeable and obvious to all.

But, for static, existing, pages either WILL get you there
without much drama.

[Computer Nerd Kev]

Post by Carlos E.R.
Interesting. I find multi-page docs confusing, I prefer single page. I
can do searches on the whole thing.

Having both is really the ideal for me. But I read the multi-page
version and resort to the single-page one if I'm reduced to text
searching the whole document. On the other hand often it's useful
to do text searches limited to paticular sections/pages too.

If I set out to read a manual from start to finish then the
single-page version is good. But for reference use it's too easy
to lose my place in them. I usually open different parts in
multiple tabs. Five minutes after I've learnt something important
from a manual I've usually forgotten it again, but I remember the
tab or what the layout of the page looked like, so I can get back
to it quickly. That doesn't work so well on a massive single
webpage.

[Theo]

Post by Computer Nerd Kev
A prime target would be the Raspberry Pi configuration
documentation, which has convinced me of the merit of multi-page
docs by how confusing it has become for me since they switched to a
https://www.raspberrypi.com/documentation/computers/configuration.html

It doesn't answer the general question, but just to note that documentation
is generated from Asciidoc, and the source files can be found in their repo:
https://github.com/raspberrypi/documentation

Presumably there's a way to build Asciidoc to generate multi-page HTML, but
if not you could just read the .adoc files - github makes a fair stab at
rendering them.

Theo

[Computer Nerd Kev]

Post by Theo

Post by Computer Nerd Kev
A prime target would be the Raspberry Pi configuration
documentation, which has convinced me of the merit of multi-page
docs by how confusing it has become for me since they switched to a
https://www.raspberrypi.com/documentation/computers/configuration.html

It doesn't answer the general question, but just to note that documentation
https://github.com/raspberrypi/documentation

That's a good point. But I'm doubtful that Asciidoc converters will
easily give me the "previous contents next" links at the top and
bottom of pages, like published page-split HTML docs usually have.
The suggested HTMLDOC program generates those navigation links, so
I'll stick with it even for the RPi docs.

Post by Theo
Presumably there's a way to build Asciidoc to generate multi-page HTML, but
if not you could just read the .adoc files - github makes a fair stab at
rendering them.

GitHub unfortunately seems to have become a Javascript-only
solution now. Files and directories in repos can no longer be
viewed without Javascript (except for the top directory and the
readme, on the first page). Very disappointing.