Site History

Upgrading from HTML 4.0 to XHTML 1.0

for file in *
do
        while not validate(file)
        do
                fix_bug(file)
                make file
                make ftp
        done
done

fix_bug(file)
{
        if find_bug(file)
                edit file
        else if find_bug(file_generating_script)
                edit file_generating_script
        else if find_bug(data_file_for_file_generating_script)
                edit data_file_for_file_generating_script
        else if find_bug(xhtmlpp)
                edit xhtmlpp
                make install
        endif
}

Changing from Xhtmlpp to WML

Of course, looking at the process below, one must wonder why I was crazy enough to change over from xhtmlpp to WML in the first place, especially considering that I was maintaining xhtmlpp myself! Well, even though WML is noticably slower in processing the files, it has a notable advantage: it's more flexible! It follows the Unix Way: instead of having one monolithic program that does almost everything adequately, call lots of smaller programs that do one thing well.

while not make site
do
        for file in *
        do
                convert_to_wml(file)
                while feature_missing(file)
                do
                        make_feature(file)
                        make file
                done
        done
done

convert_to_wml(file)
{
        if not partial_conversion_script
                write partial_conversion_script
        endif
        partial_conversion_script(file)
        edit file
}

make_feature(file)
{
        if find_full_feature(wml)
                edit include_files
                edit file
        else if find_full_feature(internet_prog)
                install internet_prog
                edit include_files
                edit file
        else if find_partial_feature(wml)
                edit wml_files
                edit include_files
                edit file
        else if find_partial_feature(internet_prog)
                install internet_prog
                while not find_full_feature(my_feature)
                do
                        edit internet_prog
                        make internet_prog
                        edit include_files
                        edit wml_files
                        edit file
                        make file
                done
        else
                while not find_full_feature(my_feature)
                do
                        edit my_feature
                        make my_feature
                        edit wml_files
                        edit file
                        make file
                done
        endif
}

Changing from WML to XML+XSLT+embedded perl

Yeah, it seems I'm never satisfied. Why change this time? Well, because XML and XSLT are emerging standards with lots of programs and modules already written for them. And I could reuse what I already have. And because I was finding that I wasn't using all the features of WML; I was mainly using the new tag creation and the embedded perl stuff. Which could be done with a combination of XSLT and actual embedded perl (there are a couple of different ones).

And with so much less going on, by golly, it was faster!

Unfortunately the process of changing over was a bit more like the Illiad than Aesop's Fables.

The process:

  1. Buy and read the O'Reilly "Perl & XML" book.
  2. Investigate different kinds of embedded perl. Decide on eperl rather than Embperl.
  3. Fail to find an O'Reilly book on XSLT in my closest two academic bookshops, so make do with "The XSL Companion" by Neil Bradley instead. Horrible book. Can't tell a story straight, a bit like Eustace. Or maybe it's just not written for programmers.
  4. Start messing around with XSLT and eperl before finish reading book. Convert a sub-page on my home test site to use XML + XSLT + eperl + my own scripts.
    1. edit file
    2. process with xsltproc
    3. process with eperl
    4. view
    5. fix bugs, which may be in the source file, the embedded perl, the XSLT stylesheet, or may be something fundamental which requires the writing of a sed script
    6. solve other problems, such as how to include global default variables in files which may be just plain HTML
    7. repeat
  5. Also mess around with extracting nicely structured XML from my DeepFields data, and at the same time work on an XSLT stylesheet to transform said data into a HTML report (with the intent of replacing my existing rather complicated module which I create my reports with). Go through a few iterations, in an attempt to make the XSLT stylesheet not have to know any of the element names.
  6. Give up on The XSL Companion, and track down a copy of the O'Reilly "XSLT" book, buy it, and read it.
  7. Realize you've done lots of things wrong, and start to rewrite everything.
  8. Realize that you can do away with a separate eperl pass altogether if one can write a suitable XSLT-XML-using perl script with a module which allows extensions. Write said script (xsltweperl).
  9. Rewrite everything again.
  10. Realize that you can use the XInclude standard to include files; enable this facility in xsltweperl; rewrite the source files again.
  11. Solve the file-splitting problem by adding crude facility to xsltweperl. Write new report-generating XSLT stylesheets. Wrestle with numerous problems about previous-next links and generating report indexes, including strange problem where the generation would take ten times longer if I was online.
  12. Rewrite everything again.
  13. Write report-generating XSLT stylesheets which group by alphabetical character.
  14. Convert rest of test site except for CGI scripts.
  15. Start writing a navbar XSLT stylesheet.
  16. Realize that it's stupid to reinvent the wheel, and instead call already-written perl functions from within the XSTL stylesheet.
  17. Fix bugs in said functions to appease fussier XML parser.
  18. Convert the CGI search_fic.cgi script, calling it search_dfd.cgi; run afoul of HTML::Template pseudo-tags; write sed script to overcome this; write simple XSLT stylesheet for displaying the results with site look+feel which has no embedded perl.
  19. Rewrite the file-splitting stuff in xsltweperl.
  20. Solve the strange on-line delay problem (it was trying to parse the DOCTYPE in files whose content didn't match).
  21. Convert test sub-page on main site; using SSI for the first time. Realize that my previous efforts with SSI went overboard and put too much as SSI and gave myself problems I could have avoided by not using SSI. This time only implement menus/navbar with SSI. Debug.
  22. Discover Text::Template and replace HTML::Template in all CGI scripts with this because it enables one to change the delimiters of the templatable information. Toss the "simple" XSLT stylesheet and use a template for the results too.
  23. Convert another sub-page and install it on web-host to see if the validator check/referrer URL works. This fails. Try using SSI built-in variables such as DOCUMENT_URI to pass the URI to validate. Run afoul of fussy XML parsing. Adapt the sed script originally written for the CGI scripts to convert "disguised" SSI commands back into their real form as a last pass.
  24. Install todo-list software and make lots of todo lists.
  25. Convert more sub-pages (but don't install). Run afoul of HTML entity parsing. Introduce separate namespaces and two different kinds of source files.
  26. Add stuff to xsltweperl to remove unneeded xmlns attributes.
  27. Add navbar-changing links to pages. Debug.
  28. Implement more of the extra tags that were in the WML version. Debug.
  29. Remove the anti-xmlns things from xsltweperl; make a separate sed script for that instead, to use as a last pass.
  30. Fix problems with side-navigation navbars.
  31. Split AsciiDB::DeepFields into AsciiDB::DeepFields and AsciiDB::DeepFields::XML; during the course of this, move the perl source directories around.
  32. Convert more sub-pages.
  33. Write sed script to help semi-automate conversion.
  34. More revisions of report-generators.
  35. Fix references to now-moved perl source directories.
  36. Get rid of redundant classes from CSS stylesheets.
  37. Give the TopNav navbar a separate class and background. Debug. Netscape 4 is unable to cope. Attempt to debug. Give up and force Netscape 4 (with SSI) to never use TopNav.
  38. Convert and customize another sub-page which uses the report-generators. Do lots of tweaking to both XSLT stylesheets and to the input data. Rename the data file extension from .ffd to .dfd
  39. Rewrite everything that used to reference .ffd files
  40. Solve some lingering problems with the border and overlap of elements in TopNav.
  41. Sit down and wonder if you will ever finish.
  42. Convert more sub-pages. Go back and re-make already-converted pages to see if they still work. Debug.
  43. Revamp CSS stylesheets to be grouped as "layout", "theme" and "theme_img", and hide theme_img from Netscape 4 by using @import for it. Try this out to see if Netscape 4 likes this version of TopNav. It doesn't. Try repeatedly. Give up and force Netscape 4 (with SSI) to never use TopNav.
  44. Update all files that reference the stylesheet variables.
  45. Combine the "theme" and "theme_img" CSS files into just "theme" ones.
  46. Update all files that reference the stylesheet variables.
  47. Discover the existance of the "use alternate styles" menu in Mozilla. Implement some alternate styles.
  48. Update the about-this-site page.
  49. Bang head against inconsistent interaction of perl scripts, included file output thereof, and namespaces, and decide to re-introduce an eperl pass for one file, with this difference; the eperl pass comes before the XSLT rather than afterwards.
  50. Convert some more sub-pages.
  51. Go the whole hog and make every "theme" style available as an alternate stylesheet. Of course this is only useful to people with advanced browsers such as Mozilla (no, IE does not have this facility!)
  52. Wrestle with pod2html while converting the tools sub-pages.
  53. Resurrect my old "Jumping On The Breeze" look by adding it to the alternative stylesheets. What a fun toy!
  54. Rush ahead and implement the a "print" stylesheet, then wonder why it doesn't work. Look up the W3 definition, and fix problems.
  55. Convert the works-related sub-pages. Wrestle with customizations of report-generation and indexes, as well as trying to change Text-To-HTML options depending on whether a file was a poem or not.
  56. Battle with the Tiles page conversion.
  57. Update the about-this-site page.
  58. make clean-all; make; make install
  59. Check to see if there's anything missing...
  60. Stop.

Changing from XML+XSLT+embedded perl to Posy

I've forgotten the details, actually.

I made a mirror-directory of my original site, moving the contents into my new Posy data directory, with a script which did automatic replacement of certain strings, calling up an editor to do the last bits, and copying the results into .html files in the right spots. I did this a directory at a time.

At the same time I was actually working on Posy, first to get the basic functionality working, then, as that was in, writing plugins to give me all the features I needed to be able to have all the old things I did with my site originally, such as navbars, CSS layouts, browser detection, table-of-contents, random quotes, anti-spambot mail obfustcation, site search, custom Not Found page, and so on.

I was also finding that certain parts of the site were better off using the original pre-generation scripts to pre-process them, and using makepp and rsync to copy the results into the Posy data directory.

There were also certain CGI scripts I needed to have run independently of Posy, so wrote more stuff to be able to have these use the Posy template files to generate their own template files, so that the site look-and-feel would match. Then I scrapped that, instead making a Posy plugin which would enable the CGI scripts to be just dropped into the Posy data tree, and their output then processed by Posy as if it came from an entry file.

Changing from Posy to EmbPerl and XSLT

Yes, I decided to go back to XHTML+XSLT+embedded perl, but with a difference. Now that I am running my own server, I can use something which works well when one can control the server, and that's EmbPerl; it can use mod_perl and thus be very fast. And I can get both embedded perl and XSLT, in a much cleaner way than I was doing it before. And it overcomes some of the irritating limitations of Posy.

The process:

  1. Put the site under subversion.
  2. Remove old cruft.
  3. Make a development branch for converting to EmbPerl, so that I can work on a branch without having to forego updating the main site; just merge the changes in.
  4. Set up an EmbPerl virtual host for testing.
  5. Move all the posy_data/docs files into the html/ directory.
  6. set up XSLT framework for processing XHTML files
    1. basic processing
    2. convert all layouts: top, side
    3. convert all themes
    4. run tidy over all HTML files to fix DOCTYPE
  7. Alter all the Makefiles in the src directory to do the same, as well as fixing them so that they actually put stuff in the new directory rather than the old one (oops!).
  8. different default themes for different directories
  9. make ks:tag tag library
  10. move the file-generation of things like Fiction back under src
  11. attempt to write a Text-To-HTML Embperl::Syntax module for converting text, give up, do the conversion in the base.epl file instead.
    1. finding the .txt file from the .html url
  12. convert some .blx files to HTML and parse the others
  13. Categories (navigation, site map)
    1. get navigation.html working
    2. breadcrumb-navigation (breadcrumb plus near and below)
  14. make index.html files for empty directories
    1. about/
    2. fandom/sentinel/cascade_times_awards/
    3. thoughts/
    4. posy/plugins/
  15. write the equivalent of Posy directory display
    1. fandom/sentinel/episodes/
    2. fandom/stardig/episodes/
    3. history/
    4. reviews/books/
    5. reviews/misc/
    6. reviews/movie_tv
    7. updates/
  16. replace all CGI scripts
    1. polls (removed)
    2. gallery - replace with a static gallery
      1. Write own gallery-generation script, khatgallery
    3. fiction show.cgi (show.epl)
    4. netfic show.cgi (show.epl)
  17. Anti-Spambot
  18. browser sniffing
  19. reproduce EntryTitle
  20. search
  21. Replace Info or make it unneeded
  22. custom error pages
  23. pre-generate all POD documentation (in src/)
  24. RandQuote
  25. ThemeCss
  26. Table-of-Contents
    1. go through and change all image-only headers
    2. enable ToC to be turned off (for things like POD)
    3. add ks:toc tag to enable TOC in other places
    4. different types of toc
  27. replace all <?perl ?> with Embperl or pre-generate said files using teperl
  28. replace or remove graphical headers which no longer match themes
  29. fix up the footer content
  30. correct things in "about" which no longer apply

Retrograding from EmbPerl and XSLT to embedded perl, XSLT and SSI

Yes, I'm going back and forth, aren't I? I decided to move away from EmbPerl for a few reasons:

  • The creator of EmbPerl was getting slower to respond to problems; this came to a head for me when an upgrade of Apache caused EmbPerl to stop working, and I got no useful help.
  • The navigation, layout and themes could just as easily be pre-generated as dynamically generated; there's no real pressing need for them to be dynamic. I could reuse my existing XSLT stylesheets (with modifications) for the layout and themes, and teperl for the embedded perl navigation stuff.
  • The few parts of the site that really need to be generated dynamically could be done by using Server Side Includes to call CGI scripts which generate the required content.

I also discovered omake, which I thought would make the process of pre-generating the content a bit easier than using GNU make.

The process:

  1. Convert the existing file pre-generation (under the "src" directory) from make to omake.
  2. Add the site HTML content to the "src" directory and set up the omake framework to do "dummy" pre-generation: that is, just copy the source file to the destination file.
  3. Also do pre-generation for text files that need to be converted to HTML.
  4. Go through the pages that use extra external .epl files to generate (some of) their content, and decide which ones can be pre-generated by using teperl, and which ones need to be really dynamic.
    1. Write the teperl (and possible scripts) for the pre-generated ones.
    2. Write (or find old versions of) CGI scripts and set up the SSI for the ones that need to be dynamically generated. (This was aided by my discovery of the XBitHack option in Apache, which enables files to be marked for needing SSI processing without having to rename them).
  5. Disable the text-to-html processing in the EmbPerl application and see if that breaks anything; fix it.
  6. XSLT:
    • Write script to do the XSLT pre-generation with the existing stylesheets.
    • Add the XSLT processing to the pre-generation process.
    • Fix up problems with DTD verification; set up catalog files for the XHTML DTDs on my system.
    • I was going to remove the XSLT processing from the EmbPerl application, but it was easier to just turn off Embperl at this stage.
  7. Embedded perl:
    • Update the XSLT stylesheets to use teperl rather than Embperl; this includes taking the global .epl scripts and making them perl scripts.
    • Put in a Text::Template pass into the pre-generation script.
    • Change the configuration setup to use Config::Context in one file rather than the Embperl config.pl perl files. (Config::Context can be set up to behave a bit like Apache config files, which means that one can have global settings which get overridden by more specific settings)
    • See what breaks; fix it.
  8. Files generated by scripts (mainly the Net-Fic reviews and the Gallery):
    • Go back and forth between post-processing these files (to insert navigation etc) and pre-processing the templates they use.
    • Post-process the Net-Fic reviews and don't post-process the Gallery.
  9. More embedded perl:
    • Break out the embedded perl stuff from the template-processing script and make it a separate script.
    • Switch the embedded perl from Text::Template to Text::NeatTemplate to try to speed things up.
    • Update the KatSpace::WebUtil module to integrate with Text::NeatTemplate rather than Embperl. Fix up some of the remaining non-navigation perl processing. (Things like email anti-spambot stuff)
    • Move the content of the navigation-generation scripts into KatSpace::WebUtil to speed things up.
  10. More XSLT:
    • Remove the special ks:entity tags which I put in to preserve entities which were converted to the actual characters by the XSLT processing. Fix the entity thing by using recode to convert the "diacritic" characters back to HTML entities. This has the added benefit of fixing up problems with &nbsp; entities. Note that this recoding is only done on pages where I know it is needed, rather than all pages, so as to speed up processing. (Alas, it's needed on the Net-Fic reviews pages, which means, with all the post-processing required on them, it takes about half an hour to generate the Net-Fic reviews pages. Oh well, can't be helped.)
    • Convert the special ks:toc tags into processing-instructions. This is mainly because htmltidy doesn't cope with special tags when doing xhtml, but processing instructions are fine; I want to be able to use htmltidy on my source files if need be.
  11. Update the About section with the new setup.

Combined Static Site and PmWiki

I didn't record the steps I did for this.

IkiWiki

A wiki compiler (written in Perl) from ikiwiki.info

[[!format Error: unsupported page format pmwiki]]
< Site FAQ | About | Site Logos >