English Banditry (Entries tagged as computers)

How to merge XML files for Supermemo CE using XSLT (especially for Chinese/Japanese)

nospam@example.com (your host) — Fri, 15 Aug 2008 11:53:14 -0700

Recently I've wanted to start adding more cards to my Supermemo CE collection. Much to my shock and disgust the only way to do it is to export XML, merge the two XML's externally, and then reimport. It's completely impractical when your collection is 3,000 plus elements and you just want to add ~50 new cards at a time. (This was MUCH easier on Palm Supermemo 5 years ago.)

This is the XSLT code (merge.xslt) that should do the trick for merging:

<xsl:transform version="1.0" 
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" 
            encoding="UTF-8"
            indent="no" />
  <xsl:template match="/">
    <SuperMemoCollection>
      <xsl:copy-of select="//SuperMemoCollection/SuperMemoElement"/>        
      <xsl:copy-of select="document('file_to_append.xml')//SuperMemoCollection/SuperMemoElement"/>        
    </SuperMemoCollection>
  </xsl:template>
</xsl:transform>

Then get some XML processing thinggy like nxslt (freeware) and just do:

nxslt.exe exported.xml merge.xslt -ascii -o reimportme.xml

The beautiful thing here is that your file_to_append.xml does NOT need to be Decimal NCR encoded. I am using a UTF-16 LE file with Japanese text right in there and merging it with the UTF-8 file that Supermemo CE exports, and nxslt converts the final output to the Supermemo CE required UTF-8 with Decimal NCR encoded Japanese automatically since I used the "-ascii" flag.

If you want to make it even easier you can use my Powershell script. Copy the contents of the zip to your SD card and then future additions to your collection will go this smooth:

Export collection in Supermemo CE
Plug you SD card into your desktop
Copy over new_stuff_to_add.xml to the SD card
Type: .\combineSMCExml.ps1 exported.xml new_stuff_to_add.xml
Import collection (exported.xml) back into Supermemo CE

The zip file containing merge.xslt, nxslt.exe, and combineSMCExml.ps1 can be downloaded here: combinesmcexml.zip.

WARNING:

I encountered a very frustrating problem with nxslt -- if there are *any* encoded HTML tags in new_stuff_to_add.xml (of the type < or >) then instead of getting the Decimal NCR encoded you need for Japanese/Chinese to show up in SMCE you will get a string of ???? question marks. ~~If anyone knows a workaround, please let me know.~~ 8/18: It seems upgrading to nxslt2 does the trick, the only problem is the '-ascii' tag disappears.

How to make your own Supermemo XML file (especially for Chinese and Japanese).

nospam@example.com (your host) — Fri, 18 Jul 2008 12:29:44 -0700

I am interested in getting items into my SRS with as little effort as possible. Every minute wasted inputting data is a minute that could have otherwise been used to learn more. I am creating templates to make JGloss (a program to add Furigana and definitions to Japanese text semi-automatically) export directly to Supermemo XML format. It can be a bitch because Supermemo gives no info why an XML file will not import when import fails. Here are some details I needed in the process:

Supermemo 2006 XML Format

I have experimentally determined that the files attached below are the MINIMUM of data needed for Supermemo 2006 to import one Question and Answer without any errors into a named folder. The text file can be in plain ASCII for English. If you want another language, the text file can be in UTF-8, UTF-16 little or big, or ASCII + Decimal NCR format. I recommend Babelpad for conversion, though notepad can save to UTF-8. The attached UTF-8, UTF-16, and NCR samples files have Japanese and Chinese text in them.

If you try to import to a template with text elements, you will get scrambled characters. You must import to a template with two html elements, one marked as the answer. (You can also 'apply' such a template after importing and the scrambled characters will be fixed.) If you do not set the template as an 'Item' template (leave it as a 'Topic' template) then you will probably have a problem -- so double check that if no Question or Answer contents shows up.

IF YOU WANT TO PUT HTML CODE BETWEEN THE <QUESTION></QUESTION> OR <ANSWER></ANSWER> TAGS, the HTML must be encoded! Plain HTML will interfere with Supermemo's XML parsing and the import will fail.

If you have any problems, my recommendations are as follows:

1. Create a brand new collection and brand new blank item. From the tool bar add two HTML elements to a blank Item (if there were any fields already on it, delete them) and then set one as the answer. Save this as the Default template and also name it. If import XML works in a new collection, you have deduced the problem lies somewhere in your existing collection.

2. Try importing all of the different sample files below. If even ASCII won't work, the problem is not Chinese or Japanese.

Sample XML files: ASCII (UTF-8) | UTF-8 | UTF-16 LE | UTF-16 BE | ASCII and Decimal NCR (UTF-8)

Supermemo CE XML Format

The minimum XML file required to import into Supermemo CE is a bit simpler as the ID element is not required. (That means you can get by without any programming -- just make a header XML, footer XML, several element XML's and then copy header.xml+1.xml+2.xml+footer.xml output.xml.) There is also no need for a extra outer topic element to create a folder as Supermemo CE is flat anyhow.

My experimentation shows: perfect results from English only (ASCII) and from Decimal NCR format when using Chinese/Japanese or a mix thereof. If there is ENCODED HTML between the answer or question tags, it will work properly if you tick the "HTML" box from in SM CE Mode menu. Results with HTML were ugly as hell though (terrible font) and there is NO SUPPORT for RUBY tags.

Sample XML files that work: ASCII (UTF-8) | ASCII and Decimal NCR (UTF-8)

Trackballs and Chinese Don't Mix

nospam@example.com (your host) — Fri, 23 May 2008 12:31:12 -0700

So I bought a trackball some months back when I started getting out of control carpal tunnel. One delectable side effect which I failed to anticipate was the sheer dumbfounding perplexity it seems to have brought upon my Chinese co-workers. And I have no idea why, they're all certified computer nerds. Now it isn't all that often that a coworker needs to come over and operate my terminal in order show me something, ... but today has to be about the twentieth time someone came over to my cubicle and scraped the entire Logitech unit across the bare surface of the desk. (Yes, the same people have done it on multiple occasions each.) It wasn't just that the girl wasn't looking -- after seeing that the cursor had failed to move as she directed it, she perplexedly looked at me and then pointed on the screen to the spot where she wished me to direct the voodoo magic cursor influencer.

Never mind how much it delights me to see their faces flash with exasperation when they try to tap something out on my Dvorak key-mapped keyboard.