Old news (Jan 09 - Jun 09)

Escuela de Ladrones

And now tape traps have been added for loading complete TAP files containing games, or other programs, off a mounted file system.

They work in the same was as in the DivIDE, or ZX-CF using ResiDOS - you tell the system which TAP file you're going to load, then type LOAD "" - and instead of being loaded off tape, in the case of the Spectranet, it's loaded off the network filesystem.

The %tapein command (which will be familiar if you have an add-on with ResiDOS - since there is a ResiDOS tape module with that command) prepares everything for the subsequent loading of the file:

• It opens the file itself, and reads a little bit of the information (gets the length of the first block, which will usually be the 17 byte header - in most cases for the BASIC program that loads the rest of the game).
• If the file can be opened and read OK, it then sets up the programmable execution trap at address 0x0562, which is in the Spectrum ROM's LD-BYTES routine - the routine that listens to the sounds coming from the tape, transferring the data into memory.

Then the user types LOAD "". Every time the CPU gets to 0x0562, the Spectranet CPLD fires an NMI. This NMI causes the Spectranet ROM to get paged, and the NMI handler runs, where it sees that the cause of the NMI was the programmable execution trap. It then passes control to the Spectranet BASIC extension module, which then reads in the next tape block over the network - once this is done, control is returned back to the ZX ROM in the SA/LD-RET routine (which handles the return of the loading and saving routines). This keeps on happening until either the program being loaded stops loading more parts, or we hit the end of the TAP file - or more usually, both happen at the same time since most TAP files end when everything has been loaded.

The upshot is it means any TAP file can be transparently loaded, and as far as the Spectrum is concerned, it's no different to loading a tape. I've loaded a few things for testing - a TAP file for Manic Miner (of course!), a few of Mister Beep's 1 bit music demos, and the 128K game that was released about 6 months ago, Escuela de Ladrones (Thieves School). The latter showed me I had my original tape trap in the wrong address, but once that was fixed it too loaded perfectly (in a couple of seconds instead of 7 minutes).

There's still work to do - there are a couple more system calls and TNFS protocol bits that need to be implemented, and some that really ought to be modified because they are unnecessarily complex. I'll also make a video if I can find the time that demonstrates all of this.

Winston 22:26, 17 June 2009 (BST)

Manic Miner

An important milestone...

Manic Miner was loaded over the network!

After a long gap of not doing any work at all on the Spectranet (thanks to being away in Scotland for a week, then having too much other work to do, then supporting my Dad in the Pre-TT classic and then in the sidecar TT races (if you don't know, motorcycle racing - the Isle of Man TT is the world's biggest and best known road race, run on public roads)...well, not much has been done since the last time I updated this page.

I've made up for it this weekend, with quite a lot of productive work done (although some of it frustrating), including:

• Finding a pretty serious bug in sendto which resulted in the source port getting set wrong and one byte of the MAC address getting overwritten, if a source port was set. (This may be the cause of the woes some have reported with DHCP)
• Turning the prototype TNFS code into a VFS ROM module. All now called via the new VFS layer. Although the TNFS module is part of the default Spectranet ROM, it's actually just another ROM module, apart from having a couple of system variables for its own use in the sysvars area in 0x3F00, it's no different to a ROM module written by a 3rd party (and therefore can act as example code, too).
• Turning the prototype BASIC extensions code for filesystems, into something that calls the proper VFS calls (and therefore will work with any filesystem, not just the TNFS).
• Then turning the BASIC extensions into a ROM module.
• Fixing a serious bug in the module loader utility (the utility that loads new ROM modules over the network and flashes them into the ROM). This bug caused any module larger than 1K in size to get corrupted.

There's still quite a bit to do, but at least you can now turn the Speccy on, go into BASIC, mount a file system, and load and save files, list directories, change directory etc.

As a proper test, I loaded all the bits of Manic Miner (the BASIC loader, the SCREEN$ and the game code) via the tape port, then saved them over TNFS - modifying the BASIC loader to use %load instead of LOAD. So if I now %load "manic" - Manic Miner loads over the network. It's a very important milestone that the first real piece of Spectrum software has now been stored on a network file system and loaded back.

There are now several layers involved. Closest to the user, are the BASIC extensions - %cat, %load, %mount etc. These parse the BASIC entered by the user, and in turn call the appropriate system call. For example, when saving a file, the BASIC interpreter sees the save command, gets the arguments to find out what's being saved - then calls open/write/close to save the file and returns to BASIC. In turn, the open/write/close system calls go via a dispatcher, which figure out what should actually receive these calls - which will depend on the mounted filesystem. Once handed off to the filesystem module, this then does whatever it needs to write the file - in our case, sending TNFS commands to the server to open, write, then close the file. The important thing here is that the dispatcher isolates the BASIC extensions from TNFS. Or any user program that uses the filesystem system calls from the actual underlying filesystem.

Winston 18:37, 7 June 2009 (BST)

VFS

As I said last time, the current job is to integrate what's been done on the network filesystem into ROM. But I want to do it "right", as in, not limit the implementation to my network filesystem.

So I've started on the framework that will allow *any* filesystem to be plugged in. It doesn't even have to be a network filesystem - indeed, the example code for the upcoming tutorial won't be a network filesystem, it'll be something rather simpler.

The VFS (virtual filesystem) layer consists of three sets of function calls, which are accessed by a jump table. The three groups are filesystem related functions - for example, mkdir, opendir, open, umount etc. Then the file descriptor level functions - read, write, seek, close, poll - then the directory related functions, readdir and closedir. Each one of these groups has some sort of "handle" associated with it, for the fS level functions, the filesystem mountpoint itself determines which ROM module gets the call. The file descriptor does the same for file level operations. Incidentally, the file descriptors are out of the same pool as are sockets - so if someone makes a filesystem with something that can be polled, you can poll everything for potential inputs without caring whether they are sockets, files, or whatever. Just like unix fcntl. Just like unix fcntl, you can then go and read() or write() to them without caring what they are, it's just a stream of bytes. (Of course, you can still take the shortcut of using send/recv on things you know are sockets, it'll save a few CPU cycles by not having to look in the file descriptor vector table).

For the purposes of keeping things simple and reasonably fast, an FS module must be one (or more) ROM modules. The ROM module itself contains a valid address for a MOUNT function (which allows itself to be identified as a filesystem module), and a jump table to all the FS related functions. The entry points at 0x3Exx are used to get there - basically, the VFS entries are actually CALL instructions rather than JP, and the VFS dispatcher figures out what to look up in the module jump table by examining the return address (and removing it from the stack because you don't actually want to return to the jump table!). File/mount/directory handles, instead of having an address, just have a ROM number associated with them - this ROM is paged into paging area B, and the appropriate jump table entry is entered. All this once again proves how useful EX (SP), HL is. It's one of those things you use so infrequently you've forgotten about it, but it's so useful in instances like this :-) (Actually, I hadn't forgotten about it, it's a useful function for writing functions called by C code in asm).

Winston 23:57, 4 May 2009 (BST)

Saving and loading over the net

Saturday saw me save the first BASIC program over ethernet, and now you can load and save both BASIC and CODE files. There are still some details to implement (such as LINE for BASIC programs). As I said last time, I adopted the TAP format for the metadata parts of files that are loaded/saved with the normal %LOAD and %SAVE commands. (%ALOAD and %ASAVE save and load raw files with no metadata, which is often a useful thing to do). After saving a BASIC program and a SCREEN$ image, I validated that the files had been created correctly, by using Fuse to load them - as they are TAP files, well, an emulator just loads them via its normal tape traps.

The TNFS stuff is still rather slow, some optimization needs to be done (such as when a block of data is being read by F_tnfs_read, it should be read directly into the memory that it's destined for - at the moment, it reads it into a buffer in Spectranet RAM, which then has to get copied one more time. However, it's not really a priority at the moment. But it means I've got enough to be going on with.

The next thing I want to do is to integrate this with the ROM; currently all this code is just being run at address 32768 so it can't really realize its full potential.

What I plan to do is implement a simple "VFS layer" (virtual filesystem), and then something that resembles "fcntl" in unix, such that you can plug in any kind of filesystem layer (whether that be TNFS, FTP, HTTP or whatever) - and code that uses the Spectranet functions for opening, reading, and writing gets directed to the right actual driver to move the data around. This will need a slight tweak to the existing socket code, so that each file descriptor has a vector table associated with it. For sockets, the vector table will only get used by using the generic read() and write() fcntl calls, given that send/recv only ever works with sockets, there should not be a performance impact on simple socket based netcode that uses send/recv rather than read/write. I may also need to shuffle some "secondary" functions out of ROM 0 to make room (I'm thinking of the 42 column print routine, given its data already lives in ROM 1). There needs to be enough room in ROM 0 to put the dispatcher routines for all the core network filesystem calls - i.e. open, close, read, write, opendir, closedir (etc).

In other news I've been invited to talk about the Spectranet at RetroEuskal '09 in Bilbao. I hope my spoken Spanish gets a bit better by then - and I intend to bring a Spectranet and Spectrum with me! (I wonder how I'll get treated by airport security when they see a rubber key Speccy in the tray instead of a laptop...)

Winston 23:17, 28 April 2009 (BST)

The BASIC loader, and some future plans

Today I succeeded in loading the first BASIC program over TNFS.

There was quite a bit of groundwork to be done to manage this - first, I had to understand how the ZX ROM sets everything up so that a BASIC program appears in the right place, with all the bits loaded. "The Complete Spectrum ROM disassembly" is extremely handy in this regard - since it explains in quite a bit of detail in the assembly comments how the various ROM routines work.

I also needed to decide how "ZX native" files should be formatted for loading. For the ALOAD command, this didn't matter, because this just loads whatever bytes happen to be in the file straight into memory completely unmolested. However, files specifically for the Spectrum need to have some metadata associated with them; normally this is contained in the tape header (or disc header, if you have some kind of disc, such as a floppy, or microdrive header, or whatever). Unlike a disc though, from the point of view of the Spectrum, a file is just a stream of bytes that is opened and read. The TNFS server sorts out how to actually find the file on disc (whatever that may be), so physical filesystem metadata is not needed. Eventually I came around to the idea of...why not just make the TNFS ZX header format a TAP file?

This has a few good points that I can think of. Lots of things can generate a TAP file, and if when saving something from a Spectrum to the network filesystem, if it gets saved as a TAP file, then any emulator can load it, already extant tools can do things to it etc. The beauty of TAP is its simplicity, and that you can do things like just concatenate the files together. Further thoughts that I have is when a file is loaded, tape traps should be enabled, so if you %LOAD "somefile", and it's a BASIC program followed by machine code (as is the usual way for something on tape) it just loads it all unmolested - so you don't have to do anything to convert a TAP file for direct use on the network filesystem, and it all loads with a single %LOAD command. (This will require some smarts in the loader, for instance, to decide whether the file should be closed, it'll have to see if there's more to come. Once all bytes are loaded, then close the file).

There's nothing stopping more headers being recognised, perhaps the raw format for a file on a ResiDOS volume, or TZX (where the standard ROM loader only is used for all parts of the TZX).

I've also been thinking about how to integrate TNFS into the ROM. Originally I had thought that the ROM would just have a bunch of very low level things - file descriptors not being common, for example - and letting the C library make it all look unified. But I'm starting to think that perhaps the Spectranet ROM should provide a sort of fcntlish like interface, so you can read/write any file descriptor without caring if it's a socket, a file or whatever. This has the advantage that it would be possible to, say, write an FTP ROM module whereby the TNFS BASIC commands work via the FTP module (or HTTP, or whatever) without needing to write new interpreter - the read or write operation would just find the real function to call via a vector table which any module could install. This way you could even use the Spectranet ROM to run a module to access a local filesystem, for example, a standard DivIDE interface, or a simple CF interface - the flash ROM and 128K RAM of the Spectranet could be used to enhance these other devices.

Still, I'm not going to worry about that until the basic TNFS implementation is complete and debugged (I'd rather not have to try and debug and develop both things at once!)

But after all this rambling, I feel like I've made an important milestone in loading a BASIC program successfully over the network, using commands just as a normal user would.

Winston 22:43, 22 April 2009 (BST)

TNFS - Loading files

I had hoped to get a great tranche of work done on the Spectranet software on the long weekend, but it was not to be. The entire weekend got eaten up doing maintenance work on various machines (not computers, machines of the oily petrol-burning variety, where you always skin your knuckles on something sharp when the socket driver slips).

However, I did get the first bit of data loaded via the BASIC commands. I implemented the simplest of the commands, the %ALOAD command (Abitrary LOAD), which works similar to LOAD "file" CODE, except it just loads whatever's on the filesystem rather than examining a header and deciding whether it's BASIC, CODE, an array etc., and where in memory to put the file. As such, an address after CODE is mandatory. Loading a couple of .SCR files downloaded from WOS showed it was working.

However, I had lots of problems with timeouts, although I think a lot of this was due to it really actually timing out because the server on the PC didn't respond in a timely fashion (I could see this happening in Wireshark). However, I have the feeling that there may also be a bug in the poll routine in ROM, too - which would go some way to explaining the DHCP problems that people have experienced. I will implement the proper timeout/retry mechanism (with a longer timeout value)

I also did a little more research on the +3 reset problem. It's very interesting - the +3 resets fine if it's not crashed, but if it's crashed it won't reset. Doing CLEAR 24999 then RAND USR 25000 from BASIC (with no program at 25000) would crash the machine (usually with attributes scrolling up the screen after a few seconds). Trying to reset the machine while the attribute vomit was scrolling up the screen didn't result in a reset - as if the Z80 hadn't got the signal at all, although the screen got cleared. It'll take the logic analyzer to figure out what is actually going on when this happens, I think. But it's bizarre nonetheless - to the RESET circuit, a crashed +3 looks no different to a perfectly running +3. The reset problem only occurs also if the crash is causing things to be drawn on the screen (attributes or paper pixels). But resetting works fine if you do it while a game is running that's drawing to the screen. It is extremely bizarre, possibly a funny interaction with the ULA when there's sufficient load on the bus (the Spectranet will put a little extra stray capacitance on all the lines that it uses or are available to the through port). I'll have to try making the +3 crash this way with a different peripheral connected and see what happens.

Winston 23:12, 14 April 2009 (BST)

Feedback: On reset

Another bit of feedback I got was from Radastan, who said that the Spectranet wasn't initializing on many resets on his +3. I have observed it not initializing on power on (perhaps one time in three) on a +3, but it's always initialized on my +3 when the reset button was pressed.

I've not been able to do anything over the last month on account of being 4000 miles from the nearest Spectranet, so it's had to wait until now. And now I'm puzzled after looking at the schematic for the +3 - the reset circuit for this computer is bizarre. Most Z80 systems (including the rubber keyed 48k) have a simple resistor/capacitor on the reset pin, which functions just fine. At most, some reset circuits additionally buffer this (especially if the reset signal is used elsewhere), like I did with the Z80 single board computer I made a couple of years ago. The resistor/capacitor are sized such that the reset pin is held low for long enough for everything to be reset. As the capacitor charges, the voltage at the junction of the resistor and capacitor rises (this is what's connected to the Z80 RESET pin) until it goes above the TTL threshold, and then the CPU starts to run. The Spectrum+ reset switch simply shorts out the capacitor to restart this process.

But the +3 has lots of extra components in its reset circuit. Not just a buffer, but extra resistors and capacitors, and I'm not sure why it's been designed that way. The principles are the same, but at first glance it looks like the Amstrad engineers made a mountain from a molehill - but it's entirely possible that I'm missing something.

One thing I've tested is the voltages as seen by the buffer (Spectrum reset circuits are all very high impedance and the CPLD causes them to stop functioning unless it's buffered, hence there's a tiny little 1 gate buffer between the Spectrum's RESET output and the CPLD), and the CPLD itself. The voltage during reset on the CPLD RESET pin is 0.1 volts, well below the 0.8 volts max V(IL) for the chip. So it's not voltage levels. In any case, the latch in the CPLD that holds the reset state initializes as logic low anyway at power up, so even if the RESET signal got missed altogether, the latch state will be correct. Even if the Z80 started running before the CPLD was ready, the first thing the CPLD would do would be to grab ROMCS, so in that case there would be at least some weirdness, rather than a total failure to initialize.

So at the moment I'm stumped.

Winston 23:37, 2 April 2009 (BST)

At last, perhaps a server that works...

After a few false starts at the data centre, the server on which the wiki and SVN is stored perhaps might actually stay up now! A long litany of bad hardware has persistently kept spectrum.alioth.net relegated to my ancient UltraSPARC system that lives in the closet. The return to proper hardware has been somewhat hampered by me being on the other side of the world for practically all of March.

But now it's on its proper hardware, with real bandwidth rather than my ADSL upstream. It should be much faster.

Also, I will make it so that those who have prototypes can commit to SVN this week. (I'm going to have to recreate everyone's accounts first).

WebSVN is also newer and shinier, and now tarball downloads are enabled from WebSVN, so you can grab a whole directory as a .tar.gz archive. (some Windows users may need to download something to unpack them, but I think later XP and Vista understands .tar.gz)

Winston 22:40, 31 March 2009 (BST)

The BASIC interpreter

I've got the skeleton of the TNFS library calls done; there's still a bit to do (testing the retry mechanism, adding a few more bits to the proof-of-concept Perl server that runs on the PC) - but files can be opened, closed, read from and written to. Directories can be opened, read and closed. Chdir is actually implemented client-side (all filenames go over the wire as an absolute path). There's a few things I think I will change, for example, I think I'll simplify the way timestamps are sent (to an ISO date/time rather than seconds since the epoch) when statting a file.

But now I want to get the BASIC extensions done for this, and once that's done I might do another video for YouTube, perhaps demonstrating loading a game from the network filesystem or something like that. At the moment though I have a few things more that I need to learn about the ZX BASIC interpreter to get the most from it.

Actually, I've not done all that much to the code for a couple of weeks, it's been very busy at work and I've just felt too frazzled to work on yet more code when I get home, so, errmmm... I've been playing Enemy Territory instead. But I have some time off, so I'll have some more time to think about things.

Winston 22:38, 26 February 2009 (UTC)

Bug fixes

The first 3rd party bug report is in, and it's fixed :-)

Miguel Guerreiro discovered the NMI menu crashed on exit in USR 0 mode and also with his DivIDE firmware, and correctly guessed the cause was the ZX ROM paging being incorrect. Since the Spectranet ROM no longer accesses ZX BASIC for key scan routines, the calls to rearrange ZX ROM paging were now actually not needed in the NMI menu, so they've been taken out. If you have one of the prototypes, I suggest you get the new ROM image.

There is a new page off the main page of this site - Downloads. The first files available are the ethup tool and the current ROM images. Instructions on flashing new ROMs are included.

Winston 22:19, 4 February 2009 (UTC)

The Tiny (Or Trivial) Network Filesystem

As has been mentioned in the past, one of the things I want to have on the Spectranet is a network filesystem - basically something that's a bit better for filesystem use than FTP, but nothing so gnarly and complicated as NFS or SMB. Not everyone who wants a Spectranet will have mass storage for their Speccy, and having a network filesystem makes it easy to load files off another computer (which, in time, I hope to be things like another Spectrum).

First, I'm making the client on the Spectrum, and a proof-of-concept server written in Perl for the server. The client, as such, is a library of calls that correspond to the sort of things you find in 'fcntl', such as open, read, write, close etc. The idea is to provide an asm interface and a z88dk interface, just as in the socket library. For TNFS, I also want to provide BASIC keywords, and possibly at some stage, ResiDOS integration (so if you have something like a ZXCF, you would be able to load a ResiDOS module which adds support for TNFS via the Spectranet, allowing the Spectrum to be able to copy files from the network to a CF card. But that's a way off).

So far, I've started the basic functionality, and done "nominal" tests (i.e. not trying to break it) with the following calls:

• mount - mount a remote filesystem
• umount - Dismount the mounted filesystem
• opendir - Open a directory for reading
• readdir - Read a directory entry
• closedir - Close a directory
• open - Open a file
• read - Read from a file
• write - Write to a file
• close - Close a file

The open, read, write and close calls directly correspond to the same in C, i.e. are "system call level" rather than stdio. The z88dk, for instance, has those calls, and the z88dk stdio uses those low level calls to implement stdio.

In other news, I'm probably going to get the "Issue 1" PCBs made very soon - I had hoped that I could have others do a bit of hardware testing before I committed to more PCBs, but the pound sterling is rapidly collapsing and the PCB maker charges in US dollars. Already this has significantly increased costs - I bought more W5100 chips just before Christmas in case the pound collapsed further, and unfortunately, it has. Reykjavík on Thames, indeed. If the Bank of England starts printing money as has been rumoured, even Euroskeptics are going to wish we were in the euro...

Winston 18:04, 24 January 2009 (UTC)

Administratrivia

A few notices. You may have already seen that the wiki and WebSVN have been unavailable pretty frequently recently. This is because the server this all runs on is dying of a suspected case of Capacitor Plague. Unfortunately, I don't have physical access to the machine.

So in the meantime, I've moved the site onto a computer at home. This means things may run a bit slow - firstly, it's having to share the pathetic 256K upstream you get with ADSL with all my other network traffic, and secondly, it's now practically running on a retrocomputer - the only spare hardware I have with sufficient memory is an old 333MHz UltraSPARC system (I don't want to run it on my PC, for one, I the fan in the new PSU on my PC is so loud I can hear it in the next room, which happens to be where I sleep). It could have been a lot worse, it could have ended up on the old VAX that I have, then you'd have had to have waited about 5 minutes for a page load!

The SVN repo isn't yet back up, but it will be shortly, so you may find some broken links in the wiki.

Winston 00:11, 14 January 2009 (UTC)

Through port success

As I said last time, the NMI problem turned out to be a software bug. Gasman kindly let me use the Open ZX ROM keyboard routines under the MIT license from his GPLd OpenZX ROM project which saved me the effort of writing and debugging one. I wanted to use someone else's as it's likely to have already been debugged, and I need to start getting some of the prototypes out soon.

So the board now works great with the ZX-CF plugged into the back of it, including the Spectranet NMI menu, so now my +3 has both access to a CF card and the network. I tested all the prototypes by loading the new ROM image off the CF card and programming them that way. This meant I had to cut slots in the through ports of all the PCBs, but the job's now done. (I only found out later that PCB Cart would cut the slots for me if I included them in the PCB outline).

I think the combination of a ResiDOS device and the Spectranet could lead to some very good things.

Winston 19:46, 3 January 2009 (UTC)

Through port testing

Happy new year!

The first job of 2009 is to test the through port. The last time I did any testing was on the prototype-for-the-prototype (the breadboarded monstrosity that eventually got turned into the prototype PCB), with the DivIDE. Since then, I have got a ZX-CF board, which has ResiDOS, and as such is a bit more complex than the DivIDE.

So I had to cut a slot in a prototype PCB (something I need to do for the rest of the boards I've made). The DivIDE works pretty much perfectly, The ZX-CF is a little more awkward. It generally works fine, but its execution trap method seems to be very aggressive and it still traps NMI even when the Spectranet traps it...no, update, scratch that -- it's actually a bug in the NMI handler. Well, not a bug as such, but after looking at the logic in the CPLD and scratching my head a bit, I remembered that to save memory in ROM page 0, I had used the ZX ROM's keyboard handler...this of course, means a call out to the Spectrum ROM, and the return comes back in via RST 8. Oops. In light of this, I probably ought to put a key scanning routine into one of the Spectranet ROM pages so it's not necessary to call the ZX ROM. I ought to do that before tinkering with TNFS any more!

I already expected the RST 8 trap on the Spectranet to need disabling, with ResiDOS running - only one board should be allowed to handle RST 8. But none of these are fatal problems. The Spectranet socket library can still be called with the ZX-CF plugged into the back of the Spectranet, due to the unique design of the library entry point instruction trapping. I tested this with the IRC client - I put Spectrum IRC onto a CF card, and loaded it with ResiDOS, then connected to IRC. All this means is with the ZX-CF at least, utility programs should be stored on the CF card and loaded with ResiDOS, rather than using the Spectranet NMI button. (It should be perfectly possible to write network aware ResiDOS modules, too).

The RESET trap works fine too, with the ZX-CF, because that's pre-triggered by the reset signal itself, and as such A15 is already being held high when the CPU begins executing instructions.

Winston 17:03, 1 January 2009 (UTC)