Current events
"Production boards"
Well - it's happened.
The first 10 production boards were sent to the PCB factory last night, I should get them within the next 3 weeks or so.
There are not many changes from the prototype PCB:
- "Disable all" jumper corrected so it actually makes a change on the appropriate CPLD pin. (The error was on the schematic, carried through to the netlist and onto the PCB). The only PCB bug that I've found.
- Four mounting holes added
I managed to get the mounting holes in (they are the same size as the plastic clips on the RJ-45 connector, and will take either a plastic clip or a small screw) without having to make the PCB any bigger. The two screw holes on the left of the board were easy - I just added plated holes of the appropriate size, and connected them to the ground plane. The other two caused tracks to get re-routed, something that always makes me nervous just in case I miss something! The PCB software helps me verify everything got reconnected - I can select a pin on one of the chips on each track I changed, and check that everything in the netlist still shows up in green. But I still think of all those expensive coasters I'll end up with if I've missed something... Those two right hand mounting holes either need a plastic clip, or if using a metal screw, an insulated washer between the PCB and the screw head due to the proximity of tracks. While the solder mask should prevent contact, it could easily wear through when tightening a screw and you then end up with a track shorted to ground.
I'm also assembling one of the last prototype PCBs I have left, and I found that I've run out of some components (specifically 150uF tantalum capacitors for the 3.3v voltage regulator), and used the last 74HCT1G125 IC that I have, so I need to re-order these so they are here when the new PCBs arrive.
I hope to have at least 5 working boards for the Vintage Computing Festival - four for the machines I'm demonstrating, and hopefully the fifth to connect to Chris Smith's Harlequin.
Winston 22:57, 25 May 2010 (BST)
Roadmap
I've not updated this in a long while - this is partly because of the troubles I've had with this web server, and partly because I've just been insanely busy. So here's a round-up of recent events, and what's to come...
Retro Rewind, April 2010
Retro Rewind, organized by sscott on the Retro Gamers forum was a great success. The venue was superb - held at Sheffield Hallam University's Student Union - lots of machines, lots of stuff on offer, and of course I had two Spectrums equipped with the Spectranet. Visitors certainly appreciated the quick and easy loading of files over the network from the file server. I wrote a menu program in BASIC, which demonstrates the BASIC extensions as well as BASIC streams working to build a menu and load games from the network filesystem. Visitors just had to type LOAD "" and they were away. Simple and fast! It was good to meet up with various forum members once again, and meet some new people (I was kindly given an Acorn A3010 by woody.cool of the RG forums, and I bought my first Amiga, an A1200 - complete with a heap of games. I've not had time to really play with it though).
Getting ready for the VCF
The next event is the Vintage Computing Festival. It's at the historic Bletchley Park, and judging by how fast the hotels are filling, is going to be a very well attended event. I hope to have four Spectrums with the Spectranet, and with a bit of luck Chris Smith's Harlequin with a Spectranet too. Again, the MicroVAX will be the file server. I'm hoping to have internet access here so people can IRC from a Spectrum, and perhaps even update Twitter from a Spectrum. I need to build two more Spectranets before this will happen (I have three working - one of them I repaired tonight, the 25 MHz crystal had gone bad which really surprised me). If you're coming to the VCF, come and see the Spectranet and have a play with it!
Making more Spectranets
After the VCF is done, I have one more thing to do on the software side (make sure that TNFS loading copes with dropped packets) and then I think it's time to actually get some Spectranets into people's hands. I intend to build 10 at first. Before I get more PCBs made I need to find a good place to put at least three plated holes in the PCB to allow it to be mounted in a box. I've talked once or twice with Rich Mellor of RWAP software, who does the DivIDE Plus - I may see if he fancies doing some "value add" and packaging it up nicely and then selling the whole device through his site. Once the VCF is over and done with it shouldn't take too long to get the first Spectranets made (I'll probably put these for sale myself as a "beta" - after all, they don't really have terribly wide testing - all I can say is they work well with my network switches and my Spectrums). I'd like to be in a position to start building these by the time I get back from RetroEuskal in July, if not before. Expected price for assembled units will depend somewhat on the dollar/pound exchange rate, the PCB being the most expensive single component (almost half the cost).
RetroAcción
I have had the great honour to be invited to join RetroAcción, a Spanish society for the preservation and demonstration of retro hardware and all things related. They organize both RetroEuskal in Bilbao and Retromañía in Zaragoza, and exhibitions like "Del pincel al pixel" (Paintbrush to Pixel) celebrating the fantastic arkwork of Alfonso Azpiri, who did a great deal of artwork "back in the day" for Spanish games on the Spectrum. So I'll be off to RetroEuskal again, and hopefully Retromañía, but this time as part of the organization, not just someone coming to chat about his project! I'm very excited to be going again - I had immense fun at RetroEuskal last year, not to mention eating far too much.
Documentation
"Documentation is the castor oil of programmers. Managers know it must do programmers good, because they hate it so much"
But if I want anyone to be able to develop for the Spectranet, I have to do more documentation, in particular, how to use BASIC - what I hope is that the BASIC extensions make network programming accessible to everyone - so tinkering with networks is not just the preserve of asm and C programmers (for whom tutorials and socket library documentation already exists). But the BASIC streams module needs documenting first. Other things are needed like at least a "Getting started" sheet to ship with the device, so people know what the jumpers do, how to configure it, etc.
Winston 21:04, 13 May 2010 (BST)
Surprises
Doing further testing of ZX BASIC stream support has been very fruitful in shaking out bugs in the core socket library.
- Edit*
I had thought this bug had existed for the lifetime of the Spectranet and further more... but no, it was a regression I'd put in a couple of weeks ago (duh!). A new check for the socket going dead during the "wait for data loop" in the buffer unload routines was put in the wrong place, with the result that it caused a race condition.
- End edit*
The regression was this. When receiving it was checking for the socket buffer being in the RST state before checking to see whether there was data to unload (and also an unbalanced stack, which what actually caused the crash - if it left due to the RST state it wasn't popping a value that had recently been pushed). In the case of a short enough piece of data, the remote end was closing down - and the resultant race was either the code would have passed the check for RST and started to check to see if there was a buffer to unload, or the RST condition became set a little bit sooner, so the code saw the status had gone to RST and exited before checking to see if there was more data to be unloaded from the ethernet buffer. Up until now, the RST flag in the hardware had always been set just after we'd checked it, so the buffer would get copied correctly. But for some reason, the W5100 was beating the code tonight (even though the code in question hadn't changed, and therefore took the same number of T-states).
The fix to the core socket library code fixed this particular problem with BASIC streams. Now to find the next bug...
Winston 22:46, 2 March 2010 (GMT)
Back to BASICs
ZX BASIC is slow. Very, very slow. But this slowness, it turns out, has its uses.
I noticed that somewhere along the way I forgot to actually get the streams module into an anywhere near finished state (while it works, it lacks certain very important features that are needed for the Spectranet to be useful to the BASIC programmer). It's been an interesting shake-down for the base socket library, which I had thought was done and dusted many months ago...
All the tests that I had used, basically the Spectrum could keep up. The tests were written in C or raw assembler. So were real programs, such as the network filesystem and the IRC client. For general networking, the Speccy does seem to cope quite happily with keeping up - it's never pulling in too much data at once, and never really gets behind. As you may have seen, the raw socket library goes fast enough to show full motion video. But BASIC is another matter entirely, and presents a number of interesting challenges.
First, messages end up piling up in the W5100's buffer as the BASIC program can't even keep up with another Spectrum. I've been doing most the testing with a server program to test the new control/status stream that's available to BASIC. This revealed a couple of bugs in the low level routines that load and unload the ethernet buffers (the "wait for data" or "wait for buffer space to fill" loops could hang, because they didn't actually check that the connection went away in the meantime). It also revealed bugs in the poll routines. The W5100 has several status bits for the interrupt register, but they all get cleared when you say "I've handled the interrupt for socket N". This meant if you used poll to check whether there was data waiting (but the socket had also closed), when you went to poll again, the closed interrupt flag had also cleared. Now in a machine code program that may check all the flags at once and know to read and then close, that's fine. But the BASIC interface only handles one flag at a time - so the read would end up clearing the close flag, and the BASIC program would never get the message that the socket had also been closed. So the poll routines now also check the status register so that they reliably return a closed flag always.
The extra challenges put up by BASIC are this. Essentially, BASIC doesn't really map onto the way the socket library really works. The socket library only cares about bytes - it has no concept of "records". But the BASIC INPUT# routine is very much oriented around the concept of a "record", or at least a carriage return terminated string. This means the streams module buffers incoming data, fills the BASIC INPUT buffer until it hits a CR, and exits. Since it may read 256 bytes (and these 256 bytes may contain what BASIC considers are a dozen actual records) we may do a recv() once, and INPUT# is actually executed several times for the one call to recv() as the buffer is emptied. As far as the socket library is concerned, if you call poll() at this stage, there's nothing waiting, and correctly it returns with all the flags cleared. Or it may have even noticed the connection has gone away, and so return the closed flag. But as far as BASIC is concerned, it's not over yet. There may not be anything more to come along the network... but there may be still stuff waiting to be emptied from the buffer. So the poll routine for the BASIC control/status channel must not just call the socket library's poll, but also check the state of all the stream buffers.
All this writing is coming to one final point: I want to get the BASIC streams support into a ...umm...basically demonstrable state before I make a video of "Progress so far". Soon, I hope, really soon...
Winston 21:16, 15 February 2010 (GMT)
Porting and optimizations
The simple portable TNFS server is (I think) pretty much done, except for an 8 bit version. I've tested it on Mac OSX (64 bit intel), Linux (32 bit intel), OpenBSD (64 bit SPARC), Windows XP (32 bit Intel). Windows of course gave all the problems, its POSIX interfaces are annoyingly ever-so-slightly-different to the Unix world (for example, "open" requires an extra flag O_BINARY to work - I notice that Python fell into this trap too), and the socket interface on Windows needs extra things that are not needed for Unix types. And all the header files are different. But at least it works now!
I've also worked a little on the optimization of the network file system routines. The read routine was essentially double buffering the data (reading the block off the network into a Spectranet buffer, then copying it to wherever it was supposed to finally land). This has been changed so that the header is first decoded, then the remainder of the data (the actual block of data read() is supposed to return) is copied directly to its destination from the W5100's buffer. This probably saves around 8,000 T-states per 512 byte block read. It certainly seems faster when loading a 128K snapshot file, although it probably only saves 2/3rds of a second (a 128K snapshot loads in about 2 secs).
Now I probably ought to make that video I've been going on about for the last 3 months...
Winston 17:18, 31 January 2010 (GMT)
Portable network fileserver
The current job is a portable fileserver program for the simple TNFS protocol.
The server program I have used to test the filesystem is something I hacked together using perl, for reasons of expediency. However, to make it simple for Windows users (and don't have a perl interpreter by default), and also as a basis for a fileserver for 8 bit systems, I'm writing a lightweight server in C. It'll also be better for "retroservers" like the MicroVAX that I have. Although the perl server runs on the VAX with adequate speed, a server written in C will run a lot faster. The idea is to have something very portable and small and simple. The architectures I intend to test with the first version will be sparc64 (big endian, 64 bit), VAX (little endian, 32 bit), Win32 (x86, little endian, 32 bit), macppc (32 bit big endian) and mac on intel (x86_64, 64 bit little endian). The endianness matters for testing because 16 and 32 bit integers are encoded as 16 and 32 bit values in the datagrams as little endian values (most 8 bit systems seem to deal with 16 or 32 bit values in a little endian manner).
My goal at the moment is to make the "simple" TNFS server. This TNFS server is for LAN use, so that someone may share files between their PC and a Spectrum. It doesn't include things like authentication and authorization. The idea is that the user sets the server up to share a specific directory tree and it works a bit like ResiDOS - it lets you access files within. A later goal is to find out what I need to use for ESXDOS on the Spectrum, and make a Speccy version of the fileserver, so it can serve files from a DivIDE. All the protocol level stuff should be portable, as will the network code. Filesystem level stuff will probably need some Speccy-specific #asm sections - given that ESXDOS is brand new it may be some time before the z88dk gets ESXDOS support for its base I/O libraries. (However, ESXDOS does have a POSIX like interface which should mean it won't be too bad to port).
Later, I may make a proper "multiuser server" (that is, it'll integrate with the host OS authentication and authorization methods). However, I'll only do this for Unix. The last time I touched Windows at this level was with NT 4.0 and lots of things have changed with Vista and newer, so someone more familiar than me with the Windows security model would have to take that up.
The snapshot manager
Unfortunately "real life" problems have kept me away from doing serious work with the Spectranet software for a week or three, so I'm well behind where I expected to be. However, I still managed to get the first pass of the snapshot manager working. At the moment it's rather rudimentary, but it does at least have a scrollable file/directory browser, and it can load and save snapshots. It is invoked via the NMI menu (and is a ROM module). I am also moving the snapshot code itself to this module - for testing, this had lived in the BASIC extensions module. Hopefully, by next week I'll have the current firmware in a state where it's worthy of making a video update on what the Spectranet does so far.
The other feature I need to add is a rename system call for filesystems in the base ROM. This will need a slight rearrangement of some internal functions because the jump table is full, and there are some functions that can be combined or moved elsewhere to free up entries (I don't really want the jump table to take up more than 256 bytes). Also I need to write a portable TNFS server (the current one is in Perl, and although this is portable enough, Windows users generally don't have a perl interpreter, and most importantly - a Spectrum will never have a perl interpreter and I want the Spectrum itself to be able to be a file server for the ultimate retro network). Then after that, get some new hardware made for those who want one!
Winston 23:05, 28 December 2009 (GMT)