Good, very productive discussion

Ok, here's the thing. For now we are going to add ALL of it, because obviously the matter cannot be completely resolved. Also, it would potentially rule out user questions about formats the games were originally dumped to and I strongly suspect it will happen almost immediately.

We are not going to change anything other than create subformats like TD0 or PSI for emulation purposes should they be missing. If we get both RAW and SCP for one game then both will be added regardless of the type of CP. There are at least three considerable advantages to that: 1) People will always know those are original dumps, 2) They won't have to bother converting anything themselves unless the format matching their own device is missing, 3) The bigger variety of formats, the more curious visitors. The disadvantage, however, is glaring: rapid FTP space consumption.

Of course, Teledisk tests are still pending, but I'm afraid whatever the results may be, that is not going to change the policy other than a few lines in the user docs. Only the future can tell what to keep and what to ditch, and I hope we are not going to run out of FTP space too soon.

Mr Creosote, I think we are pretty done here.

It depends on your definition of accurate. Is it completely identical to how it was out of the factory? Or.. Is it completely working with all of the data the same as it was out of the factory? Those aren't really the same, though they sound a bit like it. My work bench is full of TI-99 stuff right now, but once my final parts get here and I can finish it off, I'll move it out of the way & set up my vintage PC. Then I can test writing the "fixed" Ultima games back to disk & see if they work. However, from the Kryoflux results reading the written disks, I'm willing to state that they will almost definitely work fine. Not enough for me to erase my archived dump, but enough to state that they will work. Once I've proven they work, "then" I'll be deleting my archive for the cleaner/smaller created dumps

As for Hampa's comment, I don't see anything to disagree with there. I even, on a personal level, agree with his recommendation of keeping whichever format was submitted. But, if it's decided that only a single format would be used to streamline and/or simplify storage/cataloguing, then I would still suggest SCP. Better detail than Kryoflux, and I just don't know enough about PFI.

Got a reply from Hampa:

when you are ready, warn me

I'm still not sure how good it is at writing stuff back. So every Teledisk or PSI image I would additionaly supplement with one RAW or SCP counterpart, unless you have some really good news. You know, successful execution in emulator does not always mean quality writeback. I have more reason to trust KF or SCP because those are actual hardware & software suites and Teledisk is just one app.



Yes, but do we really want to give average users such extra work however light? I mean, if somebody wants KF RAW, then it's just best to have it. I've thought of it over and over and I think I'm giving up now. You have to admit, it's safer. So, basically, if we receive a KF RAW file - then so it will stay and we may only create TD0 or PSI from that as we know it's safe enough. If we get TD0 or PSI - then so they will stay and we never make KF RAW or SCP from them because, honestly, we don't know exactly what happens in the process.



We've reached a consensus. Only contrary to you I'd much rather keep as fewer formats as possible, which I now think is not discreet enough. Apparently, the more options the better, though at the expense of our time and FTP space. To get it over with, I asked flyers80 to test writeback of some of the Teledisk images we fixed, and depending on the results we shall pass the final judgment.



All will be converted to plain IMGs, which will be included in the release regardless for future reference and crack lovers similarly to many other CP-less images already archived. We'll use their hash values for verification, the rest will be ignored for the reasons already indicated. As for CP data, it will either work or not, so there is little point verifying it I suppose. And even if we must, there is a way to produce hybrid IMGs with CP data injected, but I'm not sure how accurate that is, Pheonix should know better.

How to canonicalize the files for comparability purposes?

Just did a search to make sure. Like the Kryoflux, as long as the end user did things right so that the software knows what to test for, the SCP software will perform very similar to the way the Kryoflux software does. I don't know if it can detect altered sectors the way the Kryoflux can, but corrupted data is detected just fine.

To be completely honest, though, this is just from web research as I don't actually have a SCP to test with. From the videos I've been watching, it works pretty much the same as the Kryoflux, it just has many more features. As I think I said somewhere before. I would have purchased the SCP instead of the Kryoflux except that, at that time (when I had the funds to get one,) the SCP wasn't capable of writing back to disk, while the Kryoflux was.

As for the final decision, it was my understanding that the following would be used:

IMG - All things without on disk copy protection
Its the easiest to deal with, and with no CP to deal with, it works just fine. Very easy to write back to disk.

TD0 - All CP titles it can handle
Can handle many CP disks and has solid emulator support. With the right tools can be written back to disk fairly easily. Either Transdisk on real HW or through a Kryoflux or SCP (just use HxC to convert for Kryoflux at least - don't know about SCP.)

Kryoflux/SCP - Everything else
No emulator support, only good for archival purposes. There are plenty of tools for conversion to emulator supported formats, though. Also, can be written back to real disk if you have the appropriate HW. Without the HW, can still be written back, but only after converting to another format (IMD, IMG, TD0, etc...)


The vast majority of titles would be IMG, with a large portion of the remainder being TD0. Leaving only a very few as SCP/Kryoflux. Nowever, if it was completely up to me, any donated SCP/Kryoflux files would be archived anyways, even if another format is what actually gets put up on the board. But that's just me.

It's very hard to make a final verdict at this point, because it seems the further we get into this the more pitfalls appear on every side. Apparently, my initial plan is not working out very well, because PSI cannot be written back correctly and it's best only for emulation purposes.

Furthermore, writing back Teledisk images is pretty much untested, too, and it may or may not be fully consistent. Therefore, to rid of all pitfalls, already detected and potentially impending, we, after all, need a reliable and universal format albeit indirectly which is either KF RAW or SCP, or actually both since we so far have no clear understanding how they are superior or inferior to one another.

I DON'T KNOW if we should keep both or just one of them. Searching for any related info wasn't very fruitful and it seems like a good old clash akin to Microsoft vs Apple. That's why I'm desperately looking for some intermediary format which may not be directly writable but much more compact and effectively convertible. I still haven't got a reply from Hampa with regards to PFI, though.

PSI and TD0 are lower priority as they are semiuseful. However, they are still pretty much required since converting from RAW and SCP is not a trivial task and making them readily available would probably delight many people who only want to use emulators.

Another problem is .CP2 and .TC. These are pretty much useless as neither one works with any emulator and those few CP methods supported by them are already absorbed by RAW and SCP. However, both can be written back more or less successfully and some people may still have a Copy II PC option board, which is like an antique and inferior counterpart to KryoFlux or Supercard Pro.

Furhtermore, there is no perfect way to derive either .CP2 or .TC from RAW or SCP nor to convert the former two to the latter. Yes, PCE tools can do it but so far it's at "experimental" stage. I'd rather not deal with these at all, but in certain few cases we may have to accept them either for the lack of better alternatives or because they are actually good for what they are.

Finally, the only best solution I see now is smart selection. Yes, accept everything, but perform all the necessary tests and rule out redundancies either immediately or at a later time. Besides, it seems that everybody else wants to see a variety of formats on TGOD. Well, I guess we have no choice but to please them. Let's just try and see how it goes. All I know is that we obviously need two types of formats, for instant emulation and for consistent storing/writing back which may or may not be received in conjunction.

For emulation

Default: TD0
Alternative: PSI, IMD, ...

For writing back

Default: RAW
Alternative: SCP, .ANA, .CP2, .TC, ...

Of course, this will be moderated and carefully filtered out using priority levels. I'm afraid it's the only best way to address this perplexing situation. Each copy protected release will have to receive several best candidate format sets for all practical purposes.



I really wish it's all there is to it. What about sector maps? You know, I saw on some boards that SCP may actually dump disks with lots of errors which may often go unnoticed. I don't think I like it very much. In that sense KryoFlux is at least much more reliable.

SCP might be better as it actually contains more detailed flux data. If someone submits Kryoflux, then converting to SCP doesn't lose anything. But if someone submits SCP, then you are losing a bit of detail when it's converted to Kryoflux.

So as the very technical discussion seems to be roughly concluded, what will the policy be now (summarized)? Which formats will be accepted and published with which preferences? How will the files be canonicalized in order to ensure comparability?

Ok, then I suppose we'll go with just KryoFlux since it's more popular and if someone wants SCP he can easily convert RAW to that using HxC. At this point it doesn't matter if there is one single file or many, hashing is not applicable to either format, therefore it's not a priority at all. PFI is still under question (awaiting response from Hampa) and its only other advantage is smaller size. If it's 100% compatible with both RAW and SCP, then we might use that.



Actually, I've already run into a handful of CP schemes which cannot be run as TD0. I've converted some RAW's and PSI's to TD0 and the result was non-working. Such is Prehistorik 1-2 (I think their CP tracks are the very last ones on the disk, which is weird) or lots of Capcom games.



Exactly, and this is pretty annoying. That's why I created this thread here to list all such games and possibly their CP methods in advance. If something's missing or you discover anything new - make proper changes to the thread, please. You might have to receive additional privileges from Mr Creosote, though.

Conversion to/from SCP & Kryoflux is pretty easy with HxC (just stick with 2.1.8.0 for now.) So, using the single file SCP format shouldn't pose a problem. The overall size is about the same as with Kryoflux, it's just in a single file instead. From what I've read, PFI is just a different format but basically the same as SCP. That is, raw flux streams placed into a single file. But it's easier to convert with HxC than with the PCE tool set.

As for fixing the disks. Some CP techniques may make it difficult, maybe even extremely so in combination. I'm not adverse to trying I much prefer good clean images as pristine as possible. All those modified tracks bother me at times I've discovered that most CP disks aren't that complicated most times. They generally take 1 or 2 of the techniques (occasionally 3, very rarely 4 or more,) and combine them in one way or another. Ultima I & II, for example, uses: Improper sector size & overlapping sectors. That's it. Short & simple on a single track.

I think you'll discover that most on disk CP is rather simple (though hard to copy, especially during that era.) The difficulty arises in finding documentation on the techniques used for each individual title. It isn't always clear from the dumped data what is actually being done

The hardest part comes about when CP doesn't use an entire track (for cleaning purposes.) At this time, the only method I have for preserving the CP data is to just take the cleaned disk & insert the CP track into it. There may be other ways, but I haven't discovered them yet

Ok, how about PFI then? You know, we're running out of options and I'm on verge of just accepting RAWs and SCPs as they are as this is seriously wearing me out now. Can we at least convert SCP to RAW without any problems and vice versa? Guess I should ask Hampa about PFI before we decide anything. Another problem is, if we should stick to RAW and SCP, then we also can't fix anything if it comes modified. Even though, it's possible as we have discovered, but all other issues considered it's best not to bother with it, unless the CP is simple like that in Ultima 1-3 and Sierra games. In other words, everything that runs as TD0 can be restored. The rest not.

OK, tracked down the issue. It was the latest HxC failing to properly align the track marks from the Kryoflux stream files. Switched back to a previous version and that works now. Please ignore a good chunk of what I've said previously, it may no longer hold true.

What I really need to do this properly is to publish my current list of CP techniques (not all will hold true for every system.) Then try to get a list of titles that use each technique. This will allow me to start collecting & testing titles for each one. We know, it seems, that TD0 doesn't support week bits. I can postulate that MFM doesn't support index straddling (not sure how it handles track skew, not "quite" the same thing.)

Found it in HxC... Sorry, I'm still not 100%, missed it the first several times I looked.

It does present a problem, though, with at least one disk. Ultima I when converted to MFM ends up with lots of errors that remain when converted back to Kryoflux. Haven't tried writing then re-reading, though. Right now, I'm testing other disks. Ultima I, it seems, has tracks shifted all over the place, so that the index straddles data on most tracks. Going to try & see if that can be fixed later, as I know it isn't part of the copy protection. It also prevents me from writing the dump back, as the Kryoflux cannot (at this time,) handle that (it's something they could program in, but its for later, I guess.)

I don't remember HxC doing this in the past... Maybe I changed a setting or something. Looking into that as well. This is double concerning because I know I wrote some of the CP disks back to floppy for testing on my real HW, and they worked.

Ok, I've got the answer from Hampa:



I actually thought of this PSI -> PRI -> PFI -> RAW thing, but then I failed at some point and concluded that reverse cannot be done. Anyway, apparently this is a no go, and to do it correctly we DO in fact need flux-based images. As for MFM, again, we need to test it. First we convert RAW or SCP to MFM, then do the reverse, write it back to a floppy and see if it works. Alternatively, we could try PFI.



I thought HxC emulator does it? I've already tested it, but then I don't know how good the output is.

From the viewpoint of a single smaller file that can be converted to higher/lower format for writing or use in emulation, then MFM is probably a decent choice. It may not retain all CP information though. I cannot currently test, as I don't have a way to convert to MFM yet. Having a bit of problem focusing right now, I'll do a search later for conversion tools to MFM format.

C'est la vie, they will just have to learn if necessary.



I want to have one file instead of two different, yet virtually the same data-wise. So, if MFM can easily substitute both RAW and SCP with a chance of being converted to either one, then I would always opt for that, checksums or not. Besides, it's much smaller, so it's good at least on two counts. As for IMGs, I agree and I've already considered that. I guess that's all we got.



As you know, IMGs should be included by default, so that's not a problem. Only I'd rather they were normal size actually, because at the very least people can use cracks and that wouldn't be possible having a faulty dump like that. Besides, CP either works or not, so I'm guessing there is no need to verify it. Moreover, I suspect that part could also be variable, so I'd rather it was filtered out.



You're probably right, but I intend to test it anyway. I have enough just for that and will let you know whether or not it's correct. After all those are hybrids.

I was mostly concerned with output. The PSI tools are difficult to use for many people (No GUI, have to analyze the streams, etc...)

Converting to MFM is an option. But it may face the same problem, though not as bad. Really, the only way to track checksums consistently would be to have a high level image container (IMG for example,) with any CP issues converted in the same way every time.

The best way, IMHO, for this would be to use the software for the format in question for the conversion. For CP disks, this would result in a non-working image, but it would stay consistent from dump to dump. Even different disks that have the same contents would remain consistent.

I can't speak for SCP, but Kryoflux's DTC software can create an IMG file at the same time as the stream dump files. Just put these non-working IMG files with the stream dump files & use them for the generated checksums. Just mark them as non-functional. Considering the size of the stream dump, the extra 160K to 1.44M shouldn't be that big a problem.

I don't really see how TD0 or PSI/PFDC can produce consistent checksums either. Not if they are supporting CP that involves GAP data, track skew, and other such CP techniques. This is information that can vary to an extent on non-CP disks & tracks as well. Track skew can actually vary on a read of the exact same disk. Unless the drive's rotation speed is constant well beyond normal specs.