Game:

The Spam Club

» The Spam Club - Life, The Universe and Everything - Site Issues - Disk image formats
ReplyNew TopicNew Poll
» Multiple Pages: 123

Disk image formats

Posted at 06:42 on May 1st, 2018 | Quote | Edit | Delete
Avatar
Admin
Reborn Gumby
Posts: 9071
So as the very technical discussion seems to be roughly concluded, what will the policy be now (summarized)? Which formats will be accepted and published with which preferences? How will the files be canonicalized in order to ensure comparability?
-----
Now you see the violence inherent in the system!
Posted at 06:43 on May 1st, 2018 | Quote | Edit | Delete
Avatar
Member
Master Gumby
Posts: 98
Originally posted by Moebius at 06:19 on May 1st, 2018:
Quote:
Conversion to/from SCP & Kryoflux is pretty easy with HxC (just stick with 2.1.8.0 for now.) So, using the single file SCP format shouldn't pose a problem. The overall size is about the same as with Kryoflux, it's just in a single file instead. From what I've read, PFI is just a different format but basically the same as SCP. That is, raw flux streams placed into a single file. But it's easier to convert with HxC than with the PCE tool set.


Ok, then I suppose we'll go with just KryoFlux since it's more popular and if someone wants SCP he can easily convert RAW to that using HxC. At this point it doesn't matter if there is one single file or many, hashing is not applicable to either format, therefore it's not a priority at all. PFI is still under question (awaiting response from Hampa) and its only other advantage is smaller size. If it's 100% compatible with both RAW and SCP, then we might use that.

SCP might be better as it actually contains more detailed flux data. If someone submits Kryoflux, then converting to SCP doesn't lose anything. But if someone submits SCP, then you are losing a bit of detail when it's converted to Kryoflux.
Posted at 15:40 on May 3rd, 2018 | Quote | Edit | Delete
Avatar
Global Moderator
Deceased Gumby
Posts: 1693
Originally posted by Mr Creosote at 06:42 on May 1st, 2018:
So as the very technical discussion seems to be roughly concluded, what will the policy be now (summarized)? Which formats will be accepted and published with which preferences? How will the files be canonicalized in order to ensure comparability?


It's very hard to make a final verdict at this point, because it seems the further we get into this the more pitfalls appear on every side. Apparently, my initial plan is not working out very well, because PSI cannot be written back correctly and it's best only for emulation purposes.

Furthermore, writing back Teledisk images is pretty much untested, too, and it may or may not be fully consistent. Therefore, to rid of all pitfalls, already detected and potentially impending, we, after all, need a reliable and universal format albeit indirectly which is either KF RAW or SCP, or actually both since we so far have no clear understanding how they are superior or inferior to one another.

I DON'T KNOW if we should keep both or just one of them. Searching for any related info wasn't very fruitful and it seems like a good old clash akin to Microsoft vs Apple. That's why I'm desperately looking for some intermediary format which may not be directly writable but much more compact and effectively convertible. I still haven't got a reply from Hampa with regards to PFI, though.

PSI and TD0 are lower priority as they are semiuseful. However, they are still pretty much required since converting from RAW and SCP is not a trivial task and making them readily available would probably delight many people who only want to use emulators.

Another problem is .CP2 and .TC. These are pretty much useless as neither one works with any emulator and those few CP methods supported by them are already absorbed by RAW and SCP. However, both can be written back more or less successfully and some people may still have a Copy II PC option board, which is like an antique and inferior counterpart to KryoFlux or Supercard Pro.

Furhtermore, there is no perfect way to derive either .CP2 or .TC from RAW or SCP nor to convert the former two to the latter. Yes, PCE tools can do it but so far it's at "experimental" stage. I'd rather not deal with these at all, but in certain few cases we may have to accept them either for the lack of better alternatives or because they are actually good for what they are.

Finally, the only best solution I see now is smart selection. Yes, accept everything, but perform all the necessary tests and rule out redundancies either immediately or at a later time. Besides, it seems that everybody else wants to see a variety of formats on TGOD. Well, I guess we have no choice but to please them. Let's just try and see how it goes. All I know is that we obviously need two types of formats, for instant emulation and for consistent storing/writing back which may or may not be received in conjunction.

For emulation

Default: TD0
Alternative: PSI, IMD, ...

For writing back

Default: RAW
Alternative: SCP, .ANA, .CP2, .TC, ...

Of course, this will be moderated and carefully filtered out using priority levels. I'm afraid it's the only best way to address this perplexing situation. Each copy protected release will have to receive several best candidate format sets for all practical purposes.

Quote:
SCP might be better as it actually contains more detailed flux data. If someone submits Kryoflux, then converting to SCP doesn't lose anything. But if someone submits SCP, then you are losing a bit of detail when it's converted to Kryoflux.


I really wish it's all there is to it. What about sector maps? You know, I saw on some boards that SCP may actually dump disks with lots of errors which may often go unnoticed. I don't think I like it very much. In that sense KryoFlux is at least much more reliable.
-----
Cheer up! Remember the less you have, the more there is to get.

I am on irc.freenode.org: #TGOD
-----
Edited by Moebius at 17:00 on May 3rd, 2018
Posted at 17:41 on May 3rd, 2018 | Quote | Edit | Delete
Avatar
Member
Master Gumby
Posts: 98
Just did a search to make sure. Like the Kryoflux, as long as the end user did things right so that the software knows what to test for, the SCP software will perform very similar to the way the Kryoflux software does. I don't know if it can detect altered sectors the way the Kryoflux can, but corrupted data is detected just fine.

To be completely honest, though, this is just from web research as I don't actually have a SCP to test with. From the videos I've been watching, it works pretty much the same as the Kryoflux, it just has many more features. As I think I said somewhere before. I would have purchased the SCP instead of the Kryoflux except that, at that time (when I had the funds to get one,) the SCP wasn't capable of writing back to disk, while the Kryoflux was.

As for the final decision, it was my understanding that the following would be used:

IMG - All things without on disk copy protection
Its the easiest to deal with, and with no CP to deal with, it works just fine. Very easy to write back to disk.

TD0 - All CP titles it can handle
Can handle many CP disks and has solid emulator support. With the right tools can be written back to disk fairly easily. Either Transdisk on real HW or through a Kryoflux or SCP (just use HxC to convert for Kryoflux at least - don't know about SCP.)

Kryoflux/SCP - Everything else
No emulator support, only good for archival purposes. There are plenty of tools for conversion to emulator supported formats, though. Also, can be written back to real disk if you have the appropriate HW. Without the HW, can still be written back, but only after converting to another format (IMD, IMG, TD0, etc...)


The vast majority of titles would be IMG, with a large portion of the remainder being TD0. Leaving only a very few as SCP/Kryoflux. Nowever, if it was completely up to me, any donated SCP/Kryoflux files would be archived anyways, even if another format is what actually gets put up on the board. But that's just me.
Posted at 17:43 on May 3rd, 2018 | Quote | Edit | Delete
Avatar
Admin
Reborn Gumby
Posts: 9071
How to canonicalize the files for comparability purposes?
-----
Now you see the violence inherent in the system!
-----
Edited by Mr Creosote at 17:48 on May 3rd, 2018
Posted at 18:24 on May 3rd, 2018 | Quote | Edit | Delete
Avatar
Global Moderator
Deceased Gumby
Posts: 1693
Quote:
TD0 - All CP titles it can handle


I'm still not sure how good it is at writing stuff back. So every Teledisk or PSI image I would additionaly supplement with one RAW or SCP counterpart, unless you have some really good news. You know, successful execution in emulator does not always mean quality writeback. I have more reason to trust KF or SCP because those are actual hardware & software suites and Teledisk is just one app.

Quote:
With the right tools can be written back to disk fairly easily.


Yes, but do we really want to give average users such extra work however light? I mean, if somebody wants KF RAW, then it's just best to have it. I've thought of it over and over and I think I'm giving up now. You have to admit, it's safer. So, basically, if we receive a KF RAW file - then so it will stay and we may only create TD0 or PSI from that as we know it's safe enough. If we get TD0 or PSI - then so they will stay and we never make KF RAW or SCP from them because, honestly, we don't know exactly what happens in the process.

Quote:
The vast majority of titles would be IMG, with a large portion of the remainder being TD0. Leaving only a very few as SCP/Kryoflux. Nowever, if it was completely up to me, any donated SCP/Kryoflux files would be archived anyways, even if another format is what actually gets put up on the board. But that's just me.


We've reached a consensus. Only contrary to you I'd much rather keep as fewer formats as possible, which I now think is not discreet enough. Apparently, the more options the better, though at the expense of our time and FTP space. To get it over with, I asked flyers80 to test writeback of some of the Teledisk images we fixed, and depending on the results we shall pass the final judgment.

Originally posted by Mr Creosote at 17:43 on May 3rd, 2018:
How to canonicalize the files for comparability purposes?


All will be converted to plain IMGs, which will be included in the release regardless for future reference and crack lovers similarly to many other CP-less images already archived. We'll use their hash values for verification, the rest will be ignored for the reasons already indicated. As for CP data, it will either work or not, so there is little point verifying it I suppose. And even if we must, there is a way to produce hybrid IMGs with CP data injected, but I'm not sure how accurate that is, Pheonix should know better.
-----
Cheer up! Remember the less you have, the more there is to get.

I am on irc.freenode.org: #TGOD
-----
Edited by Moebius at 19:57 on May 3rd, 2018
Posted at 18:46 on May 3rd, 2018 | Quote | Edit | Delete
Avatar
Member
Master Gumby
Posts: 127
when you are ready, warn me :D
Posted at 20:54 on May 3rd, 2018 | Quote | Edit | Delete
Avatar
Global Moderator
Deceased Gumby
Posts: 1693
Got a reply from Hampa:

Quote:
The three formats are roughly equivalent. The PFI format was
originally designed as a substitute for Kryoflux raw streams,
because those individual files are inconvenient (SCP did not
exist at the time). Now it's the format I like best, but of
course I can't be objective about this.

For long term storage I would recommend to use the native
format of the device that dumped the disk. SCP, RAW and PFI can
quite easily be converted to each other and when they are
compressed, the file sizes are very similar.
-----
Cheer up! Remember the less you have, the more there is to get.

I am on irc.freenode.org: #TGOD
Posted at 08:54 on May 4th, 2018 | Quote | Edit | Delete
Avatar
Member
Master Gumby
Posts: 98
It depends on your definition of accurate. Is it completely identical to how it was out of the factory? Or.. Is it completely working with all of the data the same as it was out of the factory? Those aren't really the same, though they sound a bit like it. My work bench is full of TI-99 stuff right now, but once my final parts get here and I can finish it off, I'll move it out of the way & set up my vintage PC. Then I can test writing the "fixed" Ultima games back to disk & see if they work. However, from the Kryoflux results reading the written disks, I'm willing to state that they will almost definitely work fine. Not enough for me to erase my archived dump, but enough to state that they will work. Once I've proven they work, "then" I'll be deleting my archive for the cleaner/smaller created dumps :)

As for Hampa's comment, I don't see anything to disagree with there. I even, on a personal level, agree with his recommendation of keeping whichever format was submitted. But, if it's decided that only a single format would be used to streamline and/or simplify storage/cataloguing, then I would still suggest SCP. Better detail than Kryoflux, and I just don't know enough about PFI.
Posted at 11:43 on May 4th, 2018 | Quote | Edit | Delete
Avatar
Global Moderator
Deceased Gumby
Posts: 1693
Ok, here's the thing. For now we are going to add ALL of it, because obviously the matter cannot be completely resolved. Also, it would potentially rule out user questions about formats the games were originally dumped to and I strongly suspect it will happen almost immediately.

We are not going to change anything other than create subformats like TD0 or PSI for emulation purposes should they be missing. If we get both RAW and SCP for one game then both will be added regardless of the type of CP. There are at least three considerable advantages to that: 1) People will always know those are original dumps, 2) They won't have to bother converting anything themselves unless the format matching their own device is missing, 3) The bigger variety of formats, the more curious visitors. The disadvantage, however, is glaring: rapid FTP space consumption.

Of course, Teledisk tests are still pending, but I'm afraid whatever the results may be, that is not going to change the policy other than a few lines in the user docs. Only the future can tell what to keep and what to ditch, and I hope we are not going to run out of FTP space too soon.

Mr Creosote, I think we are pretty done here.
-----
Cheer up! Remember the less you have, the more there is to get.

I am on irc.freenode.org: #TGOD
Posted at 13:36 on May 4th, 2018 | Quote | Edit | Delete
Avatar
Admin
Reborn Gumby
Posts: 9071
Good, very productive discussion :)
-----
Now you see the violence inherent in the system!
» Multiple Pages: 123
ReplyNew TopicNew Poll
Powered by Spam Board 5.2.4 © 2007 - 2011 Spam Board Team