Game:

The Spam Club

» The Spam Club - Life, The Universe and Everything - Site Issues - Disk image formats
ReplyNew TopicNew Poll
» Multiple Pages: 123

Disk image formats

Posted at 01:23 on April 23rd, 2018 | Quote | Edit | Delete
Avatar
Global Moderator
Deceased Gumby
Posts: 1624
Ok, so we have technically adopted the new policy for non-raw copy protected images and are ready to go any time. There is only one essential question left: which formats would you like to see on TGOD? At present moment we see future for the following:

Teledisk (.TD0)

Pros

- Works with PCE emulator and 86box
- Easily converted and verified

Cons

- Supports a limited number of CP schemes

PCE new (deprecated) format (.PSI)

Pros

- Supports a considerably greater variety of CP schemes
- Easily converted and verified

Cons

- Currently only works with PCE emulator (retracted from the latest version, though)
- Can't be created directly other than converted to, or methods are yet unknown
- Cannot be written back to a disk using any ordinary tools except KryoFlux DTC utility on post-conversion or Teledisk (.TD0) with a risk of losing unsupported CP data

Now formats which we are not so sure about, but ready to consider:

PCE standard format (.PFDC)

Pros

- Supposedly as good as .PSI (it absorbed the latter eventually), though somehow it's still not as popular
- Easily converted and verified

Cons

- Works only with PCE emulator
- Cannot be written back to a disk using any ordinary tools except KryoFlux DTC utility on post-conversion or Teledisk (.TD0) with a risk of losing CP data unsupported by the output format

KryoFlux (.RAW)

Pros

- Supports all CP schemes except a few really severe methods like laser hole

Cons

- Rather large output size (100kb~10mb)
- Doesn't work with any emulator as is
- Can only be written back to a disk using native KryoFlux device or if converted to another format with a risk of losing CP data

Supercard Pro (.SCP)

Pros

Same as KryoFlux.

Cons

Same as KryoFlux.


And so far we see very little use for other non-raw formats like Anadisk (.ANA), .CP2 (Copy II PC), ImageDisk (.IMD), PDIWrite (.PDI), Disk2FDI (.FDI), Transcopy (.TC) and so on, unless you have a good argument to the contrary.

Think carefully and choose wisely. Remember, by making certain decisions you might considerably increase our load and dependence on more software. So, whatever it is that you request it should be really worth it.
-----
Cheer up! Remember the less you have, the more there is to get.

I am on irc.freenode.org: #TGOD
-----
Edited by Moebius at 10:50 on April 24th, 2018
Posted at 06:22 on April 23rd, 2018 | Quote | Edit | Delete
Avatar
Member
Master Gumby
Posts: 127
I would adopt all those you have listed :)
Posted at 09:03 on April 23rd, 2018 | Quote | Edit | Delete
Member
Master Gumby
Posts: 121
Originally posted by flyers80 at 06:22 on April 23rd, 2018:
I would adopt all those you have listed :)


+1

I expected such an answer. :)
Posted at 09:36 on April 23rd, 2018 | Quote | Edit | Delete
Avatar
Admin
Reborn Gumby
Posts: 8986
If the answer is "all", what would be the priority list? Let's say somebody asks which format to use. What do you answer?

a) "Whatever you like"
b) "Depends on copy protection, here is the decision tree"
-----
Now you see the violence inherent in the system!
Posted at 10:14 on April 23rd, 2018 | Quote | Edit | Delete
Avatar
Global Moderator
Deceased Gumby
Posts: 1624
Ok, then I will ask you two: what will be the practical purpose of KryoFlux RAW for example if Teledisk and/or PSI can handle some copy protection just as effectively? Same goes for Supercard Pro. What are the advantages, really? I don't really mind upping them, but only if they can really play their part and not just take up space as spare alternatives. That would be redundant, no? Let's observe a situation. Suppose some game runs well as Teledisk or PSI format, what's the point uploading those bulky RAWs or SCPs which you can't even mount in any emulator? What's their overall advantage?

Also, why would you need ImageDisk or Anadisk if they are practically inferior to Teledisk alone? I could be ignorant on some matters, but as far as I know they are hardly any better than simple IMGs, except they are compressed. And while ImageDisk is at least supported by 86box, Anadisk is not. Both seem to be pretty useless to me unless you enlighten me some more.

Copy II PC and Transcopy I like even less than RAW or SCP. Both are pretty rigid and hardly convertible or verifiable. Do they support any exotic CP methods? You see, I wouldn't want to add different stuff just in case, I want to know why we keep this or that. Unfortunately, though, there is very little information on these tools, and even if present it's always somewhat incomplete.

Quote:
b) "Depends on copy protection, here is the decision tree"


That's what I thought, but there might be exceptions, however. For instance, usage in different emulators. So, I believe we could upload both Teledisk and PSI images as a matter of preference, that works. Everything else depends on other factors which are not determined yet.
-----
Cheer up! Remember the less you have, the more there is to get.

I am on irc.freenode.org: #TGOD
-----
Edited by Moebius at 10:18 on April 23rd, 2018
Posted at 10:15 on April 23rd, 2018 | Quote | Edit | Delete
Avatar
Member
Master Gumby
Posts: 127
Originally posted by Mr Creosote at 09:36 on April 23rd, 2018:
If the answer is "all", what would be the priority list? Let's say somebody asks which format to use. What do you answer?

a) "Whatever you like"
b) "Depends on copy protection, here is the decision tree"


my answer is (b)

Definetely TD0 and PSI

but if you can also SCP (Supercard Pro Dump) and kryoflux dump
Posted at 10:18 on April 23rd, 2018 | Quote | Edit | Delete
Avatar
Member
Master Gumby
Posts: 127
Originally posted by Moebius at 10:14 on April 23rd, 2018:
Ok, then I will ask you two: what will be the practical purpose of KryoFlux RAW for example if Teledisk and/or PSI can handle some copy protection just as effectively? Same goes for Supercard Pro. What are the advantages, really?


they only serve those who own these cards to recreate the floppies
Posted at 10:21 on April 23rd, 2018 | Quote | Edit | Delete
Avatar
Global Moderator
Deceased Gumby
Posts: 1624
Quote:
they only serve those who own these cards to recreate the floppies


Yes, but can't you do the same with TD0 or PSI? Both can be converted to RAW at least using PCE tools. Would there be any particular difference? Does RAW contain any particular metadata that you need?
-----
Cheer up! Remember the less you have, the more there is to get.

I am on irc.freenode.org: #TGOD
Posted at 10:51 on April 23rd, 2018 | Quote | Edit | Delete
Avatar
Member
Master Gumby
Posts: 127
Originally posted by Moebius at 10:21 on April 23rd, 2018:

Yes, but can't you do the same with TD0 or PSI? Both can be converted to RAW at least using PCE tools. Would there be any particular difference? Does RAW contain any particular metadata that you need?


I do not know, I've never tried
Posted at 19:16 on April 23rd, 2018 | Quote | Edit | Delete
Member
Master Gumby
Posts: 121
Originally posted by Moebius at 10:14 on April 23rd, 2018:
Let's observe a situation. Suppose some game runs well as Teledisk or PSI format, what's the point uploading those bulky RAWs or SCPs which you can't even mount in any emulator?


I thought "The disk image collection is primarily about preservation and cataloguing..." (Source: FAQ). ;) And therefore, also those formats doing they job very well. (At least what I read about them.)
-----
Edited by fuxxxyfloppy at 19:27 on April 23rd, 2018
Posted at 04:44 on April 24th, 2018 | Quote | Edit | Delete
Avatar
Global Moderator
Deceased Gumby
Posts: 1624
Yes, but it doesn't say we should have duplicate content only stored in different formats, does it? ;) Which in 99% cases it will be. So what I'm asking is, is it really worth having those huge ass files which on top of it are completely inapplicable unless written back?

One way to test if some CP method is valid for one or another format is to actually RUN it or write it back. How do we run them without conversion? We can't. How do we write it back without conversion? We can't, unless you own those native devices. Also, I don't know how to look inside Transcopy or Copy II PC images. Do you? Suppose someone sends you such file. What are you going to do with it? How do you know if it's good or not? Technically, PCE tools can convert them, but the following I don't find very attractive:

Quote:
cp2 The Copy II PC / Snatchit disk image format. Support for
this format is experimental. This format is only avail-
able as an input format.


Quote:
tc Transcopy dump format. Support for this format is highly
experimental. This format is only available as an input
format.


Also, be ready to answer questions like: "why is there no such and such format for such and such game?". Aaaaaand, be ready to credit people for just sending in other formats of the games which they produced using some of our files. So in the end we'll have a lot of people contributing not games, but formats. That's why I wanted to limit it to usability alone, so we don't get any pseudocontributions at least. I'm not saying "no" to any particular format, but I'd rather we add them only if there are no better alternatives, so we could at least have a good argument.

Our other problem, and we have to face it, is that we know little of CP methods and which format can handle which methods exactly. One way to deal with it would be to accept just everything as per your suggestion, but that's a little too unprofessional in my opinion and a better solution is to actually test if CP works or not by RUNNING those in emulators and to add only that which is 100% working without leaving any surprises for both ourselves and others. And if you want to have a galore, then be ready for a big headache ;)

By the way, speaking of KF RAW and SCP, something tells me depending on the disk's condition but regardless of actual content those may generate different metadata which means they can't even be properly verified hash-wise and we will have to convert them every time anyway to see if they have any matches on here. I'm ok to add RAW or SCP ONLY if they contain a very special type of CP inappropriate for any other formats. But then it doesn't mean it will be properly verifiable, the problem will still remain.

A little update: I think PSI and PFDC are now really the same, at least PCE seems to read PSI if simply renamed to PFDC. It wasn't deprecated, it simply became a new standard. However, it's best that we keep it as PSI at least to distinguish it from the old PFDC version.
-----
Cheer up! Remember the less you have, the more there is to get.

I am on irc.freenode.org: #TGOD
-----
Edited by Moebius at 06:07 on April 24th, 2018
Posted at 07:42 on April 24th, 2018 | Quote | Edit | Delete
Avatar
Member
Master Gumby
Posts: 127
mine were just ideas :)
the site is yours, you decide
however at least the psi and td0 formats must be inserted, with these two formats the majority of protections are covered
Posted at 08:55 on April 24th, 2018 | Quote | Edit | Delete
Avatar
Admin
Reborn Gumby
Posts: 8986
Clearly, if a complex format is not needed, it should not be used. So what could a priority list look like?

1. Plain IMG files are the preferred default. Other formats shall only be used if there is a good reason.

2. If disk-based copy protection exists, TD0, PSI or PFDC shall be used.

3. If still not working due to extremely exotic schemes, use whatever else is necessary.

Does this make sense?

Question would of course be what to do if somebody simply makes Kyroflux dumps by default even where it is not needed and refuses to provide other formats. Do we nevertheless accept the images or do we insist on the "minimum needed" policy. In other words, is it "shall" or "should"?
-----
Now you see the violence inherent in the system!
Posted at 10:14 on April 24th, 2018 | Quote | Edit | Delete
Avatar
Global Moderator
Deceased Gumby
Posts: 1624
Quote:
1. Plain IMG files are the preferred default. Other formats shall only be used if there is a good reason.

2. If disk-based copy protection exists, TD0, PSI or PFDC shall be used.

3. If still not working due to extremely exotic schemes, use whatever else is necessary.


Sounds about right. Not sure about PFDC yet. Even though latest version looks similar to PSI, it is still somehow different, I checked (smaller size). Probably just compressed, because it converts well with PSI.EXE, but there might be something else to it and I can't find any related information. Pheonix knows more about it, so we'll have to wait for him to see this.

Quote:
Question would of course be what to do if somebody simply makes Kyroflux dumps by default even where it is not needed and refuses to provide other formats. Do we nevertheless accept the images or do we insist on the "minimum needed" policy. In other words, is it "shall" or "should"?


Just KF RAWs are perfectly fine, because: a) We can convert them to all other formats ourselves, b) Most of them cannot be produced without KF RAWs anyway except Teledisk. So, in all actuality, KF RAW IS the bare minimum and ironically most preferable on many occasions. If we have that we know for sure nothing is missing, but I personally see no point adding it as is if the case is pretty much clear. One disk depending on the original capacity can be 30-60mb and consist of 84-168 files each of which will have to be hashed. Meaning the hash list will be enormous, too. So yes, it's perfectly acceptable, just not exactly addable. Same goes for SCP.
-----
Cheer up! Remember the less you have, the more there is to get.

I am on irc.freenode.org: #TGOD
Posted at 11:58 on April 24th, 2018 | Quote | Edit | Delete
Avatar
Admin
Reborn Gumby
Posts: 8986
So, in such a case, what would you do? Concert the Kyroflux file to one of t preferred ones and post only the latter?

What if someone just uploads a format which cannot be converted to something preferred?
-----
Now you see the violence inherent in the system!
Posted at 12:47 on April 24th, 2018 | Quote | Edit | Delete
Avatar
Global Moderator
Deceased Gumby
Posts: 1624
Quote:
So, in such a case, what would you do? Concert the Kyroflux file to one of t preferred ones and post only the latter?


If none of the preferred ones support this type of CP then it's just Kryoflux, naturally. Otherwise I would rather publish only runnable files of course, unless someone tells me what the real advantage is. At the end of the day, it's only a small CP segment which makes all the difference, and having a myriad of formats just because of that doesn't make a whole lot of sense.

Quote:
What if someone just uploads a format which cannot be converted to something preferred?


Technically, all can be converted (even .CP2 and .TC), but the real question is, would Teledisk, PSI/PFDC support this type of CP or not. It's always easy to check, and if it doesn't then it will be accepted as is. I'm not against other formats per se, only unnecessary assortment which may produce certain complications.

For instance, if something is modified and we can only fix it later on (if OEM ID or correct timestamps are rediscovered) I personally don't want to go through all those files as it's a little more troublesome than fixing some IMGs. So, by all means, the less different formats the better. And if 86box finally acquires support for .PSI, then in all honesty, I would use just that as it's apparently a lot more potent than Teledisk.

This is not really a final decision, I'm not entirely sure if we don't need RAW, SCP and such as alternatives alongside other formats, but I propose we start with smaller things first. The rest will be stored on FTP for the time being. If it's any urgent we can always add it, but right now we need more information, which, apparently, is not so easy to find.
-----
Cheer up! Remember the less you have, the more there is to get.

I am on irc.freenode.org: #TGOD
-----
Edited by Moebius at 12:50 on April 24th, 2018
Posted at 18:20 on April 24th, 2018 | Quote | Edit | Delete
Member
Master Gumby
Posts: 121
Originally posted by Mr Creosote at 11:58 on April 24th, 2018:
So, in such a case, what would you do? Concert the Kyroflux file to one of t preferred ones and post only the latter


I haven't tried it before, therefore I can't judge how reliable the conversion tools work. And every conversion entails the risk of errors. Therefore, I would publish the original files and as an addition the converted counterpart, and not vice versa.

However, I would suggest, high priority formats should be published first. Others could be added later.

Originally posted by Moebius at 10:14 on April 24th, 2018:
One disk depending on the original capacity can be 30-60mb and ...

This can only judged by Mr Creosote. However, I except it would be more a bandwidth problem than a space problem.

Originally posted by Moebius at 10:14 on April 24th, 2018:
... and consist of 84-168 files each of which will have to be hashed. Meaning the hash list will be enormous, too.

I don't see a problem here.
-----
Edited by fuxxxyfloppy at 18:20 on April 24th, 2018
Posted at 19:09 on April 24th, 2018 | Quote | Edit | Delete
Avatar
Global Moderator
Deceased Gumby
Posts: 1624
Quote:
I haven't tried it before, therefore I can't judge how reliable the conversion tools work. And every conversion entails the risk of errors. Therefore, I would publish the original files and as an addition the converted counterpart, and not vice versa.


It either works or it doesn't because the format doesn't support the CP. And the outcoming files are usually much cleaner, too. This conversion is nothing else but a re-dump really.

Quote:
This can only judged by Mr Creosote. However, I except it would be more a bandwidth problem than a space problem.


Well, I never said space is the issue, either. Perhaps at least the download speed wouldn't be the best.

Quote:
I don't see a problem here.


First off, I'm not sure the hash would be applicable here because RAWs contain a lot of different metadata related to flux stream analysis and each new disk with equal content may in all likelihood produce different results. Though, I'm not completely sure about it, I strongly suspect this to be true. But even if I'm wrong, do you think it would be very convenient to compare such large hash lists? I'm not talking about us, we can use a script, but what about other people?
-----
Cheer up! Remember the less you have, the more there is to get.

I am on irc.freenode.org: #TGOD
Posted at 06:43 on April 25th, 2018 | Quote | Edit | Delete
Avatar
Admin
Reborn Gumby
Posts: 8986
Originally posted by Moebius at 12:47 on April 24th, 2018:
Quote:
So, in such a case, what would you do? Concert the Kyroflux file to one of t preferred ones and post only the latter?


If none of the preferred ones support this type of CP then it's just Kryoflux, naturally. Otherwise I would rather publish only runnable files of course, unless someone tells me what the real advantage is.

OK, so your policy would be: unless the more complex format is absolutely necessary, you will convert the files to a preferred format and only publish the latter. The originally submitted files will only be stored internally. Correct?

Originally posted by fuxxxyfloppy at 18:20 on April 24th, 2018:
Originally posted by Moebius at 10:14 on April 24th, 2018:
One disk depending on the original capacity can be 30-60mb and ...

This can only judged by Mr Creosote. However, I except it would be more a bandwidth problem than a space problem.

As floppy images are not downloaded all that often, space will indeed likely become an issue earlier than bandwidth in this case. If we now wanted to store an extra 60MB per set we already have, it wouldn't be possible. For now, don't bother with this, though. If we really get close to running out of space, I'll raise a flag in time.
-----
Now you see the violence inherent in the system!
Posted at 07:19 on April 25th, 2018 | Quote | Edit | Delete
Avatar
Global Moderator
Deceased Gumby
Posts: 1624
Quote:
OK, so your policy would be: unless the more complex format is absolutely necessary, you will convert the files to a preferred format and only publish the latter. The originally submitted files will only be stored internally. Correct?


Yes, that's the plan for now. Anyway, if we should fix anything, original but modified RAWs would have to go. If we still keep them they would introduce discord into the game group/release. Original is modified and others are clean? And the only way to create a clean RAW is to reverse convert IMG, TD0, PSI or any other format edited prior to it, which seems completely meaningless to me, because at the very least it will lose all the originally introduced metadata. In other words, it won't be the same anymore as conversion filters it out.

We'll wait and see what newcomers have to say. If there will be a lot of people asking for RAWs and providing better reasons for its regular presence (which, honestly, I haven't received yet), we'll launch them, too. Perhaps, Kryoflux might be the best at writing the data back whereas other formats and related tools may not do it quite as correctly, and this, obviously, has nothing to do with the games running properly in emulators. Anyhow, I need a clear confirmation of that, otherwise we end up with tons of heavyweight duplicates which equals junk.
-----
Cheer up! Remember the less you have, the more there is to get.

I am on irc.freenode.org: #TGOD
» Multiple Pages: 123
ReplyNew TopicNew Poll
Powered by Spam Board 5.2.4 © 2007 - 2011 Spam Board Team