Game:

The Spam Club

» The Spam Club - Life, The Universe and Everything - Site Issues - disk image file listing or DAT
ReplyNew TopicNew Poll

disk image file listing or DAT

Posted at 19:25 on September 15th, 2018 | Quote | Edit | Delete
Member
Student Gumby
Posts: 38
Is there a consideration to add a directory listing (xml/dat) to the image browser? I am validating DAT info for TDC and want to avoid downloading an image (which I cannot right now anyway :)) every time just to verify a filename/size/date. Less overhead for you guys unless it's not an issue. Alternatively, has someone considered creating a DAT (unless I am missing something?) from the images with their content?

GAME Name, Version, Year, Publisher, Genre, etc.
DISK Number, Name, SHA1
File1, Size, Date, (MD5/SHA1)
File2, Size, Date, (MD5/SHA1)
etc.
Posted at 06:42 on September 16th, 2018 | Quote | Edit | Delete
Avatar
Admin
Reborn Gumby
Posts: 8891
Already exists: /forum/topic.php?id=3586
-----
Now you see the violence inherent in the system!
Posted at 04:08 on September 20th, 2018 | Quote | Edit | Delete
Member
Student Gumby
Posts: 38
My bad. :(

Thank you
Posted at 03:23 on September 24th, 2018 | Quote | Edit | Delete
Member
Student Gumby
Posts: 38
Just checked the DAT file. It's not quite what I had asked.

Knowing the contents of IMG files could be extremely worthwhile when comparing loose files to know where something belongs.

Has anyone done an inventory of the IMG content to get a file listing at least (i.e. 7z -l DISK.img)? At best, having checksums of those files would be akin to a complete DAT.

One way would be to use a floppy image mount program and batch images after running a checksum program against them.
Posted at 10:53 on September 24th, 2018 | Quote | Edit | Delete
Member
Master Gumby
Posts: 115
@bmanbdaman:
I calculated the checksums for files inside the images. I thought it might be a good way to analyze them but I didn't followed that path later on. It's not up-to-date and not in a DAT format. I can check my script again and If there is interest, I can rerun it.

What kind of output do you prefer?
Posted at 12:49 on September 24th, 2018 | Quote | Edit | Delete
Member
Student Gumby
Posts: 38
Short answer.

I think starting with a plaintext file (not DAT) will help realize what format is required. I could take a look at the old data you have. Probably something like this?

GAME1,DISK1.IMG,CHKSUM,DIR\FILE1,DATE,SIZE,CHKSUM
GAME1,DISK1.IMG,CHKSUM,DIR\FILE2,DATE,SIZE,CHKSUM
GAME1,DISK2.IMG,CHKSUM,DIR\FILE1,DATE,SIZE,CHKSUM
GAME1,DISK2.IMG,CHKSUM,DIR\FILE2,DATE,SIZE,CHKSUM
etc.

Having a DAT that shows the file content may help with floppy variances (different SHA1) and indicate sameness at a file level which could make validating images quicker, but may not help with verifying proper formatting or identifying a new original image.

Long answer.

The final output to consider may vary because DATs normally align with their parent content (i.e. roms), which are most often singular files. Multiple content is then usually additional listings so a grouping in an XML will suffice.

TDC (TotalDosCollection) realized the need and are using their own DAT format to catalog multiple content images like installed DOS programs with nested directories and potential overlapping content. They do not do a date comparison for completeness, though. That may be a concern.

Code:
game (
	name "IMAGENAME.IMG"
	file ( name FILE1 size 11111111 date 1979/12/31 00:00:00 crc AAAAAAAA )
	file ( name FILE2 size 22222222 date 1979/12/31 00:00:00 crc BBBBBBBB )
	file ( name DIR1\FILE3 size 333 date 1979/12/31 00:00:00 crc CCCCCCCC )
	file ( name DIR2\FILE4 size 444 date 1979/12/03 00:00:00 crc DDDDDDDD )
	file ( name FILE5 size 55555555 date 1979/12/31 00:00:00 crc EEEEEEEE )
)


More complicated and compatible options are documented for ROMCenter or CLRMamePro, but both are rigid and require content match the DAT in strict adherence. TDC has a weight and comparison option which checks other content in the DAT and relates it by percentage and allows you to select other potential references within.

The problem is that all the available tools require expanding the content as a zip and already containing checksums. IMG does not have that format so something would have to be built. TDC is looking into that.
Posted at 12:23 on September 25th, 2018 | Quote | Edit | Delete
Member
Master Gumby
Posts: 115
All that relevant information I can extract using the script.

If you code example is the needed format, I will change the output accordingly. I should have time for it in the next few days.

What do you mean by:
"The problem is that all the available tools require expanding the content as a zip and already containing checksums. IMG does not have that format so something would have to be built."
-----
Edited by fuxxxyfloppy at 12:25 on September 25th, 2018
Posted at 02:23 on September 26th, 2018 | Quote | Edit | Delete
Member
Student Gumby
Posts: 38
IMG file contents by themselves do not have a checksum so there is no way of having a current tool check them without extracting them first. If your script generates a DAT, then some tool (RC, CLRM, Sabre, TDC, etc.) would still be used to reconcile a collection unless you are writing it altogether. The folks at TDC are discussing this topic as well to try and find a working solution.
Posted at 20:33 on September 26th, 2018 | Quote | Edit | Delete
Member
Master Gumby
Posts: 115
It can look like your example:

Code:
DOSCenter (
	Name:TGOD .dat
	Description:
	Version: 
	Date:
	Author:TGOD
	Homepage:http://www.goodolddays.net/
	Comment:
)
game (
        name "000463_mega_man/disk1.img"
        file ( name CREDITS.STA size 5851 date 1990/11/05 02:42:08 crc 707C1C0E )
        file ( name DYNA.BIN size 9057 date 1990/11/02 13:57:32 crc 83F13625 )
        file ( name DYNA.BLK size 15755 date 1990/10/27 16:05:22 crc 2EC82123 )
        file ( name DYNA.FRM size 7547 date 1990/10/01 22:35:46 crc 044138B2 )
        file ( name DYNA.SCN size 4422 date 1990/10/27 16:05:22 crc 87E92F00 )
        file ( name LOGO.STA size 13478 date 1990/10/25 01:22:36 crc 7894CE5C )
        file ( name MEGA.FRM size 14044 date 1990/10/01 22:34:44 crc 11DA9A87 )
        file ( name MM.EXE size 20349 date 1990/11/29 11:31:42 crc AEA06825 )
        file ( name PASS.FRM size 10336 date 1990/11/05 02:42:20 crc 235C08E5 )
        file ( name SECUR.BIN size 1985 date 1990/10/29 18:26:44 crc 235F27BB )
        file ( name SECUR.BLK size 10249 date 1990/10/01 00:59:20 crc 785025CC )
        file ( name SECUR.FRM size 6335 date 1990/10/01 22:35:08 crc ED1EA490 )
        file ( name SECUR.SCN size 1145 date 1990/10/01 00:59:18 crc F274515D )
        file ( name SELECT.FRM size 14907 date 1990/10/01 22:35:00 crc F0AD5167 )
        file ( name SKULL.STA size 19911 date 1990/11/02 15:16:00 crc 8E963B7A )
        file ( name SONIC.BIN size 8536 date 1990/11/01 17:05:44 crc FF2D14D1 )
        file ( name SONIC.BLK size 23163 date 1990/10/01 00:56:12 crc 001CA663 )
        file ( name SONIC.FRM size 7359 date 1990/10/01 22:35:26 crc B1922277 )
        file ( name SONIC.SCN size 10630 date 1990/10/01 00:56:10 crc 0DABC266 )
        file ( name VOLT.BIN size 10061 date 1990/11/01 18:30:42 crc CD8DD4BC )
        file ( name VOLT.BLK size 29842 date 1990/10/01 00:56:30 crc D5322E7E )
        file ( name VOLT.FRM size 11262 date 1990/10/01 22:35:36 crc C4DB0553 )
        file ( name VOLT.SCN size 6845 date 1990/10/01 00:56:28 crc D81A6ED8 )
        file ( name WILEY.BIN size 12810 date 1990/11/02 13:46:32 crc 9C3BB407 )
        file ( name WILEY.BLK size 26414 date 1990/10/01 00:56:54 crc C3BF1BEB )
        file ( name WILEY.FRM size 21801 date 1990/10/01 22:36:00 crc BA42A54B )
        file ( name WILEY.SCN size 6047 date 1990/10/01 00:56:52 crc 0B39EA9F )
        file ( name WPNSCN.STA size 16670 date 1990/11/02 14:39:34 crc 17498365 )
)


Is this suitable for your use case?

Edit:
I also played a bit around with DOSCenter and this example DAT file.

Due to the fact that DOSCenter only works with zip files and does not recognize the folder structure in the name, it would be better to name it like 000463_mega_man_-_disk1.img.zip. What do you think?
-----
Edited by fuxxxyfloppy at 21:16 on September 26th, 2018
Posted at 02:21 on September 28th, 2018 | Quote | Edit | Delete
Member
Student Gumby
Posts: 38
This is fantastic. If you have this ready I'd like to test it against my floppies to see how it works in DOSCenter. Yes, the ZIP may have to be necessary for the compressor to pick it up. They may still have to code to get CRCs to work since the IMG files do not have them by default. I think they are trying to work that out. Thanks
-----
Edited by bmanbdaman at 02:23 on September 28th, 2018
Posted at 15:01 on September 28th, 2018 | Quote | Edit | Delete
Member
Master Gumby
Posts: 115
I will post it, when I'm back home in the evening
Posted at 21:36 on September 28th, 2018 | Quote | Edit | Delete | Delete Attachment
Member
Master Gumby
Posts: 115
Check the attached file.

I added a few more properties that might be good to have (at least for me, because I plan to do some analyzes with that information later) and it seems it is no problem for DOSCenter.

The dat file also contains booter disks. I haven't filtered them out yet. But they don't have a filelist.
Attachment: *****
Posted at 01:51 on October 1st, 2018 | Quote | Edit | Delete
Member
Student Gumby
Posts: 38
This works great against my old disk file copies (not IMGs). Have to convert all the ARJs to ZIP to see what I have. Shared at TDC and they are working on a format process to label this data into a working DB.

Thanks
Posted at 21:54 on October 12th, 2018 | Quote | Edit | Delete
Member
Master Gumby
Posts: 115
Originally posted by bmanbdaman at 01:51 on October 1st, 2018:
Shared at TDC and they are working on a format process to label this data into a working DB.

Pleas keep us in the loop.

And if you found/detected new disks, it would be great, if you can submit them here, too.
Posted at 23:03 on October 12th, 2018 | Quote | Edit | Delete
Member
Student Gumby
Posts: 38
I wish they were disk images. They are old ARJ distros of scene floppy archives. However, I have a few games left in boxes somewhere I will have to add.

TDC is still running a debate about how to represent the data. When they are at the catalog stage we'll have to get synced up.
Posted at 12:05 on June 10th, 2019 | Quote | Edit | Delete
Member
Master Gumby
Posts: 115
Hi bmanbdaman, I read that the TDC will start to include floppy images in their next releases. Did you/they find a way to analyse and compare them properly?
Posted at 13:12 on June 10th, 2019 | Quote | Edit | Delete
Member
Student Gumby
Posts: 38
The floppies being included are not (yet?) as thoroughly vetted as TGODs methods (i.e. validating MBR/FAT originality for tamper, etc.), but they are including the versions that are cleaned here. They are also leaning more towards KF dumps rather than images, so that is a huge difference. Their idea may be to go for accurate disk and byte equivalent representation of images, but someone still has to validate originality. Redump.org and TGOD do more of that work, where TDC does more of the collecting of source materials. I spend a lot of time cleaning DAT references for versioning, file inconsistencies, playability and functionality, and patching games for install and testing than actually collecting them.

If possible, both teams should talk and find out if their charters are compatible. I think the admin(s) here may want to talk to the admins at TDC to figure out what common ground can be had.
Posted at 08:40 on June 12th, 2019 | Quote | Edit | Delete
Member
Master Gumby
Posts: 115
Thanks for the update. I'm looking forward to it.
Posted at 15:48 on June 12th, 2019 | Quote | Edit | Delete | Delete Attachment
Member
Student Gumby
Posts: 38
As of now, the official response is that they plan to have their tools analyze decoded sector data which should produce a hash that is equivalent to an IMA (as is done here.) I cautioned that there will be many discrepancies with other collections since there is no guarantee on originality if the disk cannot be certified before accepting the dump. A KF would be impossible to match to any OBP/FAT fixed contents that are rectified, for example.

This is why I started this thread. I believe that ultimately the content will drive the identification and comparison, and not the hash of the image. That way, someone can use a comparator, much like TDC did with their DOSCenter DAT tool, that can provide an accuracy gauge to show that something is closely related to an already defined composition of a collected game. While this is available for individual files (akin to a rom manager), it has yet to be done for an image process.
Attachment: *****
ReplyNew TopicNew Poll
Powered by Spam Board 5.2.4 © 2007 - 2011 Spam Board Team