Jump to content
Sign in to follow this  
Sonarpulse

A command line Mix editor

Recommended Posts

My mix tool currently supports creating and the modifying uncompressed mixes used exclusively by TD and also by RA.

Source: https://github.com/Ericson2314/Codec.Archive.CnCMix
Linux Binary: http://www.mediafire.com/?2zb11vk44p3sqad

Windows Binary: http://www.mediafire.com/download.php?37ywuyc3ydn541g



Original Post:

So a while ago a made a thread about make a system to build RA, from scratch, and put some data up here: https://github.com/Ericson2314/CnC-Red-Alert

Well the first step for that was making a command line tool to build mixes, so I could automate things with make. Well now I am actually getting around to it, but I am starting by implementing TD mixes because they are simplest. If you click that link now you can see what I have so far (the default branch just contains my "mixer" code).

Olaf van der spek (the guy behind XCC Mixer) has specs for mixs and other C&C formats on his website, but they don't cover his extensions. If anybody knows anything about them, that would be much appreciated.

Also, anybody that uses XCC mixer a lot with especially TD mixes (looking at you, Nyerguds), is there any feature you would like? Keep in mine that I am making a console program, but since this is in a very high-level language it's a basically a library I can slap any interface on.

Edited by Sonarpulse

Share this post


Link to post

XCC is kinda open source... have you checked out https://code.google..../#svn/trunk/xcc ? It should have the XCC Mix Editor code, which should show a nice distinction between the saving options of all mix file types.

 

Personally, I just want a 32-bit app that can do all the stuff the old MMLite could do; create mix files, add files to it, delete files from it, replace files in it, and which can do those 3 actions both by filename and by file ID.

For RA mixfiles specifically, the option for enabling mixfile encryption is a must, since RAED can't read RA mixfiles without that encryption (see TFD).

 

In general, though, one option I'd love to have is for it to accept a file with bulk operations to execute on one mix file. Like a file that looks like this:

 

a scb04ea.ini "new missions\scb04ea.ini"

a scb04ea.bin "new missions\scb04ea.bin"

d expand.dat

ar htitle.pcx

ar mtitle.pcx hires\mtitle.pcx

ar xtitle.pcx hires\xtitle.pcx

 

'a' being add, 'd' being delete, 'r' being replace, and 'ar' being 'add or replace', and the add and replace ones supporting an optional extra input file path at the end. Both the filename and the input file path obviously supporting quoted names to allow long filenames.

 

Oh, and if you copy from XCC... please leave out the non-compacting save method. It's what made the TFD RA main.mix 400 mb too big X_x

Share this post


Link to post

  • Yeah, I did a svn->git and put his code on my github. but it's kindof big and c++ and hard to read, IMO.
  • Until I figure out how XCC stores filenames I can only read mix entries by ID, but I can still replace from filename by making the ID first. If you could point me to MMLite, I'd be interested to see how it does filenames. I may also experiment with "cracking" IDs into filenames, hopefully the hashing function doesn't make to many collisions.
  • I honestly have no idea how non-compacted mode works, so I won't be implementing that.
  • scripts with that assembly like syntax will be easier to parse than arguments, so you can be sure I'll do that!
  • Haskell has built-in CRC and blowfish so the encryption shouldn't be too much of a stumbling block.

Share this post


Link to post

Not sure what language you're programming in, but I got a C# version of the mix encoding algorithm for C&C1 and RA.

 

http://nyerguds.arsa...ionnamechecker/

 

That's just copied from XCC though. The hard part about the id calculation isn't understanding C++, it's understanding bit shifts and their syntax.

 

I don't see what's so hard about performing actions by filename... it's just calculating the ID of the given filename and looking it up in the mix header.

 

You may need a parameter to give the mixfile type though, to see which id calc algo to use, or attempt mix type identification the way XCC does it. Still, you'll need that parameter anyway for creating mix files.

 

Getting filenames from IDs is utterly impossible though. You can't get a 12 character filename from data reduced to 4 bytes. Even C&C's mission syntax caused a mess of collisions once I started using it to its full extent, which forced me to limit mini-campaigns (which use all mission choice suffices) to the 20-99 and 900-999 number ranges.

 

On the other hand, encoding a name and looking up if that ID exists in a mixfile is ridiculously simple. As long as you don't have to LIST filenames in a mix file, this is all you need for a command line tool.

As for saving file names, um, as far as I know XCC's filenames list is really just that; a simple file with all filename strings in it, and a header telling XCC it has the right file.

 

I honestly have no idea how non-compacted mode works, so I won't be implementing that.

Basically, when deleting a file it just deletes it from the header, and when replacing a file by a smaller version, it just overwrites the data at the file position with the new version, and adjusts the size in the header. Neither action will actually rewrite the whole mixfile to clean up the unused data. REALLY messy IMO.

Share this post


Link to post

I took a look at the "local mix database.dat" file format. As I thought, it's ridiculously simple. It contains:

 

-32 bytes: The string "XCC by Olaf van der Spek", immediately followed by bytes "1A 04 17 27 10 19 80 00".

-16 bytes: the full size of the "local mix database.dat" file itself, in big-endian of course. Padded with zeroes.

-4 bytes: the number of files in the list, again big-endian

-zero-separated filename strings. There is a zero behind the last string in the file, too.

 

The file size for the header can be precalculated as [header size (=52)] + + [number of files (for the zero bytes)]

 

I tested these mysterious header bytes on C&C1 and RA2 mixfiles, and they're exactly the same in both cases, meaning it's not mix type dependent or anything. If you support generating these (please do), just copy it and you'll do fine.

 

[edit]

 

btw, Mix Manager (including MMLite) can be found here:

http://nyerguds.arsaneus-design.com/tools/old/mman351.zip

Share this post


Link to post

Thanks so much for the information. I just fixed my ID function in fact, but thanks for the link to the c# anyways. No idea how I missed converting the filenames to IDs and then looking at the contents of the mix. I will certainly do it that way.

 

And certainly thanks so much for the "local mix database" information. As you say this is all I need to implement it so I will certainly do so. Now i don't need to see mmix as much, but it will still be interesting.

 

The language is Haskell by the way. I don't know if you have done any functional programming before, but but both Haskell itself and the dominant implementation are quite innovative.

 

One of it's features is "type classes". basically I can define a default method to, let's say, naively unpack, add a file to the list, and repack for "add file", so you can be sure I won't resort to XCC's uncompacted save.

Share this post


Link to post

hmm I noticed there are four bytes between the 16 for the size and the first filename. Any idea what those are for?

Share this post


Link to post

Oh. Hadn't noticed that. I just did a quick check, and, as I thought from the value, it's the number of filenames in the list.

 

[edit]

 

Added it into the post above, so it's still usable as reference :)

Share this post


Link to post

Ok, so this project has been getting pretty complex due to corner cases involving local mix databases. For example, what does XCC do when you try to add a file to an original (database-less) mix where XCC doesn't know all of the IDs? It normally generates a local mix database whenever you make a change, but how can it do that when it doesn't know the filenames for all files?

 

If it weren't for local mix databases, it would be pretty easy to implement "add file to mix" "remove file from mix". With the databases however, I am starting to think it's easier if I limit to complete "files to mix" "mix to files" and "extract file". Already, let's say replacing a file in the middle of a mix with a different sized file and updating database is hard to do more efficiently and safely than simply unpacking the mix to my internal

  1. format.

 

I think this would also provide everything you need, Nyerguds, If we think of a naming convention for name-less files. This should allow you to build mix files from scratch, rather than make modify-mix scripts. You have the same ease of use either way.

Share this post


Link to post

OK, well I tested XCC and I found that it simply doesn't include strings for filenames it doesn't know in local mix database. I guess to read the names it just converts all the strings to IDs and looks for matches. Not exactly efficient but robust.

 

My plan was to make an exception so the stringToId function so that names like 0x055d42 would be directly converted to IDs. This would basically fix all the problems induced by local Mix databases WITHOUT adding any special cases:

  • allows files with unknown proper names to by saved by their ID, re-imported and still keep their proper ID.
  • I typically either have a list of file-contents + ids, or file-contents + proper-long-names. This will allow me to keep the lists all one or all the other.
  • still a simple 1-1 correspondence with entries in the database, making lookup much easier.

The reason MMlight could add and remove by filename or ID so easily is that it didn't support anything like local mix database.dat. then you can add and swap files with much more freedom.

 

My program works best when it can just deserialize everything, do whatever complicated stuff you want, and then serialize it back again. And indeed, this is more efficient than MM's methods if you are replacing a bunch of files with files of different sizes (or removing files) because the latter method involves walking through all the offsets multiple times.

 

On the other hand this would at the very least confuse XCC, if not make my tool incompatible all together.

 

Edit: I could also have add the ability to export and import XCC style mixes. Maybe use a different random string of bytes to tag them too. (On a whim, I checked and no those bytes aren't used for determining which entries have filenames, they really are a random constant as you say).

Edited by Sonarpulse

Share this post


Link to post

OK, I solved most of this by storing both the ID and the filename in my file data type. This means no matter what you can still add and remove by ID or Filename. Also it means my internal data structure can handle mixed name-known and name-unknown data. Converting that to a name database will still be difficult, but at least I have somewhere to start. And yes, I did did ID-as-file name trick too, so you can still easily merge filenames without manually dealing with IDs. And to make sure that "0x..." filename will not end up in a name database, the ID generator also removes the string name from a files with such names when it gives them their ID.

 

So yeah, despite my fear in the previous post, I think I am now handling local mix databases pretty well, AND not creating to many special cases too boot. I will make it deserialize all the way by default I think, (the single case where this is excessive is a adding a nameless file to a nameless database). Even if I am extracting the file, the laziness should mean that the "deserialization" really just makes a linked-list of thunks, so that shouldn't be nearly so much extra work.

Share this post


Link to post

Umm... I don't see the problem. XCC simply LEAVES OUT the files it doesn't know when making its local mix database file. You only need to make a local mix database file if you have any identified files, and you only add those that you have identified into it, either from an older local mix database file in the mixfile, or from the filenames given by the user. You can even add new filenames to it whenever a Replace command identifies a new one.

 

I'd just do it like this:

-Make sure the header file-item object you use has an extra filename peoperty, in addition to its ID property (which should be read from the header).

-When reading the mix file, immediately look for and read the local mix database.dat file, convert all filenames in it to IDs, match those to the header data and store the filenames you identify with that into your header data.

-Any user actions like adding, replacing, deleting happen by converting the given filename to ID. No exceptions. But, whenever you add or replace a file by filename, you fill in the given filename in that file's data.

-When saving, write a new local mix database which simply includes all files from your internal files list for which the filename is filled in. If none of them have a filename, or the only one you got is the local mix database file itself, don't write the database file, and if there already is a database file, remove it from the mixfile.

 

Doesn't seem too hard to figure out, tbh. As far as I can see, this way would have no special cases whatsoever, except for the actual handling of the database file in the save logic.

 

I would include an option to specifically ignore the local mix database file, though, in case people don't want it to be added. In that case, the local mix database file isn't read or written, and, if present, is treated just like any other file. This would also allow people to delete the filenames list if they want to (which wouldn't be possible if the program writes it whenever it saves the mixfile).

 

As opposite use case, in addition to "Add", "Add or Replace", "Replace" and "Delete", you could have an "Identify" option ("i"), which does nothing except try to identify a filename in a mix file, to write into the local mix database file.

Share this post


Link to post

Oh, to support ID-based operations, I'd simply support a syntax like "id=XXXXXXXX" to be usable instead of a filename for any "internal filename" parameter.

 

Basic syntax:

command optional_internal_filename external_path

Delete and Identify only have the internal filename, of course, making it non-optional for those. Quotes are allowed around filenames to allow use of spaces (especially in paths for the last parameter, but, allow it for the first parameter too, for cases like d "local mix database.dat")

 

Example for use with IDs:

ar id=3F2E12E7 "addon british\briefings.ini"

d id=F92BD31D

 

The "Identify" command suggested above would simply work like this:

i missbrit.ini

 

(all these examples are taken from my british language addon, btw ;))

Share this post


Link to post

Oh, one special case you may want to support:

 

if the local mix database contains a filename, and an Add or Replace command for a DIFFERENT FILE NAME gives the SAME ID, give a UI warning that there has been an ID collision, and list the 2 file names on which it happened. As for actually writing, continue normally (if it was Add, it shouldn't happen, if it was "Add or Replace", or "Replace", do the replace and make sure the new filename is stored as that ID's filename)

 

If you need a test case, here's an example: SCG148EE.ini, SCG248EC.ini and SCG348EA.ini all give exactly the same ID on the C&C1 name ID algorithm.

 

CnCMissionNameChecker.png

(tool can be found here)

Share this post


Link to post

Hmm, I actually have done things just about that way. The biggest exception is instead of doing replacing strictly by ID, a I "combine" the new file (with whatever information is provided) with the old version. My combiner is pretty intelligent (empty list is "null filename", 0 is "null id" (though that shouldn't be needed), and empty "ByteString" (Arrays under the hood, but interface and efficiency more like a linked list). Adding removing and replacing and removing should all work this way with a little more work.

 

There are a few more bugs to work out, but it's all ready pretty usable. Again, I haven't implemented any interface, but you can just fire it up in the GHCi interpreter, and call the functions with that. That will frankly be a much more powerful interface than anything I can rig up.

Edited by Sonarpulse

Share this post


Link to post

um, the only 'interface' you need is console text feedback... it's a CLT after all.

Share this post


Link to post

Well for the script it's a big harder, and even with the interface to do it right you need long and short form command names in any order and a couple of other things. It's far too easy to continue putting it off doing that until I have completely finished the mix library part, and test it in the interpreter. But then I'll be the only tester unless anybody else downloads GHC.

Edited by Sonarpulse

Share this post


Link to post

yeeeah, I'd prefer the tool to not need any external systems to work :P

Share this post


Link to post

OK, I just got it to work! Well, at least my roundtrip test does. XCC will read but not modify mixes I made/modified (I think because the local mix database is not at the end), but it's now robust enough to read the names from the local mix database even if they are out of order, and apply them to the correct files. It also has "safe add/replacing" as you requested.

 

I haven't tested adding/removing, or tested a mix with TD yet, but I doubt anything will go wrong, as now everything really is done with the same list-merging primitive as I planned. That means I have no longer any excuse to avoid making the UI :)

Share this post


Link to post

OK, found a good library to through together a CLI, and implemented mix creation and extraction! It will recur into directories to get files, and it exports unnamed files and reimports them flawlessly.

 

I just compiled a Linux binary, I'll need to reboot and install Haskell on Windows to compile one there so tell me if you need that.

Linux Binary: https://github.com/downloads/Sonarpulse/CnC-Red-Alert/cncmix

Edit: ok, I want this tested so here you guys go:

Windows Binary: https://github.com/downloads/Sonarpulse/CnC-Red-Alert/cncmix.exe

 

known bugs:

  • strings in filepaths. Probably all sorts of shell-path ideosyncrasies will cause problems
  • --safe mix creation (check for id collisions) is not implemented, flag does nothing
  • --type defaults to TD, which is good because none of the others are implemented and it doesn't recognize game names ATM anways. It also does nothing but through UI errors ATM

I tested it with AUD.MIX (installer sounds) because it contains files with unknown names. XCC said the same information was there (albeit the files were in a different order).

 

Because of the import-export by ID functionality, I really don't think you need the ability to modify an existing mix, just make a new one. Neverless the ability is there in the library, and with my experience making this interface it should be easy to add.

Edited by Sonarpulse

Share this post


Link to post
Because of the import-export by ID functionality, I really don't think you need the ability to modify an existing mix

Eh? Updating mix files is what I'd do most with a clt mix editor.

 

For example, updatec.mix is where C&C95 stores most of its C&C95-specific UI files. It's also where I added the language settings and rules files. I would like to be able to update these without having to bother with all the other stuff in there.

Share this post


Link to post

We are thinking in different ways. I would keep a folder called updatec, version control it, and make a makefile like this:

%.mix : mix/%
            cncmix -O [email protected] $(MIX_FLAGS) $<

I forget if patterns can match directories but you get the idea

 

I would prefer this to editing a mix, even with a script to make my changes, because I never have to worry about what version of what is in the mix, and I can build fresh binaries whenever I want.

 

Anyways, I made a windows executable so test that. And don't worry, though I still don't see the point I'll have your features in soon.

Edited by Sonarpulse

Share this post


Link to post

The point is only keeping my own custom files outside of the mixfile, and leaving the original ones where they are, to reduce the amount of files the full project contains.

Share this post


Link to post

So then you leave the original mixes as is, and create a new one with your changes? Just cause you build the mixes from scratch doesn't mean you need to build every one every time.

 

Anyways, I now have adding removing files from a mix, and print File Names and IDs implemented. I'll post the linux binary in a second, and windows one once I reboot.

other changes/bugs:

  • Adding is done by creating a mix file with the filename of a mix that already exists, exactly like tar IIRC.
  • Adding will replace if applicable
  • --safe (-s) now works. Adding to a mix is already pretty safe without it, but using it guarantees perfect collision safty
  • quotes around paths probably still cause problems
  • currently only files the first name and first ID are actually removed. You can specify more of either.
  • You can't remove a file with no known name (I.E it doesn't convert names to IDs and match that). But if it doesn't know the file name you probably don't either, and you can always Manaully give it the ID.
  • scripts not implemented, at least when the multiple name+id bug is fixed, every conceivable operation can be done in two steps (add these, remove these)

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

×