Jump to content
Sign in to follow this  
Sonarpulse

A command line Mix editor

Recommended Posts

So then you leave the original mixes as is, and create a new one with your changes? Just cause you build the mixes from scratch doesn't mean you need to build every one every time.

No, I update my own files in the mix archive every time. Or rather, I update them whenever I changed them. There's no 'global build procedure' involved in my C&C patch.

 

On the note of patching, using the mix editor in cleanup scripts, like for The First Decade, would be really hard without an update function.

Share this post


Link to post

Wait, is "update" something you are saying is missing in this new version? Create now will create new mix files and update old ones with the same syntax.

Share this post


Link to post

huh? You said you didn't see the point of updating existing mix files. I'm just saying it's the feature I use the most.

Share this post


Link to post

Oh good then. I still stand by my old belief that updating a mix is not the best way to do things, but it's in there as I said, so test it away. Even with the bugs I mentioned it still offers some vast improvements for scripted use over XCC.

 

After putting in a burst of work these last couple days I am now going to have much less time for this for the next couple. I simply replaced the files so the old links still work. You can see and index of the downloads here: https://github.com/Sonarpulse/CnC-Red-Alert/downloads

Edited by Sonarpulse

Share this post


Link to post

Adding is done by creating a mix file with the filename of a mix that already exists, exactly like tar IIRC.

Adding will replace if applicable

Oh. So add and replace is one function, then?

And yeah, it's pretty obvious you need to rewrite the mixfile. That's how all tools do it (besides xcc mixer, with its tendency to actually add stuff to the end and rewrite the header, which wastes loads of space on Replace commands)

 

--safe (-s) now works. Adding to a mix is already pretty safe without it, but using it guarantees perfect collision safty

What exactly does this do though? Refuse to replace/add if a filename in the filenames db is different while the id for your given filename is the same?

 

quotes around paths probably still cause problems

Heh, yeah, that can be a pain. Regex can help with that kind of stuff though.

 

currently only files the first name and first ID are actually removed. You can specify more of either.

I have no idea what you mean with this...

 

You can't remove a file with no known name (I.E it doesn't convert names to IDs and match that). But if it doesn't know the file name you probably don't either, and you can always Manaully give it the ID.

Wait, what? It only looks up names in the names table? Why? All input filenames should be treated as IDs, except for the collision checking. Also, why is this different for removing, specifically?

 

scripts not implemented, at least when the multiple name+id bug is fixed, every conceivable operation can be done in two steps (add these, remove these)

What do you mean with "multiple name+id bug"?

 

As far as I can see, every action should be done like this...

 

Given parameters: internal filename, optional external file path

logic:

-convert internal filename param to ID, look up in header

-check collision by comparing internal filename with the one set for that ID as read from filenames db file.

--Give warning on collision.

--If in 'safe' mode, abort this file's processing on collision

--Else, save (add if new, replace if already there) the new filename for this ID in your header info object

-Execute action (add/update/remove) for that fileID, with that file's contents (if applicable), taken from the optional external file path, or, if that isn't given, the given internal filename parsed behind the current working directory. The Delete action removes the file's info from the header info object at this point.

 

-After processing all files, generate a new names table from the current (updated) header info.

 

Looking at the things you seem to have problems with, I really think you're making this way too complicated...

 

Personally, I'd add an extra field in the header with the "external file path" for all given files. Then, for the rewrite, you can simply loop over the header listing and check whether that field is filled in. If not, copy the file contents from the old mix file, else read them from the given external path. Since delete commands remove from the header, those aren't rewritten.

Share this post


Link to post

  1. Add, Replace, and Create are all one interface, yes. I'll probably added "append" to combine to mixes, which will be a seperate interface, else you couldn't include mix files in other mix files.
  2. --safe starts with an empty list, and adds everyfile one at a time, checking for collisions each time. If there is a collision it replaces. There difference is more subtle with adding files to a mix: because it checks for collisions with the new files either way. However if the mix before adding contains a collision, you need safe to catch it.
  3. I should really read about what stage shells manage regex and quotes. I swear escaping and meta-programming couldn't be more complicated than in shell scripting.
  4. It's actually quite simple. You specify a list of names, and/or a list of IDs to filter the list of files by. Problem is, it is only using the first ID and the first name specified to filter. The rest are ignored.
  5. The name ID bug is the bug listed above.

I've sort of hinted at in the above descriptions, but the way my tools is even more conceptually simple:

  1. Convert mix to list of files with with name (if it has one) and ID
  2. Use the list manipulating tools built into Haskell to manipulate the list
  3. Convert the list into a mix

The only reason I didn't have filenames match files with the same ID but none/a diferent name is because I didn't want to expose my ID making function to the CLI. But I realized I have already encapsulated it in a function perfect for the task. Once I use it, you will be able to freely mix names and IDs in the filter list, just prefix the IDs with 0x. (it is the same function used to get the names+IDs from real file-names).

 

The challenge for scripts is I never have more than a small part of the mix in memory. And yet If simply read an write back to real file multiple times, I end up needlessly unpacking and repacking the mix. The thing to do is to read the script as a seize of operations, compose those operation into a single operation, and then apply that operation to a real mix so it is only unpacked and repacked once. Haskell itself has an excellent method to compose operations listed serially/imperatively called Monads. The problem is using this in my own CLI.

 

BTW, if you were planning on using this tool client-side in your installer, then that would be IMO a situation were modifying an existing mix (to save bandwhith and avoid extra "update*.mixS) is the best solution. :D If that is what you were thinking of all along, my apologies.

Share this post


Link to post

  • It's actually quite simple. You specify a list of names, and/or a list of IDs to filter the list of files by. Problem is, it is only using the first ID and the first name specified to filter. The rest are ignored.
  • The name ID bug is the bug listed above.

Ohhh. You mean, if there's collision INSIDE the list of filenames you specify for adding/replacing?

 

The challenge for scripts is I never have more than a small part of the mix in memory. And yet If simply read an write back to real file multiple times, I end up needlessly unpacking and repacking the mix. The thing to do is to read the script as a seize of operations, compose those operation into a single operation, and then apply that operation to a real mix so it is only unpacked and repacked once. Haskell itself has an excellent method to compose operations listed serially/imperatively called Monads. The problem is using this in my own CLI.

Well, that's kinda what I suggested, with going over the mix header info, and writing to one new mix, each time selecting either the original mix or the external file as input source.

 

BTW, if you were planning on using this tool client-side in your installer, then that would be IMO a situation were modifying an existing mix (to save bandwhith and avoid extra "update*.mixS) is the best solution. :D If that is what you were thinking of all along, my apologies.

Hmm... never really considered that. Might be a good idea though. That could save loads of space :D

 

Share this post


Link to post

Ohhh. You mean, if there's collision INSIDE the list of filenames you specify for adding/replacing?

This issue is just for removing. Adding/replacing/creating mixes is not affected by this bug. It's supposed to see if a file matches any of the id's or names listed. The problem is simple, it only checks the first name and ID you listed as a argument. All files in the mixed are checked but because of this bug you can only match one ID and name at a time. If the multiple files match in the mix, however, they will still be removed.

 

For exmaple: If I have a bad mix with two files named "eva1.aud", and another named "rules.ini", and i tell it to remove files named "eva1.aud" or "rules1.ini", the mix will be remade with both eva1.auds removed, but rules.ini still there because it never did anything with the second name listed.

 

Well, that's kinda what I suggested, with going over the mix header info, and writing to one new mix, each time selecting either the original mix or the external file as input source.

Ah ok. I think though to simplify things I will just combine create/add/replace and remove into one interface : --add [files] --remove [names and IDs]. Adding will take place before removing, because adding will also replace as before. This also means if you are creating a new mix you can easily add all the files in a folder (specify folder name/path) except a list of files you have it filter out after.

 

Hmm... never really considered that. Might be a good idea though. That could save loads of space :D

Glad to help :D If you want I can also include a hashing function to work on file contents.

Share this post


Link to post

umm... you CAN'T have "multiple files matching in the mix". An ID in the header is always unique. If it isn't, you're screwing up when writing the mix file.

 

And I don't really get your example... Not counting on how you're working on a corrupt mix file that should never, ever exist anyway, how is that related to collision at all?

 

You can't HAVE a mixfile with 2 files named "eva1.aud". The collision is always 2 filenames giving the same ID, not the other way around... 2 identical filenames obviously always give the same IDs.

 

Ah ok. I think though to simplify things I will just combine create/add/replace and remove into one interface : --add [files] --remove [names and IDs]. Adding will take place before removing, because adding will also replace as before. This also means if you are creating a new mix you can easily add all the files in a folder (specify folder name/path) except a list of files you have it filter out after.

Well, my suggestion was simply, with the mix header read:

-add/replace: if ID doesn't exist yet, add new file entry to mix header. Add external path to the entry.

-Delete: remove mix header entry

 

The header info would simply become a tasks list to build the new mix file. Delete is irrelevant because the files are gone from the list, and thus those files' data in the old mix file is "unreferenced" for the new rebuild. You just go over all entries, and see whether to copy the data from files or from the mix. The only exception would be the need to somehow specify reading a file from a specified byte array too, for writing the local mix DB file.

 

Obviously, the ACTUAL header for the new mix file is new data, with the new offsets in the new mix file, only taking the IDs (and, possibly, file length info) of the old header.

Share this post


Link to post

Well I was purposefully using that example to show "repairing" an invalid mix. But the problem is completely seperate from collisions. It's really a very simple issue, I must have been unclear:

 

Lets say you have a mix with files a b c d, and you tell cncmix to remove files b and c. You end up with a c d in your mix, because the cncmix improperly ignored the second file to be removed. cncmix's delete command currently takes a list of file names and a list of ids:

 cncmix delete -I path/to/mix -n name1 name2 name3.... -i ID1 ID2 ID3 ID14 

unfortunately that command will only remove files matching name1 or id1. even though you told it to also remove name2 name3 name4.. and id2 id3 id4... it doesn't do anything with regard to those names and IDs.

 

My latest plan and yours for deleting and add/replacing together are very similar, except with mine you are stuck with the name the file has (else you couldn't recur into sub directories).

Edited by Sonarpulse

Share this post


Link to post

OK, I did everything I said I was going to. there is now "mod" mode which supports adding and removing, "info" mode which can print the mix type or a list of IDs, and extract mode which will extract all files. You can also make the modified mix have a different name than the original. The name+ids bug has been fixed, along with the fact where I had it so if I was merging files and the ids matched but the strings didn't, the files would not be merged. Github should be updated new binaries shortly.

 

The short story is for TD this should now be basically complete. I don't know of any bugs in it ATM. So please everybody test this.

Edit: Binaries Updated: https://github.com/Sonarpulse/CnC-Red-Alert/downloads

Edited by Sonarpulse

Share this post


Link to post

Awesome. I'll give it a go as soon as I got some time :)

Share this post


Link to post

Thanks! Feel free to fork me code or whatever, besides the bug I list below it's time to start on red alert mixes.

 

Found an error: print doesn't like the arguments you need it. I happen to be on windows, so i'll upload the fix in a second (I know what the problem is).

Edited by Sonarpulse

Share this post


Link to post

lol i don't know any obscure languages like Haskell.

Share this post


Link to post

Well, try it out! If you have never done any functional programming before, Haskell will blow your mind :)

Share this post


Link to post

I tried it a few months ago cause people keep posting about it on reddit.com/r/programming, don't really understand how it's better than C# and the syntax is way too confusing.

Share this post


Link to post

Well for one the compiler is much more advanced. Haskell binaries don't have the overhead of a virtual machine. Also unless you use some foreign code very intensionally, your code will probably be completely cross platform without any extra work.

 

Many functional programmers would probably say the style is more intuitive, though if you are used to something else Ill readily admit that is pretty negated.

 

I haven't used c# but my guess I like java it has no type inference. I did scheme first but I have grown to love static typing and type inference. I truly have grown quite lazy about testing my code because honestly, if it type checks it probably works. And type inference means you get all the benefits of type checking without having to type anything extra.

 

Anyways you are right. Haskell does have a bunch of syntactic constructs that are very foreign for imperative programmers. I would start with scheme or some other lisp dialect, which all effectively have 1 syntax. Then once you are used to functional semantics, Haskell and its ML cousins will make much more sense.

Share this post


Link to post

I've done some Scheme while going thru SICP and Scheme's syntax (Lisp too) is really intuitive. But say I see code like this (taken from Wikipedia):

 

map f [] = []
map f (first:rest) = f first : map f rest

 

I have no idea what this does, while in C# (or even C) I would get a function with a descriptive name so I know what it does. Should be the same with Scheme too. C# has a form of type inference with the the 'var' keyword, you place it before a variable and the compiler (and importantly, the IDE) figures out what type the variable needs to have.

Share this post


Link to post

Aha, that example illustrates a couple types syntactic sugar in Haskell. It really is the same map as in scheme, except you can only map over one list with it. Maybe this will help:

myMap :: (a ->  -> [a] -> [b]
myMap f lst = case lst of
 []            -> []
 (( car cdr) -> (( (f car) (map f cdr))

Here's map, also in perfectly valid Haskell, but perhaps written in a more 'scheme-like' manner. There is a tool hlint (http://community.has...org/~ndm/hlint/) that works like C's lint in that it gives you suggestions on how to shrink your code. If you were to run it repeatedly on my code there, making it's suggested changes, I almost guarantee you would arrive at Wikipedia's version.

Edited by Sonarpulse

Share this post


Link to post

Nope, still looks like gibberish to me. :/

Share this post


Link to post

(define (my-map f lst)
 (cond
   [(null? lst) '()]
   [(cons? lst) (cons (f (car lst))
              (my-map (f (cdr lst))))]))

 

myMap f lst = case lst of
 []        -> []
 (( _ _) -> (( (f (head lst)) (myMap f (tail lst)))

 

This first line a the type signature. It shows that myMap takes two arguments: a function of type a to type b and a list of type a. It also shows that it returns a list of type b. The type signature is usually optional (it is in this case for example) but sometimes it is useful of the code itself is ambiguous.

 

the patterns to the left of the "->"s are used in pattern matching. Basically it's a way to dynamically dispatch based on type. You can achieve the same thing with predicates and cond in scheme (as I have done), but pattern matching is a bit more succinct, and limiting case to type-based branching helps the compiler optimize. This is probably the most foreign concept for schemers trying Haskell, but it's a nice feature once you get used to it.

  1. [] is an empty list
  2. : is cons. You can't use any old function in pattern matching, but you can use constructors, which are implicitly defined with data definitions. so the second pattern will match a non-empty list
  3. you can also use a pattern match for variable binding. That's how car and cdr were defined in my previous Haskell code. Here though i used "_" to signify I didn't want to bind any variables, and instead used head and tail in my code (same as car and cdr in lisp) in the call.
  4. Lastly, functions containing only non-alphanumeric characters are used in infix notation by default. By wrapping ":" in parenthesis, I show I want to use them in prefix notation.

http://en.wikipedia.org/wiki/Pattern_matching

 

EDIT: does anybody know why the latest version of IPB has serious bugs with leading whitespace in

 tags?
Edited by Sonarpulse

Share this post


Link to post

Nope, still looks like gibberish to me. :/

Gotta agree with that..

Share this post


Link to post

Oh well I don't really wanted to go off-topic with that, it's also not intended as criticism of anything you've done, this tool sounds great.

Share this post


Link to post

Thanks, and it's fine. If either of you ever change your mind the code will still be there to look at. :D And at least you did some scheme and presumably liked it more than Haskell.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

×