Nyerguds 100 Posted November 28, 2012 So then you leave the original mixes as is, and create a new one with your changes? Just cause you build the mixes from scratch doesn't mean you need to build every one every time. No, I update my own files in the mix archive every time. Or rather, I update them whenever I changed them. There's no 'global build procedure' involved in my C&C patch. On the note of patching, using the mix editor in cleanup scripts, like for The First Decade, would be really hard without an update function. Share this post Link to post
Sonarpulse 1 Posted November 28, 2012 Wait, is "update" something you are saying is missing in this new version? Create now will create new mix files and update old ones with the same syntax. Share this post Link to post
Nyerguds 100 Posted November 28, 2012 huh? You said you didn't see the point of updating existing mix files. I'm just saying it's the feature I use the most. Share this post Link to post
Sonarpulse 1 Posted November 28, 2012 (edited) Oh good then. I still stand by my old belief that updating a mix is not the best way to do things, but it's in there as I said, so test it away. Even with the bugs I mentioned it still offers some vast improvements for scripted use over XCC. After putting in a burst of work these last couple days I am now going to have much less time for this for the next couple. I simply replaced the files so the old links still work. You can see and index of the downloads here: https://github.com/Sonarpulse/CnC-Red-Alert/downloads Edited November 28, 2012 by Sonarpulse Share this post Link to post
Nyerguds 100 Posted November 29, 2012 Adding is done by creating a mix file with the filename of a mix that already exists, exactly like tar IIRC. Adding will replace if applicable Oh. So add and replace is one function, then? And yeah, it's pretty obvious you need to rewrite the mixfile. That's how all tools do it (besides xcc mixer, with its tendency to actually add stuff to the end and rewrite the header, which wastes loads of space on Replace commands) --safe (-s) now works. Adding to a mix is already pretty safe without it, but using it guarantees perfect collision safty What exactly does this do though? Refuse to replace/add if a filename in the filenames db is different while the id for your given filename is the same? quotes around paths probably still cause problems Heh, yeah, that can be a pain. Regex can help with that kind of stuff though. currently only files the first name and first ID are actually removed. You can specify more of either. I have no idea what you mean with this... You can't remove a file with no known name (I.E it doesn't convert names to IDs and match that). But if it doesn't know the file name you probably don't either, and you can always Manaully give it the ID. Wait, what? It only looks up names in the names table? Why? All input filenames should be treated as IDs, except for the collision checking. Also, why is this different for removing, specifically? scripts not implemented, at least when the multiple name+id bug is fixed, every conceivable operation can be done in two steps (add these, remove these) What do you mean with "multiple name+id bug"? As far as I can see, every action should be done like this... Given parameters: internal filename, optional external file path logic: -convert internal filename param to ID, look up in header -check collision by comparing internal filename with the one set for that ID as read from filenames db file. --Give warning on collision. --If in 'safe' mode, abort this file's processing on collision --Else, save (add if new, replace if already there) the new filename for this ID in your header info object -Execute action (add/update/remove) for that fileID, with that file's contents (if applicable), taken from the optional external file path, or, if that isn't given, the given internal filename parsed behind the current working directory. The Delete action removes the file's info from the header info object at this point. -After processing all files, generate a new names table from the current (updated) header info. Looking at the things you seem to have problems with, I really think you're making this way too complicated... Personally, I'd add an extra field in the header with the "external file path" for all given files. Then, for the rewrite, you can simply loop over the header listing and check whether that field is filled in. If not, copy the file contents from the old mix file, else read them from the given external path. Since delete commands remove from the header, those aren't rewritten. Share this post Link to post
Sonarpulse 1 Posted November 29, 2012 Add, Replace, and Create are all one interface, yes. I'll probably added "append" to combine to mixes, which will be a seperate interface, else you couldn't include mix files in other mix files. --safe starts with an empty list, and adds everyfile one at a time, checking for collisions each time. If there is a collision it replaces. There difference is more subtle with adding files to a mix: because it checks for collisions with the new files either way. However if the mix before adding contains a collision, you need safe to catch it. I should really read about what stage shells manage regex and quotes. I swear escaping and meta-programming couldn't be more complicated than in shell scripting. It's actually quite simple. You specify a list of names, and/or a list of IDs to filter the list of files by. Problem is, it is only using the first ID and the first name specified to filter. The rest are ignored. The name ID bug is the bug listed above. I've sort of hinted at in the above descriptions, but the way my tools is even more conceptually simple: Convert mix to list of files with with name (if it has one) and ID Use the list manipulating tools built into Haskell to manipulate the list Convert the list into a mix The only reason I didn't have filenames match files with the same ID but none/a diferent name is because I didn't want to expose my ID making function to the CLI. But I realized I have already encapsulated it in a function perfect for the task. Once I use it, you will be able to freely mix names and IDs in the filter list, just prefix the IDs with 0x. (it is the same function used to get the names+IDs from real file-names). The challenge for scripts is I never have more than a small part of the mix in memory. And yet If simply read an write back to real file multiple times, I end up needlessly unpacking and repacking the mix. The thing to do is to read the script as a seize of operations, compose those operation into a single operation, and then apply that operation to a real mix so it is only unpacked and repacked once. Haskell itself has an excellent method to compose operations listed serially/imperatively called Monads. The problem is using this in my own CLI. BTW, if you were planning on using this tool client-side in your installer, then that would be IMO a situation were modifying an existing mix (to save bandwhith and avoid extra "update*.mixS) is the best solution. If that is what you were thinking of all along, my apologies. Share this post Link to post
Nyerguds 100 Posted November 30, 2012 It's actually quite simple. You specify a list of names, and/or a list of IDs to filter the list of files by. Problem is, it is only using the first ID and the first name specified to filter. The rest are ignored. The name ID bug is the bug listed above. Ohhh. You mean, if there's collision INSIDE the list of filenames you specify for adding/replacing? The challenge for scripts is I never have more than a small part of the mix in memory. And yet If simply read an write back to real file multiple times, I end up needlessly unpacking and repacking the mix. The thing to do is to read the script as a seize of operations, compose those operation into a single operation, and then apply that operation to a real mix so it is only unpacked and repacked once. Haskell itself has an excellent method to compose operations listed serially/imperatively called Monads. The problem is using this in my own CLI. Well, that's kinda what I suggested, with going over the mix header info, and writing to one new mix, each time selecting either the original mix or the external file as input source. BTW, if you were planning on using this tool client-side in your installer, then that would be IMO a situation were modifying an existing mix (to save bandwhith and avoid extra "update*.mixS) is the best solution. If that is what you were thinking of all along, my apologies. Hmm... never really considered that. Might be a good idea though. That could save loads of space Share this post Link to post
Sonarpulse 1 Posted November 30, 2012 Ohhh. You mean, if there's collision INSIDE the list of filenames you specify for adding/replacing? This issue is just for removing. Adding/replacing/creating mixes is not affected by this bug. It's supposed to see if a file matches any of the id's or names listed. The problem is simple, it only checks the first name and ID you listed as a argument. All files in the mixed are checked but because of this bug you can only match one ID and name at a time. If the multiple files match in the mix, however, they will still be removed. For exmaple: If I have a bad mix with two files named "eva1.aud", and another named "rules.ini", and i tell it to remove files named "eva1.aud" or "rules1.ini", the mix will be remade with both eva1.auds removed, but rules.ini still there because it never did anything with the second name listed. Well, that's kinda what I suggested, with going over the mix header info, and writing to one new mix, each time selecting either the original mix or the external file as input source. Ah ok. I think though to simplify things I will just combine create/add/replace and remove into one interface : --add [files] --remove [names and IDs]. Adding will take place before removing, because adding will also replace as before. This also means if you are creating a new mix you can easily add all the files in a folder (specify folder name/path) except a list of files you have it filter out after. Hmm... never really considered that. Might be a good idea though. That could save loads of space Glad to help If you want I can also include a hashing function to work on file contents. Share this post Link to post
Nyerguds 100 Posted November 30, 2012 umm... you CAN'T have "multiple files matching in the mix". An ID in the header is always unique. If it isn't, you're screwing up when writing the mix file. And I don't really get your example... Not counting on how you're working on a corrupt mix file that should never, ever exist anyway, how is that related to collision at all? You can't HAVE a mixfile with 2 files named "eva1.aud". The collision is always 2 filenames giving the same ID, not the other way around... 2 identical filenames obviously always give the same IDs. Ah ok. I think though to simplify things I will just combine create/add/replace and remove into one interface : --add [files] --remove [names and IDs]. Adding will take place before removing, because adding will also replace as before. This also means if you are creating a new mix you can easily add all the files in a folder (specify folder name/path) except a list of files you have it filter out after. Well, my suggestion was simply, with the mix header read: -add/replace: if ID doesn't exist yet, add new file entry to mix header. Add external path to the entry. -Delete: remove mix header entry The header info would simply become a tasks list to build the new mix file. Delete is irrelevant because the files are gone from the list, and thus those files' data in the old mix file is "unreferenced" for the new rebuild. You just go over all entries, and see whether to copy the data from files or from the mix. The only exception would be the need to somehow specify reading a file from a specified byte array too, for writing the local mix DB file. Obviously, the ACTUAL header for the new mix file is new data, with the new offsets in the new mix file, only taking the IDs (and, possibly, file length info) of the old header. Share this post Link to post
Sonarpulse 1 Posted November 30, 2012 (edited) Well I was purposefully using that example to show "repairing" an invalid mix. But the problem is completely seperate from collisions. It's really a very simple issue, I must have been unclear: Lets say you have a mix with files a b c d, and you tell cncmix to remove files b and c. You end up with a c d in your mix, because the cncmix improperly ignored the second file to be removed. cncmix's delete command currently takes a list of file names and a list of ids: cncmix delete -I path/to/mix -n name1 name2 name3.... -i ID1 ID2 ID3 ID14 unfortunately that command will only remove files matching name1 or id1. even though you told it to also remove name2 name3 name4.. and id2 id3 id4... it doesn't do anything with regard to those names and IDs. My latest plan and yours for deleting and add/replacing together are very similar, except with mine you are stuck with the name the file has (else you couldn't recur into sub directories). Edited November 30, 2012 by Sonarpulse Share this post Link to post
Sonarpulse 1 Posted December 6, 2012 (edited) OK, I did everything I said I was going to. there is now "mod" mode which supports adding and removing, "info" mode which can print the mix type or a list of IDs, and extract mode which will extract all files. You can also make the modified mix have a different name than the original. The name+ids bug has been fixed, along with the fact where I had it so if I was merging files and the ids matched but the strings didn't, the files would not be merged. Github should be updated new binaries shortly. The short story is for TD this should now be basically complete. I don't know of any bugs in it ATM. So please everybody test this. Edit: Binaries Updated: https://github.com/Sonarpulse/CnC-Red-Alert/downloads Edited December 6, 2012 by Sonarpulse Share this post Link to post
Nyerguds 100 Posted December 6, 2012 Awesome. I'll give it a go as soon as I got some time Share this post Link to post
Sonarpulse 1 Posted December 7, 2012 (edited) Thanks! Feel free to fork me code or whatever, besides the bug I list below it's time to start on red alert mixes. Found an error: print doesn't like the arguments you need it. I happen to be on windows, so i'll upload the fix in a second (I know what the problem is). Edited December 7, 2012 by Sonarpulse Share this post Link to post
Iran 12 Posted December 7, 2012 lol i don't know any obscure languages like Haskell. Share this post Link to post
Sonarpulse 1 Posted December 7, 2012 Well, try it out! If you have never done any functional programming before, Haskell will blow your mind Share this post Link to post
Iran 12 Posted December 7, 2012 I tried it a few months ago cause people keep posting about it on reddit.com/r/programming, don't really understand how it's better than C# and the syntax is way too confusing. Share this post Link to post
Sonarpulse 1 Posted December 8, 2012 Well for one the compiler is much more advanced. Haskell binaries don't have the overhead of a virtual machine. Also unless you use some foreign code very intensionally, your code will probably be completely cross platform without any extra work. Many functional programmers would probably say the style is more intuitive, though if you are used to something else Ill readily admit that is pretty negated. I haven't used c# but my guess I like java it has no type inference. I did scheme first but I have grown to love static typing and type inference. I truly have grown quite lazy about testing my code because honestly, if it type checks it probably works. And type inference means you get all the benefits of type checking without having to type anything extra. Anyways you are right. Haskell does have a bunch of syntactic constructs that are very foreign for imperative programmers. I would start with scheme or some other lisp dialect, which all effectively have 1 syntax. Then once you are used to functional semantics, Haskell and its ML cousins will make much more sense. Share this post Link to post
Iran 12 Posted December 8, 2012 I've done some Scheme while going thru SICP and Scheme's syntax (Lisp too) is really intuitive. But say I see code like this (taken from Wikipedia): map f [] = [] map f (first:rest) = f first : map f rest I have no idea what this does, while in C# (or even C) I would get a function with a descriptive name so I know what it does. Should be the same with Scheme too. C# has a form of type inference with the the 'var' keyword, you place it before a variable and the compiler (and importantly, the IDE) figures out what type the variable needs to have. Share this post Link to post
Sonarpulse 1 Posted December 8, 2012 (edited) Aha, that example illustrates a couple types syntactic sugar in Haskell. It really is the same map as in scheme, except you can only map over one list with it. Maybe this will help: myMap :: (a -> -> [a] -> [b] myMap f lst = case lst of [] -> [] (( car cdr) -> (( (f car) (map f cdr)) Here's map, also in perfectly valid Haskell, but perhaps written in a more 'scheme-like' manner. There is a tool hlint (http://community.has...org/~ndm/hlint/) that works like C's lint in that it gives you suggestions on how to shrink your code. If you were to run it repeatedly on my code there, making it's suggested changes, I almost guarantee you would arrive at Wikipedia's version. Edited December 8, 2012 by Sonarpulse Share this post Link to post
Iran 12 Posted December 8, 2012 Nope, still looks like gibberish to me. :/ Share this post Link to post
Sonarpulse 1 Posted December 8, 2012 (edited) (define (my-map f lst) (cond [(null? lst) '()] [(cons? lst) (cons (f (car lst)) (my-map (f (cdr lst))))])) myMap f lst = case lst of [] -> [] (( _ _) -> (( (f (head lst)) (myMap f (tail lst))) This first line a the type signature. It shows that myMap takes two arguments: a function of type a to type b and a list of type a. It also shows that it returns a list of type b. The type signature is usually optional (it is in this case for example) but sometimes it is useful of the code itself is ambiguous. the patterns to the left of the "->"s are used in pattern matching. Basically it's a way to dynamically dispatch based on type. You can achieve the same thing with predicates and cond in scheme (as I have done), but pattern matching is a bit more succinct, and limiting case to type-based branching helps the compiler optimize. This is probably the most foreign concept for schemers trying Haskell, but it's a nice feature once you get used to it. [] is an empty list : is cons. You can't use any old function in pattern matching, but you can use constructors, which are implicitly defined with data definitions. so the second pattern will match a non-empty list you can also use a pattern match for variable binding. That's how car and cdr were defined in my previous Haskell code. Here though i used "_" to signify I didn't want to bind any variables, and instead used head and tail in my code (same as car and cdr in lisp) in the call. Lastly, functions containing only non-alphanumeric characters are used in infix notation by default. By wrapping ":" in parenthesis, I show I want to use them in prefix notation. http://en.wikipedia.org/wiki/Pattern_matching EDIT: does anybody know why the latest version of IPB has serious bugs with leading whitespace in tags? Edited December 8, 2012 by Sonarpulse Share this post Link to post
Nyerguds 100 Posted December 9, 2012 Nope, still looks like gibberish to me. :/ Gotta agree with that.. Share this post Link to post
Iran 12 Posted December 9, 2012 Oh well I don't really wanted to go off-topic with that, it's also not intended as criticism of anything you've done, this tool sounds great. Share this post Link to post
Sonarpulse 1 Posted December 10, 2012 Thanks, and it's fine. If either of you ever change your mind the code will still be there to look at. And at least you did some scheme and presumably liked it more than Haskell. Share this post Link to post