[yt-dev] Arguments to scripts

Fri Dec 2 11:50:38 PST 2011

Hi Dave,

On Fri, Dec 2, 2011 at 1:39 PM, david collins <antpuncher at gmail.com> wrote:
>> Talking this through leads me to think we might be able to have a
>> combination of 4 and 2.  I just tested and what we *can* do is
>> actually provide the parser arguments as above, but allow for a
>> "leftovers" section for all *positional* arguments to the script.
>> This would mean for --something and -s arguments you would need to add
>> those to the parser object, but for things like what you do above, you
>> wouldn't.
>>
>> Would that work for your use case?
>
> So is option 4 something like
>
> <myscript>
> import yt
> if __name__ == "__main__":
>   yt_option_parser.add_option("--clown_hair_size_threshold")
>   do some stuff using the clow hair size
> </myscript>
>
> like a modified/subclassed optparse?  Then my command line is like
> mpirun -np 16127 python myscript --clown_hair 14 --parallel
> ?

Precisely.

>
> In your "leftovers" version of Option 2, could that just modify
> sys.argv in place, pop-ing out the bits that yt wants or wants to
> raise an error on?  Not strictly necessary, but it would mean a little
> less modification of old scripts on my part.  (Though frankly, scripts
> where I use sys.argv directly should be thrown in the toilet, so
> forcing me to re-write those tools to use some less fragile option
> parser is probably good for me.)

What we could do is have it pop everything out, then replacing
sys.argv with the contents of the leftovers.  I don't really like that
options.  What I think I like more is providing this "leftovers" list
(which can be looked at similarly to sys.args) to yt.mods.

>
> Anyhow, as long as it still retains the ability for me to pass in
> non-yt options to the script, I'm happy.  And there are certainly
> bigger concerns here than my army of dumb scripts.  The first version
> of option
> 2 wouldn't really do that, but the 'leftovers' version would.  I think
> I like your new 2 + 4.
>

Okay.  Then I'll aim for that.

Does anyone else have any feelings on this?

-Matt

> d.
>
>
>>
>> -Matt
>>
>>>
>>> Additionally, if I use optparse, similar outcomes?
>>>
>>> d.
>>>
>>>
>>> On Fri, Dec 2, 2011 at 9:30 AM, Matthew Turk <matthewturk at gmail.com> wrote:
>>>> Hi all,
>>>>
>>>> I have prepared a Pull Request to change how yt processes arguments to
>>>> scripts.  I just issued it, but I am emailing because I think
>>>> discussion of what it does warrants a bit more public hashing out.
>>>> The PR is not done yet, for the reasons I outline below, so please
>>>> don't anybody accept it yet.
>>>>
>>>> https://bitbucket.org/yt_analysis/yt/pull-request/38/overhaul-configuration-system
>>>>
>>>> This will directly affect you if you have:
>>>>
>>>> 1) Ever written "from yt.config import ytcfg; ytcfg[...."
>>>> 2) Ever put your *own* command-line parser into yt.
>>>> 3) Gotten annoyed with configuration files.
>>>>
>>>> What I've done is create a new file, startup_tasks.py, that gets
>>>> imported whenever yt.mods gets imported, and only the first time that
>>>> happens.  It sets up an argument parser (using argparse, which is
>>>> Python 2.7 only) that parses looking for:
>>>>
>>>> --parallel
>>>> --paste
>>>> --paste-detailed
>>>> --detailed
>>>> --rpdb
>>>> --parallel
>>>>
>>>> One of the things this does is that it also provides --help, so you
>>>> can see what is available.  Furthermore, I've added a --config option,
>>>> so that from the command line you can set configuration options.  For
>>>> instance:
>>>>
>>>> --config serialize=False
>>>>
>>>> and so on.  This is pretty cool I think and will go a long way toward
>>>> making things nicer.  However, the way this works is still up for a
>>>> few more problems.  There are basically two ways this can work:
>>>>
>>>>  * Parse the entirety of sys.args and accept all arguments that yt
>>>> finds, rejecting and throwing an error on unrecognized ones (i.e.,
>>>> typos or things you might pass in to a script your write on the
>>>> command line).  This will be an exclusive operation.
>>>>  * Parse *non-exclusively*, allowing unrecognized arguments to pass
>>>> through.  However, the old arguments will still be there: so any
>>>> script that has issues with things like --parallel and whatnot will
>>>> now see there, whereas it did not before because yt (totally un-cool!)
>>>> stripped them out of the sys.args variable.  I don't want to do this
>>>> anymore.
>>>>
>>>> The way I have implemented this for the yt command line tool is to set
>>>> a flag that says, "We're also inside the command line, so don't parse
>>>> anything, we'll handle adding new options to the parser and then we'll
>>>> parse everything at the end."  This way you can pass both --parallel
>>>> and whatever option the yt command line utility wants.  This works
>>>> because startup_tasks creates a "parser" object, adds arguments to
>>>> that parser object, then delays actually conducting the parsing until
>>>> all the arguments from teh command line tool have been added.
>>>>
>>>> There are four ways this can work.  I have presented them in order of
>>>> my increasing preference.  (Coincidentally, on the astropy mailing
>>>> list they discussed this this week, as I was thinking about my
>>>> feelings on it as well, and they are moving away from parsing args in
>>>> the library; I think that works for them because AstroPy is designed
>>>> to be used much more inside larger frameworks, whereas yt is somewhat
>>>> more insular.)
>>>> 1) Don't do any argument parsing if not called through a yt-specific
>>>> script runner.  This means if you want to pass --parallel, you have to
>>>> run with something like "yt run my_script.py --parallel".  Same for
>>>> --config and so on.
>>>> 2) Parse all arguments any time yt.mods is imported, do not allow for
>>>> additional arguments.  This breaks scripts that have their own
>>>> parsing.
>>>> 3) Parse *some* of the arguments, but not all.  All typos would
>>>> succeed and this could lead to confusion for the user.
>>>> 4) Provide a yt-specific mechanism for adding new arguments.  So if
>>>> you want to add new arguments, you do it at the top of your script,
>>>> rather than the bottom, and at the bottom inside the construction "if
>>>> __name__ == '__main__'" you'd inspect the values.
>>>>
>>>> Anyway, I'm inclined to go for #4, simply because it would be the
>>>> simplest mechanism for ensuring an explicit method of getting
>>>> arguments into user-written scripts.
>>>>
>>>> Thoughts?
>>>>
>>>> -Matt
>>>> _______________________________________________
>>>> yt-dev mailing list
>>>> yt-dev at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>>
>>>
>>> --
>>> Sent from my computer.
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
>
> --
> Sent from my computer.
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org