[yt-dev] Arguments to scripts

Fri Dec 2 12:36:39 PST 2011

Hi Dave,

On Fri, Dec 2, 2011 at 3:31 PM, david collins <antpuncher at gmail.com> wrote:
>> something like yt.argv.  That way, anyone who has been pull arguments from
>> the command line can simply replace sys.argv with yt.argv, or something like
>> that.  As long as the substitution is intuitive, people won't have to go
>> parsing through source or documentation when they go back to an old script
>> and find it doesn't work anymore.
>
> If we go with 2+4 and "leftovers," what would be the actual behavior of
>
> from yt.mods import *
> import sys.argv
> print sys.argv
> ?
>
> Would that still give me the same thing as before, and additionally
> create yt.optparse and yt.leftovers, with yt getting its info from
> these features?  Or will sys.argv get replaced with something else?

It'll have everything in it, including options that have been parsed
by the option parser.

>
> d.
>
>
>
>
>>
>> Britton
>>
>>
>> On Fri, Dec 2, 2011 at 2:57 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>>
>>> That's the big problem, yes.  I would be fine with yt taking over
>>> things like "--help" and whatnot, which are reserved.  But I would
>>> also note that we're actually *changing* the functionality, not
>>> *removing* it if we go with 2+4.  It says, if you want your script to
>>> be able to run in parallel yt, or to use these yt commands, here's how
>>> you interface with the yt argument parsing.
>>>
>>> On Fri, Dec 2, 2011 at 2:51 PM, Britton Smith <brittonsmith at gmail.com>
>>> wrote:
>>> > I think I still prefer option 3.  I'm in favor of making the system of
>>> > parsing yt args cleaner, but options 1, 2, and 4 feel to me like we're
>>> > saying in order to use yt, you have to trade in some python
>>> > functionality
>>> > that you've been using.  Is the only reason we wouldn't go with option 3
>>> > that it will not catch typos in flags that we intended to be for yt?
>>> >
>>> > Britton
>>> >
>>> > On Fri, Dec 2, 2011 at 1:39 PM, david collins <antpuncher at gmail.com>
>>> > wrote:
>>> >>
>>> >> > Talking this through leads me to think we might be able to have a
>>> >> > combination of 4 and 2.  I just tested and what we *can* do is
>>> >> > actually provide the parser arguments as above, but allow for a
>>> >> > "leftovers" section for all *positional* arguments to the script.
>>> >> > This would mean for --something and -s arguments you would need to
>>> >> > add
>>> >> > those to the parser object, but for things like what you do above,
>>> >> > you
>>> >> > wouldn't.
>>> >> >
>>> >> > Would that work for your use case?
>>> >>
>>> >> So is option 4 something like
>>> >>
>>> >> <myscript>
>>> >> import yt
>>> >> if __name__ == "__main__":
>>> >>   yt_option_parser.add_option("--clown_hair_size_threshold")
>>> >>   do some stuff using the clow hair size
>>> >> </myscript>
>>> >>
>>> >> like a modified/subclassed optparse?  Then my command line is like
>>> >> mpirun -np 16127 python myscript --clown_hair 14 --parallel
>>> >> ?
>>> >>
>>> >> In your "leftovers" version of Option 2, could that just modify
>>> >> sys.argv in place, pop-ing out the bits that yt wants or wants to
>>> >> raise an error on?  Not strictly necessary, but it would mean a little
>>> >> less modification of old scripts on my part.  (Though frankly, scripts
>>> >> where I use sys.argv directly should be thrown in the toilet, so
>>> >> forcing me to re-write those tools to use some less fragile option
>>> >> parser is probably good for me.)
>>> >>
>>> >> Anyhow, as long as it still retains the ability for me to pass in
>>> >> non-yt options to the script, I'm happy.  And there are certainly
>>> >> bigger concerns here than my army of dumb scripts.  The first version
>>> >> of option
>>> >> 2 wouldn't really do that, but the 'leftovers' version would.  I think
>>> >> I like your new 2 + 4.
>>> >>
>>> >> d.
>>> >>
>>> >>
>>> >> >
>>> >> > -Matt
>>> >> >
>>> >> >>
>>> >> >> Additionally, if I use optparse, similar outcomes?
>>> >> >>
>>> >> >> d.
>>> >> >>
>>> >> >>
>>> >> >> On Fri, Dec 2, 2011 at 9:30 AM, Matthew Turk <matthewturk at gmail.com>
>>> >> >> wrote:
>>> >> >>> Hi all,
>>> >> >>>
>>> >> >>> I have prepared a Pull Request to change how yt processes arguments
>>> >> >>> to
>>> >> >>> scripts.  I just issued it, but I am emailing because I think
>>> >> >>> discussion of what it does warrants a bit more public hashing out.
>>> >> >>> The PR is not done yet, for the reasons I outline below, so please
>>> >> >>> don't anybody accept it yet.
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>> https://bitbucket.org/yt_analysis/yt/pull-request/38/overhaul-configuration-system
>>> >> >>>
>>> >> >>> This will directly affect you if you have:
>>> >> >>>
>>> >> >>> 1) Ever written "from yt.config import ytcfg; ytcfg[...."
>>> >> >>> 2) Ever put your *own* command-line parser into yt.
>>> >> >>> 3) Gotten annoyed with configuration files.
>>> >> >>>
>>> >> >>> What I've done is create a new file, startup_tasks.py, that gets
>>> >> >>> imported whenever yt.mods gets imported, and only the first time
>>> >> >>> that
>>> >> >>> happens.  It sets up an argument parser (using argparse, which is
>>> >> >>> Python 2.7 only) that parses looking for:
>>> >> >>>
>>> >> >>> --parallel
>>> >> >>> --paste
>>> >> >>> --paste-detailed
>>> >> >>> --detailed
>>> >> >>> --rpdb
>>> >> >>> --parallel
>>> >> >>>
>>> >> >>> One of the things this does is that it also provides --help, so you
>>> >> >>> can see what is available.  Furthermore, I've added a --config
>>> >> >>> option,
>>> >> >>> so that from the command line you can set configuration options.
>>> >> >>>  For
>>> >> >>> instance:
>>> >> >>>
>>> >> >>> --config serialize=False
>>> >> >>>
>>> >> >>> and so on.  This is pretty cool I think and will go a long way
>>> >> >>> toward
>>> >> >>> making things nicer.  However, the way this works is still up for a
>>> >> >>> few more problems.  There are basically two ways this can work:
>>> >> >>>
>>> >> >>>  * Parse the entirety of sys.args and accept all arguments that yt
>>> >> >>> finds, rejecting and throwing an error on unrecognized ones (i.e.,
>>> >> >>> typos or things you might pass in to a script your write on the
>>> >> >>> command line).  This will be an exclusive operation.
>>> >> >>>  * Parse *non-exclusively*, allowing unrecognized arguments to pass
>>> >> >>> through.  However, the old arguments will still be there: so any
>>> >> >>> script that has issues with things like --parallel and whatnot will
>>> >> >>> now see there, whereas it did not before because yt (totally
>>> >> >>> un-cool!)
>>> >> >>> stripped them out of the sys.args variable.  I don't want to do
>>> >> >>> this
>>> >> >>> anymore.
>>> >> >>>
>>> >> >>> The way I have implemented this for the yt command line tool is to
>>> >> >>> set
>>> >> >>> a flag that says, "We're also inside the command line, so don't
>>> >> >>> parse
>>> >> >>> anything, we'll handle adding new options to the parser and then
>>> >> >>> we'll
>>> >> >>> parse everything at the end."  This way you can pass both
>>> >> >>> --parallel
>>> >> >>> and whatever option the yt command line utility wants.  This works
>>> >> >>> because startup_tasks creates a "parser" object, adds arguments to
>>> >> >>> that parser object, then delays actually conducting the parsing
>>> >> >>> until
>>> >> >>> all the arguments from teh command line tool have been added.
>>> >> >>>
>>> >> >>> There are four ways this can work.  I have presented them in order
>>> >> >>> of
>>> >> >>> my increasing preference.  (Coincidentally, on the astropy mailing
>>> >> >>> list they discussed this this week, as I was thinking about my
>>> >> >>> feelings on it as well, and they are moving away from parsing args
>>> >> >>> in
>>> >> >>> the library; I think that works for them because AstroPy is
>>> >> >>> designed
>>> >> >>> to be used much more inside larger frameworks, whereas yt is
>>> >> >>> somewhat
>>> >> >>> more insular.)
>>> >> >>> 1) Don't do any argument parsing if not called through a
>>> >> >>> yt-specific
>>> >> >>> script runner.  This means if you want to pass --parallel, you have
>>> >> >>> to
>>> >> >>> run with something like "yt run my_script.py --parallel".  Same for
>>> >> >>> --config and so on.
>>> >> >>> 2) Parse all arguments any time yt.mods is imported, do not allow
>>> >> >>> for
>>> >> >>> additional arguments.  This breaks scripts that have their own
>>> >> >>> parsing.
>>> >> >>> 3) Parse *some* of the arguments, but not all.  All typos would
>>> >> >>> succeed and this could lead to confusion for the user.
>>> >> >>> 4) Provide a yt-specific mechanism for adding new arguments.  So if
>>> >> >>> you want to add new arguments, you do it at the top of your script,
>>> >> >>> rather than the bottom, and at the bottom inside the construction
>>> >> >>> "if
>>> >> >>> __name__ == '__main__'" you'd inspect the values.
>>> >> >>>
>>> >> >>> Anyway, I'm inclined to go for #4, simply because it would be the
>>> >> >>> simplest mechanism for ensuring an explicit method of getting
>>> >> >>> arguments into user-written scripts.
>>> >> >>>
>>> >> >>> Thoughts?
>>> >> >>>
>>> >> >>> -Matt
>>> >> >>> _______________________________________________
>>> >> >>> yt-dev mailing list
>>> >> >>> yt-dev at lists.spacepope.org
>>> >> >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> --
>>> >> >> Sent from my computer.
>>> >> >> _______________________________________________
>>> >> >> yt-dev mailing list
>>> >> >> yt-dev at lists.spacepope.org
>>> >> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >> > _______________________________________________
>>> >> > yt-dev mailing list
>>> >> > yt-dev at lists.spacepope.org
>>> >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Sent from my computer.
>>> >> _______________________________________________
>>> >> yt-dev mailing list
>>> >> yt-dev at lists.spacepope.org
>>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > yt-dev mailing list
>>> > yt-dev at lists.spacepope.org
>>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>
>
>
> --
> Sent from my computer.
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org