[yt-dev] Arguments to scripts

Matthew Turk matthewturk at gmail.com
Tue Dec 20 13:56:46 PST 2011 

Hi all,

I have revised my pull request:

https://bitbucket.org/yt_analysis/yt/pull-request/38/overhaul-configuration-system

This now includes all the arguments in sys.argv, including the
arguments parsed out by the internal yt system.  The arguments
remaining are all inside yt.startup_tasks.leftover_args.

There's a line included that shows how to switch to the behavior where
we modify sys.argv.  I have come around a bit and I think this would
be okay, since we have been doing it for some time.  The new system is
much easier to modify, and it also allows us to specify configuration
options on teh command line using --config serialize=False for
instance.  If somebody wouldn't mind pulling this down and testing it
out, that'd be great.  Works for me okay though, and I like the new
look of the system -- you can now actually get help on the arguments!

-Matt

On Fri, Dec 2, 2011 at 4:08 PM, j s oishi <jsoishi at gmail.com> wrote:
> Agreed. I am also on board.
>
> j
>
> On Fri, Dec 2, 2011 at 1:03 PM, david collins <antpuncher at gmail.com> wrote:
>>> It'll have everything in it, including options that have been parsed
>>> by the option parser.
>>
>> Perfect.  I'm on board with all that you've proposed.
>>
>> d.
>>
>>>
>>>>
>>>> d.
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>> Britton
>>>>>
>>>>>
>>>>> On Fri, Dec 2, 2011 at 2:57 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>>>>>
>>>>>> That's the big problem, yes.  I would be fine with yt taking over
>>>>>> things like "--help" and whatnot, which are reserved.  But I would
>>>>>> also note that we're actually *changing* the functionality, not
>>>>>> *removing* it if we go with 2+4.  It says, if you want your script to
>>>>>> be able to run in parallel yt, or to use these yt commands, here's how
>>>>>> you interface with the yt argument parsing.
>>>>>>
>>>>>> On Fri, Dec 2, 2011 at 2:51 PM, Britton Smith <brittonsmith at gmail.com>
>>>>>> wrote:
>>>>>> > I think I still prefer option 3.  I'm in favor of making the system of
>>>>>> > parsing yt args cleaner, but options 1, 2, and 4 feel to me like we're
>>>>>> > saying in order to use yt, you have to trade in some python
>>>>>> > functionality
>>>>>> > that you've been using.  Is the only reason we wouldn't go with option 3
>>>>>> > that it will not catch typos in flags that we intended to be for yt?
>>>>>> >
>>>>>> > Britton
>>>>>> >
>>>>>> > On Fri, Dec 2, 2011 at 1:39 PM, david collins <antpuncher at gmail.com>
>>>>>> > wrote:
>>>>>> >>
>>>>>> >> > Talking this through leads me to think we might be able to have a
>>>>>> >> > combination of 4 and 2.  I just tested and what we *can* do is
>>>>>> >> > actually provide the parser arguments as above, but allow for a
>>>>>> >> > "leftovers" section for all *positional* arguments to the script.
>>>>>> >> > This would mean for --something and -s arguments you would need to
>>>>>> >> > add
>>>>>> >> > those to the parser object, but for things like what you do above,
>>>>>> >> > you
>>>>>> >> > wouldn't.
>>>>>> >> >
>>>>>> >> > Would that work for your use case?
>>>>>> >>
>>>>>> >> So is option 4 something like
>>>>>> >>
>>>>>> >> <myscript>
>>>>>> >> import yt
>>>>>> >> if __name__ == "__main__":
>>>>>> >>   yt_option_parser.add_option("--clown_hair_size_threshold")
>>>>>> >>   do some stuff using the clow hair size
>>>>>> >> </myscript>
>>>>>> >>
>>>>>> >> like a modified/subclassed optparse?  Then my command line is like
>>>>>> >> mpirun -np 16127 python myscript --clown_hair 14 --parallel
>>>>>> >> ?
>>>>>> >>
>>>>>> >> In your "leftovers" version of Option 2, could that just modify
>>>>>> >> sys.argv in place, pop-ing out the bits that yt wants or wants to
>>>>>> >> raise an error on?  Not strictly necessary, but it would mean a little
>>>>>> >> less modification of old scripts on my part.  (Though frankly, scripts
>>>>>> >> where I use sys.argv directly should be thrown in the toilet, so
>>>>>> >> forcing me to re-write those tools to use some less fragile option
>>>>>> >> parser is probably good for me.)
>>>>>> >>
>>>>>> >> Anyhow, as long as it still retains the ability for me to pass in
>>>>>> >> non-yt options to the script, I'm happy.  And there are certainly
>>>>>> >> bigger concerns here than my army of dumb scripts.  The first version
>>>>>> >> of option
>>>>>> >> 2 wouldn't really do that, but the 'leftovers' version would.  I think
>>>>>> >> I like your new 2 + 4.
>>>>>> >>
>>>>>> >> d.
>>>>>> >>
>>>>>> >>
>>>>>> >> >
>>>>>> >> > -Matt
>>>>>> >> >
>>>>>> >> >>
>>>>>> >> >> Additionally, if I use optparse, similar outcomes?
>>>>>> >> >>
>>>>>> >> >> d.
>>>>>> >> >>
>>>>>> >> >>
>>>>>> >> >> On Fri, Dec 2, 2011 at 9:30 AM, Matthew Turk <matthewturk at gmail.com>
>>>>>> >> >> wrote:
>>>>>> >> >>> Hi all,
>>>>>> >> >>>
>>>>>> >> >>> I have prepared a Pull Request to change how yt processes arguments
>>>>>> >> >>> to
>>>>>> >> >>> scripts.  I just issued it, but I am emailing because I think
>>>>>> >> >>> discussion of what it does warrants a bit more public hashing out.
>>>>>> >> >>> The PR is not done yet, for the reasons I outline below, so please
>>>>>> >> >>> don't anybody accept it yet.
>>>>>> >> >>>
>>>>>> >> >>>
>>>>>> >> >>>
>>>>>> >> >>> https://bitbucket.org/yt_analysis/yt/pull-request/38/overhaul-configuration-system
>>>>>> >> >>>
>>>>>> >> >>> This will directly affect you if you have:
>>>>>> >> >>>
>>>>>> >> >>> 1) Ever written "from yt.config import ytcfg; ytcfg[...."
>>>>>> >> >>> 2) Ever put your *own* command-line parser into yt.
>>>>>> >> >>> 3) Gotten annoyed with configuration files.
>>>>>> >> >>>
>>>>>> >> >>> What I've done is create a new file, startup_tasks.py, that gets
>>>>>> >> >>> imported whenever yt.mods gets imported, and only the first time
>>>>>> >> >>> that
>>>>>> >> >>> happens.  It sets up an argument parser (using argparse, which is
>>>>>> >> >>> Python 2.7 only) that parses looking for:
>>>>>> >> >>>
>>>>>> >> >>> --parallel
>>>>>> >> >>> --paste
>>>>>> >> >>> --paste-detailed
>>>>>> >> >>> --detailed
>>>>>> >> >>> --rpdb
>>>>>> >> >>> --parallel
>>>>>> >> >>>
>>>>>> >> >>> One of the things this does is that it also provides --help, so you
>>>>>> >> >>> can see what is available.  Furthermore, I've added a --config
>>>>>> >> >>> option,
>>>>>> >> >>> so that from the command line you can set configuration options.
>>>>>> >> >>>  For
>>>>>> >> >>> instance:
>>>>>> >> >>>
>>>>>> >> >>> --config serialize=False
>>>>>> >> >>>
>>>>>> >> >>> and so on.  This is pretty cool I think and will go a long way
>>>>>> >> >>> toward
>>>>>> >> >>> making things nicer.  However, the way this works is still up for a
>>>>>> >> >>> few more problems.  There are basically two ways this can work:
>>>>>> >> >>>
>>>>>> >> >>>  * Parse the entirety of sys.args and accept all arguments that yt
>>>>>> >> >>> finds, rejecting and throwing an error on unrecognized ones (i.e.,
>>>>>> >> >>> typos or things you might pass in to a script your write on the
>>>>>> >> >>> command line).  This will be an exclusive operation.
>>>>>> >> >>>  * Parse *non-exclusively*, allowing unrecognized arguments to pass
>>>>>> >> >>> through.  However, the old arguments will still be there: so any
>>>>>> >> >>> script that has issues with things like --parallel and whatnot will
>>>>>> >> >>> now see there, whereas it did not before because yt (totally
>>>>>> >> >>> un-cool!)
>>>>>> >> >>> stripped them out of the sys.args variable.  I don't want to do
>>>>>> >> >>> this
>>>>>> >> >>> anymore.
>>>>>> >> >>>
>>>>>> >> >>> The way I have implemented this for the yt command line tool is to
>>>>>> >> >>> set
>>>>>> >> >>> a flag that says, "We're also inside the command line, so don't
>>>>>> >> >>> parse
>>>>>> >> >>> anything, we'll handle adding new options to the parser and then
>>>>>> >> >>> we'll
>>>>>> >> >>> parse everything at the end."  This way you can pass both
>>>>>> >> >>> --parallel
>>>>>> >> >>> and whatever option the yt command line utility wants.  This works
>>>>>> >> >>> because startup_tasks creates a "parser" object, adds arguments to
>>>>>> >> >>> that parser object, then delays actually conducting the parsing
>>>>>> >> >>> until
>>>>>> >> >>> all the arguments from teh command line tool have been added.
>>>>>> >> >>>
>>>>>> >> >>> There are four ways this can work.  I have presented them in order
>>>>>> >> >>> of
>>>>>> >> >>> my increasing preference.  (Coincidentally, on the astropy mailing
>>>>>> >> >>> list they discussed this this week, as I was thinking about my
>>>>>> >> >>> feelings on it as well, and they are moving away from parsing args
>>>>>> >> >>> in
>>>>>> >> >>> the library; I think that works for them because AstroPy is
>>>>>> >> >>> designed
>>>>>> >> >>> to be used much more inside larger frameworks, whereas yt is
>>>>>> >> >>> somewhat
>>>>>> >> >>> more insular.)
>>>>>> >> >>> 1) Don't do any argument parsing if not called through a
>>>>>> >> >>> yt-specific
>>>>>> >> >>> script runner.  This means if you want to pass --parallel, you have
>>>>>> >> >>> to
>>>>>> >> >>> run with something like "yt run my_script.py --parallel".  Same for
>>>>>> >> >>> --config and so on.
>>>>>> >> >>> 2) Parse all arguments any time yt.mods is imported, do not allow
>>>>>> >> >>> for
>>>>>> >> >>> additional arguments.  This breaks scripts that have their own
>>>>>> >> >>> parsing.
>>>>>> >> >>> 3) Parse *some* of the arguments, but not all.  All typos would
>>>>>> >> >>> succeed and this could lead to confusion for the user.
>>>>>> >> >>> 4) Provide a yt-specific mechanism for adding new arguments.  So if
>>>>>> >> >>> you want to add new arguments, you do it at the top of your script,
>>>>>> >> >>> rather than the bottom, and at the bottom inside the construction
>>>>>> >> >>> "if
>>>>>> >> >>> __name__ == '__main__'" you'd inspect the values.
>>>>>> >> >>>
>>>>>> >> >>> Anyway, I'm inclined to go for #4, simply because it would be the
>>>>>> >> >>> simplest mechanism for ensuring an explicit method of getting
>>>>>> >> >>> arguments into user-written scripts.
>>>>>> >> >>>
>>>>>> >> >>> Thoughts?
>>>>>> >> >>>
>>>>>> >> >>> -Matt
>>>>>> >> >>> _______________________________________________
>>>>>> >> >>> yt-dev mailing list
>>>>>> >> >>> yt-dev at lists.spacepope.org
>>>>>> >> >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>> >> >>
>>>>>> >> >>
>>>>>> >> >>
>>>>>> >> >> --
>>>>>> >> >> Sent from my computer.
>>>>>> >> >> _______________________________________________
>>>>>> >> >> yt-dev mailing list
>>>>>> >> >> yt-dev at lists.spacepope.org
>>>>>> >> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>> >> > _______________________________________________
>>>>>> >> > yt-dev mailing list
>>>>>> >> > yt-dev at lists.spacepope.org
>>>>>> >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> --
>>>>>> >> Sent from my computer.
>>>>>> >> _______________________________________________
>>>>>> >> yt-dev mailing list
>>>>>> >> yt-dev at lists.spacepope.org
>>>>>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > _______________________________________________
>>>>>> > yt-dev mailing list
>>>>>> > yt-dev at lists.spacepope.org
>>>>>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>> >
>>>>>> _______________________________________________
>>>>>> yt-dev mailing list
>>>>>> yt-dev at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> yt-dev mailing list
>>>>> yt-dev at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from my computer.
>>>> _______________________________________________
>>>> yt-dev mailing list
>>>> yt-dev at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>>
>> --
>> Sent from my computer.
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

More information about the yt-dev mailing list