[yt-dev] Arguments to scripts

[j s oishi]

Agreed. I am also on board.

j

On Fri, Dec 2, 2011 at 1:03 PM, david collins <antpuncher at gmail.com> wrote:
>> It'll have everything in it, including options that have been parsed
>> by the option parser.
>
> Perfect.  I'm on board with all that you've proposed.
>
> d.
>
>>
>>>
>>> d.
>>>
>>>
>>>
>>>
>>>>
>>>> Britton
>>>>
>>>>
>>>> On Fri, Dec 2, 2011 at 2:57 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>>>>
>>>>> That's the big problem, yes.  I would be fine with yt taking over
>>>>> things like "--help" and whatnot, which are reserved.  But I would
>>>>> also note that we're actually *changing* the functionality, not
>>>>> *removing* it if we go with 2+4.  It says, if you want your script to
>>>>> be able to run in parallel yt, or to use these yt commands, here's how
>>>>> you interface with the yt argument parsing.
>>>>>
>>>>> On Fri, Dec 2, 2011 at 2:51 PM, Britton Smith <brittonsmith at gmail.com>
>>>>> wrote:
>>>>> > I think I still prefer option 3.  I'm in favor of making the system of
>>>>> > parsing yt args cleaner, but options 1, 2, and 4 feel to me like we're
>>>>> > saying in order to use yt, you have to trade in some python
>>>>> > functionality
>>>>> > that you've been using.  Is the only reason we wouldn't go with option 3
>>>>> > that it will not catch typos in flags that we intended to be for yt?
>>>>> >
>>>>> > Britton
>>>>> >
>>>>> > On Fri, Dec 2, 2011 at 1:39 PM, david collins <antpuncher at gmail.com>
>>>>> > wrote:
>>>>> >>
>>>>> >> > Talking this through leads me to think we might be able to have a
>>>>> >> > combination of 4 and 2.  I just tested and what we *can* do is
>>>>> >> > actually provide the parser arguments as above, but allow for a
>>>>> >> > "leftovers" section for all *positional* arguments to the script.
>>>>> >> > This would mean for --something and -s arguments you would need to
>>>>> >> > add
>>>>> >> > those to the parser object, but for things like what you do above,
>>>>> >> > you
>>>>> >> > wouldn't.
>>>>> >> >
>>>>> >> > Would that work for your use case?
>>>>> >>
>>>>> >> So is option 4 something like
>>>>> >>
>>>>> >> <myscript>
>>>>> >> import yt
>>>>> >> if __name__ == "__main__":
>>>>> >>   yt_option_parser.add_option("--clown_hair_size_threshold")
>>>>> >>   do some stuff using the clow hair size
>>>>> >> </myscript>
>>>>> >>
>>>>> >> like a modified/subclassed optparse?  Then my command line is like
>>>>> >> mpirun -np 16127 python myscript --clown_hair 14 --parallel
>>>>> >> ?
>>>>> >>
>>>>> >> In your "leftovers" version of Option 2, could that just modify
>>>>> >> sys.argv in place, pop-ing out the bits that yt wants or wants to
>>>>> >> raise an error on?  Not strictly necessary, but it would mean a little
>>>>> >> less modification of old scripts on my part.  (Though frankly, scripts
>>>>> >> where I use sys.argv directly should be thrown in the toilet, so
>>>>> >> forcing me to re-write those tools to use some less fragile option
>>>>> >> parser is probably good for me.)
>>>>> >>
>>>>> >> Anyhow, as long as it still retains the ability for me to pass in
>>>>> >> non-yt options to the script, I'm happy.  And there are certainly
>>>>> >> bigger concerns here than my army of dumb scripts.  The first version
>>>>> >> of option
>>>>> >> 2 wouldn't really do that, but the 'leftovers' version would.  I think
>>>>> >> I like your new 2 + 4.
>>>>> >>
>>>>> >> d.
>>>>> >>
>>>>> >>
>>>>> >> >
>>>>> >> > -Matt
>>>>> >> >
>>>>> >> >>
>>>>> >> >> Additionally, if I use optparse, similar outcomes?
>>>>> >> >>
>>>>> >> >> d.
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> On Fri, Dec 2, 2011 at 9:30 AM, Matthew Turk <matthewturk at gmail.com>
>>>>> >> >> wrote:
>>>>> >> >>> Hi all,
>>>>> >> >>>
>>>>> >> >>> I have prepared a Pull Request to change how yt processes arguments
>>>>> >> >>> to
>>>>> >> >>> scripts.  I just issued it, but I am emailing because I think
>>>>> >> >>> discussion of what it does warrants a bit more public hashing out.
>>>>> >> >>> The PR is not done yet, for the reasons I outline below, so please
>>>>> >> >>> don't anybody accept it yet.
>>>>> >> >>>
>>>>> >> >>>
>>>>> >> >>>
>>>>> >> >>> https://bitbucket.org/yt_analysis/yt/pull-request/38/overhaul-configuration-system
>>>>> >> >>>
>>>>> >> >>> This will directly affect you if you have:
>>>>> >> >>>
>>>>> >> >>> 1) Ever written "from yt.config import ytcfg; ytcfg[...."
>>>>> >> >>> 2) Ever put your *own* command-line parser into yt.
>>>>> >> >>> 3) Gotten annoyed with configuration files.
>>>>> >> >>>
>>>>> >> >>> What I've done is create a new file, startup_tasks.py, that gets
>>>>> >> >>> imported whenever yt.mods gets imported, and only the first time
>>>>> >> >>> that
>>>>> >> >>> happens.  It sets up an argument parser (using argparse, which is
>>>>> >> >>> Python 2.7 only) that parses looking for:
>>>>> >> >>>
>>>>> >> >>> --parallel
>>>>> >> >>> --paste
>>>>> >> >>> --paste-detailed
>>>>> >> >>> --detailed
>>>>> >> >>> --rpdb
>>>>> >> >>> --parallel
>>>>> >> >>>
>>>>> >> >>> One of the things this does is that it also provides --help, so you
>>>>> >> >>> can see what is available.  Furthermore, I've added a --config
>>>>> >> >>> option,
>>>>> >> >>> so that from the command line you can set configuration options.
>>>>> >> >>>  For
>>>>> >> >>> instance:
>>>>> >> >>>
>>>>> >> >>> --config serialize=False
>>>>> >> >>>
>>>>> >> >>> and so on.  This is pretty cool I think and will go a long way
>>>>> >> >>> toward
>>>>> >> >>> making things nicer.  However, the way this works is still up for a
>>>>> >> >>> few more problems.  There are basically two ways this can work:
>>>>> >> >>>
>>>>> >> >>>  * Parse the entirety of sys.args and accept all arguments that yt
>>>>> >> >>> finds, rejecting and throwing an error on unrecognized ones (i.e.,
>>>>> >> >>> typos or things you might pass in to a script your write on the
>>>>> >> >>> command line).  This will be an exclusive operation.
>>>>> >> >>>  * Parse *non-exclusively*, allowing unrecognized arguments to pass
>>>>> >> >>> through.  However, the old arguments will still be there: so any
>>>>> >> >>> script that has issues with things like --parallel and whatnot will
>>>>> >> >>> now see there, whereas it did not before because yt (totally
>>>>> >> >>> un-cool!)
>>>>> >> >>> stripped them out of the sys.args variable.  I don't want to do
>>>>> >> >>> this
>>>>> >> >>> anymore.
>>>>> >> >>>
>>>>> >> >>> The way I have implemented this for the yt command line tool is to
>>>>> >> >>> set
>>>>> >> >>> a flag that says, "We're also inside the command line, so don't
>>>>> >> >>> parse
>>>>> >> >>> anything, we'll handle adding new options to the parser and then
>>>>> >> >>> we'll
>>>>> >> >>> parse everything at the end."  This way you can pass both
>>>>> >> >>> --parallel
>>>>> >> >>> and whatever option the yt command line utility wants.  This works
>>>>> >> >>> because startup_tasks creates a "parser" object, adds arguments to
>>>>> >> >>> that parser object, then delays actually conducting the parsing
>>>>> >> >>> until
>>>>> >> >>> all the arguments from teh command line tool have been added.
>>>>> >> >>>
>>>>> >> >>> There are four ways this can work.  I have presented them in order
>>>>> >> >>> of
>>>>> >> >>> my increasing preference.  (Coincidentally, on the astropy mailing
>>>>> >> >>> list they discussed this this week, as I was thinking about my
>>>>> >> >>> feelings on it as well, and they are moving away from parsing args
>>>>> >> >>> in
>>>>> >> >>> the library; I think that works for them because AstroPy is
>>>>> >> >>> designed
>>>>> >> >>> to be used much more inside larger frameworks, whereas yt is
>>>>> >> >>> somewhat
>>>>> >> >>> more insular.)
>>>>> >> >>> 1) Don't do any argument parsing if not called through a
>>>>> >> >>> yt-specific
>>>>> >> >>> script runner.  This means if you want to pass --parallel, you have
>>>>> >> >>> to
>>>>> >> >>> run with something like "yt run my_script.py --parallel".  Same for
>>>>> >> >>> --config and so on.
>>>>> >> >>> 2) Parse all arguments any time yt.mods is imported, do not allow
>>>>> >> >>> for
>>>>> >> >>> additional arguments.  This breaks scripts that have their own
>>>>> >> >>> parsing.
>>>>> >> >>> 3) Parse *some* of the arguments, but not all.  All typos would
>>>>> >> >>> succeed and this could lead to confusion for the user.
>>>>> >> >>> 4) Provide a yt-specific mechanism for adding new arguments.  So if
>>>>> >> >>> you want to add new arguments, you do it at the top of your script,
>>>>> >> >>> rather than the bottom, and at the bottom inside the construction
>>>>> >> >>> "if
>>>>> >> >>> __name__ == '__main__'" you'd inspect the values.
>>>>> >> >>>
>>>>> >> >>> Anyway, I'm inclined to go for #4, simply because it would be the
>>>>> >> >>> simplest mechanism for ensuring an explicit method of getting
>>>>> >> >>> arguments into user-written scripts.
>>>>> >> >>>
>>>>> >> >>> Thoughts?
>>>>> >> >>>
>>>>> >> >>> -Matt
>>>>> >> >>> _______________________________________________
>>>>> >> >>> yt-dev mailing list
>>>>> >> >>> yt-dev at lists.spacepope.org
>>>>> >> >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>> >> >>
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> --
>>>>> >> >> Sent from my computer.
>>>>> >> >> _______________________________________________
>>>>> >> >> yt-dev mailing list
>>>>> >> >> yt-dev at lists.spacepope.org
>>>>> >> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>> >> > _______________________________________________
>>>>> >> > yt-dev mailing list
>>>>> >> > yt-dev at lists.spacepope.org
>>>>> >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Sent from my computer.
>>>>> >> _______________________________________________
>>>>> >> yt-dev mailing list
>>>>> >> yt-dev at lists.spacepope.org
>>>>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>> >
>>>>> >
>>>>> >
>>>>> > _______________________________________________
>>>>> > yt-dev mailing list
>>>>> > yt-dev at lists.spacepope.org
>>>>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>> >
>>>>> _______________________________________________
>>>>> yt-dev mailing list
>>>>> yt-dev at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> yt-dev mailing list
>>>> yt-dev at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>
>>>
>>>
>>>
>>> --
>>> Sent from my computer.
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
>
> --
> Sent from my computer.
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org