[yt-dev] Arguments to scripts

david collins antpuncher at gmail.com
Fri Dec 2 13:03:49 PST 2011 

> It'll have everything in it, including options that have been parsed
> by the option parser.

Perfect.  I'm on board with all that you've proposed.

d.

>
>>
>> d.
>>
>>
>>
>>
>>>
>>> Britton
>>>
>>>
>>> On Fri, Dec 2, 2011 at 2:57 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>>>
>>>> That's the big problem, yes.  I would be fine with yt taking over
>>>> things like "--help" and whatnot, which are reserved.  But I would
>>>> also note that we're actually *changing* the functionality, not
>>>> *removing* it if we go with 2+4.  It says, if you want your script to
>>>> be able to run in parallel yt, or to use these yt commands, here's how
>>>> you interface with the yt argument parsing.
>>>>
>>>> On Fri, Dec 2, 2011 at 2:51 PM, Britton Smith <brittonsmith at gmail.com>
>>>> wrote:
>>>> > I think I still prefer option 3.  I'm in favor of making the system of
>>>> > parsing yt args cleaner, but options 1, 2, and 4 feel to me like we're
>>>> > saying in order to use yt, you have to trade in some python
>>>> > functionality
>>>> > that you've been using.  Is the only reason we wouldn't go with option 3
>>>> > that it will not catch typos in flags that we intended to be for yt?
>>>> >
>>>> > Britton
>>>> >
>>>> > On Fri, Dec 2, 2011 at 1:39 PM, david collins <antpuncher at gmail.com>
>>>> > wrote:
>>>> >>
>>>> >> > Talking this through leads me to think we might be able to have a
>>>> >> > combination of 4 and 2.  I just tested and what we *can* do is
>>>> >> > actually provide the parser arguments as above, but allow for a
>>>> >> > "leftovers" section for all *positional* arguments to the script.
>>>> >> > This would mean for --something and -s arguments you would need to
>>>> >> > add
>>>> >> > those to the parser object, but for things like what you do above,
>>>> >> > you
>>>> >> > wouldn't.
>>>> >> >
>>>> >> > Would that work for your use case?
>>>> >>
>>>> >> So is option 4 something like
>>>> >>
>>>> >> <myscript>
>>>> >> import yt
>>>> >> if __name__ == "__main__":
>>>> >>   yt_option_parser.add_option("--clown_hair_size_threshold")
>>>> >>   do some stuff using the clow hair size
>>>> >> </myscript>
>>>> >>
>>>> >> like a modified/subclassed optparse?  Then my command line is like
>>>> >> mpirun -np 16127 python myscript --clown_hair 14 --parallel
>>>> >> ?
>>>> >>
>>>> >> In your "leftovers" version of Option 2, could that just modify
>>>> >> sys.argv in place, pop-ing out the bits that yt wants or wants to
>>>> >> raise an error on?  Not strictly necessary, but it would mean a little
>>>> >> less modification of old scripts on my part.  (Though frankly, scripts
>>>> >> where I use sys.argv directly should be thrown in the toilet, so
>>>> >> forcing me to re-write those tools to use some less fragile option
>>>> >> parser is probably good for me.)
>>>> >>
>>>> >> Anyhow, as long as it still retains the ability for me to pass in
>>>> >> non-yt options to the script, I'm happy.  And there are certainly
>>>> >> bigger concerns here than my army of dumb scripts.  The first version
>>>> >> of option
>>>> >> 2 wouldn't really do that, but the 'leftovers' version would.  I think
>>>> >> I like your new 2 + 4.
>>>> >>
>>>> >> d.
>>>> >>
>>>> >>
>>>> >> >
>>>> >> > -Matt
>>>> >> >
>>>> >> >>
>>>> >> >> Additionally, if I use optparse, similar outcomes?
>>>> >> >>
>>>> >> >> d.
>>>> >> >>
>>>> >> >>
>>>> >> >> On Fri, Dec 2, 2011 at 9:30 AM, Matthew Turk <matthewturk at gmail.com>
>>>> >> >> wrote:
>>>> >> >>> Hi all,
>>>> >> >>>
>>>> >> >>> I have prepared a Pull Request to change how yt processes arguments
>>>> >> >>> to
>>>> >> >>> scripts.  I just issued it, but I am emailing because I think
>>>> >> >>> discussion of what it does warrants a bit more public hashing out.
>>>> >> >>> The PR is not done yet, for the reasons I outline below, so please
>>>> >> >>> don't anybody accept it yet.
>>>> >> >>>
>>>> >> >>>
>>>> >> >>>
>>>> >> >>> https://bitbucket.org/yt_analysis/yt/pull-request/38/overhaul-configuration-system
>>>> >> >>>
>>>> >> >>> This will directly affect you if you have:
>>>> >> >>>
>>>> >> >>> 1) Ever written "from yt.config import ytcfg; ytcfg[...."
>>>> >> >>> 2) Ever put your *own* command-line parser into yt.
>>>> >> >>> 3) Gotten annoyed with configuration files.
>>>> >> >>>
>>>> >> >>> What I've done is create a new file, startup_tasks.py, that gets
>>>> >> >>> imported whenever yt.mods gets imported, and only the first time
>>>> >> >>> that
>>>> >> >>> happens.  It sets up an argument parser (using argparse, which is
>>>> >> >>> Python 2.7 only) that parses looking for:
>>>> >> >>>
>>>> >> >>> --parallel
>>>> >> >>> --paste
>>>> >> >>> --paste-detailed
>>>> >> >>> --detailed
>>>> >> >>> --rpdb
>>>> >> >>> --parallel
>>>> >> >>>
>>>> >> >>> One of the things this does is that it also provides --help, so you
>>>> >> >>> can see what is available.  Furthermore, I've added a --config
>>>> >> >>> option,
>>>> >> >>> so that from the command line you can set configuration options.
>>>> >> >>>  For
>>>> >> >>> instance:
>>>> >> >>>
>>>> >> >>> --config serialize=False
>>>> >> >>>
>>>> >> >>> and so on.  This is pretty cool I think and will go a long way
>>>> >> >>> toward
>>>> >> >>> making things nicer.  However, the way this works is still up for a
>>>> >> >>> few more problems.  There are basically two ways this can work:
>>>> >> >>>
>>>> >> >>>  * Parse the entirety of sys.args and accept all arguments that yt
>>>> >> >>> finds, rejecting and throwing an error on unrecognized ones (i.e.,
>>>> >> >>> typos or things you might pass in to a script your write on the
>>>> >> >>> command line).  This will be an exclusive operation.
>>>> >> >>>  * Parse *non-exclusively*, allowing unrecognized arguments to pass
>>>> >> >>> through.  However, the old arguments will still be there: so any
>>>> >> >>> script that has issues with things like --parallel and whatnot will
>>>> >> >>> now see there, whereas it did not before because yt (totally
>>>> >> >>> un-cool!)
>>>> >> >>> stripped them out of the sys.args variable.  I don't want to do
>>>> >> >>> this
>>>> >> >>> anymore.
>>>> >> >>>
>>>> >> >>> The way I have implemented this for the yt command line tool is to
>>>> >> >>> set
>>>> >> >>> a flag that says, "We're also inside the command line, so don't
>>>> >> >>> parse
>>>> >> >>> anything, we'll handle adding new options to the parser and then
>>>> >> >>> we'll
>>>> >> >>> parse everything at the end."  This way you can pass both
>>>> >> >>> --parallel
>>>> >> >>> and whatever option the yt command line utility wants.  This works
>>>> >> >>> because startup_tasks creates a "parser" object, adds arguments to
>>>> >> >>> that parser object, then delays actually conducting the parsing
>>>> >> >>> until
>>>> >> >>> all the arguments from teh command line tool have been added.
>>>> >> >>>
>>>> >> >>> There are four ways this can work.  I have presented them in order
>>>> >> >>> of
>>>> >> >>> my increasing preference.  (Coincidentally, on the astropy mailing
>>>> >> >>> list they discussed this this week, as I was thinking about my
>>>> >> >>> feelings on it as well, and they are moving away from parsing args
>>>> >> >>> in
>>>> >> >>> the library; I think that works for them because AstroPy is
>>>> >> >>> designed
>>>> >> >>> to be used much more inside larger frameworks, whereas yt is
>>>> >> >>> somewhat
>>>> >> >>> more insular.)
>>>> >> >>> 1) Don't do any argument parsing if not called through a
>>>> >> >>> yt-specific
>>>> >> >>> script runner.  This means if you want to pass --parallel, you have
>>>> >> >>> to
>>>> >> >>> run with something like "yt run my_script.py --parallel".  Same for
>>>> >> >>> --config and so on.
>>>> >> >>> 2) Parse all arguments any time yt.mods is imported, do not allow
>>>> >> >>> for
>>>> >> >>> additional arguments.  This breaks scripts that have their own
>>>> >> >>> parsing.
>>>> >> >>> 3) Parse *some* of the arguments, but not all.  All typos would
>>>> >> >>> succeed and this could lead to confusion for the user.
>>>> >> >>> 4) Provide a yt-specific mechanism for adding new arguments.  So if
>>>> >> >>> you want to add new arguments, you do it at the top of your script,
>>>> >> >>> rather than the bottom, and at the bottom inside the construction
>>>> >> >>> "if
>>>> >> >>> __name__ == '__main__'" you'd inspect the values.
>>>> >> >>>
>>>> >> >>> Anyway, I'm inclined to go for #4, simply because it would be the
>>>> >> >>> simplest mechanism for ensuring an explicit method of getting
>>>> >> >>> arguments into user-written scripts.
>>>> >> >>>
>>>> >> >>> Thoughts?
>>>> >> >>>
>>>> >> >>> -Matt
>>>> >> >>> _______________________________________________
>>>> >> >>> yt-dev mailing list
>>>> >> >>> yt-dev at lists.spacepope.org
>>>> >> >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>> >> >>
>>>> >> >>
>>>> >> >>
>>>> >> >> --
>>>> >> >> Sent from my computer.
>>>> >> >> _______________________________________________
>>>> >> >> yt-dev mailing list
>>>> >> >> yt-dev at lists.spacepope.org
>>>> >> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>> >> > _______________________________________________
>>>> >> > yt-dev mailing list
>>>> >> > yt-dev at lists.spacepope.org
>>>> >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Sent from my computer.
>>>> >> _______________________________________________
>>>> >> yt-dev mailing list
>>>> >> yt-dev at lists.spacepope.org
>>>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>> >
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > yt-dev mailing list
>>>> > yt-dev at lists.spacepope.org
>>>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>> >
>>>> _______________________________________________
>>>> yt-dev mailing list
>>>> yt-dev at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>>
>>>
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>
>>
>>
>> --
>> Sent from my computer.
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org



-- 
Sent from my computer.

More information about the yt-dev mailing list