[yt-dev] Arguments to scripts

[Matthew Turk]

Hi Dave,

The key thing here is the sys.argv access.

On Fri, Dec 2, 2011 at 11:53 AM, david collins <antpuncher at gmail.com> wrote:
> Hey, Matt--
>
> I'm a little slow, so let me make sure I understand what your
> suggestions are, or at least put it into context that I use.
>
> Say I have a dumb script that does
>
> <dumb>
> import sys
> pf = load(sys.argv[1])
> pc = PlotCollection(pf,center=[0.5]*3)
> pc.add_projection("ClownDensity",axis=2)
> pc.save("Clown2PerCm2")
> </dumb>
>
> I would typically do something like
> "mpirun -np 4013 python2.7 dumb.py path --parallel"
>
> For option 1, use of this would only work if I called
> "mpirun -np 4013 yt run dumb.py path --parallel" instead

Yes.

>
> For option 2, this wouldn't work at all?

Accessing sys.args wouldn't work in the way you currently have it.
However, we could add a "leftovers" or something variable to the
yt.mods namespace that includes all positional arguments not already
parsed.  I just tested and this does work.

>
> For option 3, this would work as "normal",

Yes, but if you mistyped "--paralleeel" it would not fail (nor does it
fail now.)

>
> and option 4 I'd have to put "path" inside of dumb.py?

Yes.

Talking this through leads me to think we might be able to have a
combination of 4 and 2.  I just tested and what we *can* do is
actually provide the parser arguments as above, but allow for a
"leftovers" section for all *positional* arguments to the script.
This would mean for --something and -s arguments you would need to add
those to the parser object, but for things like what you do above, you
wouldn't.

Would that work for your use case?

-Matt

>
> Additionally, if I use optparse, similar outcomes?
>
> d.
>
>
> On Fri, Dec 2, 2011 at 9:30 AM, Matthew Turk <matthewturk at gmail.com> wrote:
>> Hi all,
>>
>> I have prepared a Pull Request to change how yt processes arguments to
>> scripts.  I just issued it, but I am emailing because I think
>> discussion of what it does warrants a bit more public hashing out.
>> The PR is not done yet, for the reasons I outline below, so please
>> don't anybody accept it yet.
>>
>> https://bitbucket.org/yt_analysis/yt/pull-request/38/overhaul-configuration-system
>>
>> This will directly affect you if you have:
>>
>> 1) Ever written "from yt.config import ytcfg; ytcfg[...."
>> 2) Ever put your *own* command-line parser into yt.
>> 3) Gotten annoyed with configuration files.
>>
>> What I've done is create a new file, startup_tasks.py, that gets
>> imported whenever yt.mods gets imported, and only the first time that
>> happens.  It sets up an argument parser (using argparse, which is
>> Python 2.7 only) that parses looking for:
>>
>> --parallel
>> --paste
>> --paste-detailed
>> --detailed
>> --rpdb
>> --parallel
>>
>> One of the things this does is that it also provides --help, so you
>> can see what is available.  Furthermore, I've added a --config option,
>> so that from the command line you can set configuration options.  For
>> instance:
>>
>> --config serialize=False
>>
>> and so on.  This is pretty cool I think and will go a long way toward
>> making things nicer.  However, the way this works is still up for a
>> few more problems.  There are basically two ways this can work:
>>
>>  * Parse the entirety of sys.args and accept all arguments that yt
>> finds, rejecting and throwing an error on unrecognized ones (i.e.,
>> typos or things you might pass in to a script your write on the
>> command line).  This will be an exclusive operation.
>>  * Parse *non-exclusively*, allowing unrecognized arguments to pass
>> through.  However, the old arguments will still be there: so any
>> script that has issues with things like --parallel and whatnot will
>> now see there, whereas it did not before because yt (totally un-cool!)
>> stripped them out of the sys.args variable.  I don't want to do this
>> anymore.
>>
>> The way I have implemented this for the yt command line tool is to set
>> a flag that says, "We're also inside the command line, so don't parse
>> anything, we'll handle adding new options to the parser and then we'll
>> parse everything at the end."  This way you can pass both --parallel
>> and whatever option the yt command line utility wants.  This works
>> because startup_tasks creates a "parser" object, adds arguments to
>> that parser object, then delays actually conducting the parsing until
>> all the arguments from teh command line tool have been added.
>>
>> There are four ways this can work.  I have presented them in order of
>> my increasing preference.  (Coincidentally, on the astropy mailing
>> list they discussed this this week, as I was thinking about my
>> feelings on it as well, and they are moving away from parsing args in
>> the library; I think that works for them because AstroPy is designed
>> to be used much more inside larger frameworks, whereas yt is somewhat
>> more insular.)
>> 1) Don't do any argument parsing if not called through a yt-specific
>> script runner.  This means if you want to pass --parallel, you have to
>> run with something like "yt run my_script.py --parallel".  Same for
>> --config and so on.
>> 2) Parse all arguments any time yt.mods is imported, do not allow for
>> additional arguments.  This breaks scripts that have their own
>> parsing.
>> 3) Parse *some* of the arguments, but not all.  All typos would
>> succeed and this could lead to confusion for the user.
>> 4) Provide a yt-specific mechanism for adding new arguments.  So if
>> you want to add new arguments, you do it at the top of your script,
>> rather than the bottom, and at the bottom inside the construction "if
>> __name__ == '__main__'" you'd inspect the values.
>>
>> Anyway, I'm inclined to go for #4, simply because it would be the
>> simplest mechanism for ensuring an explicit method of getting
>> arguments into user-written scripts.
>>
>> Thoughts?
>>
>> -Matt
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
>
> --
> Sent from my computer.
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org