[yt-dev] chunk size undefined before fill

[Matthew Turk]

Hi Doug,

On Thu, Mar 14, 2013 at 2:16 PM, Douglas Harvey Rudd <drudd at uchicago.edu> wrote:
>
>> Yup, you're right.  Sorry about that.  I want to take the opportunity
>> also to thank you for your work on this -- as I've said before, opaque
>> data object support is going to be a huge win for us, and I appreciate
>> you sorting through all the myopic, "Matt didn't think this would be a
>> problem" sections.  :)
>>
>
> Of course, changing code in this way will always uncover implicit assumptions,
> and these are very difficult to pick up in unit tests (since most of those tests will
> reflect the same assumptions).
>
>>
>> Yes, something like this, but is there going to be a case where
>> _current_chunk is None?  Or will this always be called after
>> _identify_base_chunk?  Incidentally, I am now wondering if we should
>> just get rid of reliance on .size in the first place.  It's not clear
>> to me that there is a case when we absolutely need it, except *inside*
>> the IO handlers, and those should all be able to guess or not rely on
>> the size anyway.  Right?
>>
>
> I don't know, but I also didn't know what to do in that situation, so I let it fall
> back to returning an undefined quantity (_size is None).  We can identify a
> base chunk, but then it will simply fail due to the size being queried before
> the read.

That sounds good to me.

>
>>> def _chunked_read(self, chunk):
>>>        # There are several items that need to be swapped out
>>>        # field_data, size, shape
>>>        old_field_data, self.field_data = self.field_data, YTFieldData()
>>>        old_chunk, self._current_chunk = self._current_chunk, chunk
>>>        self._size = None
>>>        old_locked, self._locked = self._locked, False
>>>        yield
>>>        self.field_data = old_field_data
>>>        self._current_chunk = old_chunk
>>>        self._size = None
>>>        self._locked = old_locked
>>>
>>> That may just push the problem further down the call stack, but at least means this code won't ask for .get_size until needed.
>>
>> I actually don't see the diff here, what changed in this function?
>>
>
> I removed the code that copied the old size and restored it after the yield.  Instead, the _size is always
> invalidated any time _current_chunk is changed, and then will be repopulated as needed.  I'm assuming
> here that size is trivially computed from the chunks.

Ah, great.  I agree with this.

>
> Doug
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org