On 05/08/2015 5:20 PM, Matthew wrote:
> I have 23 individual matrices each are 385x781x365. They are saved as
> .mat files, each are ~100mb. I would like to take the mean of all 23
> along the third dimension. Unfortunately, there is too much data to load
> all 23, concatenate them, and use mean(concatenated_array,3).
> As an alternative, I have resorted to a loop (dummy data for your use):
>
> data = randn(385,781,365,23);
If you can do the above, you should be able to concatenate the arrays.
I've only got 32-bit so it exceeds the allowable size but just sayin'...
> for n = 1:23
> if n == 1
> data_mean = data(:,:,:,n)./23;
> else
> data_mean = data_mean + data(:,:,:,n)./23;
> end
> end
>
> data_mean2 = squeeze(mean(data,4));
>
> If you look at the data, you will see that data_mean and data_mean2 are
> equivalent (isequal will return 0 because of rounding, though).
> So, this works ok. However, a problem arises because the 23rd matrix is
> actually 385x781x274. Thus, on the final iteration the loop fails
> because the matrices are not the same size.
You don't need to concatenate at all, anyway, just compute a running sum
over the third dimension and you're done. The question is then only how
you want to treat the last partial set -- if it's just a mean of the
give data then the divisor is simply 22*365+274 = 8304.
data_mean=[];
for n = 1:23
<load data % nth file here...>
data_mean = data_mean + sum(data,3);
end
data_mean = data_mean/(22*364+274);
...
> Any ideas how to fix? Ideally a solution that does not require the loop.
You've got to have a loop to read the files anyway, so what difference
does it make? Just don't try to store data you don't actually need; run
the computation at the same time.
The only other alternative I'd have would require converting the storage
from .mat files to stream ("binary") files and then you could use
memmapfile() to get at the data but still reduces to the same thing just
without an explicit load().
You _might_ find the File Exchange submission
<http://www.mathworks.com/matlabcentral/fileexchange/9723-FILEFUN:%20APPLY%20A%20FUNCTION%20TO%20FILES>
of some interest to abstract the computation to a somewhat higher level;
I've not used it much so not sure how well it actually fits this
particular case but it is kinda' handy to have, irregardless.
--
> I have 23 individual matrices each are 385x781x365. They are saved as
> .mat files, each are ~100mb. I would like to take the mean of all 23
> along the third dimension. Unfortunately, there is too much data to load
> all 23, concatenate them, and use mean(concatenated_array,3).
> As an alternative, I have resorted to a loop (dummy data for your use):
>
> data = randn(385,781,365,23);
If you can do the above, you should be able to concatenate the arrays.
I've only got 32-bit so it exceeds the allowable size but just sayin'...
> for n = 1:23
> if n == 1
> data_mean = data(:,:,:,n)./23;
> else
> data_mean = data_mean + data(:,:,:,n)./23;
> end
> end
>
> data_mean2 = squeeze(mean(data,4));
>
> If you look at the data, you will see that data_mean and data_mean2 are
> equivalent (isequal will return 0 because of rounding, though).
> So, this works ok. However, a problem arises because the 23rd matrix is
> actually 385x781x274. Thus, on the final iteration the loop fails
> because the matrices are not the same size.
You don't need to concatenate at all, anyway, just compute a running sum
over the third dimension and you're done. The question is then only how
you want to treat the last partial set -- if it's just a mean of the
give data then the divisor is simply 22*365+274 = 8304.
data_mean=[];
for n = 1:23
<load data % nth file here...>
data_mean = data_mean + sum(data,3);
end
data_mean = data_mean/(22*364+274);
...
> Any ideas how to fix? Ideally a solution that does not require the loop.
You've got to have a loop to read the files anyway, so what difference
does it make? Just don't try to store data you don't actually need; run
the computation at the same time.
The only other alternative I'd have would require converting the storage
from .mat files to stream ("binary") files and then you could use
memmapfile() to get at the data but still reduces to the same thing just
without an explicit load().
You _might_ find the File Exchange submission
<http://www.mathworks.com/matlabcentral/fileexchange/9723-FILEFUN:%20APPLY%20A%20FUNCTION%20TO%20FILES>
of some interest to abstract the computation to a somewhat higher level;
I've not used it much so not sure how well it actually fits this
particular case but it is kinda' handy to have, irregardless.
--